Another fun one is disabling the network interface on a remote server. An acquaintance did that by mistake on a cloud VM running some core services, and the cloud provider had no virtual console for some reason. Ended up having to write off the VM and restore from backup. Fun day at the office.
Long ago, I succeeded once to cut my own access through SSH to a remote server, after some firewall changes. That of course has required a long trip to the server, for physical access.
However that was good, because after that I have always been extra careful at any changes that could affect the firewall in any way. (That is not restricted to changes in firewall rules, because there are systems where the versions of the firewall program and of the kernel must be correlated, so an inconsistent update may make the firewall revert to its default state of denying all connections.)
I previously managed a firewall via scripts which would automatically revert your update in 20 seconds unless interrupted. So if you botched it and lost access, you just had to sit tight for 20 seconds.
Hah, I once did “netplan try” on a prototype production machine. The new config wasn’t quite right (although not catastrophic in any respect) so I told it to roll back. Bye bye new machine.
Fortunately this was an exercise and we had BMC access, so no big deal. Except that we got yet another datapoint suggesting that netplan is not a high quality piece of software.
Last I checked, if you non-forcibly reboot a GCE instance via console or API and it does not shut itself down in a timely manner, there was literally no way to force it to turn off or hard-reboot so that your block storage instances get released. IIRC the last time I encountered this the process timed out eventually after some silly long wait.
