AWS - Crash and recovery
While Ennie and I were off getting some dinner and I get a message from my Wordpress monitoring service saying my site (the one you're reading now) is offline.:-/Normally this isn't a big deal. Either restarting MySql or rebooting the server generally fixes the problem. Usually MySql just goes insane or something. It's infrequent enough that I don't really bother much with it.On the odd occasion the disk gets full of MySql logs. Again, no big deal. Delete and move on.This time... nope.I couldn't even ssh into it. It was just dead. Rebooting it didn't help either.At this point I was getting nervous.I was able to spin up a new instance and remote into that so it's not the core EC2 stuff...Here's what I did to solve it.
- Shut down my old non-working instance (stop, do not terminate!)
- Make a snapshot of the drive
- Turn that snapshot into an AMI (Amazon Machine Image) (*)
- Start a new instance using that AMI
- I started with a t2.micro instance and have to turn on the hardware assisted virtualization when I was creating the AMI. Without this I couldn't start a new t2.micro instance.
This got it working. :-DThen I made a backup.Which reminds me -- make backups!
- = -
I'll probably check to see if there are any problems at work. No one likes to see how sausage is made... but sometimes it's nice working in the sausage factory.