47ronin.com: AMI instance disaster!

 

Long story short: Be careful with your AMI instance

I wiped out my website again. January 29, I decided to SSH into the 47ronin.com AMI instance and upgrade everything —including the distro. Almost every server admin in the world will tell you, “if it ain’t broke, don’t fix it.” Yeah, well…

Short story long

I tried to upgrade from Ubuntu 14.10  to 15.x over a remote connection and oh boy did I screw things up. The EC2 EBS volume didn’t have enough free space and a big chunk of that was taken up by swap. I decided to reboot the instance, it got stuck, and I forced a restart from the Amazon AWS console. Except… it came up with no SSH nor web access. Locked out. Spent my entire weekend trying to revive the server salvage the data. Thankfully, someone named “Donald C.” at Amazon support walked me through how to properly connect a “marketplace” AMI EBS volume to another instance and rescue my stuff. I had to:

1. Boot a new instance
2. Relabel the boot device
3. Configure GRUB to boot from the relabeled drive
4. Shut down the instance and connect the corrupted EBS
5. Boot the instance and voila! Just enough access to SFTP my website off the old drive
6. Spin up a new WordPress AMI, import the website database, files
7. Fix DNS to use the VPS elastic IP
8. Reissue my SSL certificate and install onto the new instance

Lesson learned

Use an LTS version of a server distro and only upgrade to another LTS. In between those times, back up the db and filesystem regularly (I’ll figure out a proper way to do this easily). Oddly, I found no problem doing this with all the servers I was in charge of at the Port of San Diego (back when the intranet and public website ran on Joomla/Apache/PHP/MySQL). I ran a cron that ran every 12 hours and backed up the MySQL database, gzipped it, and FTP’d the daily snapshot file down to a local server. The cron also SCP’d an rsync of the public documents —would be silly to copy the whole repository over if only two or three files changed. Duh. However, with my own website I don’t have a local server (one with unlimited space, anyway). I don’t feel like filling up my local drives. So, for now I’m just making EBS snapshots and labeling them (primitive but my brain has been fried ever since the rescue). Big operations on Amazon AWS are stupid simple, I gotta admit. Love it.

Am I ready to dump proprietary marketplace AMIs? I was very, very close to switching to a community AMI but in the end— I don’t have the patience anymore to do everything from scratch.

Below was the post I was about to publish before I inadvertently trashed my server for a whole weekend.

…

I’m three weeks into the new year, and it looks like not much happened on the web-dev front here. On the contrary, a lot has been planned and painstakingly thought-out —in my head.

Challenges

• NGINX: I’m probably not going to re-do the website on it at this point because of my SSL configuration. I want to keep the project simple, and having to deal with re-working my encryption might be a huge chore. Besides, this Bitnami appliance was built with Apache. If I truly wanted to change everything, I might as well spin up an NGINX-WordPress-PHP7-MariaDB appliance and save myself the headaches.
• PHP 7.x: I will likely do this first. From my experience, WordPress is right at home in this config and hopefully will be no more complex than `sudo apt-get install…` Getting NGINX and PHP7 on Mac OS X was a breeze using Homebrew and slightly modded version of this tutorial.
• MariaDB: This one I’ll probably do second. It’s just a drop-in replacement from what I’ve read, and you know me —I’m always looking to get myself into trouble by fixing what’s not broken.
• oEmbed: I’m really excited about WordPress embracing oEmbed. I just wish I could go back and fix my old posts  that are stuck using Embedly. In order to prevent odd behavior in future publishing, I may have to disable the Embedly plugin and live with old posts looking funky. We’ll see.

Other projects

It’s been a joy working with the talented folks at Open San Diego, my local Code for America brigade. I’ve been volunteering my time filming and publishing videos of their events, and I have been helping them out with their website and Github projects.

I had a great time shooting and editing these pieces. Hoping for better gear someday to improve the quality!