Homebrewed live server migration

By Andrew Smith

I mentioned that I’ll talk about the software migration from the old littlesvr.ca hardware to the new machine. The neat thing is – I accomplished it in less than a minute of downtime while preserving all my data/metadata.

Here’s the long story (shorter version at the bottom):

  1. First step was to install the OS on the new hardware. This was a full install (just like the old one) of the newest available Slackware version.
  2. At this point I had two servers running, on different internal IPs, both claiming to be littlesvr.ca but only one (the old one) beeing accessible from the internet.
  3. Then I had to remember/relearn how to use rsync (-avx).
  4. My first sync was from the entire root of the old server into a directory on the home partition on the new one. I’ve used this tree to set up the services the way I wanted them. Most services I reconfigured manually rather than using the old config files – partly because I was expecting the newer versions to have different options (which was sort of true with Apache) and partly because I wanted to make sure I’ve done it right the first time (mostly I have).
  5. Don’t underestimate the step above, that was a lot of work. Things I have completely forgotten about such as my aliases.db file and the stunnel config had to be accounted for.
  6. Originally I was going to keep all the keys from the old server, but instead I’ve decided to consolidate the keys and now I have one set for most of the services I use. Yeah, yeah, whatever.
  7. I also needed to migrate my MySQL databases (of which I have a few). It turned out that just copying /var/lib/mysql isn’t enough, so I had to make an SQL dump of the old database and restore it on the new server. That approach had these problems:
    • The old database wasn’t as secure as I liked, it still had the test db in it, and though I’m sure I went through the users thoroughly, I wasn’t sure enough.
    • The dump included the “mysql” database, which had some tables that changed slightly in the newer version. So mysql refused to work properly.
  8. So even after doing a dump, and transferring the dump over to the new server, and importing it, I still needed to run a couple of commands to secure the databases and make MySQL happy.
  9. The second rsync was more complex. Here I had to sync my home directories (lots of static/dynamic data), the SQL, and /var/spool/mail.
  10. And now the magic:

This is the short version, and the interesting part, here’s what I did:

  1. Opened up my router web configuration page in the browser, navigated to the port forwarding page, and changed all the IPs for forwarded services from the old server’s LAN address to the new one’s. But didn’t save the changes yet.
  2. Stopped all the relevant services on the new server (simple script).
  3. Ran the second rsync again, this completed much quicker than the first time because most of the data was unchanged.
  4. Restarted all the relevant services.
  5. Pressed save on my router config.

The trick worked so well I amazed myself :) I happend to be tailing my apache logs on both the old and new server while doing the final steps of the migration, and the second I saved my new port forwarding settings I saw the logs stop on the old server and start on the new one. It was an awesome feeling.

I’m sure I must have said in the past that rsync is a pain in the ass. I don’t necessarily take it back – but I will say I appreciated having such a powerful tool that day.