multi-db: successful move

FYI, I want to say thanks to Andrew, and the rest of you for helping me on my move to multi-db.

One thing that made this process a little easier is that since we run on AWS/EC2, we were actually able to test this process on a separate instance of our install without having to worry about where to get a second box with the same specs. Once we had the new environment working, we were able to flip our public IP to point to the new instance and the upgrade was complete.

We ended up going with the 4096 hash, and we're running a "single" mysql instance for accessing these databases. Actually we run a master/slave pair, we use the slave to do backups from once an hour, so our master stays live the whole time.

Our total maintenance window took just over 3 hours to do this move. The restore/migrate being the biggest chunk of time as it took about 1 hour 15 minutes for all of the migration from a single directory to 4096 directories to occur. Also since we needed to set up the master and the slave data areas, we had some extra time there. The 3 hours included taking it slow and double checking all our steps. We used a pilot/co-pilot mode with myself and another engineer live on the boxes and working together on IM as we executed each step of our deployment plan.

For what it's worth, here's the basic deployment plan we followed:

++++++++++++++++
Pre-Upgrade:
++++++++++++++++
* Make copy scripts
* Get multi-db configs built
* Change backup scripts to handle new db structure
* Fire up new instance of our server config
* set up multi-db config on new instance
* Verify that it’s working

++++++++++++++++
ON OLD MACHINE:
++++++++++++++++
* Display Under Maintenance site
* Shut down mysql
* run backup
* stop crons/backups, etc.

++++++++++++++++
ON NEW MACHINE:
++++++++++++++++
* Verify mysql is not running
* remove existing (test) mysql database data images
* Verify backup is complete
* run restore & migrate
* verify restore is complete
* start up mysql
* Make sure to grant privs to mysql/apache user account - this is an important step since we've added new databases from the migration process
* Verify new site is working
* startup crons
* redirect IP address to new instance