1. Acknowledge MySQL issues in the current grid server platform.
2. Talk about their old load balancing system for MySQL, and why it did not work.
3. Discuss their new VPS based MySQL solution that will be switched on in March.

Scaling databases on a shared multi-tenant system has to be one of the most challenging exercise for system administrators, and I am sure the Media Temple would agree. Because of the way web applications are usually written these days, scaling your PHP apps can be trivial. PHP’s share nothing architecture makes scalability native to the language itself, and you can simply add more boxes to take care the load on the web servers.

However, with “share-nothing”, the design is that all states are pushed into the database, and the assumption is that database is scalable. Is the database scalable? That alone is worth a separate discussion article. To cut the story short, you cannot simply add more hardware to scale up a busy database server. It usually requires the application to optimise the way data are queried. Setting up replication. Distribute heavy queries. Partition the dataset. Rewrite big queries into smaller ones in case of MySQL. Etc etc.

With shared hosting, you’ll get even more problems. As a web hosting provider you need to optimise your database to cater for your customers who might install random or home-made apps with badly designed queries.

Media Temple’s old design was pretty interesting — a load balancer for MySQL database clusters. From the description, it looks like it constantly monitor all MySQL servers in the system, and live-migrate busy databases onto servers that are less loaded. However it does not work as hostile DB users can still bring the entire database server to its knees, where anyone else on the same server will suffer.

Media Temple’s new approach is even more interesting — they will put MySQL into “containers”, i.e. a VPS-like environment with restricted CPU and RAM. From the description I think the new MySQL container approach will be even easier to manage than the current load balancing cluster approach.

Each account would have its own MySQL server running inside a VPS. Instead of continuously monitoring the load and move the offending DBs around, they can just rely on the virtualisation software to keep each instance in check. A busy user will only crash his own MySQL container and won’t affect other accounts. Likewise, a light user should always receive guaranteed CPU and IO.

That’s quite revolutionary for shared hosts. Well done! It also demonstrated a great use case for VPS — partition otherwise shared resources.

D’oh

When I looked at the databases and tables, they all had default charset set to latin1, as they were created in MySQL 4.0.x. However I would like them to be ut8. These are the steps I took to fix up my databases.

1. Back up all databases!!! This is very important. Just in case anything goes wrong, at least we can still restore from the database.

   $ mysqldump --opt --allow-keywords --hex-blob --all-databases -qc > backup.sql
   

   Or you can back up on database by database level to make restoring easier.

2. Change the charset of your databases to utf8. If your databases were created prior to MySQL 4.1, the character set will be latin1. Use the command line MySQL client, and issue the following command to change charset:

   mysql> ALTER DATABASE [database name] DEFAULT CHARACTER SET utf8
       -> COLLATE utf8_general_ci;
   

   Or use the following shell script to change character set for all databases:

   for database in `echo 'SHOW DATABASES;' | mysql -s`
   do
       echo "ALTER DATABASE \`$database\` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;"
   done | mysql
   

   Now all database should have default charset as utf8, but tables and fields are still in latin1 format.

3. Dump database in latin1 and restore in utf8. This is the tricker part. As mysqldump is default to utf8, but we want to dump the tables in latin1 because that’s what the character sets were. You can do this on database by database level.

   $ mysqldump --opt --allow-keywords --hex-blob \
   > --default-character-set=latin1 -qc [database name] > database.sql
   

   Now, open database.sql up and changet all tables/fields declaration from latin1 to utf8! A global search and replace might do, or use some sed magic.

   After the edited SQL dump has been saved, just reload them back into the database. Update: Added --default-character-set option just in case MySQL client is not in utf8, which is default for Gentoo (but might not be in other distributions).

   $ mysql --default-character-set=utf8 [database name] < database.sql
   

   And now you have a database that should be utf8 everywhere and works with MySQL 4.1.

Hopefully that works for you as well. I am pretty sure that there are easier ways to achieve it, and I would love to know them as well. If it doesn’t work out right… Hmm. You still got the backup?!

Update 2006-05-25: Robin wrote me his MySQL unicode fixes to his Zope-based application.

Luckily I found out what the main problem was. Setting “default-character-set=latin1” in the [client] section of /etc/mysql/my.cnf made sure all my legacy apps that don’t support unicode get latin1 data to them. Seems like a lot of PHP applications assume the data stream is in latin1. The default-character-set solution did the trick for b2Evolution, phpBB, Mambo and a lot of other applications. For Zope it wasn’t that easy, because MySQLdb supported unicode and the latest version of ZMySQLDA didn’t. Found a patch for ZMySQLDA in the mailmanager repository on SourceForge that added unicode support to it. After installing that, activating unicode support in all the connections in the Zope admin interface I was back on track.

The good thing is now I can migrate my applications over to Unicode/UTF8 in an orderly fashion as soon as the applications get native support for Unicode directly with MySQL.

Thanks for the tip, Robin!