Wasn’t their database compromised, offsite backup erased, and hashed password stolen about two weeks ago? So I headed to WebHostingTalk, and got the iNET’s web page instead with status updates. It’s not looking good.

What data was compromised?
At this point, we know that the hacker compromised and has publicly posted credit card information from our self-service billing system currently used for sticky posts (located at http://myinet.inetinteractive.com). This system was also used for display (banner) advertising in prior to December 2007.

You’ll get a lot more discussion on this matter from other web hosting related forums. Some have been able to download the entire creditcard table with name, card number, CCV number, etc attached. Some have claimed that some number have already been used in China. With hashed-password stolen, it’s already quite possible to run dictionary attack on those foolish enough to reuse passwords across multiple sites (LxLabs’ billing system got compromised for example). With subscribers’ CC details stolen — it must suck to be iNET right now.

On Saturday Dec 20, the hard drive on TPN’s server suddenly died. We are in the process of restoring and re-building all of our sites and will have all of the shows back online asap.

Hard drive crashes — it’s not if but when, and when that actually happens, are you prepare for it? Especially when it keeps every file of your online business, how much down time can you afford to loose, and how much are you willing to pay to reduce the downtime?

If you do not run mission critical applications and have a very low budget (like me), here are some of simple things that you can do that do not cost a lot to implement.

1. Have a Contingency Recovery Plan

Even when you have top of line hardware with redundant power supply and RAID’ed disk arrays, there is still a possibility that you’ll loose all your files due to an accident (natural disasters, security breach, or fat fingered sysadmin typed in rm -rf /). So always having a recovery plan in mind might be a good thing. For those working on Amazon EC2, not having a persistent local storage is something taken from granted, and smart ways to preserve data and to provide fast server recovery sprung naturally. Maybe all sysadmins and website owners should have the same attitude.

It’s a good idea to come up and document a check list of “todo’s” when disaster happens, so you won’t miss out something during the panic.

Cost: $0 (but lots of thinking)

2. Backups — as Frequent as You Can Afford

It’s something that have been emphasised again and again — make sure you have backups! Moreover, NEVER rely on your hosting provider to provide backups for you, if you are on shared hosting or VPS. Because (1) you can never be assured that the exact files you want to be backup has been backed up (2) backups are usually within the same data centre (or even on the same computer God forbids), which is useless if access to that provider has been cut (3) it’s much faster to restore data if you can DIY instead of firing a few support requests.

How frequent is frequent enough? No less than once per day in my case. You definitely do not want to restore from a database that’s more than 2 weeks old. Some backup tools like rdiff-backup and rsnapshot also let you keep a few rolling backups, so you can have something like “daily backups from the last 7 days”.

For me, I use rsync to my DreamHost account’s backup user. Since I already have a DreamHost account, it does not cost me anything extra for backups 50GB or less. There are many alternate rsync backup storage providers. I have personally used BQInternet, they are pretty good at reasonable price and rdiff-backup is supported there.

Finally thing about backups — make sure you check the status regularly and make sure you can restore from backups. You might wish to run something on weekly basis to verify that everything you need has been backed up. Data restoration procedure should also be part of (1) Contingency Recover Plan — no point of keeping regular backups if you can’t restore them. Not just restoring — but restoring the data quickly on new servers in case of disaster, so your downtime is limited.

Cost: 10G — $5/month (BQInternet).

3. Serving Large Media Files from the Cloud

I don’t do podcasting nor video casting because I sounds crap and looks like an idiot in front of a camera. However in the case of The Podcast Network, they ought to serve their podcast MP3′s from those cloud-storage like Amazon S3/CloudFront or Mosso Cloud Files. If you have 100′s of GB of media files, it makes more sense to have them served directly from the cloud, which probably would have replicated your files multiple times already. Instead of restoring them from off-site backups (which can take weeks), your big media files can continue to be served on the clouds even when your main server has been rebuilt from scratch.

That means you can probably get away without backing up those files (which will make backing up/restoring much faster). Or just push them up to two different cloud service providers.

Cost: $0.15/GB/month storage + $0.10i/$0.17o/GB data transfer (Amazon S3).

4. DR Servers — Ready but not Deployed

Paying for one server can be expensive for some people, and having to pay for a live data redundancy server would be unthinkable. Building a DR solution that has two servers always in sync with each other is another complex topic to look at, and it will probably stay unavailable to most amateur webmasters.

However, it is still a good idea to have some providers that can instantly (or very very quickly, depending on how much in panic you are) provision a new servers when you need to execute your recovery plan. That’s where VPS shines — many virtual private server providers can instantly provision a server when you sign up, so your downtime is minimised.

Cost: $0 (but I recommend Linode and SliceHost)

5. Constant Server Monitoring

What we are trying to do here is to quickly re-deploy the backups when you realise that an unrecoverable disaster had happened to your website/server. But how do you know that a site is down? It happened to me before that my site was down for 10+ hours and I have only realised that it was down when one of my users emailed me. You should be the first one to be notified when your site is down, so you can quickly determine the cause and decide whether to execute your recovery plan.

There are many site monitoring services and I have used both Pingdom and Site24x7. Both provides good service. If you have quite a few sites/servers to monitor, then Pingdom might be the cheaper choice.

Cost: $9.95/month (Pingdom).

That’s pretty much the minimum you need to do if you have your business websites online. Any good tips on reducing the downtime cheaply?

However DreamHost is now changing the game. In their August 2008 newsletter, Josh Jones has announced their new feature — 50GB personal backup space for all web hosting users.

A new item “Backup User” has been added to the User menu to let you manage your backup user. The help text reads:

At DreamHost, you may only keep website-related content on your regular users.

You do, however, get one user per account where anything legal may be stored; your Backups User.

This user cannot have any websites pointed to it, nor may you share files via it… it is only to be used as an off-site backup for your personal files.

As such, we keep no backups of files on this account. These are already supposed to be your backups… not your only copy!

(Of course, you should always keep your own copies of all data stored with us.. we make no guarantees!)

Every full DreamHost Hosting plan includes 50GB of backups space!

(Additional usage will be charged at the rate of 10 cents / GB a month: the best backup deal on the net!)

It then let you manage your special backup user on a different server than your web server. Two methods of access — either FTP or SFTP.

What’s Good

A few good points that might tempt myself to use DreamHost’s backup user:

• It comes with your web hosting account, free of charge (if you are already a DreamHost user).

• 50GB free storage is huge, and 10 cent / GB makes it very competitive in pricing. Amazon for example charges 15 cent / GB. BQ Internet, which I currently use to back up all my servers, costs 15 – 50 cents / GB if you use all the space allocated to your plan. Joyent’s Bingo Disk is cheap at 4.25 cent / GB (their $49/year 25GB plan), but WebDAV?

• I don’t know about how well DreamHost runs, but they look like a big cashflow positive hosting company that is not going to all the sudden going out of business (like many Web 2.0′ish online storage companies). I trust that they can make sure their disks have enough space.

• LA servers — I love LA servers because they are fast enough for me (from Australia).

Well. I am almost sold on DreamHost’s backup plans, and I actually plan to use more than just the 50GB provided, because their 10 cent / GB overage is just too cheap. I am currently on $5/month backup plan with BQ Internet and while Scott @ BQ Internet runs a good job there, I occasionally still have access issues and would switch over to DreamHost backup if it supports rsync… Except it doesn’t.

What’s Not So Good

A few issues when I tested out DreamHost backup.

1. Only FTP and SFTP access are provided. FTP is definitely a no-no. SFTP is not too bad — you can mount it using FUSE on Linux or Mac. Not too bad on Windows either. But if you have 50GB of data that needs to be backed up, and are only changing 10MB everyday — rsync is still much preferred.

2. No backup on backup users. I found this is a bit ambiguous. Actually I found it less than assuring. I do hope they have RAID storage that does patrol read to against data loss. The last thing you want during data discovery is finding your backup files are totally corrupted.

Looks like a show stopper for me. Hopefully DreamHost can address these issues because there are actually people who are willing to pay for quality backup storage service.

Update

2008-11-05 — looks like DreamHost is now allowing rsync and scp. Well done!

Just a tip — DreamHost backup service seems to be taking DSA keys but not RSA keys, just in case you are wondering why you cannot log into their backup server using your private key. Also the shell is protected by rssh, allowing only scp, sftp and rsync.

Moreover, the cost of running a home server is in fact less than what Jeremy has calculated. My home server (a 1Ghz Duron + 3 smaller disks) uses less juice, but more importantly, it needs to be running anyway regardless whether I am using it to perform backups or not, as it also provides a few other services. Like, acting as a file server for my home network.

My home file server backup is another matter. Currently all files are sitting on an RAID1, with home directories rsync’ed to another drive on daily basis. But for big media files (my photo archives, home videos, etc), there is no live backup at moment.

I really need to backup those files. My daughter‘s photos and videos are way more valuable than all my websites combined. I need to have them on some storage, somewhere, as long as it’s not at home.

So I decided to gave Amazon’s Simple Storage Service a try, since there are so many good reviews about them.

I basically installed two clients — S3Fox and JungleDisk. One is a Firefox extension that lets me managing S3 buckets and files. The other one is a local WebDAV server that pushes data to S3. However, an evening of attempts later, I decided to give it up. No way I am going to use S3 to backup my data.

• Middle-ware makes things “indeterministic”. I am trying to copy a folder of 1,000 photos to JungleDisk, so it can translate directory structure to S3′s schema, and upload the files for me. However, half way through I am getting a few internal errors from Amazon, but JungleDisk kept on going. Now, is that photo backed up? No sure. Having a middle-man like JungleDisk that translates WebDAV calls to S3′s service API calls does make things a little bit “indeterministic” sometimes.
• Uploading is Slow. From Sydney Australia to Amazon’s servers, uploading seems to max out at around 30Kbytes/sec. That’s slow especially when you are considering backing up photos and videos. So I simply gave up without waiting for photos to upload.
• Extra complexity. For the scripts and apps I have seen so far, everyone is trying to implement a filesystem or a pseudo file system on top of S3′s bucket system. Moreover, between products there is few compatibility. Directory tree uploaded by s3sync.rb cannot be understood by JungleDisk, for example.

I know it is cheap as you only pay for the amount you have used. I know it is cool to use a web services to back up your files. I know Amazon can really take care of my files securely. But there is just too much work for me.

So I ended up backing up my files to my DreamHost account, via scp and rsync. Since I already have a very under-utilising DreamHost account, so I am not really paying another $9.95 per month for an account. Moreover,

• rsync and ssh is faster, much more reliable and easier to understand. They are also easily scriptable.
• Uploading across the Pacific to DreamHost’s server in LA at 95Kbytes/sec — a big improvement over S3!

See my Cacti graph. I think it maxed out my ADSL2+ modem’s upload.

Looks like I have just found myself an offsite backup solution.