It started at around 1:40pm AEST (Sydney time). Both of my VPS with VPSLink went offline. One of them, an OpenVZ Link-3, came back 90 minutes later. The other one, a Xen Link-4 hosting this very website, did not come back online until around 6:30pm in the evening. Pretty bad day for HostingFu and around 15 other websites hosted on this VPS.

Bandwidth graph — blank bit are the down-time.

Again, the first thing I did when the VPS went down is to search on Twitter and saw it wasn’t just me who have lost contact with his servers. VPSLink has done a good job to use this offsite media to communicate & report updates regarding the outage. Turns out to be a power failure.

There was actually some complication to bring my 2nd VPS back online. It did come back later that afternoon, but it had some problem and sshd wasn’t running. It turns out that my /dev is completely borked, as udevd wasn’t running. It did not run as I upgraded to Debian Squeeze, and its udev package does not work with para-virtualized kernel which was in very outdated 2.6.18. So I had to use Xen console to get onto my VPS, downgrade the udevd, so I can restart ssh and other affected service. However in order to use Xen console, I had to wait until VPSLink’s console panel to come back up, which wasn’t available until around 6pm…

Hopefully everything is resolved now and we will see a formal explanation from VPSLink soon.

On another note, I also managed to sign up LayerBoom‘s GeoVPS — a Linux KVM based virtual server company with its own customised panel. Very interesting product, and hopefully you’ll see a review here soon.

Update: VPSLink has posted the response to downtime on their forums.

From SliceHost to VPSLink

I have migrated from SliceHost, i.e. the RackspaceCloud, to VPSLink for hosting this blog. See my previous VPSLink review here.

SliceHost has served me very well for the last 3 years. In fact at one stage I had > 450 days of uptime, before one of my Python script crashed the slice by using up too much memory. Performance has been great. Absolutely tops in stability. Highly recommended if you are looking for the “Rackspace of Xen VPS host” (which it literally is).

I am however switching to VPSLink for this site for a few reasons (together with about 10 other sites I run). Because,

• It costs me less. From heavy discount & referral system, I am getting a Xen Link4 for ~$251/12 months ($29.36/month x 12 – 10% coupon code discount – $66 referral bonus), which has 512MB RAM, 20GB disk space & 500GB/month data transfer. It’s actually free to me because of past referral credit.
• Sydney to Seattle is around 20ms less than Sydney to St. Louis
• VPSLink has 32bit templates, which saves memory for my LAMP stack.
• Accessing to all 8 cores of an Xeon E5420 rocks (not that I need it).
• Did I say it’s free?

So due to simple economics I made the change to migrate my sites from SliceHost to VPSLink. Thanks for the service!

From Drupal to WordPress

Another big change that I have made is switching from Drupal to WordPress as the blogging platform.

Man I love Drupal. It is flexible and very customisable. It is a developer’s dream CMS because of its flexibility. It is powering my main hobby site at the moment (which takes way too much of my time) serving over 4 million page views a month.

However what I am doing here is just blogging. I am not running a community, nor trying to organise some structured content. I am just posting things here a few times a month, and the flexibility of Drupal has actually became a curse. Drupal 7 is almost out, but I cannot even upgrade to Drupal 6 without heavily tweaks on my modules.

So I gave up. Might as well run WordPress and keep things simple.

I ended up spending a day developing a converter to migrate all my old posts + comments from Drupal to WordPress and retain all the permalinks. So far so good.

Other Updates

A few other updates around the place.

• Updated serverstats to 0.2. Ended up rewriting the tokenizer to get rid of dependency on libpcre so it uses even less memory.
• Updated phpmonitor.py as Linode cleaned up their pastebin thus removed my code.

Their website and the forums are apparently down right now. VPS control panel is up — but a bit useless as it kept on throwning me error messages. @NeosurgeHosting said that there’s a power outage, but someone suggested that there’s an DDoS attack. Waiting for the official statement when it comes back online…

Updates

• Discussion thread at WebhostingTalk.com

• VPSLink Datacenter Incident 5/18/2009 approx 18:30 PST

  An electrical failure at Seattle City Light impacted our datacenter this evening – multiple hardware nodes (including internal VPSLink services) were brought offline while our system administrators and electrical technicians worked to establish the point of failure and correct the issue.

  So yes the power outage has been confirmed by VPSLink. Summer is coming in Northern Hemisphere and I think power might be an issue soon…

Before I go on, I feel that I might need to make some disclaimer. VPSLink has also been a regular sponsor of this blog since June this year (in case you have not spotted their skyscraper ads on the right), but I will try to keep my review unbiased :) I was going to give their Xen beta program a try any way when it was announced 2 months ago, but was too busy to do so.

The Product

VPSLink launched their Xen VPS hosting product a month ago after one month of public beta testing. You can read about their Xen Virtual Private Server here:

New Features available in XEN:

• Swap space
• Full control of all iptables modules
• Loadable kernel modules (please note you can not run a fully custom kernel)
• Access to remote console for troubleshooting

XEN differs greatly than OpenVZ and there are several factors to keep in mind when selecting a XEN based VPS account.

Then it listed out a few things to consider to choose between Xen and OpenVZ. None of those matters to me anyway, and it failed to explain how Xen and OpenVZ are in fact intrinsically different, especially how memory model differences will turn a perfectly fine application on Xen to instability on OpenVZ, which I sort-of explained here.

The plan price is nowhere near low-end, however they are very cheap consider VPSLink/Spry’s reputation. My test VPS is a “Link-3″, which comes with:

• 10GB disk space
• 300GB bandwidth
• 1 dedicated IP
• 256MB dedicated RAM

It costs between $20.79 – $24.95, depending on the length of your billing cycle.

Signing Up and Setting Up

Signing up is the easy bit — it has all been done for me when I asked Cameron for a test VPS! However from my previous sign up experience (over 17 months ago), the provisioning is instantaneous (if you pass through their automated fraud check). You should be receiving instructions within minutes of payment.

VPSLink uses its internally developed control panel for both OpenVZ and Xen VPS. It is easy to navigate and the layout is pretty much task-driven. The first things I do were logging into control panel, select my new VPS, click on “Manage OS” and then “Install OS”. Ubuntu 7.10 came out around 2 weeks ago, and it was already available on the list of OS images to install. “Why not give the latest Ubuntu a try as well?” so I clicked on Submit and waited.

It took only a few minutes for VPSLink to rebuild the server with a base Ubuntu installation (which is one thing I really like about VPS). An email is sent out containing the root password, and I am in with a root shell in minutes!

Software and Hardware

As it is paravirtualisation and you cannot run your own kernel, VPSLink dictates the kernel version on all the Xen VPS. Here is the uname -a output.

# uname -a
Linux vpslink 2.6.18-8.1.15.el5xen #1 SMP Mon Oct 22 09:33:52 EDT 2007 i686 GNU/Linux

Looks like it is running the kernel from CentOS 5 (or RHEL 5). You can build your own kernel modules but if you are not running CentOS 5, you’ll need to grab the Linux 2.6.18 kernel source and compile your kernel module against that.

Hardware wise it is not that fancy. Basically VPSLink has two classes of servers — dual core single proc for Link-1 to Link-3 customers, and dual core dual proc Xen for Link-4 to Link-6 customers. So what do I get with my Link-3 account?

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz
stepping        : 2
cpu MHz         : 1994.998
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc up pni monitor ds_cpl est tm2 cx16 xtpr lahf_lm
bogomips        : 4989.56

Basically an Intel Core 2 Duo E4400 CPU with 2MB L2 cache running at 2.0Ghz, i.e. “low-end”. Moreover, my VPS has only been assigned to one of the two CPU cores so it is impossible to burst my CPU usage to the full potential of the CPU. A single E4400 core is still plenty of power for a 256MB VPS and my neighbours will be happy to know that my “make -j4″ will not affect the whole node.

Still, while a single E4400 core packs enough power for a low-end VPS (and VPSLink probably keeps the number of domU low), it is nothing to be boasted about. I have no CPU performance problem though over these two weeks, as web-serving VPS usually have their bottleneck in disk IO.

There are plenty of disk IO bench mark programs, but I just use something that is readily available — hdparm.

# hdparm --direct -t /dev/sda1

/dev/sda1:
Timing O_DIRECT disk reads:  486 MB in  3.00 seconds = 161.88 MB/sec

# hdparm -T /dev/sda1

/dev/sda1:
Timing cached reads:   2330 MB in  2.00 seconds = 1165.09 MB/sec

Direct disk read at 161 MB/sec — that’s pretty impressive! Reading from disk cache at 1.1GB/sec — Jeez! It probably has the best disk read speed comparing with my other VPS.

Network

VPSLink is in Seattle, inside Spry’s own data centre. Network performance has been great around the clock, and there has not been any downtime that has been detected (polling at 5 minutes interval).

But you know what is good about hosting in Seattle? Three words — Amazon Web Services. Amazon is also in Seattle, and if you are a heavy user of services like Amazon S3, or want a static IP front-end for your fleet of Amazon EC2 slaves, the latency and bandwidth between your servers and Amazon can make a huge difference.

Check out these benchmarks.

# tcptraceroute s3.amazonaws.com
Selected device eth0, address 209.40.199.168, port 39897 for outgoing packets
Tracing the path to s3.amazonaws.com (207.171.181.225) on TCP port 80 (www), 30 hops max
 1  64.79.219.1  0.671 ms  0.370 ms  0.433 ms
 2  64.79.223.1  0.429 ms  0.458 ms  0.470 ms
 3  ge1-4.cr01.sea02.mzima.net (72.37.232.33)  9.481 ms  0.933 ms  0.466 ms
 4  xe0-1.cr01.sea01.mzima.net (216.193.255.193)  1.470 ms  10.979 ms  0.912 ms
 5  ge-6-2.car3.Seattle1.Level3.net (4.71.152.1)  0.931 ms  0.943 ms  0.996 ms
 6  ae-12-55.car2.Seattle1.Level3.net (4.68.105.131)  0.951 ms  0.965 ms  0.976 ms
 7  * * *
 8  185-33.amazon.com (207.171.185.33)  1.459 ms  0.953 ms  0.973 ms
 9  177-159.amazon.com (207.171.177.159)  0.976 ms  1.449 ms  0.968 ms
10  207-171-181-225.amazon.com (207.171.181.225) [open]  0.982 ms  0.946 ms  0.960 ms

# wget http://s3.amazonaws.com/ec2/assets/aws_console_screencast_1.mov
--06:45:35--  http://s3.amazonaws.com/ec2/assets/aws_console_screencast_1.mov
           => `aws_console_screencast_1.mov'
Resolving s3.amazonaws.com... 207.171.183.113
Connecting to s3.amazonaws.com|207.171.183.113|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11,004,220 (10M) [video/quicktime]

100%[==========================================================================>] 11,004,220    11.36M/s

06:45:36 (11.34 MB/s) - `aws_console_screencast_1.mov' saved [11004220/11004220]

Yes, read that! Less than 1ms latency to Amazon S3, and pulls a 10MB movie file down at 11.36MB/sec! Awesome.

Still with Bugs

Great performance aside, somehow I feel that VPSLink’s Xen VPS is still not 100% ready. There has been a few issues with their console panel and OS images.

• Somehow /etc/fstab wasn’t set up properly on the Ubuntu 7.10 image. It is emptied out, so commands like df and mount do not show the root directory is mounted, thus you can’t find out how much space has been used. It wasn’t an issue with Debian 4 OS image that I’ve tried.

• Getty wasn’t initialised on console for my Ubuntu 7.10 image either, but then console access from control panel does not work for me either (always got connection timeout). The screen cast with CentOS shows it’s working fine so it might just be Ubuntu OS image again.
• Other glitches like hostname is not persistent.

None of that are show stoppers, and VPSLink is actively working on resolving these issues. Hopefully they will be resolved soon.

Conclusion

Xen VPS at VPSLink provides solid performance and great network connectivity especially to Amazon Web Services. The low-end VPS might not get state of the art hardware, but disk IO is certainly not the issue in my own experiment.

Moreover, it has a great community and good support. Recommended if you are looking for a West coast VPS.

Xen and OpenVZ — What’s The Difference?

While both Xen and OpenVZ are open source server virtualisation technology, there exists some big differences between the two. I think potential VPS customers might need to check the applications that need to be hosted to determine which one is the preferable virtualisation technology.

On one hand you have Xen, a para-virtualisation platform that gives you much of the dedicated server behaviour. You run your own instance of Linux kernel, you can load your own kernel modules, you have properly virtualised memory, IO and scheduler, and it’s stable and predictable. On the other hand you have OpenVZ, an operating-system level virtualisation system that is just a thin layer on top of the underlying OS. It is simple to understand, has lower overhead, which usually translates to better performance.

VPSLink is offering both OpenVZ and Xen VPS of similar specs at the same price. For example, I have a Link-3 OpenVZ VPS running CentOS 4.5, and it has 256MB memory, 10GB storage space and 300GB data transfer per month. VPSLink has also provided me a Link-3 Xen VPS for my review running the latest Ubuntu 7.10 ,and it too has 256MB memory, 10GB storage and 300GB data transfer per month. Same price for both — $24.95/month if you pay monthly. Now, which one should I buy?

OpenVZ Memory Model

First of all, when VPSLink’s OpenVZ Link-3 says “256MB guaranteed”, it actually means around 232MB of “privvmpages”, 14MB of “kmemsize” and other miscellaneous resources. When an application calls malloc(), the allocated amount will be added to “privvmpages”. However when “privvmpages” hits the limit, malloc() will fail with a NULL. When the host server ran out of memory, then processes in VE (virtual environment, OpenVZ’s term for a VPS) that exceeded “oomguarpages” will be terminated, although I do not think it will ever apply to VPSLink.

There are a few problems and a few advantages with OpenVZ’s approach to memory management. One of the biggest problem is the amount of memory an application uses and the amount of memory an application allocates is actually different, and the difference can vary a lot depending on the application. Take Java for example, it usually allocates a huge chunk of memory — usually everything it can see the host node has — but it might only use/commit a small fraction of allocated memory. It can usually render a Java program unusable as you will pretty much hit the privvmpages limit straight away. A bit of tweaking on bootstrapping parameters might fix it, but it is definitely not as clean as Xen or a dedicated server. In fact, almost all applications that use internal memory allocator suffer from this issue under OpenVZ.

Then there are issues associated with /proc/meminfo itself. While OpenVZ has already provided a way to virtualise it, “free” command on my OpenVZ VPS at VPSLink still shows the memory size of the host node. It makes some tasks, like running Java or heavy compilation with gcc almost impossible on a small VPS.

The advantage of OpenVZ’s memory model is that it is simple to understand — you are pretty much limited by only privvmpages on a VPSLink OpenVZ VPS. Unlike a dedicated server or Xen, your disk cache and your buffered pages are not counted against your overall memory usage. Therefore on an under-sold OpenVZ system with lots of cache and buffer memory on the host server, it might actually perform better than a similar spec’ed Xen VPS.

Xen Memory Model

Memory model for Xen VPS is much easier to explain. A 256MB Xen VPS is just like a 256MB dedicated server — that segment of memory is reserved for this VPS only, and no other VPS nodes can touch it. Like a real dedicated server, it counts only resident pages, i.e. only the memory blocks are allocated and used.

Moreover, what happens when you run out of memory? You VPS starts to swap. Each VPSLink Xen VPS comes with a swap partition that is twice the size of memory. When your application requires more memory, least-often used pages will be swapped out to make more rooms. Therefore a 256MB Xen VPS actually has 768MB of total memory (256MB RAM + 512MB swap), and believe me, swap space is very useful to handle that sudden spike of demand.

So Xen is always much better than OpenVZ? Not quite. While your 256MB VPS can theoretical use up to 768MB of memory, in reality

1. Kernel, cache, buffer — they all take up memory.
2. Swapping kills performance.

Yes, you can tune the swappiness so you can keep on reducing cached and buffers without touching swap, however performance will suck. On the other hand you can bump up memory usage on an OpenVZ VPS all the way to the privvmpages limit without much degrading of performance, provided the host node still has room to spare. It is a good thing but can also a bad thing, which I will explain later.

Performance vs. Predictability

At the end it comes down to performance and predictability, and my preference has always been with Xen.

While Xen has more overhead thus possibly slower VPS, its out-of-memory behaviour is much more predictable than OpenVZ, and this predictability won me over. As I have already said, that OpenVZ will continue to perform well when its memory usage approaches the limit. However, if privvmpages has been exhausted, the next malloc() will return a NULL pointer, and depending on how the applications handle NULL pointer, they either die gracefully or die with a segmentation fault (usually the later). It is like driving at 100km/hour and then suddenly hits a brick wall. You do not even know that there is a memory shortage issue because everything just sails so smoothly.

On the other hand, when a Xen VPS used up free memory, it will start taking memory from buffers and cached pages. Then it will start swapping. And then it will finally die when the last bit of swap partition gets exhausted. Performance will start to degrade when it started swapping. The load will go up, and the server will get less and less responsive. Your Xen VPS will spend more of its time swapping pages in and out than actually handling tasks. Even if it dies at the end, it will be a very noticeable long struggle than a head-on smash…

… just like a dedicated server!

My preference? Predictable performance. I’ll rather have my sites slowing down to its knees, than having it crash and burn when the memory is exhausted.

What About Burstable Memory in OpenVZ?

“Burstable memory” in OpenVZ is overrated IMHO, as it makes the behaviour of your VPS even less predictable. It is often advised to set privvmpages (burstable amount) at twice the amount as vmguarpages (guaranteed amount) as allocated amount vs. used amount is usually 2:1. However it is not always the case. At work where we had lots of Java development it’s a bit like 5:1, but on my VPSLink OpenVZ VPS where it is mostly Lighttpd, MySQL and PHP, the ratio is about as low as 1.45:1.

Therefore out-of-memory could still happen when you have burstable (privvmpages) set to twice the guaranteed (vmguarpages). On the other hand, VPSLink’s “no burst, no swap” policy, i.e. making burstable amount the same as guaranteed amount, actually gives each VPS less memory to play with, at least it guarantees that when OOM occurs, no VE will be held responsible as all of them will be under their guaranteed limit.

Conclusion

By looking at OpenVZ and Xen’s memory model and their handling of out of memory situation, I am leaning towards Xen, if I am going out to buy a VPS now. Especially when they are sold at the same price per month at VPSLink, I personally cannot see why someone would prefer OpenVZ over Xen.

Except if there is a discount on OpenVZ VPS :)

I have seen a few promotions coupons from VPSLink (last one being 31% discount from Halloween) but none of them can be used on VPSLink’s Xen VPS. I for one will switch from OpenVZ to their Xen VPS in a heart beat, except I am still on the initial 50% discount for my OpenVZ VPS that is just too hard to let go :)

Alright. Enough Xen vs. OpenVZ. I will be reviewing VPSLink’s new Xen VPS hosting in my next post, on what’s good (+ what’s not so good) about them.

The disk array error fiasco at VPSLink, has revealed some dark side of shared servers — or when the server hardware setup is out of your control. How would you know that your hosting company is using RAID 0 with SATA drives that is probably having double the failure rate as the drives are stripped?

Tough luck.

First of all, I am currently a VPSLink customer simply because of the unbeatable pioneer discount (<$13/month for a 256Mb OpenVZ VPS). So far the performance is quite acceptable for its price, and I can rest easy knowing that they are backed by a big hosting company (Spry.com) and not some garage operations.

However, although I’ve acknowledged that VPSLink provides budget VPS hosting, I was quite surprised that RAID 0 was used to provide performance and sacrifice reliability. Further more, the performance was actually not that great (VPS was often starving on IO), showing how many VPS they have packed into one server to bring down the price.

Fortunately, my VPS was not affected by this incident. A few things to think about though.

RAID 0 Stripping on VPS? Bad Idea

IO contention is quite often the bottleneck of VPS. However, I do not think using RAID 0 is a feasible solution to this problem.

With RAID Level 0, you basically split up your file system into lots of tiny slices, and then put alternate slices onto each of the two drives in the RAID set up. The advantage is speed gain in both reading and writing from/to the disk drives, as operations on consecutive blocks can be done in parallel.

However it is also at the expense of reliability. If one drive fails, you pretty much will loose the entire volume, as it is impossible to extract out files when you only have alternate blocks. And believe, disk drives do fail very frequently especially when you have a large sample. Even Google has no exception.

In web hosting world, loosing a drive in RAID 0 usually means extensive down-time — taking the server down, fsck the file system, scavenge the /lost+found, restore from backups, etc — which often translates to loosing customers. In fact, I would not host my websites on a host with less than redundancy provided by RAID 1, where there is always a hot mirror. With RAID 1 mirroring, you get only half the overall capacity. You still get the same reading speed, but writing will be only as fast as with a single drive. But I will rather write my files back slowly, than risking my entire file system fried.

How Storage is Handled at Your Shared Server?

That brings us to the next question — how is the storage handled at your shared server? With a dedicated server provider, it is easy. You tell them how you like it to be done! With my other two VPS hosts, SliceHost and GPLHost, at least it was stated clearly on the website that RAID1 is implemented.

However, there are many VPS and shared hosts out there who never mention their storage arrangement on their website. Some just say “RAID” (in VPSLink’s case). Some just give you a size in number of gigabytes. That alone should make you worried.

Backup, Whose Responsibility?

VPSLink claims to have nightly backups (from their RAID 0 drive array to a separate SATA drive), however they seem to have issues restoring the backups and applies incremental changes.

Sometimes I do wonder, should we rely on hosts to backup our servers for us?

The answer is a big NO. While the host is responsible for performing regular backups, it will be considered stupid not to have your own off-site backups. How are you doing it, it is up to you (I have briefly talked about it previously). However, make sure you do it! An earthquake can hit US east coast and sub-merge the entire Seattle into the Pacific Ocean — and you cannot tell VPSLink/Spry that they need to somehow find some divers to rescue the hard disks containing your data, can you?

And let’s face it. Web hosting is a commodity, despite denial from those in the industry. Not happy with the current host? Can’t get the site back up in time? As long as you keep your own sets of backups, you are free to just pack up and leave. Find another host, upload your files, migrate the DNS, and you are good to go.

Should I Stick with VPSLink?

Well. They are cheap, and I am getting the service I expected. Moreover I prepaid to June so I am not in hurry to leave before then.

However my VPS there is really under utilised — I am only running a few low traffic sites and cron jobs on it. I might as well get rid of it unless there is any new project that I need to off-load onto it. Not likely with my current schedule…

Anyone interested in buying a cheap 256Mb VPS off me that’s on a RAID 0 disk array? :)

So I logged into VPSLink’s control panel, seeing that my VPS is still up. So it is probably a network issue. It wasn’t the first time network outage has occurred since I had this VPS with them. Network was down for about 90 minutes a month ago due to some routing issue. So not surprisingly there’s an outage thread on VPSLink’s forum. Seems I am not the only one that has been affected. That’s comforting, not!

What I have noticed is the lack of communication during these outages. Servers appeared to be down. Routing is not working. Customers submitted tickets and reported onto their forums. People started to speculate what was happening. Furious ones threatened to change host and ask for money back. No one has any idea what’s going on — because there simply lacks information feeding the desperate customers. Sure, there’s an announcement after outage has been resolved, but I believe customers would much prefer up-to-date communication throughout the outage! A simple public acknowledgement stating “we have identified that there is an issue” would be sufficient.

Not that I have important sites there though. Off-line for two hours ain’t going to hurt me — I probably won’t even notice if it is not at this time of the day. However I am sure there are many who have their revenue-generating sites on these VPS, and to those whose livelihood depends on their online presence, “downtime” would be a totally different story. Frequent communication would surely help in these situations.

On the topic of openness and communication during outage, Matt from SliceHost impressed me with this blog post. I have no idea how the communication is taken place, as the outage only lasted 30 minutes. However he was totally open on what was going on and what had been done. I am sure all his customers would appreciate that.

Now, web hosting companies — do utilise your blogs to keep everyone up to date, and forums are not just a place for customers to chat amongst themselves. Most people are fine with occasional outage, especially when they have been acknowledged that these have been taken care of, and they are in safe hands.