Web24 — Virtuozzo VPS in Australia

Back in early this year I talked about writing a review on Web24.com.au, which I used to replace a VPS I got from GPLHost, which I terminated last December as I was using too much bandwidth (and was too cheap to pay :) The VPS I got from Web24 was their Silver package, a Virtuzzo Linux VPS running Ubuntu Linux, with 384MB guaranteed memory, over 6+GB of privvmpages (burstable memory), 50GB/month data transfer, and was located in Fujitsu data centre in Melbourne (see their profile on VPSAU). All these for under AUD$50/month — very affordable for my little sites.

Great Network, Great Uptime, Great Customer Service

Other than the Web24-wide connectivity issue back in February, their network has been rock solid. I am getting around 35ms from my home ADSL2+ to my VPS, and have used 30-40GB a month without issue.

My VPS there also had great uptime. It was up for the entire period — almost 120 days without rebooting. It has certainly demonstrated the stability of Virtuozzo. Hardware wise Web24 was using a 2.4Ghz Xeon with 4x physical core and 8GB of physical memory. However, cpulimit of my VPS has been set to 160% so I can only use around 960Mhz of each CPU core (and 4 cores at the same time), which is not bad at all. Web applications are rarely constrained by CPU power anyway.

Customer service wise — top notch, all the way from pre-sale to support. I have sent in a few support tickets and all of them have been answered in timely manner. GPLHost still has a better service though in my opinion because they are not only solving your problems, reporting back their progress, but also following up after the issues have been resolved.

Then There was Performance Issue

The main reason I am moving away from Web24 is performance, or lack of it. As I have said previously that the server CPUs have more than enough power, but my website over there has suffered much degraded performance throughout February and March, due to large amount of I/O wait.

First of all, what am I hosting there? A small-medium sized Drupal-based community site with around 5,000-7,000 visits per day, 800+k page views and 7.5 million hits per month. Lots of customisation and optimisation, and lots of fragment caching. Lighty on the front-end with 5x PHP+xcache/FastCGI processes (which is more than enough). MySQL 5 with a fat key buffer and query_cache_size to optimise on read requests. I am also trying to stay way below the memory limitation just to be on the safe side. In fact I usually use no more than 200MB under privvmpages. And I thought I was safe.

Not really.

Unfortunately I am not able to reproduce all the communications between Web24 support and myself (as I have a bad habit of deleting old emails), but most support issues I raised were performance related.

Swapping Swapping and Constantly Swapping

20 December 2007 evening, my VPS suddenly stopped responding. Yes it still responded to ICMP pings. Yes you can still connect to port 80, but PHP/FastCGI backend won’t load. SSH in took forever, and once I was in I checked the loadavg it went skyrocket. Type in free -m and I was shocked to discover that all the swap space has been exhausted. free, buffers and cached are all on single digit. Note that on Virtuozzo 3 UBC this is from meminfo of the host server, and has nothing to do with my own VPS. The host server was breathlessly swapping to cope with memory shortage. I cannot remember how long it lasted but it did resolve itself at the end, probably with OOM and some process slaughtering. It wasn’t pretty.

So in the middle of my panicking, I fired a support request. It was escalated and a few hours later someone responded saying it was a CPU issue rather than memory issue, and was caused by someone else’s VPS getting compromised. The swap space on the host server then got increased from 2GB to 12GB. I was trying to argue that it WAS a memory/swapping issue, and my processes cannot get enough CPU time because everyone was busy paging in and out! Nope. Web24 denied about it. The problem was with the CPU they said, which was definitely not what vmstat told me.

I/O Wait Issue Got Worse

Performance irregularity continued, and got worse around February and March. During peak hours (which for my site is 9am-11pm AEST), the 15-minute load average might go up to 4-5 for a period of 5-10 minutes every now and then, and my site would completely stall. 15-minute load average doesn’t drop back down below 0.5 during peak hours, and simple commands line ls feels sluggish — a sure sign of I/O wait issue. Swap usage on the host server has been ranging from 4GB-6GB (out of 12GB total swap space), and sometimes some of my key processes got swapped out even though I am only using half the guaranteed memory.

So I posted this question at WebHostingTalk (without mentioning who the provider was), and asked for opinions.

The question I would like to ask is — is it possible to pin-point why the VPS is slow? My apps are obviously not CPU bounded, and as iostat is not really working in Virtuozzo I cannot tell whether it is IO-bounded inside my VPS either. Is it possible to find out whether it is due to excessive paging on the physical server, so I can go to my provider and say, “hey you are overselling and you should not pack that many VE into a physical server”.

And the responses I’ve got:

… when a VPS node is using ANY swap at all that’s your first warning sign that the server is being pushed to its extremes.
– seankoons at Zone.net

My take: It’s vastly oversold…
– TheWiseOne at TekTonic

If 5GB of SWAP is being used, I’d say they are overselling…
– devonblzx at Reseller3k

Well. I guess I got te message. Despite claims that “We do not overcommit on our VPS infrastructure”, looks like Web24 has just jammed too many people onto that host node. Or maybe my hypothesis with UBC-based VPS hosting is true — by providing 6+GB of burstable memory, you are basically inviting everyone to use as much as they want, and OOM won’t kick in until swap space has been exhausted — which would be probably too late. In that case they might not be overselling if they calculated with the amount of guaranteed memory, but poor UBC settings can still make you look bad. Not that I am going to be hosted with another UBC-based VPS provider anyway.

Moved to Linode

As traffic to my website has also been slowly growing, I was facing two choices — move up to the next plan, or move elsewhere (again). With the performance issues at Web24, I do not think moving up to the 512MB guaranteed plan will do any wonder. AUD$50/month is all I am willing to pay anyway, which means I won’t be able to find any virtual servers in Australia that will fit in my budget. With over 85% of traffic coming from Down Under, it makes sense to also host my site in Australia. But when budget becomes a constraint — well I guess I can live with a bit of latency.

As I still kept a Xen VPS at Linode after my review in January (yes, I actually became a customer), it is at Fremont CA which is “close enough” to Australia for me, I decided to make a migration from Web24 to Linode at the end of March.

So one evening I changed the DNS TTL to 15 minutes, put up a maintenance notice on my website, copy all the files across (which was less than 200MB), set everything up at Linode, make sure everything works, and update the DNS records to let them propagate. The next morning — traffic as usual, and everything “just works”. You do feel the lag typing inside a SSH session, but you cannot actually tell much difference with page serving. I guess the poor performance at Web24 sort of cancels out the benefit of low latency.

My community site has now been running for a month now at Linode and the performance has been great. In April it has been 8,000+ visitors/day and used around 62GB of data. Linode Platform Manager shows 4% of average CPU utilisation last month, and my Linode 360 uses little swap and always has 200+MB of (free + buffers + cached). 15 minute loadavg rarely goes above 0.1… 200GB/month of data transfer means it will probably last me a while.

Verdict

A few concluding points:

• Don’t always believe what the support says. Gather your own evidence. Even a vmstat dump can point out roughly where the bottleneck is.

• Xen >> Virtuozzo. Yes I know Matt is advertising here, but personally I still much prefer Xen than Virtuozzo. At least you get a virtualised block device where iostat can tell you how much I/O you are doing.

• When you have high load on a Virtuozzo/OpenVZ VPS, adding more memory have no value by itself (despite many have suggested this at WHT). If you are CPU-bound (not likely) — check whether there’s a cpulimit on your VPS and ask your provider to remove it. If you are IO-bound (very likely), then you can go and get more memory PLUS implement aggressive caching throughout the stack so hopefully there is less load on the database, if you are the trouble maker.

  If you are not the one causing the problem, or maybe optimisation is not your cup of tea, then prepare to jump ship.

• Oversold VPS exists.

Web24 is coloc at Fujitsu data centre in Melbourne, and a quite traceroute check with mtr shows around 20-40% packet loss at Fujitsu data centre.

$ mtr -r www.web24.com.au
HOST: localhost                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.xxx                 0.0%    30    0.3   0.3   0.3   0.7   0.1
   ...
  7. 0.so-0-2-0.XT1.SYD2.ALTER.NE  3.3%    30   19.2  22.4  17.6  73.6  11.2
  8. so-0-2-0.XR2.MEL1.ALTER.NET   0.0%    30   29.0  34.2  28.3 125.5  17.6
  9. 411.AT-6-0-0.GW4.MEL1.ALTER.  0.0%    30   87.0  35.5  29.2 104.3  16.6
 10. fuji865-mel-gw.aspac.custome 13.3%    30   32.9  50.6  31.5 231.3  47.9
 11. v13.csr02.nobp01.fujitsu.com 10.0%    30   33.9  34.0  31.6  38.9   1.8
 12. 216.14.207.85                23.3%    30   35.6  37.0  32.0  50.1   5.0
 13. 203.16.60.201                23.3%    30   34.2  35.3  33.3  38.9   1.4
 14. www.web24.com.au             23.3%    30   35.0  35.3  32.3  43.2   2.5

I have posted the same issue on Whirlpool Forums and so far we have got no response from the provider. Web24.com.au and their HSPcomplete panel are hosted inside the same network thus neither are working at the moment. I can’t even get onto the support page to get the contact details (lucky there is Google cache).

Not happy Jen. Have to wait and see… Maybe it is a sign that I need to go to sleep now :)

Update 9AM AEST 27 Feb

I got a reply back from Web24 support this morning detailing the communication between Web24, Fujitsu and their up stream providers. Great job! Some paraphrased details:

• Packet loss issue identified at 10:15pm on 26 February, and later found out it was an upstream router issue.
• Web24 has been working with Fujitsu and Verizon in identifying and resolving the issue, and by 1:18am 27 February, the main web24 site failed over to US network (very good idea).
• All the traffic are routing through secondary links by 4:45am, and still waiting for feedbacks on primary provider.

Pingdom still reports my site was down until around 8:20am. This is the graph generated from Cacti:

Note that the network does not appear to be down from 10pm onwards because the querying packets are small enough to get through. I guess network connectivity is going to be patchy this morning.