Bye Bye VPSLink

I migrated HostingFu from SliceHost to VPSLink last November, however HostingFu has actually been running on a Linode (aff link) in Dallas for a few months now, after the sale of VPSLink/Spry to Endurance International. I previously had pretty good review on their Xen VPS, and had 2 VPS with them — one OpenVZ and one Xen. Cameron and Dan have been great to work with, and they ran a great setup in Seattle.

However the sale ruined it. Migration of all servers to Boston is not well executed. After all I do not think I can recommend them any more. I used to have $1000+ of referral credit under my account but it disappeared into nowhere after the transition. My 8-core Link-4 Xen VPS now only can access one CPU core. So I decided not to renew, and a sample of their dying forums is also full of similar stories of people migrating elsewhere. It’s sad to see how a once great company has fallen.

Hello Linode

Well. I am no stranger to Linode — I became a customer when I first reviewed them almost 3 years ago, and had hosted my biggest site with them — until this week but more about that later. So when I decided to move HostingFu away from VPSLink, and was too busy to find a new host, I turned to Linode. I used that opportunity to write an article on how to deploy a Linode with their API, and that newly deployed node ended up running HostingFu.

It had been smooth sailing since May (as expected), and Linode even gave a free RAM upgrade back in June (a bit unexpected). $19.95/month for a Xen VPS running Ubuntu 10.04 with 512MB of memory, 16GB of disk space and 200GB/month of data — not the cheapest but probably no competitor in stability and flexibility at this price point. It’s now also running all my git repositories and a private Trac instance. Two thumbs up.

I also have a pair of Linode in Fremont running a busy Drupal site — and the web node (Linode720, Debian 5) is having uptime of 319 days. On September 12 my database node in Fremont went down (Linode1024, Ubuntu 10.04). By the time I noticed my Pingdom alert, Danny Ariti from Linode (their Australian support guy) has already created a support ticket and kept me informed throughout the 1 hour down time. Top quality support I’ll say. Linode rocks.

Bye Bye Linode, and Hello Crucial Paradigm

Then last week I made an important decision to move my busy Drupal site from the pair of Linode in Fremont to Crucial Paradigm in Sydney Australia. I have been thinking of moving that site for quite a while because it does not make sense not to, as 95% of visitors are coming from Australia. Why then should I deprive them with 200ms of latency across the Pacific Ocean? I went with Crucial Paradigm because I have reviewed them before — again due to my lack of time to do more research so I have to trust whom I have previously worked with.

There were some big disadvantages moving from Linode to Crucial Paradigm though. First of all it’s the cost. A pair of Linode1024 costs USD$80/month with 800GB of data, and that’s $81.63 AUD at the moment. On the other hand I am paying a lot more — AUD$400/month for one single VPS in Australia. It has better spec (6x E5620 cores, 6GB memory, 300GB storage — way more than I need), but lacking in data (600GB/month, after Aaron W generously doubled that for my package).

While Crucial Paradigm is also a great company, it still does not have the flexibility Linode gives me. Because my requirement of Ubuntu 10.04 they have to put me on a Citrix XenServer node (rather than HyperVM node). Which means rebooting need tickets, and I don’t get OOB console either. Obviously I do not have the flexibility to quickly deploy a new node to test something and decommission it (and costs me only the time it’s up). Private IP? API access? Rescue mode? Backup service?

Well, I’ll see how it goes. I am hoping a better experience for my visitors would offset the deficiencies of moving away from Linode. Hopefully.

I am also using Memcached for various caching, and to share variables between my PHP 5 Drupal setup and my Tornado server. Move from Debian 5 to Ubuntu 10.04 means moving from Memcached 1.2.2 to 1.4.2, which supports new UDP and binary protocol. Right after turning the service back on I am getting errors such as this every 1-2 seconds:

MemcachePool::get(): Server localhost (tcp 11211, udp 0) failed with: Network timeout (0) in /var/www/my-script.php on line 123

New server is running Ubuntu 10.04 x86_64 on Xen HVM, with 6GB of memory and access to 6 cores of Xeon E5620, which should be significantly faster than the host I migrated from. I could not figure out the actual cause (too many opinions and “fixes” from Google that do not work). However it turns out with PHP memcache module, using pconnect() rather than connect() almost eliminate the issue.

  $memcache = new Memcache;
- $memcache->connect('localhost', 11211) or die('Cannot connect to memcached');
+ $memcache->pconnect('localhost', 11211) or die('Cannot connect to memcached');

It’s not completely eliminating the issue as I am still getting around one time out every 3-4 hours. It’s not heavy usage either — only around 50 get/set commands per second.

You can get the documentation for both storage (Swift) and computing Nova). Very Pythonic. According to Hacker News, Swift is pretty much production ready as it’s the code behind Rackspace CloudFiles. I wonder whether Nova actually resembles CloudServer code, which would be related to SliceHost’s manager/console?

Well. I am doing it a bit differently today, and decided to do it via API. I went with Ataraxia Consulting’s Python binding as it’s a language I am fluent in. These are the steps that I took.

Downloading Python Binding

The Python binding is hosted with git so I basically just clone it. It comes with a nice shell.py that initialises and turns it into a Python shell with tab completion. You will also need your API Key ready, which can be found on your Linode profile page (generate one if it does not yet exist).

$ git clone http://git.atxconsulting.com/linode
Initialized empty Git repository in linode/.git/
got 7424e15c260d1150025e9909a23bfce698e6c592
...
...
$ cd linode/
$ ./shell.py
Enter API Key: (enter in your Linode API key here)
Python 2.6.4 (r264:75706, Apr 27 2010, 11:22:46)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(LinodeConsole)
>>>

Okay. Let’s get rocking!

Creating New Linode

What I am trying to do is to create a new Linode 360 in their Dallas location (always check Linode’s availability though, although available Linodes have been abundant lately). Since I am keeping it long term, I want to prepay for 12 months. So the API I really want to call is linode.create. Under the shell I will get the data centre ID and plan ID via API.

>>> [dc['DATACENTERID'] for dc in linode.avail_datacenters()
...   if 'Dallas' in dc['LOCATION']]
[2]
>>> [p['PLANID'] for p in linode.avail_linodeplans() if '360' in p['LABEL']]
[1]
>>> linode.linode_create(DatacenterID=2, PlanID=1, PaymentTerm=12)
{u'LinodeID': 12345}

Then to rename it to something else (instead of linode12345).

>>> linode.linode_update(LinodeID=12345, Label='MyNewVPS')
{u'LinodeID': 12345}

Creating Disks

I can’t seem to be able to find an API equivalent to the Deploy a Linux Distribution functionality on the control panel, so I guess I have to go through the long way by creating disks, creating configs, etc before I can boot the Linode. I want to create a Linode with 256MB of swap, and then everything else goes into one partition. I also need to find out the Distribution ID for the 32bit Ubuntu 10.04.

>>> [d['DISTRIBUTIONID'] for d in linode.avail_distributions()
... if ('Ubuntu 10.04' in d['LABEL']) and (not d['IS64BIT'])]
[64]
>>> linode.linode_disk_createfromdistribution(
...   LinodeID=12345, DistributionID=64, Label='Root Partition',
...   Size=(16384-256), rootPass='passw0rd', 
...   rootSSHKey=open('../.ssh/id_dsa.pub').read())
{u'DiskID': 12345, u'JobID': 12345}
>>> linode.linode_disk_create(LinodeID=47347, Label='Swap Partition',
...   Type='swap', Size=256)
{u'DiskID': 54321, u'JobID': 54321}

You have to note down the actual DiskID because we will need them later when we create the config. Also note that I am using a random password + my SSH public key when I set up the root partition.

Create Config

The next task is to create a boot configuration, which needs to include (1) the kernel to boot with (2) the disk arrangement. We need to use the latest 32bit paravirt kernel.

>>> [_['KERNELID'] for _ in linode.avail_kernels()
... if ('Latest 2.6 Paravirt' in _['LABEL']) and ('x86_64' not in _['LABEL'])]
[110]
>>> linode.linode_config_create(LinodeID=12345, KernelID=110,
... Label='Ubuntu 10.04 Default', DiskList='12345,54321')
{u'ConfigID': 12345}

Now the configuration is done — ready to boot!

Booting the Node

Booting a linode using a existing config is pretty trivial. You would also want to get the IP address of your new Linode so you can SSH in.

>>> linode.linode_boot(LinodeID=12345, ConfigID=12345)
{u'JobID': 54321}
>>> linode.linode_ip_list(LinodeID=12345)[0]['IPADDRESS']
u'69.164.xxx.yyy'

Done! Now log off the Python console, and SSH into your new Ubuntu 10.04 “Lucid Lynx” Linode :)

>>> exit()
$ ssh root@69.164.xxx.yyy
Linux li109-129 2.6.32.12-linode25 #1 SMP Wed Apr 28 19:25:11 UTC 2010 i686 GNU/Linux
Ubuntu 10.04 LTS

Welcome to Ubuntu!
 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@linode12345:~#

/me pat on my back. Ought to put all that into a single script…

3) Will the service remain free?

While we don’t 100% have the answer to that yet, we will not be making any changes to the service you are currently receiving for the foreseeable future. We will be discontinuing signups in the near future but existing accounts will remain active and fully functional.

Looks like the reality is slightly different. Just got another email from Dyn Inc with regarding to the transition.

First off, anyone who has donated to David and EveryDNS since 2001 will be grandfathered into free Custom DNS hosting with DynDNS.com…

So not everyone will be migrated but those who donated. Well I did donate and got unlimited domains + unlimited records from EveryDNS (although I only have ~10 domains there). The Custom DNS hosting offer is valued at $30/year so that’s the saving a previous donator. However that’s only for 1 zone.

I have to say that DynDNS has pretty good reputation in hosted DNS space, and having 100% DNS availability is arguable even more important than your having your web servers online all the time. Still a hefty price to pay though. With EditDNS going to pay model, EveryDNS migrating to DynDNS and lots of issues with ZoneEdit. Which other free DNS provider would you put your trust on?

Basically, my conclusion about looking at an OpenVZ hosting plan is — just look at the burstable memory and (most of the time) you don’t have to worry about the guaranteed memory.

I know it’s a pretty bold statement because in almost every thread people asking about memory usage on WHT, the suggestion has always been just focusing on the guaranteed memory and you should not rely on the burstable memory. I think those advices came from misunderstanding of how OpenVZ works. After all, you may just call “guaranteed memory” and “burstable memory” marketing terms for virtual servers. In real OpenVZ terms, these are roughly mapped to oomguarpages (for guaranteed memory) and privvmpages (for burstable memory) in user beancounters.

According to OpenVZ wiki, the so called “guaranteed memory” –

oomguarpages

The guaranteed amount of memory for the case the memory is “over-booked” (out-of-memory kill guarantee).

Oomguarpages parameter accounts the total amount of memory and swap space used by the processes of a particular container.

If the current usage of memory and swap space (the value of oomguarpages) plus the amount of used kernel memory (kmemsize) and socket buffers is below the barrier, processes in this container are guaranteed not to be killed in out-of-memory situations.

Emphasis are mine. Note that what is guaranteed is not as simple as “your portion of physical memory” like how a Xen VPS with a guaranteed memory. What is guaranteed is here that your virtual environment (VE) will not have random processes killed, if your VE’s committed memory (physical + swap) is less than your oooguarpages barrier, when the hardware node (HN) is in out-of-memory (OOM) situation. That means

• You can use as much memory as possible, as long as the hardware node does not run out of memory. However, often you will find that your VE is still limited by the maximum allowed allocation amount, i.e. privvmpages parameter, i.e. the so-called “burstable memory”.

• You run the risk of having your processes killed when the HN has exhausted its both its physical memory and swap, if your committed memory is greater than oomguarpages (guaranteed memory).

• Well, for HN to run out all its memory it needs to use up all its swap. If a hardware node has 2 part of swap memory for every part of physical member, it means by the time OOM occurs, 2/3 of your process pages are deep in the swap. And if you are actually using your VPS to host websites, your processes would already been swapping like crazy by then. A quick death (killed by OOM) or dragging painful one (constantly swapping)? I’ll take the former :)

Seriously. There is no point for providers to advertise “guaranteed memory”. By the time it is actually used, during an OOM event, the hardware node is already way oversold. Providers should just use the “burstable memory” in their plan pages as that’s about the only meaningful way to cap a VE’s memory usage. Or just get rid of the word “burstable” as it just introduces more confusion.

And as it was recorded on the Hosting Gospel,

¹There were some present at that very time who told him about Coding Horror and Blog.StackOverflow’s 100% data lost. ²And he answered them, “Do you think that Jeff Atwood was a worse sinner than all the bloggers and web masters, because his blogs suffered in this way? ³ No, I tell you; but unless you implement a real backup strategy and verify it regularly, your websites will all likewise perish.

Yup. Sorry for mis-quoting Luke 13, but whenever I read about this kind of disasters, let it be a reminder that I should also make sure my back up works.

Summarizing the benchmarks gives us one clear winner: Linode. 32-bit gave the best results on the Unixbench runs while 64-bit was fastest on the Django and database tests. Since Linode also has the highest included bandwidth I have a hard time recommending any of the other providers if performance and price is most important for you.

It has been posted on various startup/programming community sites (Hacker News, Reddit, etc) and got a mention on Linode’s blog. A small thread has been started on Linode Forums on migrating from SliceHost to Linode, when SliceHost, Linode’s direct competitor, appears to perform worse and costs more in the performance comparison.

I have been a Linode customer for almost 2 years and they are hosting my main community site. Two thumbs up for them in terms of performance and stability. I have been a SliceHost customer for more than 3 years until I migrated out last month. Two thumbs up for them as well over stability and customer support. Performance wise it was not as good as my slice was running an old Opteron. Nor was the price — although I have been happy to pay extra for the “premium” until recently. I’ll say I am happy with both providers. However when you put them side-by-side to choose only one out of the two for a new project, it’s hard to go pass Linode.

Many on SliceHost certainly feel the same.

And may I say the forums feel quite a bit different from the good ol’ startup days when Matt and Jason would participate in many discussions. Anyway, the above forum thread was pretty one sided until Paul (PickledOnion) joined the discussion a few days later. Interesting to read through the comments from SliceHost, especially those toward their “competitors” (although it’s clear that some are very specifically toward Linode). A few good points:

• Major announcement in February 2010. Although no suggestion on price drop or size up.
• The uniqueness of SliceHost is (1) service (2) fast deployment and instant scaling.

Although by stating that “Want to spin up a dozen large Slices with us? … Need another dozen on Monday?” and “Want world class support while your Facebook app is experiencing 10,000% growth at 4am on Christmas Day?” the argument might not be convincing to either general public or developer community. Sure I want to have 100x traffic on my Facebook app — in the same way that I want my stocks to have 100x return by Christmas. It might happen to some, but definitely not the majority of the users. The growth of my websites have all been pretty steady and predictable (possibly because I have not done much marketing :) So is SliceHost saying that they are not the best choice for apps with steady growth? Many just want to have a stable platform to run a few WordPress sites — at least it was my case when I was with SliceHost.

Maybe we will have to wait until February to find out more.

Well. Let’s compare them side by side, from my perspective:

Well. Please Google put a resolver somewhere in Sydney! Otherwise a local cache + forward is still preferred. But for now running a cheap virtual server with a badly configured resolver from the provider, I am more likely to jump on Google Public DNS because

• It’s just much easier to remember 8.8.8.8 than 208.what?.
• NXDOMAIN works by default — there is no need for me to log into OpenDNS to set up subnet rules under my account.

OpenDNS does have one advantage for developers though — CacheCheck, which allows you to request the cache to be flushed. Very useful when you have just changed some records, and would like to see that applied to the whole OpenDNS cluster. Google on the other hand gives NIL functionality except something listening on port 53.

For enterprise users it could be a different story though. Having ability to fine tune the behavuour of NXDOMAIN handling, blocking certain domains, phishing/malware/botnet protection, etc — these would be much more useful for an organisation. Will Google gradually roll out similar tools? No idea — just like we have no idea that Google is entering into the public resolver market.

Let’s wait & see.

About GeoVPS and Layerboom

Initially Howard & team have planned to develop just a VPS hosting company with an awesome control panel, and have servers hosted in Canada, to provide alternate to US-based virtual servers but want to benefit from being under a different jurisdiction. However exactly is having servers in Canada better than in US I do not know (IANAL), but I’ve been told that it’s better in some circumstances :)

However it turns out that it is more than just Linode or SliceHost under the maple leaves, but instead they ended up developed a solution for the VPS providers, and the company is called Layerboom, which according to Peer1′s CloudXcelerator Cloud Program, is:

Layerboom provides hosting companies with a comprehensive solution which enables them to build and sell virtual private server clouds. After installing Layerboom software, hosting companies can manage their physical and virtual server inventory, customer accounts, define virtual machine sizes, packages, and pricing, as well as customise our hosted dashboard to maintain consistent branding.

Sounds like installable version of HyperVM + fully integrated WHMCS, but in a single beautiful package.

On the other hand, GeoVPS, a subsidiary of Layerboom, is the live demonstration of Layerboom software’s capability, in an actual VPS hosting provider. The dedicated servers are with ServerBeach in San Antonio in Texas, with future plan to provide Canadian VPS.

If you are a hosting provider looking at a full suite control and deployment tool, you can read about how Layerboom works on their website. However let us shift our focus back to GeoVPS which I will be reviewing.

Plans, Pricing & Ordering

One major benefit of virtual servers is reducing cost, so let’s start looking at GeoVPS from its VPS plans & their pricing. You can find out about plans on their pricing page.

3 plans basically, from 256MB to 1GB of dedicated memory, from $20/month to $80/month. First of all, these are Canadian dollars which is around USD$0.95 at the moment, although it could get worse if USD continues to depreciates. As GeoVPS has also purchased a display ad here at HostingFu, you can purchase & test out the 256MB VPS plan for $10 using promo code hostingfu.

In some sense it is even dearer than SliceHost when you go up the plan (USD$70/month for exactly the same spec Xen VPS from SliceHost), and SliceHost has also been criticised for their stale pricing from 3 years ago! GeoVPS is definitely not competing with the budget VPS providers. SliceHost is the (literally) the Rackspace of VPS market, and while GeoVPS sits on ServerBeach and Peer1, it might be too young and unproven to demand such premium.

Moreover, the plans peaked at 1GB RAM and 400GB/month data transfer — which might not be enough for your database server. I am sure GeoVPS would be able to work out packages for customers demanding bigger slice and more traffic, but it would be nice to be assured from having them available on the plan/pricing page.

Ordering was easy — although I did encounter some bugs on the ordering page. Fortunately Howard was on IM when I attempted the order so he managed to get the developers to fix the issue. Only credit card is accepted — no PayPal, Google Checkout, etc.

After the VPS has been paid for, a welcome email containing IP address & root password is emailed in within minutes. My very first KVM-based VPS :)

GeoManager, The Control Panel

Before digging into the actual server, here are some screenshots of the control panel. Dashboard — first page after you log in:

Click to get the full image. As you can see it list out some essential info & a list of virtual servers you have. Here is a screenshot after you click through one of the virtual servers:

It provides graph on CPU, bandwidth and disk IO usage, plus some operations you can do to the VM, like rebooting, rebuilding, change root password, etc.

Very simple control panel (compare to what Linode offers), but has every essentials that you need to manage a server. Currently the selection of Linux distributions is quite limited (CentOS 5.3, Debian 5.0, Ubuntu 8.04 and Ubuntu 9.04) so those who are looking for Gentoo or Slackware builds might be disappointed.

Server Setups

As I said before, I have no prior experience with a KVM VPS. All my previous virtual servers are either Xen, OpenVZ or Virtuozzo (in the order of preference). I know Linux KVM is going to be big as it is already in the kernel and RedHat is backing it (well, really depends on which vendor do you ask), it is still a relatively immature product in comparison with Xen. It however, does full virtualisation like VMWare, whereas most Xen VPS providers do para-virtualisation.

That is indeed the first thing I noticed when I logged into the system, a 64 bit Debian 5.0 Lenny build — it is running stock Debian kernel.

$ uname -a
Linux hostingfu 2.6.26-2-amd64 #1 SMP Wed Aug 19 22:33:18 UTC 2009 x86_64 GNU/Linux

Another weirdness is the network interface. While the VPS has a public IP address at 69.172.xxx.yyy, eth0 is bound to a private IP address.

$ /sbin/ifconfig eth0
eth0      Link encap:Ethernet   HWaddr 54:00:01:f5:55:e1
          inet addr:10.1.mm.nn  Bcast:10.1.255.255  Mask:255.255.0.0
          inet6 addr: fe80::5600:1ff:fef5:55e1/64 Scope:Link
          ...

Not good for the moment when I am on the VPS and can’t remember what’s my public IP :) Root directory is mounted with ext3, and the VPS has access to 4 cores of 2.5GHz “QEMU Virtual CPU”.

While it is a standard Debian build, Layerboom did add a few startup tricks to /etc/rc.local that:

                     INDEX VALUES
TEST                                        BASELINE     RESULT      INDEX

Dhrystone 2 using register variables        376783.7 20956179.5      556.2
Double-Precision Whetstone                      83.1     1796.7      216.2
Execl Throughput                               188.3     1563.5       83.0
File Copy 1024 bufsize 2000 maxblocks         2672.0   158404.0      592.8
File Copy 256 bufsize 500 maxblocks           1077.0    39005.0      362.2
File Read 4096 bufsize 8000 maxblocks        15382.0  1549344.0     1007.2
Pipe-based Context Switching                 15448.6   401468.5      259.9
Pipe Throughput                             111814.6  5401051.4      483.0
Process Creation                               569.3     2675.3       47.0
Shell Scripts (8 concurrent)                    44.8      561.6      125.4
System Call Overhead                        114433.5  8261016.9      721.9
                                                                 =========
     FINAL SCORE                                                     288.5