It could be the hosting gods punishing me for migrating from SliceHost to VPSLink. Just a week after I wrote that blog post, there is a big outage at VPSLink today.
It started at around 1:40pm AEST (Sydney time). Both of my VPS with VPSLink went offline. One of them, an OpenVZ Link-3, came back 90 minutes later. The other one, a Xen Link-4 hosting this very website, did not come back online until around 6:30pm in the evening. Pretty bad day for HostingFu and around 15 other websites hosted on this VPS.
Sorry about lack of updates here. A lot of happening in life and it has been quite a hectic year. I have also made some significant changes under the skin here at HostingFu. I said “under the skin” because I have actually kept the old theme and look & feel of the site, but…
From SliceHost to VPSLink
I have migrated from SliceHost, i.e. the RackspaceCloud, to VPSLink for hosting this blog. See my previous VPSLink review here.
SliceHost has served me very well for the last 3 years. In fact at one stage I had > 450 days of uptime, before one of my Python script crashed the slice by using up too much memory. Performance has been great. Absolutely tops in stability. Highly recommended if you are looking for the “Rackspace of Xen VPS host” (which it literally is).
Tried to set up a new VPS last night — a Xen VPS with stock kernel 2.6.18. Picked Debian 5 Lenny as operating system, and then upgrade to Squeeze using
apt-get dist-upgrade. Smooth sailing so far. Until I tried to pull some of the toolchains I was trying to build from a remote subversion repository.
For example to pull the latest WordPress.
$ svn co http://core.svn.wordpress.org/trunk/
svn: OPTIONS of 'http://core.svn.wordpress.org/trunk': could not connect to server (http://core.svn.wordpress.org)
Definitely not a network issue as every single Subversion repository over HTTP is returning the same problem. A bit of gooling seems to suggest that neon is to blame (neon is the HTTP/WebDAV client library used by Subversion). Doing strace shows that:
close(4) = 0
socket(PF_INET, 0x80001 /* SOCK_??? */, IPPROTO_TCP) = -1 EINVAL (Invalid argument)
write(2, "svn: OPTIONS of 'http://core.svn."..., 115svn: OPTIONS of 'http://core.svn.wordpress.org/trunk': could not connect to server (http://core.svn.wordpress.org)
) = 115
exit_group(1) = ?
Well. According to
/usr/include/bits/socket.h, 0×80000 is
SOCK_CLOEXEC (Atomically set close-on-exec flag for the new descriptor(s).), which is not supported on Linux kernel older than 2.6.27. Here comes the problem of para-virtualisation and operating system jails — you are still running the virtualised kernel supplied by your vendor. Almost all of my VPS — Xen or OpenVZ — runs on Linux kernel 2.6.18 as it’s the kernel of choice for RHEL 5 where many virtualisation vendors want to support.
Well. Instead of rebuilding subversion with an older neon library, here is a much simpler work-around posted on Debian’s mailing list.
$ echo 'http-library=self' >> ~/.subversion/servers
Done! Subversion should now be working on the older kernels.
Did the following command on one of my Australian VPS this morning.
$ mtr -n --report www.aussiehq.com.au
HOST: my-vps-in-bne Loss% Snt Last Avg Best Wrst StDev
1. 202.60.89.xx 0.0% 10 0.0 0.0 0.0 0.1 0.0
2. 118.127.9.xx 0.0% 10 267.6 262.3 256.3 267.6 4.3
3. 220.127.116.11 0.0% 10 270.8 265.5 258.5 275.2 5.4
4. 18.104.22.168 0.0% 10 274.9 268.7 262.4 274.9 4.1
5. 22.214.171.124 10.0% 10 271.8 268.0 262.8 271.8 3.1
My VPS is hosted with someone who has servers with DedicatedServers.com.au in Brisbane, doing a traceroute to the home page of AussieHQ in Canberra, one of the largest hosting companies in Australia. I believe the first hop is the host node for thise OpenVZ VE, and the second hop would be the router at DedicatedServers.
It’s interesting to observe the fluctuation of latency between the host node and the router throughout the day. It’s usually pretty bad in the morning — all the way up to 300ms, and then the round-trip latency drops to a few milliseconds in the evenings. Massively oversold bandwidth I guess (won’t name the names yet). Even my sites hosted in California feels faster in Australia.
Via Hacker News, here is a blog post from cloudkick — Three sysadmin mistakes start-ups make.
While being a start-up with scaling issues is a sign that things are going well, sometimes such a small team does not have the expertise to make sure all their servers are in order. We wanted to share a couple pitfalls that we have helped diagnose in hopes to prevent other start-ups from doing the same thing.
It pointed out that
- Switching out your Apache with an alternative when you are in trouble is bad.
- Don’t use SQLite in production.
fork(2) is one of the most expensive system calls (for some operating systems), and should not be used on every request.
The emphasis is mine, because I actually do not agree entirely on the argument.
I have been using CouchDB recently to build some of my new projects. It’s a new schema-less, document-modelled, highly concurrent, Erlang powered “database” where you wrote map reduce functions in design views to create queries and indices. It is also very neat to bootstrap a project when you have not worked out the relational model of your database yet, due to its schema-less nature. I am still getting my head around it (coming from 15 years of SQL experience), but so far so good.
I guess one issue with CouchDB is that it’s HTTP-based interface is still not fixed, and there are quite a few differences between versions. Nor is there any upgrade utilities to convert database created in an older version to the current one, although it’s trivial to write one. Debian 5, which is my current distro of choice, is still stuck at 0.8.0 for CouchDB, when most documentation online have moved to 0.9.0. I can’t seem to be able to find ready-made CouchDB-0.9.0 Debian packages for Lenny, so I made my own from Squeeze source.
Looking at my history of past reviews and found that I have not done a hosting review for the last 8 months! Meanwhile, I probably have used half a dozen different hosting providers since my last review (my review on Gandi.net’s Xen VPS, which I will hopefully write about them again sometime soon). So, it’s the last day of August, and let me do a quick review on one of the VPS hosting company that I have been using for the last 3 months and are quite happy with — Fivebean Media.
I first came across Fivebean Media on LowEndBox.com of their $5/month VPS deal (which I do not think is currently on offer). According to the about page, the company started as a “technology/web consulting company” back in 2004 (XlogicGroup, a service/consulting company), and Fivebean Media was created in 2008 to offer web hosting. It is still a relatively young company, and it has a weird name for a hosting company, but I do like their clean design :)
Update November 2009
If you find this review helpful and wish to sign up with VPS products from Fivebean Media, feel free to use my referral code (if there’s no better discount currently available). Here’s my referral link:
You can also quote the coupon code FB20 to get 20% recurring discount — although you can usually find better discount on their website (40-60% off VPS products).
After the FsckVPS fiasco and tragic involving KT Ligesh two months ago, I blogged about the potential future of products of LxLabs. More specifically — whether HyperVM and Kloxo are going to be open sourced. A lot of development has been happening over the last two months. A “consortium” has been created, and a new website LxCenter has been set up (and currently hosted at DMEHosting) to further the discussion. Although most of the talks are still happening at the forums.
I know I have been abandoning this blog for the last month and half due to stress at work (very busy preparing for the new version at the end of this month), and one nut case involving cyber-stalking, threatening emails and abusing phone calls (still on-going but hopefully the justice system will deal with it). Now I am coming back with some spammy blog posts :)
One of my alter ego is running probably one of the biggest bargain sharing sites in Australia (working on it like 1-2 hours a day, hosted with none other than the great Linode serving 2.5-3 million pages a month). One of pages that I recently put it on is…
Cheap .au domain names — list of registrars and resellers offering .com.au for under AUD$30/2 years.