For example to pull the latest WordPress.

$ svn co http://core.svn.wordpress.org/trunk/
svn: OPTIONS of 'http://core.svn.wordpress.org/trunk': could not connect to server (http://core.svn.wordpress.org)

Definitely not a network issue as every single Subversion repository over HTTP is returning the same problem. A bit of gooling seems to suggest that neon is to blame (neon is the HTTP/WebDAV client library used by Subversion). Doing strace shows that:

...
close(4)                                = 0
socket(PF_INET, 0x80001 /* SOCK_??? */, IPPROTO_TCP) = -1 EINVAL (Invalid argument)
write(2, "svn: OPTIONS of 'http://core.svn."..., 115svn: OPTIONS of 'http://core.svn.wordpress.org/trunk': could not connect to server (http://core.svn.wordpress.org)
) = 115
exit_group(1)                           = ?

Well. According to /usr/include/bits/socket.h, 0×80000 is SOCK_CLOEXEC (Atomically set close-on-exec flag for the new descriptor(s).), which is not supported on Linux kernel older than 2.6.27. Here comes the problem of para-virtualisation and operating system jails — you are still running the virtualised kernel supplied by your vendor. Almost all of my VPS — Xen or OpenVZ — runs on Linux kernel 2.6.18 as it’s the kernel of choice for RHEL 5 where many virtualisation vendors want to support.

D’oh.

Well. Instead of rebuilding subversion with an older neon library, here is a much simpler work-around posted on Debian’s mailing list.

$ echo 'http-library=self' >> ~/.subversion/servers

Done! Subversion should now be working on the older kernels.

The same has also been said in this SliceHost vs. Linode comparison — x86_64 simply uses more memory than the plain old x86. So now you not only paying the same for less memory ($20/month gets you 256MB on SliceHost vs. 384MB on Linode), your applications are also using more memory due to its 64 bit architecture — enough to force you to step up the plan. One of my friends has his WordPress website running at SliceHost. It’s a typical LAMB setup with no control panel, but originally I thought a 256MB Slice would suffice. Apparently not, and the amount of swapping due to possibly bad-optimisation and fat 30+MB Apache processes force him to upgrade to a 512MB slice. But 512MB just to run a low-medium WordPress site? That’s pathetic.

I too have similar experience. Two Ubuntu boxes. One 32 bit at VPSLink and one 64 bit at SliceHost. Both running pretty much my standard LAMP stack serving WordPress and Drupal sites (except Nginx instead of Apache). No opcode cache loaded. Minimum number of extensions. A php-cgi process is around 35MB VSZ and 14MB RSS on 32 bit, but 120MB VSZ and 25MB RSS on 64 bit. That means I can run almost twice the number of FastCGI processes, which can be beneficial on a busy site.

I guess for a standard web app stack, unless I really have a specific need for 64 bit, I’ll probably stick to that plain old 32 bit.

The Debian Project is pleased to announce the official release of Debian GNU/Linux version 5.0 (codenamed “Lenny”) after 22 months of constant development. Debian GNU/Linux is a free operating system which supports a total of twelve processor architectures and includes the KDE, GNOME, Xfce, and LXDE desktop environments. It also features compatibility with the FHS v2.3 and software developed for version 3.2 of the LSB.

Great news. While I switched from Gentoo to Ubuntu on production servers last year, recently I have been deploying (and migrating from Ubuntu!) Debian 5 servers and IMHO they are more stable than Ubuntu 8.04 Hardy Heron, which I have been deploying over the last 9 months or so.

As most my sites run PHP and off-the-shelf free/open source software, one of the biggest gripe about Ubuntu 8.04 LTS (which is supposed to have 5 year security updates) is its inclusion of PHP 5.2.4 that has enumerous amount of FastCGI related issues. Most of my sites run on Nginx 0.5/0.6 connecting to PHP backend via FastCGI, and under moderate load (50k-80k page view/day) the PHP FastCGI server will just crash and burn, leaving these messages in Nginx’s error log:

(104: Connection reset by peer) while reading response header from upstream

The only way to recover is by restarting the entire PHP FastCGI backend (after site is down for a couple of minutes). Later on it would occur multiple times a day, and I actually have to write a script monitoring Nginx’s error log to automatically restart PHP FastCGI server to reduce the down time. That sucks — especially when I know PHP 5.2.5 fixed many FastCGI related issues but if I stick to Ubuntu 8.04 LTS I am stuck with PHP 5.2.4 for a long time (unless I build my own PHP which I was not willing to do).

Instead of upgrading to Ubuntu 8.10, I went with Debian 5 Lenny instead. Same command line interface. Same file system layout — but it just feels more stable when it was still marked as “testing” back then. PHP 5.2.6 included was much more solid and FastCGI server hasn’t crashed once. Problem solved!

Now, something I have been doing over the last couple of days is to upgrade my Ubuntu servers to 8.04 Hardy Heron, which was “officially” released last Thursday. Now it has been almost two months since I wrote my last blog post, which was about switching from Gentoo to Ubuntu, and now most servers/VPSs that I am personally responsible for (except those at work) are running Ubuntu. Hardy Heron is a LTS (Long Term Support) release which I am hoping to build most my apps on for the next 2 weeks. Upgrading to it from previous Ubuntu releases is surprisingly trivial.

# apt-get update
# apt-get upgrade
# apt-get install update-manager-core
# do-release-upgrade
[blah blah blah]

The first two steps are only there to ensure you already have latest updates for the current release. It’s quite possible that “update-manager-core” has already been installed. “do-release-upgrade” does all the bulky work — checking whether a new release is available, checking how many packages need to be updated, download, unpackage and install all packages + resolving potential conflicts, etc. And at the end it just reboots your server. Wait for a minute and two, connect back in and hopefully you will be running 8.04 Hardy Heron. I was lucky that it worked on all my Ubuntu boxes.

Do note that the upgrading script, which was written in Python, does chew up quite a lot of memory. I have one tiny 64MB (+256MB swap) VPS that almost got killed with OOM. So be prepared, but YMMV.

So far as a server I haven’t experienced with too much differences. PostgreSQL 8.3 was in but Firebird 2.1 wasn’t (although it should be included “soon“). Now, back to more code hacking.

Over the last 3-4 months I have also gradually moved my Gentoo based servers to either Ubuntu or Debian (prefer the latest Ubuntu if available). In fact I have just deleted my 18 month old Gentoo slice at SliceHost, and moved all content to a new slice running Ubuntu 7.10 last month. Now I am happy to say that all of my live servers/VPS are now running either Ubuntu or Debian, and it has changed my Monday morning (my usual mass-update morning) from:

1. # emerge --sync
2. # emerge -avD world
3. Starring at compilation messages scrolling across the screen.
4. Trying to figure out why some packages are blocking, some packages do not emerge, and why some packages I upgraded last week is now down-grading again.
5. … 20 minutes later I finally got my root prompt back!
6. Restart all services that I have emerged, finger crossed hoping that nothing breaks, otherwise revdep-rebuild while reading special upgrading instruction on PAM, MySQL, or OpenSSL at Gentoo.org.

To:

1. # apt-get update
2. # apt-get upgrade

Upgrading all the packages in the Gentoo Portage system can be very time consuming, and it gets worse when you have quite a few servers to upgrade!

However, I still love my Gentoo and still use it on my desktop and my home server, continuously updated over the last 3-4 years. We still use it at work because of how configurable it is, and how easy it is to write an ebuild script. Portage, IMHO, is still the best thing since slice bread, but unfortunately it is not the best thing for my VPS at slice host. Building takes too long, it is too CPU and IO intensive that I am afraid I am hurting my neighbours’ performance. Moreover, if something breaks my application due to upgrading (far less than uncommon in the Gentoo world), it will take ages to revert back to the previous version (especially heavy builds like MySQL upgrades) — when my service is down!

Great for development boxes, but not so great for production boxes hosting services that people might want to access 24/7.

Ubuntu is constantly improving since the last time I gave it a try. apt-get is a joy to use comparing to yum on CentOS/Fedora. It has almost all the packages I need, and Debian package control files are not that hard to write either. One thing I have not yet tried is dist-upgrade, which is probably even more scary than emerge world. HardyHeron will (hopefully) be released next month so I guess I’ll be able to find out how easy dist-upgrade on a VPS is.

It is great to see hosts taking security seriously (especially those providing SSH access to shared hosting accounts). Looking at my list of hosting accounts:

• SliceHost — not vulnerable as they run Linux 2.6.16.
• Linode — new kernel images created within 24 hours of security alert.
• VPSLink — can’t find any discussion on their forums. My VPS there was running 2.6.9 so it should be secure, but I heard there are servers running newer 2.6.18 kernels from OpenVZ.
• DreamHost — “What security issue? We are still running 2.4 kernels!”
• NearlyFreeSpeech — “What security issue? FreeBSD ftw!”

A bad bad week for Linux for sure. Time to press that panic button.

I am a long time Gentoo Linux man, which is not hard to figure out from reading my posts here. However, recently I am thinking about giving Ubuntu a try. Source-based Linux distributions like Gentoo is great if you like to tinker, and have lots of time on your hands. However when you have multiple servers and VPS to administer, and what you are supposed to do is to focus on software development — managing all those Gentoo boxes can just be too time consuming.

So I was thinking, maybe I should pick up a binary distribution. Debian? Fedora? CentOS? Well, definitely Ubuntu, which is apparently so hot even Mac OS X nerds are migrating.

And I saw earlier this week that Ubuntu is one of two Linux distributions certified to run on SunFire T1000/T2000 servers. Yup. Those cool thread 8 core UltraSparcs, so now we know which Linux distribution is Sun adoring. Let along the rumoured partnership between Oracle and Ubuntu. Feels like RedHat in the late 90′s.

So back to Isabel’s question -

Oracle’s backing would certainly enhance Ubuntu’s exposure, but it’s getting plenty of attention on its own. So my question is, why hasn’t there been more Ubuntu uptake among dedicated server providers?

From her list of dedicated server providers, CentOS seems to be the most commonly supported distribution. I guess CentOS providers a stable platform, well supported by control panel vendors, long term continuous updates — these make them great OS to install on production boxes, run regular yum update and never need to be touched it again.

However it won’t excite a developer with its list of old packages. Python 2.3? PHP 4.3? Apache 2.0.52? Anyone still develops for these things? But they are part of CentOS 4.4 released 2 months ago. I’ve seen many cases where an in-house developed software failed to run after uploaded to the shared hosting environment, because ISP is running one of those rock-solid enterprise Linux distribution that is just way too old. Feeling familiar?

Ubuntu, on the other hand, seems to be always on the cutting edge, providing the latest tools needed by your bleeding edge Web 2.0 applications. 6.10 edgy has just been released, and Python 2.5 is there!

Ubuntu is picking up momentum. The big boys want to flirt with her. Developers love her. Zealots dumped Mac OS X for her. Time for dedicated server providers to offer her more support.

But what is so essential about SSH?

I can’t live without the shell!

First of all, uploading files via FTP sucks. Period.

Plain vanilla FTP is not secure. While many have accused those who telnet around for sending plain text password through the wire, many people are still relying on unencrypted FTP access to their web host. Hosts should at least have FTP over SSL/TLS, and actively encouraging their users to explicitly demand SSL/TLS when FTPing.

Moreover, FTP to upload files is slow. I am not talking about uploading one or two files. Many sites now consist of many files — server-side scripts, images, Javascript, stylesheets, etc. The easiest way to do is by ensuring the live copy of the site is synchronised with your local copy, so that only those files that have been updated will be uploaded.

Personally I have not found an efficient solution that utilises FTP, to ensure my live copy of the site is synchronised with my local copy. All the FTP synchronisers require traversing through directories to move files up and down. It takes ages (and large amount of bandwidth and CPU time wasted) to ensure two copies are synchronised, and it only works one way.

With SSH, you have a choice to use rsync or related friends over SSH tunnel. It only sends out the bits that have changed, very efficiently. Very useful if you have hundreds or thousands of files.

Better still — this is my personal favourite — you can use the Unison File Synchronizer over SSH tunnel. Unlike rsync which will only synchronise one way, unison does two-way sync and will prompt you to resolve conflicts, in case the same file in both live and local copies have been modified since the last sync. Every now and then I do tinker with my live site just to test things out, and unison ensures my changes also get merged into my local copy.

With efficient tools like rsync and unison where you can upload files efficiently and securely over SSH tunnel, I do not think I can go with a web host that offers only plain FTP.

Debugging Live Scripts

Although it is Linux, but there is rarely identical Linux setup across web hosts. Moreover, the web host environment will probably be very different from your local setup, if you are developing on Windows. The differences in setup will sometimes cause your perfectly working server-side scripts to stop working under live hosting environment. And sometimes it is necessary to do your debugging on the live site — although it is not recommended (especially if you have habit of breaking stuff), but sometimes necessary when code-upload-test-debug cycle just takes too long.

SSH is essential, when you need to log onto the live site to inspect things like error_log or PHP logs in real time.

I do not even know how you are supposed to do if you do not have SSH access to the raw error log files. FTPing in the modified code and FTPing the error log file out every time you made a small adjustment? Get real.

Shell = Power

Well, I won’t touch on this one. But remember what the wise one used to say, “with great power comes great responsibility”. Don’t abuse it :)

Who’s got shell?

Plenty of them. But make sure you check out their plan features and ensure SSH is on the table. Of course, if you have a dedicated server or VPS, you not only have SSH access to your box, you also have the root privileged. However, even in shared hosting not all shells are equal.

For example, with two of my shared hosting accounts, DreamHost and Jumba, you get very different experience when you log in. Take Jumba for example, as it is a cPanel account, you get jailshell with limited visibility (although I can still see the full process list — might be a bug on their side). Moreover, development tool is limited. Python 2.3, Perl 5.8.3, Ruby 1.8.2 and there is no access to gcc.

(Note: this kind of jailed limited environment is probably what you’ll expect from most cPanel hosting that provides SSH. Jumba has in fact provided SSH under a great price — AUD$35.40/USD$27.40 per annual shared hosting in my case. I have also heard that they also provide SSH access to their free hosting clients. However they will only offer on request, and you need a static IP for their firewall rules.)

On the other hand, DreamHost, gives me a great shell. You are still under some kind of jail — at least the visibility to other processes are limited, but you do get to see many other directories and files in the FS (remember: great power – great responsibility). They have 3 different versions of Python installed. Moreover, you have gcc at your disposal. Many DreamHost wiki entries actually suggest you to compile your own version of PHP, Python and other tools, in case they are not provided by the Debian install. That certainly has made your hosting account much much more useful.

What if the compiler is not provided? Fortunately you can know very well what kind of architecture your host is on, and if you have access to a local Linux box, provided your package is not too complicated, you can easily compile a static version (or one that depends on only glibc for example) of binary and upload them. Compiling unison is tedious, as you need to compile and install another bugger. So I just upload the binary I used at home into my hosting account, and it just works.

Final Remark

Still looking for a Linux shared hosting? Besides storage, bandwidth, cost, etc, do look for whether Secure Shell access is provided. It makes your hosting account much more useful.

(Note: This blog entry assumes that you have already acquired the skill to operate inside a un*x shell. I am a software developer, and I have been a Linux user since ’95 so it is all very natural to me…)

For me I have always used AllowUsers directive in /etc/ssh/sshd_config to limit the users that can login. In my setup, I have

AllowUsers root@home-IP my-regular-login

It allows root ssh login, but only from my home ADSL connection with static IP address so I can automate backups. Then it also includes a user ID that I regularly use to log into this VPS. If I need to do some system administration, I’ll use either su or sudo once I am inside.

However I found it is also ideal to slow down the attack when the infested host started to brute force the SSH authentication. There are many scripts/user-land daemons that perform monitoring and blocking. However in a resource limited VPS, I prefer to use something that has less demand in memory/CPU usage. IPTables recent module provides a kernel level solution with little overhead.

This is what I have in my iptables rules:

iptables -N SSH_CHECK
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_CHECK
iptables -A SSH_CHECK -m recent --set --name SSH
iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

What it does is:

1. Create a new chain SSH_CHECK, and all incoming SSH connection (TCP port 22) will go into this chain to test the condition.
2. Condition is, for any source IP address there cannot be more than 3 SSH connection attempts within a 60 seconds window.
3. If condition has been met, then all packets from that source IP address will be dropped.
4. That source IP can only connect again if condition is cleared again, i.e. there has been 60 seconds of quiet time.

I found it quite effectively and dramatically reduce bot attacks on SSH port. Still, it is important to remove shell access from users that no longer require it, and choose sensible random password that is difficult to guess.