Nginx came with this HTTP stub status module that you can get the current server status in a HTML page. However it’s not pretty comparing to Lighttpd’s and Apache’s. Moreover, the information it provides is a bit cryptic.

For example, when you browse to the stub status page (please refer to the wiki document linked earlier on how to set it up), all you get is 4 lines of text:

$ curl http://myserver/nginx_status
Active connections: 183
server accepts handled requests
 914384 914384 2725561
Reading: 3 Writing: 2 Waiting: 178

Not very meaningful. Instead, I wrote a small Python script (download here: nginxstats.py (2.3kb)) that does something like this:

$ ./nginxstats.py http://myserver/nginx_status
Conn     Conn/s     Request/s  Read  Write Wait
-------- ---------- ---------- ----- ----- -----
     157       9.57      31.07     0     2   155
     140      10.20      36.13     0     1   139
     147       9.33      33.60     0     4   143
     189      12.60      40.07     1     4   184
     164      13.07      41.53     5     2   157

Basically it fetches the data from the stub status page every 30 seconds (configurable in the script) and then do some calculation to find out how many connections have been established within the time frame, and what’s the average connections/second and requests/second. Similar to tools like vmstat, it basically runs continuously (until interrupted).

Very useful when you are getting slashdotted :)

I have a mystery and I’m not sure how to solve it. My 360 Linode is running several WordPress-based sites using nginx. Everything is great, except I find that, every few days, fastcgi fails and I start getting 502 gateway errors when I try to post a comment or upload an image, etc. Restarting fastcgi fixes the problem.

It looks like a very similar issue I had with Ubuntu 8.04 (and PHP 5.2.4), which was fixed when I migrated to Debian 5 Lenny. It appears that the OP is indeed running the problematic Ubuntu 8.04. Well. I still have no idea what the real issue was (thus do not know how to report a bug), except my PHP FastCGI process will crash multiple times during busy time of the day (but it’s only 2-3 requests/second average).

Before switching to Lenny, I have tested multiple methods to keep my PHP FastCGI processes alive, including running check/restarting script every 5 minutes, etc. However at the end even 5 minutes downtime felt a bit excessive, so I wrote a simple daemon in Python to keep an eye on Nginx’s error log, and restart PHP/FastCGI processes whenever something bad has been detected. The problem here is that the PHP/FastCGI processes actually do not die — Nginx simply cannot communicate with them. So I need to restart PHP whenever Nginx splits a dummy.

I put the code on Linode Pastebin here. Linode has appeared to remove that script from their Pastebin, but you can Download it here, modify some variables in the code (for example the email addresses, start/stop script location, etc), and then run it as root.

What it does:

• It will damonize first.
• Scanning /var/log/nginx/error.log every 2 seconds, using similar algorithm as tail -f.
• If “104: Connection reset by peer” or “111: Connection refused” is detected in the log,
  • Run killall -9 php-cgi
  • Run /etc/init.d/spawn-fcgi-php start to restart the PHP FastCGI processes

It should use minimum memory + CPU/IO (around 3.5MB RSS, and stat calls are pretty fast). At least it keeps me peace at night that I know if my PHP processes are crashing, there’s a guardian to bring them back up. It still runs on my VPS at Linode — even though with Debian 5 I am getting crashes once every 2 weeks instead of 4-5 times a day.

Hopefully it will be useful to some.

So, like any lazy-programmers who do not wish to spend 2 minutes solving a borning problem, I simply spent 2 hours coding up an EveryDNS client API library for Python, so I can add/remove domains/records programatically. This is inspired by the EveryDNS PHP API. It basically scrapes EveryDNS’s web interface.

Downloads

If you just want the library, there’s only a single PY file to download.

• everydnslib.py

Just put it somewhere in your PYTHONPATH. It is developed on Python 2.5 and sorry no guarantee that it will work on any other Python version :)

Usage

Some sample usage of this library:

$ python
Python 2.5.2 (r252:60911, Dec 12 2008, 10:25:58)
Type "help", "copyright", "credits" or "license" for more information.
>>> import everydnslib
>>> edns = everydnslib.EveryDNS(username, password)
>>> edns.list_domains()
[{'did': '...', 'domain': 'example.com', 'type': 'primary'},
 {'did': '...', 'domain': 'foobar.com', 'type': 'primary'},
 ...
 {'did': '...', 'domain': 'testing1.com', 'type': 'webhop', 'webhop': 'http://testing2.com/'}]
>>> edns.list_records('example.com')
[{'rid': '...', 'host': 'example.com', 'type': 'A', 'value': '123.123.123.123', 'mx': '', 'ttl': '43200'},
 {'rid': '...', 'host': 'example.com', 'type': 'MX', 'value': 'mx.example.com', 'mx': '1', 'ttl': '43200'},
 ...
 {'rid': '...', 'host': 'www.example.com', 'type': 'CNAME', 'value': 'example.com', 'mx': '', 'ttl': '7200'}]
>>> edns.add_domain('example2.com')
>>> edns.add_record('example2.com', 'www.example2.com', 'A', '123.123.123.123')
>>> edns.del_records('example.com', rtype='MX')
>>> edns.del_domain('example.com')

I have also added some helper functions to set up Google-related services.

$ python
Python 2.5.2 (r252:60911, Dec 12 2008, 10:25:58)
Type "help", "copyright", "credits" or "license" for more information.
>>> import everydnslib
>>> edns = everydnslib.EveryDNS(username, password)
>>> everydnslib.setup_gmail(edns, 'example.com')
>>> everydnslib.setup_adsense_for_domain(edns, 'example.com', 'pub-1234567890')

Unfortunately EveryDNS still does not have SRV record support so it is impossible to get Google Talk to work on your domain.

Command Shell

As a test case for the API, I have also created a simple shell to access domains/records on your EveryDNS account. Download the following PY file:

• everydnscmd.py

Put the file in the same directory as everydnslib.py. chmod it to executable, and then run it.

$ ./everydnscmd.py username password
[username@everydns] ls
TYPE       DOMAIN
primary   example.com
primary   foobar.org
...
webhop    testing1.com
[username@everydns] ls example.com
           HOST TYPE  VALUE             TTL     MX
    example.com A     123.123.123.123 43200
    example.com MX    mx.example.com  43200      1
...
www.example.com CNAME example.com      7200
[username@everydns] add example2.com
[username@everydns] add example2.com www.example2.com A 123.123.123.123
[username@everydns] del example.com example.com MX
[username@everydns] del example.com
[username@everydns] help

Documented commands (type help ):
========================================
add  del  ls

Undocumented commands:
======================
EOF  exit  help

That’s it!

Hopefully Python coders who use EveryDNS will find this module useful.

Updates 20 June 2009

Thanks to Brian Hartvigsen from EveryDNS for sending me a patch including the following updates:

• Allows listing, adding, deleting dynamic domain’s records
• Remove code that duplicates functionality in the backend
• Fixed regex for domains that are marked as not listing our nameservers
  Nothing in Python API indicates this status currently, but this could be added
• Updated for forthcoming updates. Compatible w/ existing site too
• Webhops had domain/webhop reversed

While this is not an EveryDNS officially supported API, they are trying not to break it so big thanks to them for providing me updates.

Again, the files are hosted on a Subversion server. If you would to send me patches, do

$ svn checkout http://svn.fucoder.com/fucoder/pyeverydns/
$ ...
$ svn diff > unified.diff

And feel free to send that diff to me.

Python is a great language, and there are many great Python web frame-works out there (which is one of its strength but also weakness). However, we do not see heaps of ready-to-install open-source web applications written in Python, unlike PHP. One of the biggest issue is difficulties to deploy — PHP is pretty much everywhere, but many web hosts don’t even support Python.

Here I am going to share about getting your Python-based website/web-application hosted.

Diversity of Gateway Interfaces

One complexity involving developing and deploying Python-based web applications is its diversity of gateway interfaces with web servers. For example:

• CGI — more universally supported by the hosting companies. Slow because of folking Python interpreter processes.
• Fast-CGI — another old technology that are getting more supports from hosting companies, thanks to the popularity of Ruby on Rails. Used by frameworks such as web.py and Pylons.
• SCGI — a light-weight alternative to FastCGI, has a better Apache module, but rarely supported by hosting companies. Frameworks such as Quixote support it.
• Mod_Python — how’s running Python interpreter right inside Apache? Probably the most powerful way to deploy your app, but probably don’t expect having this privilege in a shared hosting environment. Preferred by Django and TurboGears.
• Proxy — making your web-app a long-running process, listening on a specific port for HTTP request, and get your main web server to proxy/forward that request. Preferred by Zope.

As there are probably a few dozen more Python frameworks out there, I think there might be other deployment options available. However we also neeed to note that some frameworks allows multiple gateway interfaces. Django for example, can be deployed as either FastCGI or mod_python. If your framework or application is written onto of WSGI, it should also support whatever interface that also has a WSGI implementation.

Note that I am not saying deployment should govern the Python web framework you use. No — use whatever framework that you feel comfortable with, whatever that fits your way of thinking. Just that your choice of framework might severe limit your hosting options (if you are also limited by budget).

Python Versions and Third-Party Modules

While Python 2.5 has just been released, many shared hosts still support only Python 2.3 (for example running a stable Debian or CentOS). Some frameworks have gone to the length to ensure they can run on older Python versions, but many require at least Python 2.4.

Also while Python claims to be “battery included”, you’ll find not all batteries required to develop a web application is provided by the host. Many of them can be easily downloaded and distributed along side with your own apps, but some of them require extension libraries to be compiled on the host. Lack of C compiler and related include files might make such task impossible.

You’ll also find many packages come in Python eggs and require setuptools to run. Expect a bit of trouble if setuptools is not available on your host.

So, which host is suitable to host my Python application?

You need “root”

The easiest option is to host on your own box, where you are root who can install whatever Python, 3rd party module and web server extension you want. Mod_Python? No problem. apt-get libapache2-mod-python or emerge mod_python, and few minutes later you are ready to deploy. Everything can be done with sudo — no more pleading emails to ask sysadmin of the hosting company to install certain package for you.

Getting a dedicated server or a virtual private server is probably the easiest way to have this privilege. Get a VPS over dedicated if you have lower budget — but you probably still need around $20/month to get a good performing VPS with sufficient memory and enough CPU slices.

Specialised Hosting

Being “root” is not for everyone, and managing a dedicated or virtual private server can be more stressful than prototyping a web application in Django. Sometimes you just want to let someone else handling all these system administration tasks, and you just want somewhere to drop you application in and make it online.

After a bit of googling, there is a surprisingly large number of hosting companies that are specialised in Python web-app hosting. What you should probably do is go to your framework’s home page, and check whether they have listed hosting companies that are friendly to that specific framework. Many of them provide mod_python in your own Apache process, or allow you to run a persistent process and set up proxy rule to forward the requests. For example:

• Django-friendly web hosts
• TurboGears hosting companies
• Zope Solution Providers (some of them are not hosting related)

Googe it also helps, at least you’ll get to see who is the most popular hosting company the is specialised in Python applications. I found WebFaction who has shared hosting plans from $14.50/month with 1 long-running process. Looks like they are pretty Python centric, and have very easy TurboGears deployment.

Shared Hosting

Running Python web applications on shared hosting spells trouble. However if you are on a very low budget, and have plenty of time to kill, then you might as well have hack up a work around and publish a HOWTO on running XYZ framework on some cheap shared hosting.

Personally I have experience with only two shared hosts.

• DreamHost — there are plenty of howto’s on their wiki, explaining how to get Django and Trac to run on their shared hosting plans. Many suggest compiling your own Python.

  That’s right. DreamHost provides access to C/C++ compilers, and allows you to compile your own Python executable. At least it is an option when the Python 2.3/2.4 installed on the server refuse to work. On my server Python 2.4 lacks many modules and C headers to compile extensions with, which renders it little use.

  Despite having access to compilers, getting Python to run properly on DreamHost still requires a bit of patience and sometimes rubber chicken sacrifices. The CGI speed is also very slow as their server is usually overloaded.

• Jumba — they are cPanel based and Python support is pathetic. Actually in their support forum they said explicitly that Python is not supported.

  However Python 2.4 has been installed, but almost no 3rd party module is available — not even MySQLdb. All compilers are locked down so you can’t get your own C modules compiled.

  At the end it is only useful to run sysadmin scripts and nothing else.

If you are thinking about signing up a shared hosting account for your Python applications, make sure you ask whether Python is actively supported. What versions are they running, what extension modules, and very importantly whether there’s a money back guarantee if you can’t get your app working in their environment.

Conclusion

Let me summarise. Python hosting is not as available as PHP, and you need to watch out for the gateway interface your framework uses, Python version and 3rd party module requirements.

While Python encourages “there is only one way”, there are many hosting options.

Best bet is dedicated or VPS. Find a specialised Python hosting company if you do not want to attend sysadmin part of managing a box. Finally, deploying of shared host is hard — make sure you ask a lot of questions before signing up to one.