I have a mystery and I’m not sure how to solve it. My 360 Linode is running several WordPress-based sites using nginx. Everything is great, except I find that, every few days, fastcgi fails and I start getting 502 gateway errors when I try to post a comment or upload an image, etc. Restarting fastcgi fixes the problem.

It looks like a very similar issue I had with Ubuntu 8.04 (and PHP 5.2.4), which was fixed when I migrated to Debian 5 Lenny. It appears that the OP is indeed running the problematic Ubuntu 8.04. Well. I still have no idea what the real issue was (thus do not know how to report a bug), except my PHP FastCGI process will crash multiple times during busy time of the day (but it’s only 2-3 requests/second average).

Before switching to Lenny, I have tested multiple methods to keep my PHP FastCGI processes alive, including running check/restarting script every 5 minutes, etc. However at the end even 5 minutes downtime felt a bit excessive, so I wrote a simple daemon in Python to keep an eye on Nginx’s error log, and restart PHP/FastCGI processes whenever something bad has been detected. The problem here is that the PHP/FastCGI processes actually do not die — Nginx simply cannot communicate with them. So I need to restart PHP whenever Nginx splits a dummy.

I put the code on Linode Pastebin here. Linode has appeared to remove that script from their Pastebin, but you can Download it here, modify some variables in the code (for example the email addresses, start/stop script location, etc), and then run it as root.

What it does:

• It will damonize first.
• Scanning /var/log/nginx/error.log every 2 seconds, using similar algorithm as tail -f.
• If “104: Connection reset by peer” or “111: Connection refused” is detected in the log,
  • Run killall -9 php-cgi
  • Run /etc/init.d/spawn-fcgi-php start to restart the PHP FastCGI processes

It should use minimum memory + CPU/IO (around 3.5MB RSS, and stat calls are pretty fast). At least it keeps me peace at night that I know if my PHP processes are crashing, there’s a guardian to bring them back up. It still runs on my VPS at Linode — even though with Debian 5 I am getting crashes once every 2 weeks instead of 4-5 times a day.

Hopefully it will be useful to some.

For example,

fastcgi.server = (
    ".php" => (
        "localhost" => (
            "bin-path"  => "/usr/bin/php-cgi",
            "host" => "127.0.0.1",
            "port" => 1026
        )
    )
)

When lighttpd starts, lighttpd process itself will spawn multiple php-cgi processes, and communicate with them on port 1026 using the FastCGI protocol. Whenever a PHP script needs to be executed, one of spawned php-cgi child will take over the execution.

It is all fine — until the restart time.

There are some issues with lighttpd’s restarting/reloading. First of all, in my experience lighttpd does not always handle reloading properly via the standard SIGHUP. The init script on Gentoo nor implemented “reload” — logrotate script basically relies on the service to restart (Update: logrotate is now using SIGHUP after lighttpd 1.4.11 has been emerged). However, due to the way lighttpd implemented graceful restart, sometimes I found the FastCGI processes (in this case php-cgi) do not get restarted together with lighttpd. I think it is how it happens.

1. Run “/etc/init.d/lighttpd restart“, which sends a SIGINT to the old process, and then fire up a new lighttpd process.
2. Old lighttpd process received SIGINT, unbind the HTTP port to stop accepting new connections, however it cannot be terminated straight away because some of its PHP sub-processes are still finishing up their requests.
3. New lighttpd process started, and bind itself to HTTP/HTTPS port which are now available. However, it cannot spawn the php-cgi FastCGI process, because port 1026 is still occupied by the php-cgi processes from the old lighttpd instance.
4. New, old lighttpd process died after all its php-cgi children have finished their tasks. However the new lighttpd process is still left child-less. All new PHP requests will return 500 Internal Error.

I had that happening this morning, and this VM stopped serving PHP scripts for almost 4 hours, until I restarted lighttpd again! It has actually happened quite a few times before, but being procrastinating in nature, I only managed to “do something about it” today.

The solution, for me at least, is to use the spawn-fcgi script that also gets installed when you emerged lighttpd. It starts php-cgi independently from lighttpd, so that restarting lighttpd process do not have to restart all PHP FastCGI processes.

The configuration file can be found in /etc/conf.d/spawn-fcgi, and its default setting is quite reasonable for PHP. It starts 5 PHP FastCGI instance when it starts, and binds to TCP port 1026. First of all, you need to modify your lighttpd’s mod_fastcgi.conf file so that it would not try to look for local executable. All you need to do is to comment out the line where bin-path is specified. For example,

fastcgi.server = (
    ".php" => (
        "localhost" => (
#           "bin-path"  => "/usr/bin/php-cgi",
            "host" => "127.0.0.1",
            "port" => 1026
        )
    )
)

And then you need to restart lighttpd. spawn-fcgi comes with its own init script, so you’ll also need to do

localhost ~ # /etc/init.d/spawn-fcgi start
 * Starting spawn-fcgi ...        [ ok ]
localhost ~ # rc-update add spawn-fcgi default
 * spawn-fcgi added to runlevel default
 * rc-update complete.
localhost ~ #

Yup — remember to use rc-update to add spawn-fcgi to your start up script.

Now starting/stopping lighttpd would not affect starting and stopping of PHP FastCGI processes.