9A few days ago I spotted this forum post on Linode Forum — someone’s Nginx to PHP FastCGI set up kept on throwing 502 Gateway Error after a few days.
I have a mystery and I’m not sure how to solve it. My 360 Linode is running several WordPress-based sites using nginx. Everything is great, except I find that, every few days, fastcgi fails and I start getting 502 gateway errors when I try to post a comment or upload an image, etc. Restarting fastcgi fixes the problem.
It looks like a very similar issue I had with Ubuntu 8.04 (and PHP 5.2.4), which was fixed when I migrated to Debian 5 Lenny. It appears that the OP is indeed running the problematic Ubuntu 8.04. Well. I still have no idea what the real issue was (thus do not know how to report a bug), except my PHP FastCGI process will crash multiple times during busy time of the day (but it’s only 2-3 requests/second average).
Before switching to Lenny, I have tested multiple methods to keep my PHP FastCGI processes alive, including running check/restarting script every 5 minutes, etc. However at the end even 5 minutes downtime felt a bit excessive, so I wrote a simple daemon in Python to keep an eye on Nginx’s error log, and restart PHP/FastCGI processes whenever something bad has been detected. The problem here is that the PHP/FastCGI processes actually do not die — Nginx simply cannot communicate with them. So I need to restart PHP whenever Nginx splits a dummy.
I put the code on Linode Pastebin here. Linode has appeared to remove that script from their Pastebin, but you can Download it here, modify some variables in the code (for example the email addresses, start/stop script location, etc), and then run it as root.
What it does:
- It will damonize first.
- Scanning
/var/log/nginx/error.logevery 2 seconds, using similar algorithm astail -f. - If “104: Connection reset by peer” or “111: Connection refused” is detected in the log,
- Run
killall -9 php-cgi - Run
/etc/init.d/spawn-fcgi-php startto restart the PHP FastCGI processes
- Run
It should use minimum memory + CPU/IO (around 3.5MB RSS, and stat calls are pretty fast). At least it keeps me peace at night that I know if my PHP processes are crashing, there’s a guardian to bring them back up. It still runs on my VPS at Linode — even though with Debian 5 I am getting crashes once every 2 weeks instead of 4-5 times a day.
Hopefully it will be useful to some.
I might have to try this at some point, but I wonder how many php threads you have spawned?
We had one blog site that was having the GateWay error and it turned out that only one cgi thread was being created. Once we increased it to 4 or 5 it was able to handle it a lot better.
I have found some of the init scripts don’t work as expected and only start one thread. You can also edit the init script and increase the amount of threads (if it works anyway). Its possible it could be a timeout value too, if something keeps taking up all of the cgi processing time and you go to load the site you get the error.
I have 5 running that busy site of mine — more than enough I found.
Have you used php-fpm at all?
Hi, am having this annoying issue (502 bad gateway) so I’d like to give your script a try, but it seems to be no longer available on Linode’s pastebin.
Could you kindly post it?
Also, how can I adapt it for use with php-fpm?
Thanks!
Sorry guys. I have finally put it on my own server which you can now download the script here.
How do you run this script? Im on ubuntu 8.04, nginx, and fastcgi…. This 502 bad gateway is driving me crazy….. It happens like every week, especially when the site is very busy….
I am not an admin or anything like that, so if you can help with little elaboration, that will be GREAT! :D
YES! This problem drove me batty for weeks, until I bit the bullet, and compiled php-fpm. Glad to see that the problem is being documented!
[...] buscar, llegué hasta un blog que hablaba del tema. El auto se había programado un script en python que, básicamente, escaneaba por errores en el [...]
hello please provide some info on how to use this script… I’ve download it but erally don’t know how to use it