Here is probably one of the biggest/fasting growing outage thread on WebHostingTalk — started by someone reporting FsckVPS connectivity issue yesterday morning. When I went to bed last night there was around 50+ pages of discussions, and now it has grown to 80+ pages. FsckVPS, one of the VAServ companies, was offering low-cost full-automated OpenVZ VPS with instant activation. And it turns out that HyperVM, a virtualization control panel for both OpenVZ and Xen developed by LxLabs, has some serious vulnerabilities. Someone could exploit it and gain super user privilege on the physical node, including performing highly malicious operations. It appears to be the case with FsckVPS and someone managed to exploit HyperVM and wiped out data for 100,000 websites (according to The Register).
“We were hit by a zero-day exploit” in version 2.0.7992 of the application, he (Rus Foster of VAServ) said. “I’ve heard from other people they’ve been hit by the same thing.”
Because of unmanaged nature of FsckVPS, they do not actually carry any backup of their clients’ VPS (which is a fair call from a providers’ point of view). Many might not actually have offsite backup anywhere — and now that’s pretty bad.
It has also been reported that LxLabs have received vulnerability reports at least 2 weeks ahead of this incident. While they did push out a security update on HyperVM and Kloxo (formally known as LxAdmin) a few days ago, it does not actually fix all the issues. I did a quick review on LxAdmin 2 years ago. While it’s not too bad, it does not feel that it has been designed and I am not too surprised to see its big list of vulnerabilities. I have no experience with HyperVM but I think the reports on their bad security practises won’t be too far fetched. Actually from the security discussions like this on LxLabs’ forums, you’ll be wondering how much do they understand about the security (especially after previous incident where used the same password for their billing system as the one on WHT).
Out of 3 different VPS providers that I use that are relying on HyperVM, two of them have the web-based control panel completely blocked, and another one (The NY NOC) still has the control panel running but unable to connect to the slave instances (possible blocked). I guess only at this kind of crisis, people would really appreciate companies that developed their own virtualization manage solutions like SliceHost, Linode and VPSLink. There are also a few threads on WHT started by the providers to discuss whether should ditch LxLabs/HyperVM and whom else to go to. As for those who suggested that they need to raise price + hire more staff to look after these issues — personally I found defensive coding does not happen over night. Now we all know HyperVM is tainted — maybe it’s really a good time to look into something that’s more secure and open source (GPLHost’s DTC has been suggested a few times as an alternative as control panel for Xen).
This could spell the very end of LxLabs if not handled correctly (and so far not much response from them). Not only potential law suits, but also loss of trust from service providers which mean reduced number of licenses. It might also spell the end for budget < $10/month low end VPS market (or at least a big shrinkage), as they cannot afford higher priced Virtuozzo and they do not have the skill set to implement their own control panels.
As for end users, the old saying is still the same. Backup backup backup! Even a large 100,000 website provider like VAServ cannot dodge this one due to messed up upstream provider (in this case a software vendor). There is simply no excuse for relying only on providers to safe-keep your valuable data.
Update — thanks to Carlos at WebbyNode for notifying this related news on WHT.
On Monday morning, software company owner K T Ligesh, 32, was found hanging in his house. (India Times)
It does put a very sad ending to the outages that FsckVPS is having. Loosing your data due not being diligent in backups is one thing. Loosing your business due incompetent upstream provider is on a bigger scale. However loosing your precious life is something totally different and incomparable. All the best to those who are still trying to recover data, and RIP to K T Ligesh.