Read an interesting blog post the other day on Network Distance — When Sydney is closer than Sao Paulo
, where it talks about how vulnerable long-haul Internet is. Extract from the post:

Let’s imagine a perfectly balanced world-wide network. You have a business based in San Francisco. Your people are in SF, your technology suppliers are in SF, most of your customers are in the United States. You have a small but growing customer base in Hong Kong and China. We have a perfect ‘net, so there are no silly congestion problems and there is just as much bandwidth across oceans as between cities. Rack space in HK is twice the price as in SF.

So what is the no-brainer place to put your next server farm, hire people to maintain it, set up office space, pay property and business taxes, etc? Correct. Hong Kong. No matter how good the internet gets it will never be faster than a small fraction of the speed of light.

Again it comes to my dilemma of hosting in Australia or overseas. Hosting a site at Fremont rather than Melbourne is much more economical to me at tolerable degrade of performance, but what if one of the cross-Pacific link gets cut due to an earthquake? How would that impact my site which 90% of visitors are from Australia?

It is also a reason why I would not use some of the hosts — there is nothing wrong with their service, but just their data centre location which makes them less-than-usable due to excessive latency. European providers for example — they are always 300+ ms from Sydney Australia. Earlier this year I was offered a beta Xen VPS testing account at Gandi.net. Great performance, Gandi AI is actually very innovative, and a 256MB slice with 5mbps is only $7.50! However from Sydney to France takes route across Pacific, across America, across Atlantic, pass UK and then finally Frace at 350ms ping time…

I have also been sampling a Xen VPS from VPS Media this week (which a review will be up next week). Great value. Good network, and the company is doing everything right setting up a new hosting company with a strong community — but the servers are at Florida. From Sydney to Miami going through Verizon + Level3 fluctuates between 250ms-300ms on Saturday night. Better than Europe, but still feels laggy from those of us on the other side of Pacific.

I guess the worst experience this week would be connecting to a South African server during business hours. The company I worked for bought a South African software company late last year, and have sent down one engineer there to assist them with development. However that guy was on holiday this week, they have a demo on, and they couldn’t get the server to set up properly — so I ended up trying to remote access that South African server from Sydney Australia.

From Sydney to Durban is around 10,000km, which is less than from Sydney to Fremont of 11,900km. However for my packet to travel to Durban, it needs to take route across Pacific, across America, across Atlantic to Europe, and then come all the way down to Durban. 500ms. Yuk. Worse that it is a Windows box and I need to use Remote Desktop on it.

So the rule of thumb is to host as close to your market as possible, assume the rule of the wallet is not in effect…

Web24 is coloc at Fujitsu data centre in Melbourne, and a quite traceroute check with mtr shows around 20-40% packet loss at Fujitsu data centre.

$ mtr -r www.web24.com.au
HOST: localhost                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.xxx                 0.0%    30    0.3   0.3   0.3   0.7   0.1
   ...
  7. 0.so-0-2-0.XT1.SYD2.ALTER.NE  3.3%    30   19.2  22.4  17.6  73.6  11.2
  8. so-0-2-0.XR2.MEL1.ALTER.NET   0.0%    30   29.0  34.2  28.3 125.5  17.6
  9. 411.AT-6-0-0.GW4.MEL1.ALTER.  0.0%    30   87.0  35.5  29.2 104.3  16.6
 10. fuji865-mel-gw.aspac.custome 13.3%    30   32.9  50.6  31.5 231.3  47.9
 11. v13.csr02.nobp01.fujitsu.com 10.0%    30   33.9  34.0  31.6  38.9   1.8
 12. 216.14.207.85                23.3%    30   35.6  37.0  32.0  50.1   5.0
 13. 203.16.60.201                23.3%    30   34.2  35.3  33.3  38.9   1.4
 14. www.web24.com.au             23.3%    30   35.0  35.3  32.3  43.2   2.5

I have posted the same issue on Whirlpool Forums and so far we have got no response from the provider. Web24.com.au and their HSPcomplete panel are hosted inside the same network thus neither are working at the moment. I can’t even get onto the support page to get the contact details (lucky there is Google cache).

Not happy Jen. Have to wait and see… Maybe it is a sign that I need to go to sleep now :)

Update 9AM AEST 27 Feb

I got a reply back from Web24 support this morning detailing the communication between Web24, Fujitsu and their up stream providers. Great job! Some paraphrased details:

• Packet loss issue identified at 10:15pm on 26 February, and later found out it was an upstream router issue.
• Web24 has been working with Fujitsu and Verizon in identifying and resolving the issue, and by 1:18am 27 February, the main web24 site failed over to US network (very good idea).
• All the traffic are routing through secondary links by 4:45am, and still waiting for feedbacks on primary provider.

Pingdom still reports my site was down until around 8:20am. This is the graph generated from Cacti:

Note that the network does not appear to be down from 10pm onwards because the querying packets are small enough to get through. I guess network connectivity is going to be patchy this morning.