Caffeine Machine -- one of better ways to mod your PC cases! Jeff Atwood blogged about Google building their own servers instead of buying off-the-shelf boxes, probably to justify his over-the-top workstation at work. The argument is,

  • Google built their own servers on plywood back in 1999.
  • Google is still building their own servers today, 450,000 of them!
  • Therefore, build your own PC is good! Pre-built might only work for typical users, and programmers are not typical users!

Therefore Jeff concluded,

The best way to truly understand the commodity PC is to gleefully dig in and build one yourself. Get your hands dirty and experience the economics of computer hardware first hand– the same economics that have shaped the software industry since the very first line of code was stored in memory.

I disagree.

A bit of my own history

I guess “almost” every computer science/engineering graduate has gone through a period of “tinkering your own computer”. My family got our first computer back in early 80’s, and it was a Commodore VIC-20. However the fun really started when we got our IBM-compatible XT a few years later. Then a 386DX. Then I bought myself a 90Mhz Pentium. Then…

I used to love the excitement of choosing my own motherboard, buying the most compatible RAM sticks, picking the most cost-effective CPU, and not forgetting to get a bigger fan and heat-sink so you can run it faster than what it was labelled for. I was pretty happy back in 1999 to run 2x Celeron 400Mhz at 600Mhz on the legendary Abit BP-6, although it did take me a few days to dig enough holes on my beige case to create enough airflow to keep the CPUs cool.

Even when we started our company back in 2001, two programmers, myself being one of them, could spend the entire day building new servers from bare components. You feel that you have the total control over the choice of the hardware, and put them together so tidy like no other local shops would. We deliberately picked full tower cases for our servers so there was plenty of room to work on, although they were fit with at most two drives. In the process you also learnt which NIC Linux hates, and which one rules.

Oh the crazy days of startups.

No, it was not economical

Dell Servers -- by far cheaper to run in long run than build your own. But when I recalled all these a few years later — that was just so inefficient and uneconomical! Yes, you might save $50 by buying components from various online shops, but it was at the expense of time of two programmers to put the computers together — at $30-$40 an hour! Worth it?! Not at all! They (or should I say, I myself?) should have been getting the coding done instead.

Plenty of other reasons why for a small enterprise, it is better to buy than build.

  • Cost of labour. It was actually my previous point. How much does it cost for a programmer to build his computer for work? You have to count in the cost of the individual components, separate delivery cost if bought from multiple places, and the cost of time for that programmer to build it.

    How fast can a programmer build his/her PC? It can take me up to half a day depending on how tidy the cables need to be and how tiny the case is. That is still a lot of expense even for a junior programmer! And it will definitely not faster than the factory workers in China, Taiwan, Malaysia or Indonesia, where each person might assemble hundreds of desktops a day!

    The cost of assemble your own simply does not justify — not just because it was manufactured in South East Asia, but also because they can do it so efficiently.

  • Economies of scale. Over the last 10 years I have seen the trend — pre-built boxes in many cases are in fact cheaper than buying individual components, and I am not even counting the labour cost. Retailers like Dell, HP, Acer and Lenova build large number of computers, every single day! If someone is getting a big discount from the OEM component companies, who would that be? Dell or your favourite online store? They also directly deal with a smaller set of OEM companies, whereas computer hardware stores usually need to stock up items from various different distributors.

    Economies of scale drive down price, and you can easily see that from how cheap the pre-built boxes these days.

  • Volume discount. Now my company (and the parent company who bought us) uses Dell computers exclusively — from servers to laptops to monitors to PDAs. We have hundreds of Dell desktops/laptops sitting in our offices, including those fussy developers’. Even more servers deployed in data centres and clients’ premises.

    So do we still pay the listed price on Dell’s website? Do we hunt for coupon codes when we need to buy a new PowerEdge? No way! Moreover, we would have our own client manager who can sort all our hardware buying issues. Buying a Dell for us is way cheaper than buying components and then build our own.

  • Warranty claims. How long does that take you to replace a faulty hard drive you bought from NewEgg? Can you afford to have a developer’s box sitting idle for a week because the power supply is no longer working? Computers do fail, especially when you have lots of them!

    I am not saying Dell’s service is all excellent for every single customer, but when you are a business customer paying for 3 year business onsite support, you can certainly feel the difference. My colleague’s LCD for his Latitude D600 no longer works the other day, and Dell Australia sent a guy in the next day with a brand new LCD panel and fixed it on the spot! They go to data centre with you to make sure your server hardware issues are fixed.

    Try that on your local computer parts shop next time.

Everyone loves to talk about the Total Cost of Ownership (TCO). When you factor in a few things, build your own is no longer economical. Not only it can be potentially more expensive to buy up front, it will also cost more in the long run if you try to build every single one of your fleet of servers.

But, Google built their own servers!

Google's first storage solution -- not for the faint hearted. Yup. And they also run their own customised version of Linux. They also happen to own their data centres!

It is the economics of scale again. If you are just starting out, have no money and can only scrape server parts from university labs, then by all means and build your own servers in shoeboxes or on plywood! If you happen to be the biggest search engine in the world, have multiple data centres scattered all over the world, manage 450,000 servers — then you might as well build your own computers to cut the middle man, because you might have the same bargaining power as Dell or HP to OEM component companies.

Moreover, we see that Google’s servers are all purposely built. Power is one of their biggest cost, so they obviously want to design and build something more efficient. It will soon be a bigger concern to companies with only hundreds of servers as well. We shall not be surprised to see Dell, Sun or HP coming out with more power efficient servers designed to be cohabited in the data centres in the near future.

Meanwhile, buying off the shelf pre-built computers still makes more sense, and it applies to most companies, big or small.

Hmm. What about the programmers?!

Yeah. What about the programmers? They all want big evil looking cases with liquid cooling, transparent panel and LED lights?

Or have I mistaken programmers with PC case moders? And how does that correlate to understanding the hardware?!

I do agree that understanding the hardware is important to software developers, especially when you are working on code pieces that severely rely on hardware-dependent optimisation. However I shall also argue that hardware gets less and less relevant to majority of software developers these days.

Who is still writing tight loops so they can fit into the L1 code-cache inside the CPU to reduce latency? Who is still writing their code to synchronise with the hardware clock to produce smooth sprites? Any Web 2.0 developer out there doing those hardware-dependent optimisations?

Instead, most developers are focusing on optimised algorithms, scalability, and the “core architecture” of their software, and try to be as much hardware agnostic as possible. Don’t even have time for these? Just throw in more hardware — they are much cheaper than hiring a gun developer who knows how to optimise.

Not that I think “putting that card into the PCI-x slot” an useful exercise in understanding the computer hardware anyway.

The only mod we have on our developers’ boxes is fancier graphics cards. Not that they can get 100FPS in Counter Strike Source, but they can hook up more than one monitors — which is always more beneficial to developers.

And Josh Jones is right

When I was writing this, Josh Jones of DreamHost posted a new blog post — Buy or Build, which makes much more sense from an enterprise point of view.

For a company though, the decision is a little harder. Clearly, there are tons of things you need to produce your product or service that just don’t make sense to build. They’re just too difficult to build, or require too big an investment of money/time for how much of them you need.

I know it is a bit out of context — Josh is mainly talking about not building everything your service relies on just for the sake of saving money, or because we think we can do a better job ourselves. However the logic is, if you should not build everything that is part of the production, then why build your own work desktop that is not even part of the production?!

I have not built a new PC for the last 3-4 years, and I do not miss it at all. I am happy to go to Dell or HP, select and customise the computer I wish to order, and save my time to do something more worthwhile — like writing this useless post :)