At the same time, a slow or broken DNS system can be very frustrating, even when you are just hosting a service. Little did I realise that how much a typical hosting server depends on domain name services. If you are on a Linux/Unix box, try to remove /etc/resolv.conf from your server for 5 minutes, and see how it can cope.

OpenDNS came to rescue, and its aim is to provide fast and reliable domain name service to everyone — not just those who browse the web, but it can also benefit those who host a few servers. See my post on how OpenDNS saved my day. It is a fast recursive DNS with a gigantic cache, and it rarely goes down — unlike the DNS server at your ISP.

In this article I will be looking at how to use OpenDNS to speed up DNS queries on your hosted servers. Moreover, how you can employ a local recursive cache to further improve the domain name resolution performance.

Step 1: Signing up OpenDNS

No, you don’t have to sign up an OpenDNS account. You can simply use OpenDNS by pointing to their IP addresses. However there are a few benefits if you have an OpenDNS account, and they are actually quite useful if you are hosting a server.

• Query statistics! You’ll get the daily queries + unique domains in pretty graphs. Everyone likes pretty graphs. No?
• Disable typo correction. By default OpenDNS will perform typo correction if the domain does not exist, however it is not desirable in about everything but web browsers. Having an account lets you turn off typo correction for your IP range.

If you still decided against setting up an account there, then go straight to step 3. Otherwise, create an account here, and all you need is a username, your email address and password.

Step 2: Manage Your Network at OpenDNS

After your account has been set up, you should be able to add networks under your OpenDNS account. OpenDNS will send you verification emails, and request you to visit a web page with a special key from that IP address to claim that network.

Now, add your server’s IP address there. You then will be able to manage your network/IP address after it has been verified. Go in there and turn off all features! You definitely do not want typo correction for your server, as it is necessary for programs to get back NXDOMAIN when the domain name is indeed non-existing.

Save preference, and you are done.

Step 3: Test Name Resolution on Your Server

SSH into your server, and start testing OpenDNS’s service by putting their anycast servers in your /etc/resolv.conf:

nameserver 208.67.220.220
nameserver 208.67.222.222

Then try to resolve a domain name.

$ host hostingfu.com
hostingfu.com has address 63.76.232.80

Yup. Looks like it is working.

Step 4: Installing a Local Recursive Cache

If your round-trip time to OpenDNS’s servers is small, i.e. you happen to be in the same data centre, then I won’t really worry about having a local cache. If you happen to have your servers across the pacific, taking 160ms to get to their Palo Alto resolvers, then a local recursive resolver/cache will make significant improvement. They will cache the resolving result according to TTL in the records, so subsequent queries do not need to go back to the root servers (or in our case, forward to OpenDNS).

For example, your server might send 30 emails to john@example.com during a 12 hour period. Instead of forwarding to a remote DNS server 30 times for MX records, it will only be done once, and the rest will be served from cache, if TTL on the MX record is greater than 12 hours.

There are many open source recursive DNS servers available, and these are the ones that I have experience with:

• Dnsmasq
• MaraDNS
• Dnrd

Dnsmasq

Dnsmasq is probably the most feature rich and memory efficient of the bunch. It was designed to be deployed on a small LAN so it comes with DHCP capability and can act as authoritative DNS for your local network. I use it for my home network, but because of its small memory footprint, I also use it on my memory impaired VPS (it’s around 650kb RSS). Of course the DHCP feature is turned off on hosted servers.

However I am having issues with Dnsmasq on my OpenVZ VPS, and have not yet had a chance to dissect the code to figure out what’s going on.

Installation is easy if you are on Gentoo or Ubuntu, as the packages are already available.

# emerge dnsmasq

Or

# apt-get install dnsmasq

Now use the following configuration in /etc/dnsmasq.conf.

domain-needed            # Never forward plain names
bogus-priv               # Never forward addresses in the non-routed address spaces
no-resolv                # Don't use /etc/resolv.conf
no-poll                  # Don't poll /etc/resolv.conf
server=208.67.222.222    # Forward to OpenDNS's servers
server=208.67.220.220
listen-address=127.0.0.1 # Listen and bind to only localhost
bind-interfaces

Now start the service.

# /etc/init.d/dnsmasq start

MaraDNS

MaraDNS is a small, multi-threaded and easy to use DNS system, and is suitable as local recursive DNS cache. It can also act as an authoritative DNS server, although the rule of thumb usually is — don’t use the same server for authoritative and recursive DNS. I did not use it for my servers because of its memory usage, which is slightly larger than the others, and the fact that it spawns multiple processes when it starts.

Installation is also trivial, as it is available on Gentoo as well as Unbuntu (Edgy). It is however keyword-masked on Gentoo at the moment.

# emerge maradns

Or

# apt-get install maradns

Then put the follow configuration into /etc/mararc.

ipv4_bind_addresses = "127.0.0.1"
chroot_dir = "/etc/maradns"
recursive_acl = "127.0.0.1"
upstream_servers = {}
upstream_servers["."] = "208.67.222.222, 208.67.220.220"

It basically makes MaraDNS to bind to localhost, chroot to /etc/maradns and forward requests to OpenDNS. Now to get the daemon up and running:

# /etc/init.d/maradns start

Dnrd

Domain Name Relay Daemon is a caching, forwarding DNS proxy server. It is small, fast and have just enough feature to do what we want to do here. It does not even need a configuration file!

Installation on Gentoo is easy, as there’s already an ebuild for it.

# emerge dnrd

And put the following configuration in /etc/conf.d/dnrd:

DNRD_OPTS="-b -s 208.67.222.222 -s 208.67.220.220 -a 127.0.0.1"

If you are on other Linux distro, you will need to download and build the app yourself. As release 2.20.1 is the latest, here’s an example (I am using Optus Australia’s Sourceforge mirror):

# wget http://optusnet.dl.sourceforge.net/sourceforge/dnrd/dnrd-2.20.1.tar.gz
# tar zxvf dnrd-2.20.1.tar.gz
# cd dnrd-2.20.1
# ./configure --prefix=/usr/local --disable-master
# make && make install
# mkdir -p /usr/local/etc/dnrd

The last step (mkdir) is necessary as make install does not seem to create the directory where dnrd will chroot to. Now the binary will be installed as /usr/local/sbin/dnrd. We’ll also make an init script for it, and put it in /etc/init.d/dnrd.

#!/bin/sh
. /lib/lsb/init-functions

DNRD_OPTS="-b -s 208.67.222.222 -s 208.67.220.220 -a 127.0.0.1"
case "$1" in
start)
  log_begin_msg "Starting domain name relay daemon..."
  start-stop-daemon -S -q -p /var/run/dnrd.pid -n dnrd -a /usr/local/sbin/dnrd -- $DNRD_OPTS
  log_end_msg $?
  ;;
stop)
  log_begin_msg "Stopping domain name relay daemon..."
  start-stop-daemon -K -q -p /var/run/dnrd.pid -n dnrd
  log_end_msg $?
  ;;
restart)
  log_begin_msg "Restarting domain name relay daemon..."
  start-stop-daemon -K -R 5 -q -p /var/run/dnrd.pid -n dnrd
  start-stop-daemon -S -q -p /var/run/dnrd.pid -n dnrd -a /usr/local/sbin/dnrd -- $DNRD_OPTS
  log_end_msg $?
  ;;
*)
  log_success_msg "Usage: /etc/init.d/dnrd {start|stop|restart}"
  exit 1
  ;;
esac
exit 0

Don’t forget to start it!

# /etc/init.d/dnrd start

BIND

I guess if you are running your server as reseller web hosting service, you will probably have BIND running. You can obviously run a recursive/cache name server on your existing instance of BIND, who might be currently acting as an authoritative server. However, it is not recommended, quoting ISC’s documentation:

Name Servers in Multiple Roles

The BIND name server can simultaneously act as a master for some zones, a slave for other zones, and as a caching (recursive) server for a set of local clients.

However, since the functions of authoritative name service and caching/recursive name service are logically separate, it is often advantageous to run them on separate server machines. A server that only provides authoritative name service (an authoritative-only server) can run with recursion disabled, improving reliability and security. A server that is not authoritative for any zones and only provides recursive service to local clients (a caching-only server) does not need to be reachable from the Internet at large and can be placed inside a firewall.

Now, to run on forwarding, add the following configuration to the options section of your BIND’s configuration file (/etc/named/bind.conf on Gentoo).

options {
  ...

  allow-recursion {
    127.0.0.1;
  };
  forward only;
  forwarders {
    208.67.220.220;
    208.67.222.222;
  };
}

Restart BIND to make the configuration change effective.

# /etc/init.d/named restart

Step 5: Switching the Resolver

Alright. It is the final step, I promise! Now you have OpenDNS set up and customised for your IP, local recursive DNS running, and the final step to make is to switch over to your local resolver — use 127.0.0.1 in your /etc/resolv.conf, i.e.

# echo nameserver 127.0.0.1 > /etc/resolv.conf

Now, try to resolve some domain names (host hostingfu.com), and check whether it is working. If it doesn’t work, i.e. does not bring back an IP address, you should

1. Revert the content of /etc/resolv.conf (to the one in step 3, for example).
2. Use netstat -uan | grep 53 to check whether your local recursive DNS is indeed running.

If everything is up and running, you should be getting fast and stable domain name resolution on your server.

Potential Issues

I have been using OpenDNS since last July, and have been using the setup I have stated on 3 of my VPSs for the last two months. Everything is fine — name resolution is always fast, and OpenDNS has been very stable. However there are some potential issues on depending on a centralised resolver, although OpenDNS’s servers are geographical isolated in US and Europe.

I think “single point of failure” is still the biggest issue, especially when your servers are not running in North America. If an earth quake occurs and cuts off all pipes between Sydney and Palo Alto, my Sydney VPS will fail to resolve any domain name, although a routing change shift queries to other OpenDNS locations might take place. Whereas with the typical non-forwarding recursive DNS setup, it will still be able to get to other root servers so only the domains in the effected area will fail to resolve.

It will gradually be less an issue when OpenDNS deploys more servers around the world. Hong Kong is planned from what I’ve read, so Asia/Oceania would be better covered. Until then, I’ll have my finger crossed :)

Similar event happened to me today. No, it did not save my job — it was not work related. Let’s just say that OpenDNS has saved my day.

What Happened?

Earlier today when I logged onto my VPS at GPLHost, the server somehow did not feel right. It did not take long to figure out that my VPS cannot resolve any domain name. Bigger surprise was when I discovered that my mail server has a huge queue of emails spooled because it could not figure out where to send those emails to. From the log it appeared that domain name resolution has stopped working for more than 12 hours!

It turned out that the recursive DNS server at GPLHost’s Sydney facility was no longer contactable, and it was the DNS server pre-configured for my VPS. So I quickly fired an email to GPLHost about the situation, and tried to figure out whether there is any remedy.

OpenDNS!!

It was the first public recursive DNS service that pops up in my head. So I quickly updated my resolv.conf to point to OpenDNS’s 2 name servers. Bingo! Problem solved, and domains were resolving again. It did take a while to clear the queue in my mail spool, but I was happy.

OpenDNS saved my day.

I actually used OpenDNS for all my US-based servers. You can now claim IP networks/addresses that belong to your account, and then customise the resolving behaviour. For servers I just want fast resolution — no need for phishing protection or other goodies — and I can easily turn them off from my account page in OpenDNS once I have claimed those IP addresses.

I did not use OpenDNS for my Australian VPS initially as I was afraid of the high latency. It turned out it was not too bad at all. It automatically picked up their DNS servers at Palo Alto, which is around 160ms return trip from my Sydney VPS. Combining that with a local cache/forward-only DNS server (I use Dnsmasq for its light weight), DNS resolution can be very fast from this end of planet (note to myself: need to write a how to on this one Update: done).

They have also recently announced CacheCheck, which allows you to force OpenDNS to refresh its DNS records even before TTL expires. It allows you to quickly test DNS changes. You usually have to restart local recursive server after you have made some changes to flush the cache. Now you can just set your recursive server to always forward requests to OpenDNS, and flush the zone from their web interface.

Thanks OpenDNS.