These days Open DNS resolvers are getting quite popular. With Open DNS resolver I mean resolvers including OpenDNS as well as Google Public DNS.

One of major issues these resolvers suffer is failure of integration with CDN providers like Akamai, Limelight etc. In this post I will analyse sample client site of Akamai – Malaysia Airlines website – http://www.malaysiaairlines.com.  

Looking at OpenDNS, Google Public DNS and my ISP (BSNL’s) DNS resolver for its DNS records:

OpenDNS 

;; QUESTION SECTION:
;www.malaysiaairlines.com. IN A

;; ANSWER SECTION:
www.malaysiaairlines.com. 12169 IN CNAME www.malaysiaairlines.com.edgesuite.net.
www.malaysiaairlines.com.edgesuite.net. 12169 IN CNAME a1456.b.akamai.net.
a1456.b.akamai.net. 20 IN A 125.252.225.158
a1456.b.akamai.net. 20 IN A 125.252.225.151

Google Public DNS

;; QUESTION SECTION:
;www.malaysiaairlines.com. IN A

;; ANSWER SECTION:
www.malaysiaairlines.com. 12312 IN CNAME www.malaysiaairlines.com.edgesuite.net.
www.malaysiaairlines.com.edgesuite.net. 12318 IN CNAME a1456.b.akamai.net.
a1456.b.akamai.net. 10 IN A 58.27.22.154
a1456.b.akamai.net. 10 IN A 58.27.22.138

BSNL’s DNS resolver

;; QUESTION SECTION:
;www.malaysiaairlines.com. IN A

;; ANSWER SECTION:
www.malaysiaairlines.com. 20410 IN CNAME www.malaysiaairlines.com.edgesuite.net.
www.malaysiaairlines.com.edgesuite.net. 20410 IN CNAME a1456.b.akamai.net.
a1456.b.akamai.net. 20 IN A 117.239.141.35
a1456.b.akamai.net. 20 IN A 117.239.141.10

Notice different IP’s coming when asked from different DNS resolvers. 

OpenDNS passes me 125.252.225.151 which is announced by Singtel in Singapore.
Google passes me  58.27.22.154 which is announced by Tmnet in Malaysia.
BSNL’s DNS resolver passes me  117.239.141.35 announced by BSNL-NIB itself is within India (yay!) 🙂

This results in latency of 300ms for www.malaysiaairlines.com when using OpenDNS & Google while 60ms when using ISP’s default resolver. 

How and why this is happening?

The answer lies on underlying DNS layer which is doing this magic. In all cases www.malaysiaairlines.com. is a cname (alias record) to www.malaysiaairlines.com.edgesuite.net.  Further www.malaysiaairlines.com.edgesuite.net. is a cname to a1456.b.akamai.net. Real magic comes here – “b.akamai.net.” itself is a DNS zone. Let’s look at this zone from all 3 DNS resolvers:

anurag@laptop:/$ dig b.akamai.net. ns +short @208.67.222.222
n6b.akamai.net.
n7b.akamai.net.
n1b.akamai.net.
n2b.akamai.net.
n4b.akamai.net.
n3b.akamai.net.
n5b.akamai.net.
n0b.akamai.net.

anurag@laptop:/$ dig b.akamai.net. ns +short @8.8.8.8
n1b.akamai.net.
n4b.akamai.net.
n8b.akamai.net.
n3b.akamai.net.
n2b.akamai.net.
n6b.akamai.net.
n5b.akamai.net.
n0b.akamai.net.
n7b.akamai.net.

anurag@laptop:/$ dig b.akamai.net. ns +short @10.0.0.1
n0b.akamai.net.
n1b.akamai.net.
n2b.akamai.net.
n3b.akamai.net.
n4b.akamai.net.
n5b.akamai.net.
n6b.akamai.net.
n7b.akamai.net.
n8b.akamai.net.

All identical names. Let’s pick one randomly and analyse:

n0b.akamai.net

anurag@laptop:/$ dig n0b.akamai.net a @208.67.222.222 +short
124.155.223.36

anurag@laptop:/$ dig n0b.akamai.net a @8.8.8.8 +short
202.175.5.150

anurag@laptop:/$ dig n0b.akamai.net a @10.0.0.1 +short
124.124.201.156

All different IPs!
At this stage everything seems very confusing.

Let’s revise what we have till now

www.malaysiaairlines.com. is CNAME to www.malaysiaairlines.com.edgesuite.net. and www.malaysiaairlines.com.edgesuite.net. is cname to a1456.b.akamai.net. Now a1456.b.akamai.net. is a absolute hostname under DNS zone “b.akamai.net” which is giving different IPs when checked from different DNS resolvers. b.akamai.net DNS zones has several DNS servers and I randomly pick one of them n0b.akamai.net. We see n0b.akamai.net itself gives different A records and thus I am going back to parent zone which is akamai.net to further find how this is happening.

Let’s see DNS servers of akamai.net:

To avoid further confusion due to interesting DNS lookups, let’s use whois record of akamai.net domain to see what authoritative DNS servers it is using rather then a DNS query:

anurag@laptop:~$ whois akamai.net

Whois Server Version 2.0

Domain names in the .com and .net domains can now be registered
with many different competing registrars. Go to http://www.internic.net
for detailed information.

Domain Name: AKAMAI.NET
Registrar: TUCOWS.COM CO.
Whois Server: whois.tucows.com
Referral URL: http://domainhelp.opensrs.net
Name Server: NS1-1.AKAMAITECH.NET
Name Server: NS2-193.AKAMAITECH.NET
Name Server: NS3-193.AKAMAITECH.NET
Name Server: NS4-193.AKAMAITECH.NET
Name Server: NS5-193.AKAMAITECH.NET
Name Server: NS6-193.AKAMAITECH.NET
Name Server: NS7-193.AKAMAITECH.NET
Name Server: ZC.AKAMAITECH.NET
Name Server: ZD.AKAMAITECH.NET
Name Server: ZE.AKAMAITECH.NET
Name Server: ZG.AKAMAITECH.NET
Name Server: ZH.AKAMAITECH.NET
Name Server: ZI.AKAMAITECH.NET
Status: clientTransferProhibited
Status: clientUpdateProhibited
Updated Date: 18-jun-2012
Creation Date: 03-mar-1999
Expiration Date: 03-mar-2022

>>> Last update of whois database: Sun, 28 Oct 2012 16:56:03 UTC <<<

Now again let’s pick one randomly – NS1-1.AKAMAITECH.NET and see what it tells us for hostname “n0b.akamai.net” 

anurag@laptop:~$ dig @NS1-1.AKAMAITECH.NET n0b.akamai.net +short
123.201.147.5

 

Wow! Akamai’s DNS setup can make a boring Sunday evening very interesting. 😉

Now since NS1-1.AKAMAITECH.NET. itself is on a different domain name (and so different DNS zone), let’s do bit more effort to get to the core of it. NS1-1.AKAMAITECH.NET. is simply an A record on DNS servers of AKAMAITECH.NET. zone.

Let’s look at that zone now:

anurag@laptop:/$ dig AKAMAITECH.NET ns +short
zh.AKAMAITECH.NET.
ns3-193.AKAMAITECH.NET.
ns2-193.AKAMAITECH.NET.
zm-1.AKAMAITECH.NET.
zg.AKAMAITECH.NET.
zb.AKAMAITECH.NET.
ze.AKAMAITECH.NET.
zf.AKAMAITECH.NET.
ns5-193.AKAMAITECH.NET.
zd.AKAMAITECH.NET.
zi.AKAMAITECH.NET.
ns4-193.AKAMAITECH.NET.
za.AKAMAITECH.NET.
zc.AKAMAITECH.NET.

Again, let’s pick – zh.AKAMAITECH.NET. and query for NS1-1.AKAMAITECH.NET.

anurag@laptop:/$ dig NS1-1.AKAMAITECH.NET. @zh.AKAMAITECH.NET.  +short
193.108.88.1

Finally some consistent result (YAY!). So is server with IP 193.108.88.1 playing game? Remember in 2nd last step this server was giving different IPs for hostname NS1-1.AKAMAITECH.NET. I SMELL ANYCASTING! 🙂

Let’s do a traceroute to 193.108.88.1 from my location (BSNL Haryana), Airtel Delhi node & my Europe server (where this blog is hosted!):

BSNL

traceroute to 193.108.88.1 (193.108.88.1), 30 hops max, 60 byte packets
1 10.0.0.1 (10.0.0.1) [AS1] 0.644 ms 1.022 ms 1.150 ms
2 117.220.160.1 (117.220.160.1) [AS9829] 19.467 ms 20.335 ms 21.824 ms
3 218.248.169.122 (218.248.169.122) [AS9829] 27.180 ms 29.092 ms 30.510 ms
4 115.254.1.138 (115.254.1.138) [AS18101] 61.354 ms 63.244 ms 64.209 ms
5 115.255.239.53 (115.255.239.53) [AS18101] 68.160 ms 68.907 ms 69.847 ms
6 115.248.226.21 (115.248.226.21) [AS18101] 72.336 ms 54.497 ms 54.633 ms
7 203.101.100.213 (203.101.100.213) [AS9498/AS7617] 80.766 ms 82.390 ms 83.732 ms
8 AES-Static-010.194.22.125.airtel.in (125.22.194.10) [AS24560/AS9498] 87.199 ms 88.580 ms 90.314 ms
9 * * *
10 * * *

Europe server

traceroute to 193.108.88.1 (193.108.88.1), 30 hops max, 60 byte packets
1 gw.giga-dns.com (91.194.90.1) [AS51167] 0.639 ms 0.637 ms 0.623 ms
2 host-93-104-204-33.customer.m-online.net (93.104.204.33) [AS8767] 0.600 ms 0.592 ms 0.585 ms
3 xe-1-1-0.rt-decix-2.m-online.net (82.135.16.102) [AS8767] 7.784 ms 7.740 ms 7.727 ms
4 xe-1-1-0.rt-decix-2.m-online.net (82.135.16.102) [AS8767] 7.464 ms 7.461 ms 7.452 ms
5 decix-fra6.netarch.akamai.com (80.81.192.28) [AS6695] 8.434 ms 8.916 ms 8.407 ms
6 * * *
7 * * *
8 * * *

Here we go! Surely anycasting. 193.108.88.1 is coming from prefix 193.108.88.0/24 announced by Akamai AS21342 announced at different locations.

Summary:

Let’s go in forward mode now:

Akamai CDN provider has a interesting DNS setup with mix of anycasting DNS servers where “edge servers” carry different A record for a given hostname. E.g at core Akamai has set of anycasted DNS servers like zh.AKAMAITECH.NET which hold A record for another set of DNS servers like NS1-1.AKAMAITECH.NET. which act as DNS server for akamai.net domain name. Next, these DNS servers hold different values for another set of DNS servers like n0b.akamai.net which are hold the delegation for a subzone like b.akamai.net which holds the hostname like a1456.b.akamai.net to which hostnames like www.malaysiaairlines.com.edgesuite.net. point to! 🙂 

Why Akamai is having such complex setup?

My strong guess here is that multiple zones and cross dependency here is simply to spread load and avoid single point failure. The important thing here is that at core of DNS Akamai uses anycasting but for serving content from these web servers there’s no anycasting. E.g I am getting IP 117.239.141.10 for Akamai’s client site why is a unicated IP from BSNL 117.239.128.0/20 prefix announcement. Akamai is NOT using anycasting on edge distribution and my strong guess for that is that it’s way too easy for Akamai to manage things in current rather then putting caching servers on anycasting IPs. E.g if in current situation Akamai node on BSNL is choked up, they can simply distribute traffic by modifying DNS server to pass A record to BSNL 1 out of 4 times and rest of time pass the IP of caching node on Airtel. In case of anycasting that is not possible. It will simply follow short AS/hop path and distribution of load partially is not possible. Again that’s my guess. 🙂

Time for me to change DNS resolver in my router now!