07 Mar

Confusing traceroutes and more

And here goes my first post for 2017. The start of this year did not go well as I broke my hand in Jan and that resulted in a lot of time loss. Now I am almost recovered and in much better condition. I just attended HKNOG 4.0 at Hong Kong followed by APRICOT 2017 at Ho Chi Minh, Vietnam. an event and I enjoyed the both. Here’s my presentation from APRICOT 2017.

I recently I came across some of crazy confusing traceroutes as passed by one of my friends. I cannot share that exact traceroute on this blog post but can produce the same effect about which I am posting by doing a trace from one of large network like Telia London PoP to one of the Indian destinations via their looking glass

Example traceroute:



Here’s trace is as London (AS1299) > London (AS15412) > Mumbai (AS15412) >>>> Somewhere in India (AS9498) > destination (AS132933)


So traffic enters India via Reliance and next handed off to Airtel and reaches the destination. Let’s check BGP table view of same PoP for this prefix:


So out of both available routes both are 15412 > 18101 > 132933 direct and there are no AS9498 while Airtel (AS9498) does appear in the traceroute. 2nd last hop in the trace is and that indeed belongs to Airtel.

If we trust routing table as well as the fact that usually Airtel and Reliance exchange domestic traffic only and typically we do not see AS15412 pushing traffic via Airtel. This means trace is wrong and it indeed is. Before we get to on why it’s wrong to let’s try to understand how exactly traceroute works.


Working of traceroute

The way traceroute works is by using TTL i.e Time to live on packets the tool is sending out. IP headers carry TTL to prevent them for looping forever. So for instance, if router R1 sends some traffic to router R2 and R2 is not learning that route from anywhere while has a default back to R1 then traffic will start looping between R1 and R2. IP routing prevents this by using TTL and IP packets are sent with certain TTL value and as soon as they cross a router, TTL is decreased. When TTL is zero a router is supposed to drop the traffic and not carry them any further. When a router drops traffic it is supposed to reply back with error “TTL exceeded”.

Now the way “traceroute tool” works is by sending packets with increasing TTL one after other. It sends first one with TTL 1. Router directly connected to it gets the packet. It reduces TTL (and 1 – 1 so it becomes zero) and since next TTL is now zero it just drops prefix instead of sending it further. And as a part of dropping it replies back to a system running a trace with “TTL time exceeded error” revealing it’s IP to the tool. Next, another packet will be sent with TTL 2 and it will cross 1st router & would drop on a 2nd router with “TTL time exceeded” revealing it’s IP.



Back to our problem…

Now, so that was about the working of traceroute. Now going back to the case I was discussing. Think of routing between two networks when routing is not symmetric. With asymmetric routing, I mean that source & destinations may be carried via different paths.


Say e.g here A is sending traffic to B via R1-R2 and B is replying back to A via R3. Now if A does a trace to B, R1 & R2 may appear fine but what source IP B uses to convey the message of TTL exceeded can confuse things. When packets reach B with TTL 1, B decrements TTL and drops them. Next to send that “TTL timeout exceeded message” B has two options:

  1. B can reply back from IP address on the interface connected to R2. Remember I am talking about B just using source IP for TTL exceeded error.  Actual reply path, of course, is via R3
  2. B can reply back from IP of address of the interface connected to R3 using the usual logic of how packets go out – use the source IP of the interface of the best path installed in the router


What logic B uses has it’s own advantages and disadvantages. If B follows #1 i.e sends TTL exceeded from the same interface which is connected to R2 then it will give very logical traceroute output. But if network R3 is filtering packets based on BCP38, it will just drop the traffic coming from B from R2’s IP. While if B follows #2 it won’t cause any issues with BCP38 but will confuse the traceroute replies as suddenly one hop in trace will appear from entirely another network. That is what exactly has been happening in the trace I shared above. Let’s read trace again.


Here router right before destination i.e on hop7 is connected to Reliance & Airtel. It’s announcing the prefix covering the destination to Reliance and Reliance is bringing traffic but it’s using Airtel to send traffic out back to London router of Telia. While replying for “TTL exceeded” router 7 is using source IP of Airtel and thus we see the PTR record pointing to Airtel. This can be referred as “Random factoid” behaviour in traceroute. This comes from RFC1812 which suggests “ICMP source must be from the egress iface” and  Richard Steenbergen puts its very nicely in his presentation at NANOG here.


So that’s all about it for now!

30 Jul

Welcome Amazon AWS AS16509 to India!

Today I spotted some routes from Amazon AWS Cloud services –  AS16509 in Indian tables. AS16509 was originating prefixes while sitting in downstream of Tata-VSNL AS4755 and Reliance AS18101. I almost missed Amazon AWS’s announcement on their blog about Indian PoPs for their DNS service – Route53 and CDN service – Cloudfront.


New PoP’s of Amazon in India are at Mumbai and Chennai and I see pretty much consistent BGP announcements to Tata and Reliance from these locations. Prefixes I have seen so far:


Unicast prefixes originated in India (for Cloufront CDN):


Anycast prefixes (for anycasted DNS route 53)


Note: I pulled these prefixes by looking at upstream peers in India (which is Tata and Reliance) and running simply sh ip bgp regexp 4755 16509sh ip bgp regexp 18101 16509 on Oregon routeviews & few other major data collection points of global IPv4 table. 

I can’t see any upstream from Airtel AS9498 or any other major Indian telco. Also at NIXI prefixes are available partially. I see prefiex at NIXI Mumbai carried by Tata VSNL. At NIXI Chennai prefixes are present with one degree prepend (AS4755 AS4755 twice) making route less preferable. While at NIXI Delhi there seems no route at all for Amazon’s prefixes (Tata follows regional route policy at NIXI). 


So now big question here – which datacenter is that? 

I doubt it would be Tata or Reliance since they are core competitiors and run datacenters pretty much on their own networks with almost zero carrier neutral options (few exceptions are there). My strong guess is that it’s Netmagic’s datacenter in Mumbai and Chennai with direct upstream links (bypassing Netmagic’s network). Just my guess. Cannot verify it from record of AS16509 on peeringdb.net – http://www.peeringdb.com/view.php?asn=16509


With that being said here’s a trace to cdn.anuragbhatia.com (which I use via Amazon Cloudfront):

Anurags-MacBook-Pro:~ anurag$ traceroute -a cdn.anuragbhatia.com
traceroute: Warning: cdn.anuragbhatia.com has multiple addresses; using
traceroute to ddlfp4nmkhyfr.cloudfront.net (, 64 hops max, 52 byte packets
1 [AS1] ( 1.152 ms 0.765 ms 0.627 ms
2 [AS10223] ( 1.460 ms 2.906 ms 1.569 ms
3 [AS9829] ( 16.339 ms 17.905 ms 15.704 ms
4 [AS9829] ( 94.835 ms 29.628 ms 118.135 ms
5 [AS4755] ( 60.472 ms 61.304 ms 59.103 ms
6 [AS0] ( 84.706 ms 87.201 ms 85.640 ms
7 [AS4755] ( 82.327 ms 83.276 ms 81.583 ms
8 [AS16509] server-54-230-189-204.maa3.r.cloudfront.net ( 85.261 ms 85.185 ms 84.269 ms
Anurags-MacBook-Pro:~ anurag$


Always nice to maa in all these nodes at Chennai. Basically most of companies (including Google) use 3 digit airport code in name of node (in rDNS PTR record of router’s WAN IP). For Chennai (which used to be known as Madras) airport code is still MAA and this is why you will see maa in Chennai nodes and BOM on Mumbai based nodes. 🙂


Time to get back to work. Have a good week ahead! 🙂