07 Mar

Confusing traceroutes and more

And here goes my first post for 2017. The start of this year did not go well as I broke my hand in Jan and that resulted in a lot of time loss. Now I am almost recovered and in much better condition. I just attended HKNOG 4.0 at Hong Kong followed by APRICOT 2017 at Ho Chi Minh, Vietnam. an event and I enjoyed the both. Here’s my presentation from APRICOT 2017.

I recently I came across some of crazy confusing traceroutes as passed by one of my friends. I cannot share that exact traceroute on this blog post but can produce the same effect about which I am posting by doing a trace from one of large network like Telia London PoP to one of the Indian destinations via their looking glass

Example traceroute:

 

 

Here’s trace is as London (AS1299) > London (AS15412) > Mumbai (AS15412) >>>> Somewhere in India (AS9498) > destination (AS132933)

 

So traffic enters India via Reliance and next handed off to Airtel and reaches the destination. Let’s check BGP table view of same PoP for this prefix:

 

So out of both available routes both are 15412 > 18101 > 132933 direct and there are no AS9498 while Airtel (AS9498) does appear in the traceroute. 2nd last hop in the trace is 182.72.29.222 and that indeed belongs to Airtel.

If we trust routing table as well as the fact that usually Airtel and Reliance exchange domestic traffic only and typically we do not see AS15412 pushing traffic via Airtel. This means trace is wrong and it indeed is. Before we get to on why it’s wrong to let’s try to understand how exactly traceroute works.

 

Working of traceroute

The way traceroute works is by using TTL i.e Time to live on packets the tool is sending out. IP headers carry TTL to prevent them for looping forever. So for instance, if router R1 sends some traffic to router R2 and R2 is not learning that route from anywhere while has a default back to R1 then traffic will start looping between R1 and R2. IP routing prevents this by using TTL and IP packets are sent with certain TTL value and as soon as they cross a router, TTL is decreased. When TTL is zero a router is supposed to drop the traffic and not carry them any further. When a router drops traffic it is supposed to reply back with error “TTL exceeded”.

Now the way “traceroute tool” works is by sending packets with increasing TTL one after other. It sends first one with TTL 1. Router directly connected to it gets the packet. It reduces TTL (and 1 – 1 so it becomes zero) and since next TTL is now zero it just drops prefix instead of sending it further. And as a part of dropping it replies back to a system running a trace with “TTL time exceeded error” revealing it’s IP to the tool. Next, another packet will be sent with TTL 2 and it will cross 1st router & would drop on a 2nd router with “TTL time exceeded” revealing it’s IP.

 

 

Back to our problem…

Now, so that was about the working of traceroute. Now going back to the case I was discussing. Think of routing between two networks when routing is not symmetric. With asymmetric routing, I mean that source & destinations may be carried via different paths.

 

Say e.g here A is sending traffic to B via R1-R2 and B is replying back to A via R3. Now if A does a trace to B, R1 & R2 may appear fine but what source IP B uses to convey the message of TTL exceeded can confuse things. When packets reach B with TTL 1, B decrements TTL and drops them. Next to send that “TTL timeout exceeded message” B has two options:

  1. B can reply back from IP address on the interface connected to R2. Remember I am talking about B just using source IP for TTL exceeded error.  Actual reply path, of course, is via R3
  2. B can reply back from IP of address of the interface connected to R3 using the usual logic of how packets go out – use the source IP of the interface of the best path installed in the router

 

What logic B uses has it’s own advantages and disadvantages. If B follows #1 i.e sends TTL exceeded from the same interface which is connected to R2 then it will give very logical traceroute output. But if network R3 is filtering packets based on BCP38, it will just drop the traffic coming from B from R2’s IP. While if B follows #2 it won’t cause any issues with BCP38 but will confuse the traceroute replies as suddenly one hop in trace will appear from entirely another network. That is what exactly has been happening in the trace I shared above. Let’s read trace again.

 

Here router right before destination i.e on hop7 is connected to Reliance & Airtel. It’s announcing the prefix covering the destination to Reliance and Reliance is bringing traffic but it’s using Airtel to send traffic out back to London router of Telia. While replying for “TTL exceeded” router 7 is using source IP of Airtel and thus we see the PTR record pointing to Airtel. This can be referred as “Random factoid” behaviour in traceroute. This comes from RFC1812 which suggests “ICMP source must be from the egress iface” and  Richard Steenbergen puts its very nicely in his presentation at NANOG here.

Checko

So that’s all about it for now!

4 thoughts on “Confusing traceroutes and more

  1. excellent article!!! knowledge sharing!

    Is there any good site to check the latencies between destinations with india,

    I have a request, could you please write a article on Indian routing setup, like how major ISP route traffic as well about the smaller ISP route the traffic within India.

    • Thanks vv

      You can check latency with Indian networks using various RIPE Atlas probes deployed across India.

      anurag@devops1:~$ ripe-atlas probe-search –country IN –status 1

      Filters:
      Status: 1
      Country: IN

      ID Asn_v4 Asn_v6 Country Status
      ===========================================
      305 24560 in Connected
      385 9498 in Connected
      1083 45942 45942 in Connected
      4913 9829 in Connected
      6107 33480 33480 in Connected
      11832 24560 6939 in Connected
      11978 38457 in Connected
      13558 24560 in Connected
      17013 9498 in Connected
      17190 23872 in Connected
      17590 24560 in Connected
      20814 24560 in Connected
      21538 10199 in Connected
      23419 24560 in Connected
      23697 17625 in Connected
      24844 133648 in Connected
      24887 24309 in Connected
      24912 18207 in Connected
      24916 133661 in Connected
      25093 9829 in Connected
      25111 9829 9829 in Connected
      25160 4755 in Connected
      25161 58457 in Connected
      25164 24560 in Connected
      25185 9829 in Connected
      ===========================================
      Showing 25 of 41 total probes

      anurag@devops1:~$

      Following are the connected RIPE Atlas probes deployed across India.

  2. Mainly Wrong direction routes are spotted on Tier1 ISP, After Cogentco’s(AS174) Tens of confusing routes anathor Tier1 Telia(AS1299) then TATA(6453) AND TATA India (4755)

    Traceroute from Hongkong AS1299 to Haryana AS133647 via AS6453 and As4755

    Route Goes from Hongkong_Los Angeles_Ashburn_Newark_London_Paris_Marseille_Chennai_Mumbai_Chennai_New Delhi
    Insted of this They Have a simple Route from Hongkong Interconnect

    Hongkong_Singapore_Chennai_New Delhi

    Router: Hong Kong
    Command: traceroute inet 110.172.172.213 as-number-lookup

    traceroute to 110.172.172.213 (110.172.172.213), 30 hops max, 40 byte packets
    1 las-b21-link.telia.net (213.155.134.194) 164.511 ms 164.520 ms las-b21-link.telia.net (213.155.136.44) 159.076 ms
    2 las-b22-link.telia.net (213.155.130.125) 158.867 ms las-b22-link.telia.net (62.115.116.174) 164.222 ms las-b22-link.telia.net (213.155.131.75) 154.060 ms
    3 ix-ae-23-0.tcore1.LVW-Los-Angeles.as6453.net (66.110.59.65) 144.364 ms 143.248 ms 161.673 ms
    4 if-ae-2-2.tcore2.LVW-Los-Angeles.as6453.net (66.110.59.2) [AS 6453] 422.959 ms * *

    5 if-ae-36-2.tcore2.AEQ-Ashburn.as6453.net (216.6.87.110) [AS 6453] 429.002 ms * 424.227 ms

    6 if-ae-3-2.thar2.NJY-Newark.as6453.net (216.6.87.10) [AS 6453] 418.376 ms if-ae-10-2.tcore1.L78-London.as6453.net (216.6.87.105) [AS 6453] 425.560 ms if-ae-11-2.thar2.NJY-Newark.as6453.net (216.6.87.169) [AS 6453] 445.127 ms

    7 if-ae-1-3.thar1.NJY-Newark.as6453.net (216.6.57.1) [AS 6453] 420.195 ms 419.518 ms if-ae-3-2.tcore1.PYE-Paris.as6453.net (80.231.154.184) [AS 6453] 425.178 ms

    8 if-ae-4-2.tcore1.L78-London.as6453.net (80.231.130.33) [AS 6453] 432.549 ms 432.864 ms if-ae-8-2.tcore1.LDN-London.as6453.net (66.198.70.174) [AS 6453] 440.452 ms

    9 if-ae-23-2.tcore1.PYE-Paris.as6453.net (80.231.154.42) [AS 6453] 432.848 ms if-ae-3-2.tcore1.PYE-Paris.as6453.net (80.231.154.184) [AS 6453] 430.132 ms if-ae-17-2.tcore1.L78-London.as6453.net (80.231.130.129) [AS 6453] 422.296 ms

    10 if-ae-8-1600.tcore1.WYN-Marseille.as6453.net (80.231.217.5) [AS 6453] 439.425 ms if-ae-8-1600.tcore1.WYN-Marseille.as6453.net (80.231.154.2) [AS 6453] 423.122 ms *

    11 180.87.36.58 (180.87.36.58) [AS 6453] 421.881 ms if-ae-8-1600.tcore1.WYN-Marseille.as6453.net (80.231.154.2) [AS 6453] 426.481 ms 180.87.36.58 (180.87.36.58) [AS 6453] 442.479 ms

    12 if-ae-7-2.tcore1.CXR-Chennai.as6453.net (180.87.36.34) [AS 6453] 416.910 ms if-ae-5-2.tcore1.MLV-Mumbai.as6453.net (80.231.217.30) [AS 6453] 440.339 ms *

    13 if-ae-7-2.tcore1.CXR-Chennai.as6453.net (180.87.36.34) [AS 6453] 415.049 ms 180.87.36.58 (180.87.36.58) [AS 6453] 427.375 ms if-ae-7-2.tcore1.CXR-Chennai.as6453.net (180.87.36.34) [AS 6453] 442.381 ms
    14 180.87.36.58 (180.87.36.58) [AS 6453] 445.418 ms 424.647 ms 447.920 ms
    15 14.140.113.242.static-Delhi-vsnl.net.in (14.140.113.242) [AS 4755] 440.681 ms static-170.249.93.111-tataidc.co.in (111.93.249.170) [AS 45820] 433.489 ms 14.140.113.242.static-Delhi-vsnl.net.in (14.140.113.242) [AS 4755] 461.085 ms
    16 14.140.113.242.static-Delhi-vsnl.net.in (14.140.113.242) [AS 4755] 448.015 ms * 450.423 ms
    17 * * 103.43.5.90 (103.43.5.90) [AS 133647] 437.327 ms
    18 103.43.5.94 (103.43.5.94) [AS 133647] 436.354 ms 447.385 ms static-170.249.93.111-tataidc.co.in (111.93.249.170) [AS 45820] 442.365 ms
    19 103.43.5.90 (103.43.5.90) [AS 133647] 441.386 ms 461.511 ms 103.237.172.234 (103.237.172.234) [AS 133647] 441.810 ms
    20 15.15.15.2 (15.15.15.2) [AS 7430] 466.799 ms 45.115.252.70 (45.115.252.70) [AS 133647] 444.419 ms 443.786 ms
    21 45.115.252.70 (45.115.252.70) [AS 133647] 451.759 ms 450.803 ms 103.237.172.234 (103.237.172.234) [AS 133647] 445.897 ms
    22 15.15.15.2 (15.15.15.2) [AS 7430] 448.913 ms 110.172.172.213 (110.172.172.213) [AS 133647] 441.931 ms 15.15.15.2 (15.15.15.2) [AS 7430] 447.714 ms

Leave a Reply