17 Nov

Measuring latency to endpoints with blocked ICMP

And a blog post after a while. Last few months went busy with RPKI. After my last post about RPKI and the fact that India was lacking a little bit on RPKI ROA front, we started with a major push by a set of like-minded folks like us. For now, Indian signed table has jumped from 12% since Aug to 32% now in Oct. Detailed graphs and other data can be found here on the public Grafana instance.

In terms of absolute percentage, India now has the highest number of absolute signed prefixes in this region. 13972 Indian prefixes have a valid ROA and nearest to that is Taiwan at 6824. Though 13972 results in just 32% of the Indian table while 6824 results in 91% of Taiwanese table. So a long way to go for us.

If you are a network operator in India and reading this, consider joining our RPKI webinar which is planned at 3 pm (IST) on 18th Nov 2020. You can register for the event here. Or buzz me to talk about RPKI!


Catching Covid-19

Besides RPKI push I also caught up with Covid19 along with family members. Luckily for us, it went fine and wasn’t that painful. The impact was mild and everyone has recovered. Phew!
I hope readers of this blog post are well.

TraceroutePing in Smokeping

Coming to the topic for today’s blog post. I recently came across this excellent Smokeping plugin which solves a very interesting problem. There are often nodes we see in the traceroute/MTR which is either not routed or simply block ICMP/TCP/UDP packets which are addressed to them. This can include routers which have a pretty harsh firewall dropping everything addressed to them as well as cases where we have IX or any other non-routed IP in the traceroute. It becomes tricky to measure latency to those. Someone used the simple idea of incremental TTLs as used in traceroute to get a reply from these middle nodes of “TTL time exceeded error” and based on that a way to plot latency.

Let’s look at a real-world case: One of ISP serving my home is IAXN AS134316 and they peer with my ex-employers network Spectra AS10029 at Extreme IX in Delhi. Let’s see how traceroute to Spectra’s anycast DNS looks from my home.

traceroute -P icmp 180.151.151.151
traceroute to 180.151.151.151 (180.151.151.151), 64 hops max, 72 byte packets
 1  router01.rtk.anuragbhatia.com (172.16.0.1)  2.818 ms  1.876 ms  4.274 ms
 2  10.10.26.6 (10.10.26.6)  4.258 ms  4.301 ms  5.953 ms
 3  10.10.26.5 (10.10.26.5)  5.490 ms  5.916 ms  5.257 ms
 4  10.10.26.29 (10.10.26.29)  11.349 ms  9.246 ms  9.430 ms
 5  as10029.del.extreme-ix.net (45.120.248.51)  10.628 ms  8.802 ms  9.609 ms
 6  resolver1.anycast.spectranet.in (180.151.151.151)  8.446 ms  9.113 ms  10.699 ms

Now hop 5 here is likely Spectra’s Delhi router’s interface which has Extreme IX IP – 45.120.248.51. Let’s see what we get when we ping it.

ping -c 5 45.120.248.51
PING 45.120.248.51 (45.120.248.51): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

--- 45.120.248.51 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

I cannot ping it. Let’s look at the trace to it to see where it drops.

traceroute -P icmp 45.120.248.51
traceroute to 45.120.248.51 (45.120.248.51), 64 hops max, 72 byte packets
 1  router01.rtk.anuragbhatia.com (172.16.0.1)  3.196 ms  1.790 ms  4.421 ms
 2  10.10.26.6 (10.10.26.6)  5.514 ms  3.624 ms  5.323 ms
 3  10.10.26.5 (10.10.26.5)  5.252 ms  4.043 ms  3.671 ms
 4  * * *
 5  * * *
 6  nsg-static-77.249.75.182-airtel.com (182.75.249.77)  14.221 ms  10.963 ms  11.574 ms
 7  116.119.68.58 (116.119.68.58)  146.531 ms  147.899 ms  146.065 ms
 8  * * *
 9  * * *
10  * * *

Now, this is an interesting and not very unexpected result. Basically, my ISP – IAXN AS134316 does not has any route in it’s routing table for 45.120.248.51 and hence passing it to default route towards it’s upstream Airtel. BGP wise IAXN is not supposed to have any route belonging to IX peering IP anyways and that’s expected. Likely their router which peers with Extreme IX is different from the router which serves me and is possibly missing sharing of connected routes via IGP and hence the unexpected path. As soon as traffic hits Airtel router with a full routing table & no default route, it drops it.

In this setup, I cannot reach Spectra’s interface connected to the Extreme IX (45.120.248.51) directly if I try to send packets to it. But I do know from the first trace that it comes in middle when I try to send packets to 180.151.151.151. This option can be used where packets can be sent with incremental TTL and latency can be measured and even graphed. This concept can be used even if there’s a use of private IPs before the destination.

So this goes to my Probe config

+ TraceroutePing

binary = /usr/bin/traceroute # mandatory
binaryv6 = /usr/bin/traceroute6
forks = 5
offset = 50%
step = 300
timeout = 15

and this goes to my Target’s config

++SpectraExtremeIXInterface
probe = TraceroutePing
menu = Spectra via Extreme IX
title = Spectra via Extreme IX
host = 45.120.248.51
desthost = 180.151.151.151
maxttl = 15
minttl = 5
pings = 5
wait = 3

How it works?

A reminder on working on traceroute!

Remember the concept of TTL in IP routing. TTL is time to live and basically whenever the router passes the packets, it decreases TTL by 1 and when TTL reaches 0, the router just drops it. This ensures loops aren’t as dangerous in layer 3 as we see in layer 2. Now when a router drops packets with TTL 0, it replies back to the source saying “TTL exceeded” and the reply packets have the router’s source IP address. That way traceroute can send 1st packet with TTL 1, 1st router in the chain gets it, reduces TTL by 1 and (now that TTL is 0) drops it with a reply from its source IP. Next, another packet is sent with TTL 2 and so on.

Note: Thanks to networking folks from OVH Cloud who replied me with this probe on Twitter. It wasn’t what I was looking for but quite fascinating and useful!
Time to go back into the routing world! 🙂

12 Dec

Skype Vs Gmail phone: Quick check on latency with IP-PSTN gateways

I usually find that Gmail phone performs lot better then Skype for calls to PSTN. Usually latency on Skype is high as compared to Gmail phone in India (in US it was almost no difference).

 

I looked around today and here’s bit of test data I collected by calling couple of numbers from Skype & Gmail phone & monitored UDP traffic to collect information about destination servers which is very likely IP-PSTN gateway.

 

Phone calls made from Skype

US calls (outgoing): 216.49.203.243
US calls (incoming):  4.53.80.106
UK calls (outgoing): 213.244.170.88

 

While gateway IP for Gmail phone was same always and it was 209.85.175.126

 

As I experienced, latency was bit high on phone calls made from Skype, here’s a quick check to routing to these IP’s from my system on BSNL Network (AS9829) in India.

 

 

MTR to Skype US nodes:

HOST: anurag-laptop Loss% Snt Last Avg Best Wrst StDev
1.|– home-router 0.0% 10 1.7 3.6 1.5 11.5 3.5
2.|– 117.206.176.1 0.0% 10 23.1 23.8 22.8 30.8 2.5
3.|– 218.248.169.122 0.0% 10 28.7 29.8 28.6 34.9 2.1
4.|– 115.114.59.169.static-Mumbai.vsnl.net.in 0.0% 10 71.2 72.5 71.1 80.6 2.9
5.|– if-0-100.tcore1.MLV-Mumbai.as6453.net 0.0% 10 182.7 186.1 181.5 214.8 10.1
6.|– if-2-2.tcore2.MLV-Mumbai.as6453.net 0.0% 10 189.5 190.3 188.1 195.2 2.2
7.|– if-6-2.tcore1.L78-London.as6453.net 0.0% 10 209.4 209.5 193.1 221.5 10.2
8.|– Vlan704.icore1.LDN-London.as6453.net 60.0% 10 197.0 199.0 196.8 201.9 2.5
9.|– Vlan533.icore1.LDN-London.as6453.net 0.0% 10 189.5 190.9 188.4 201.1 3.7
10.|– ae-52-52.csw2.London1.Level3.net 0.0% 10 196.7 199.5 195.9 214.8 5.8
11.|– ae-58-223.ebr2.London1.Level3.net 0.0% 10 195.0 194.6 193.5 196.9 1.0
12.|– ae-41-41.ebr1.NewYork1.Level3.net 0.0% 10 288.7 274.7 271.9 288.7 4.9
13.|– ae-81-81.csw3.NewYork1.Level3.net 0.0% 10 281.8 276.6 270.3 283.1 5.1
14.|– ae-82-82.ebr2.NewYork1.Level3.net 0.0% 10 271.1 271.2 269.6 274.4 1.6
15.|– ae-2-2.ebr4.SanJose1.Level3.net 10.0% 10 340.9 341.7 340.5 343.2 0.9
16.|– ae-2-2.ebr3.LosAngeles1.Level3.net 10.0% 10 339.0 339.1 338.4 340.1 0.5
17.|– ae-73-73.csw2.LosAngeles1.Level3.net 10.0% 10 343.5 344.5 343.4 346.2 1.0
18.|– ae-22-70.car2.LosAngeles1.Level3.net 10.0% 10 339.2 339.8 337.9 347.9 3.1
19.|– unknown.Level3.net 10.0% 10 338.6 339.4 338.6 340.4 0.7
20.|– LAX-C10-C7613-1.ivanet.net 10.0% 10 344.4 346.6 344.3 357.0 4.0
21.|– LAX-H07-HP365-2.ivanet.net 10.0% 10 344.1 345.3 343.9 348.8 1.5

 

HOST: anurag-laptop Loss% Snt Last Avg Best Wrst StDev
1.|– home-router 0.0% 10 1.6 2.1 1.4 6.3 1.5
2.|– 117.206.176.1 0.0% 10 22.6 23.3 22.6 24.4 0.6
3.|– 218.248.169.122 20.0% 10 47.7 58.4 28.8 135.7 41.6
4.|– if-14-1.mcore4.NYY-NewYork.as6453.net 0.0% 10 387.5 370.5 303.5 453.8 49.9
5.|– if-8-0-2-28.tcore1.NYY-NewYork.as6453.net 0.0% 10 258.4 260.6 258.4 270.3 3.7
6.|– Vlan24.icore1.NTO-NewYork.as6453.net 0.0% 10 265.3 262.5 256.9 270.1 4.7
7.|– ae9.edge1.NewYork.Level3.net 0.0% 10 259.7 266.2 258.6 324.1 20.3
8.|– vlan51.ebr1.NewYork2.Level3.net 0.0% 10 258.9 260.6 258.3 272.6 4.2
9.|– ae-3-3.ebr2.Washington1.Level3.net 0.0% 10 262.3 262.8 261.8 264.2 0.6
10.|– ae-62-62.csw1.Washington1.Level3.net 10.0% 10 266.3 263.0 261.9 266.3 1.3
11.|– ae-11-60.car1.Washington1.Level3.net 10.0% 10 262.9 262.7 261.6 263.9 0.7
12.|– 4.53.80.106 10.0% 10 265.4 265.3 264.3 266.1 0.5

 

 

And latency with Europe based node

HOST: anurag-laptop Loss% Snt Last Avg Best Wrst StDev
1.|– home-router 0.0% 10 1.5 1.8 1.5 3.6 0.7
2.|– 117.206.176.1 0.0% 10 22.6 22.7 22.0 23.5 0.5
3.|– 218.248.169.122 10.0% 10 28.5 34.8 28.5 67.9 13.0
4.|– if-14-1.mcore4.NYY-NewYork.as6453.net 0.0% 10 342.1 300.9 255.6 360.4 46.7
5.|– if-8-0-2-28.tcore1.NYY-NewYork.as6453.net 0.0% 10 283.3 262.3 257.7 283.3 7.8
6.|– Vlan24.icore1.NTO-NewYork.as6453.net 10.0% 10 265.3 262.0 257.1 269.9 5.0
7.|– ae9.edge1.NewYork.Level3.net 0.0% 10 260.1 264.0 258.8 275.9 6.8
8.|– vlan51.ebr1.NewYork2.Level3.net 10.0% 10 259.7 259.6 258.7 260.7 0.7
9.|– ae-4-4.ebr1.NewYork1.Level3.net 0.0% 10 259.7 260.8 259.7 263.7 1.2
10.|– ae-42-42.ebr2.London1.Level3.net 0.0% 10 248.5 247.3 246.3 248.5 0.7
11.|– ae-48-48.ebr2.Amsterdam1.Level3.net 0.0% 10 256.7 257.2 256.1 257.9 0.5
12.|– ae-59-224.csw2.Amsterdam1.Level3.net 10.0% 10 257.6 257.4 256.7 258.3 0.5
13.|– ae-2-52.edge4.Amsterdam1.Level3.net 10.0% 10 273.8 264.2 257.6 294.3 12.4
14.|– 213.244.170.88 10.0% 10 257.9 257.8 257.0 258.5 0.5

 

 

 

When I tested on Gmail phone, IP address was always: 209.85.175.126

This IP address belongs to Google’s ASN 15169 and covers in two block announcements by Google:

209.85.128.0/17 & 209.85.174.0/23

 

Quick traceroute to IP

HOST: anurag-laptop Loss% Snt Last Avg Best Wrst StDev
1.|– home-router 0.0% 10 1.7 1.8 1.5 2.5 0.3
2.|– 117.206.176.1 0.0% 10 23.6 23.1 22.6 24.0 0.4
3.|– 218.248.169.122 10.0% 10 30.4 34.5 27.9 76.8 15.9
4.|– 115.114.59.169.static-Mumbai.vsnl.net.in 0.0% 10 72.0 72.1 70.9 75.9 1.5
5.|– 121.240.1.38 0.0% 10 71.8 73.5 71.0 85.2 4.3
6.|– 209.85.241.52 0.0% 10 69.6 69.8 68.8 72.6 1.1
7.|– 209.85.251.95 0.0% 10 126.3 100.5 96.7 126.3 9.1
8.|– 66.249.94.38 0.0% 10 132.1 133.5 129.7 157.7 8.5
9.|– 209.85.242.125 0.0% 10 135.3 134.6 130.8 137.5 1.7
| `|– 209.85.241.173
10.|– 66.249.94.126 0.0% 10 138.2 136.8 130.8 142.7 4.1
| `|– 66.249.94.158
11.|– nx-in-f126.1e100.net 10.0% 10 135.8 137.1 135.8 138.3 0.9

 

 

137ms latency – very likely Singapore. Infact it’s confirmed also from my tests from various looking glasses.

 

 

Summary of results

  1. Skype seems using multiple gateways for PSTN calls and most of them are located in US and Europe. 
  2. Latency with these Skype gateways & Indian ISP networks varies from 250-350ms.
  3. Gmail seems using Singapore gateway for Indian users and latency is just 135ms.
  4. Latency & performance is very for Google since they have their own PoP in India and they carry International traffic themselves rather then replying on Public Internet infrastructure or upstream ISP’s.
  5. Calls beyond these gateways have usually very low latency as they reply on dedicated circuits & route over circuit switched network.
And hence Gmail phone performance better (in India!) 🙂