29 Jan

Amazon India peering check

And here goes first blog post of 2018. Last few months went busy with some major changes in personal life. 🙂

I looked into Amazon’s India connectivity with various ASNs tonight. Here’s how it looks like. (Note: Jump to bottom most to skip traces and look at the summary data).

 

Amazon India to Vodafone India

traceroute to 118.185.107.1 (118.185.107.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 21.861 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 19.244 ms 19.233 ms
 2 100.64.2.200 (100.64.2.200) 14.789 ms 100.64.0.200 (100.64.0.200) 20.731 ms 100.64.3.12 (100.64.3.12) 13.187 ms
 3 100.64.0.193 (100.64.0.193) 14.418 ms 100.64.3.69 (100.64.3.69) 15.469 ms 100.64.3.67 (100.64.3.67) 15.946 ms
 4 100.64.16.67 (100.64.16.67) 0.343 ms 100.64.17.165 (100.64.17.165) 0.312 ms 100.64.17.199 (100.64.17.199) 0.313 ms
 5 52.95.67.213 (52.95.67.213) 1.942 ms 52.95.67.209 (52.95.67.209) 1.967 ms 52.95.67.213 (52.95.67.213) 1.935 ms
 6 52.95.66.218 (52.95.66.218) 4.998 ms 4.694 ms 52.95.66.130 (52.95.66.130) 4.650 ms
 7 52.95.66.67 (52.95.66.67) 1.752 ms 52.95.66.89 (52.95.66.89) 1.850 ms 1.806 ms
 8 52.95.217.183 (52.95.217.183) 3.111 ms 3.102 ms 3.088 ms <- Amazon India
 9 182.19.106.204 (182.19.106.204) 3.426 ms 4.547 ms 4.537 ms <- Vodafone India
10 118.185.107.1 (118.185.107.1) 2.035 ms 2.059 ms 2.039 ms

 

Amazon India to IDEA

traceroute to 223.196.83.1 (223.196.83.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 16.133 ms ec2-52-66-0-132.ap-south-1.compute.amazonaws.com (52.66.0.132) 14.232 ms ec2-52-66-0-130.ap-south-1.compute.amazonaws.com (52.66.0.130) 21.267 ms
 2 100.64.3.10 (100.64.3.10) 16.418 ms 100.64.1.76 (100.64.1.76) 20.674 ms 100.64.1.206 (100.64.1.206) 26.416 ms
 3 100.64.3.129 (100.64.3.129) 18.727 ms 100.64.3.135 (100.64.3.135) 13.790 ms 100.64.3.133 (100.64.3.133) 20.787 ms
 4 100.64.16.101 (100.64.16.101) 0.323 ms 100.64.16.195 (100.64.16.195) 0.296 ms 100.64.16.135 (100.64.16.135) 2.358 ms
 5 52.95.67.213 (52.95.67.213) 1.879 ms 2.452 ms 2.465 ms
 6 52.95.66.174 (52.95.66.174) 2.717 ms 52.95.66.130 (52.95.66.130) 8.568 ms 52.95.66.64 (52.95.66.64) 6.004 ms
 7 52.95.66.183 (52.95.66.183) 4.361 ms 52.95.66.73 (52.95.66.73) 11.474 ms 52.95.66.51 (52.95.66.51) 11.414 ms
 8 52.95.217.201 (52.95.217.201) 3.585 ms 3.627 ms 3.420 ms
 9 223.196.6.237 (223.196.6.237) 3.704 ms 3.832 ms 3.826 ms
10 223.196.15.26 (223.196.15.26) 6.155 ms 223.196.2.141 (223.196.2.141) 8.472 ms *

 

Amazon India to Reliance Communications

traceroute to 115.252.64.1 (115.252.64.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-132.ap-south-1.compute.amazonaws.com (52.66.0.132) 18.567 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 19.024 ms 19.020 ms
 2 100.64.0.72 (100.64.0.72) 22.429 ms 100.64.0.200 (100.64.0.200) 19.483 ms 100.64.2.142 (100.64.2.142) 18.679 ms
 3 100.64.1.67 (100.64.1.67) 11.717 ms 100.64.0.7 (100.64.0.7) 30.194 ms 100.64.2.195 (100.64.2.195) 11.700 ms
 4 100.64.16.195 (100.64.16.195) 0.353 ms 100.64.16.67 (100.64.16.67) 0.398 ms 100.64.17.163 (100.64.17.163) 0.400 ms
 5 52.95.67.209 (52.95.67.209) 1.847 ms 1.879 ms 1.891 ms
 6 52.95.66.108 (52.95.66.108) 5.567 ms 52.95.67.108 (52.95.67.108) 11.119 ms 52.95.66.174 (52.95.66.174) 9.029 ms
 7 52.95.66.141 (52.95.66.141) 2.938 ms 52.95.67.99 (52.95.67.99) 1.755 ms 52.95.67.165 (52.95.67.165) 1.755 ms
 8 54.239.43.171 (54.239.43.171) 18.605 ms 54.239.44.63 (54.239.44.63) 18.034 ms 54.239.43.171 (54.239.43.171) 18.522 ms
 9 52.93.19.126 (52.93.19.126) 63.893 ms 52.93.19.82 (52.93.19.82) 23.508 ms 52.93.19.62 (52.93.19.62) 20.812 ms
10 52.93.19.139 (52.93.19.139) 19.302 ms 52.93.19.133 (52.93.19.133) 18.600 ms 52.93.19.25 (52.93.19.25) 18.365 ms
11 115.110.161.61.static.vsnl.net.in (115.110.161.61) 17.966 ms 18.201 ms 115.110.161.65.static.vsnl.net.in (115.110.161.65) 17.948 ms
12 121.244.23.6.STATIC.Chennai.vsnl.net.in (121.244.23.6) 18.391 ms * *
13 115.255.253.69 (115.255.253.69) 27.234 ms 27.455 ms 27.652 ms
14 115.255.28.129 (115.255.28.129) 27.836 ms 121.244.23.6.STATIC.Chennai.vsnl.net.in (121.244.23.6) 18.906 ms 18.809 ms
15 115.255.232.1 (115.255.232.1) 27.524 ms 115.255.253.69 (115.255.253.69) 27.971 ms 115.255.232.1 (115.255.232.1) 27.777 ms
16 220.227.241.217 (220.227.241.217) 27.841 ms * 27.886 ms
17 115.255.232.1 (115.255.232.1) 27.628 ms 27.920 ms 27.965 ms
18 220.227.241.217 (220.227.241.217) 29.835 ms 220.224.230.177 (220.224.230.177) 28.402 ms *

 

Amazon India to Powergrid Corp

traceroute to 43.227.132.1 (43.227.132.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-132.ap-south-1.compute.amazonaws.com (52.66.0.132) 15.971 ms ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 11.558 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 19.813 ms
 2 100.64.0.14 (100.64.0.14) 18.356 ms 100.64.2.200 (100.64.2.200) 15.390 ms 15.390 ms
 3 100.64.2.1 (100.64.2.1) 21.122 ms 100.64.3.3 (100.64.3.3) 19.208 ms 100.64.1.1 (100.64.1.1) 21.094 ms
 4 100.64.17.165 (100.64.17.165) 0.283 ms 100.64.16.39 (100.64.16.39) 0.323 ms 100.64.17.103 (100.64.17.103) 0.341 ms
 5 52.95.67.215 (52.95.67.215) 1.638 ms 52.95.67.211 (52.95.67.211) 0.702 ms 52.95.67.215 (52.95.67.215) 0.749 ms
 6 52.95.67.64 (52.95.67.64) 1.103 ms 52.95.67.86 (52.95.67.86) 4.012 ms 3.986 ms
 7 52.95.67.51 (52.95.67.51) 0.599 ms 52.95.67.29 (52.95.67.29) 0.662 ms 52.95.67.51 (52.95.67.51) 0.651 ms
 8 52.95.219.49 (52.95.219.49) 7.935 ms 8.111 ms 8.675 ms
 9 kalwa-edge-177.230.218.103.powergrid.in (103.218.230.177) 1.452 ms 1.641 ms 1.169 ms
10 43.227.132.65 (43.227.132.65) 49.309 ms 49.233 ms 49.296 ms
11 43.227.132.1 (43.227.132.1) 58.909 ms 58.975 ms 58.893 ms

 

Amazon India to BSNL

traceroute to 218.248.235.130 (218.248.235.130), 30 hops max, 60 byte packets
 1 ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 15.199 ms 15.173 ms ec2-52-66-0-130.ap-south-1.compute.amazonaws.com (52.66.0.130) 16.306 ms
 2 100.64.2.140 (100.64.2.140) 18.831 ms 100.64.2.202 (100.64.2.202) 18.159 ms 100.64.2.78 (100.64.2.78) 15.493 ms
 3 100.64.1.1 (100.64.1.1) 16.829 ms 100.64.1.129 (100.64.1.129) 16.823 ms 100.64.2.193 (100.64.2.193) 16.813 ms
 4 100.64.16.193 (100.64.16.193) 0.403 ms 100.64.17.133 (100.64.17.133) 0.290 ms 100.64.17.99 (100.64.17.99) 0.289 ms
 5 52.95.67.215 (52.95.67.215) 0.803 ms 52.95.67.211 (52.95.67.211) 0.672 ms 0.702 ms
 6 52.95.66.130 (52.95.66.130) 7.382 ms 7.107 ms 52.95.67.174 (52.95.67.174) 2.292 ms
 7 52.95.66.179 (52.95.66.179) 1.819 ms 52.95.67.91 (52.95.67.91) 0.658 ms 52.95.66.91 (52.95.66.91) 1.798 ms
 8 52.95.217.42 (52.95.217.42) 0.534 ms 0.505 ms 52.95.217.38 (52.95.217.38) 2.010 ms
 9 182.79.178.199 (182.79.178.199) 1.938 ms 182.79.178.63 (182.79.178.63) 1.936 ms 182.79.178.201 (182.79.178.201) 1.926 ms
10 * * aes-static-042.105.144.59.airtel.in (59.144.105.42) 3.179 ms
11 218.248.235.157 (218.248.235.157) 3.106 ms 3.230 ms *
12 218.248.235.129 (218.248.235.129) 40.928 ms 40.433 ms 40.853 ms
13 218.248.235.130 (218.248.235.130) 40.307 ms 40.384 ms 40.280 ms

 

Trace from Amazon India to Google’s anycasted 8.8.8.8:

traceroute -A 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) [AS38895/AS16509] 12.662 ms ec2-52-66-0-130.ap-south-1.compute.amazonaws.com (52.66.0.130) [AS38895/AS16509] 20.008 ms ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) [AS38895/AS16509] 21.983 ms
2 100.64.0.142 (100.64.0.142) [AS22773] 15.525 ms 100.64.0.204 (100.64.0.204) [AS22773] 16.081 ms 100.64.1.206 (100.64.1.206) [AS22773] 19.696 ms
3 100.64.0.131 (100.64.0.131) [AS22773] 12.944 ms 12.893 ms 100.64.2.5 (100.64.2.5) [AS22773] 15.071 ms
4 100.64.16.103 (100.64.16.103) [AS22773] 0.318 ms 100.64.17.39 (100.64.17.39) [AS22773] 0.325 ms 100.64.17.69 (100.64.17.69) [AS22773] 0.291 ms
5 52.95.67.215 (52.95.67.215) [AS16509] 1.374 ms 5.725 ms 4.294 ms
6 52.95.67.130 (52.95.67.130) [AS16509] 4.896 ms 52.95.67.86 (52.95.67.86) [AS16509] 2.021 ms 52.95.67.108 (52.95.67.108) [AS16509] 5.735 ms
7 52.95.67.111 (52.95.67.111) [AS16509] 0.705 ms 0.672 ms 0.670 ms
8 52.95.218.225 (52.95.218.225) [*] 0.542 ms 0.570 ms 0.505 ms
9 108.170.248.161 (108.170.248.161) [AS15169] 0.853 ms 108.170.248.193 (108.170.248.193) [AS15169] 0.804 ms 108.170.248.177 (108.170.248.177) [AS15169] 0.770 ms
10 108.170.234.9 (108.170.234.9) [AS15169] 1.449 ms 209.85.248.147 (209.85.248.147) [AS15169] 1.725 ms 1.728 ms
11 google-public-dns-a.google.com (8.8.8.8) [AS15169] 0.460 ms 0.744 ms 0.449 ms

 

Amazon India to Microsoft Azure India

traceroute to 52.172.52.79 (52.172.52.79), 30 hops max, 60 byte packets
 1 ec2-52-66-0-132.ap-south-1.compute.amazonaws.com (52.66.0.132) 11.434 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 14.968 ms ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 14.694 ms
 2 100.64.2.8 (100.64.2.8) 15.301 ms 100.64.3.14 (100.64.3.14) 14.541 ms 100.64.2.10 (100.64.2.10) 15.289 ms
 3 100.64.0.131 (100.64.0.131) 19.954 ms 100.64.2.199 (100.64.2.199) 20.491 ms 100.64.1.135 (100.64.1.135) 20.473 ms
 4 100.64.16.67 (100.64.16.67) 6.650 ms 100.64.16.65 (100.64.16.65) 0.306 ms 100.64.16.133 (100.64.16.133) 0.307 ms
 5 52.95.67.213 (52.95.67.213) 1.917 ms 52.95.67.209 (52.95.67.209) 1.931 ms 1.980 ms
 6 52.95.66.130 (52.95.66.130) 3.015 ms 52.95.66.196 (52.95.66.196) 10.939 ms 52.95.66.86 (52.95.66.86) 11.644 ms
 7 52.95.66.91 (52.95.66.91) 1.802 ms 52.95.66.177 (52.95.66.177) 7.303 ms 52.95.66.199 (52.95.66.199) 3.435 ms
 8 52.95.218.61 (52.95.218.61) 16.785 ms 52.95.218.63 (52.95.218.63) 1.700 ms 1.674 ms
 9 ae0-0.bom02-96cbe-1b.ntwk.msn.net (104.44.226.17) 1.952 ms 1.897 ms ae14-0.bom30-96cbe-1b.ntwk.msn.net (198.206.164.159) 2.238 ms
10 ae14-0.bom30-96cbe-1b.ntwk.msn.net (198.206.164.159) 2.150 ms 2.172 ms ae5-0.maa30-96cbe-1a.ntwk.msn.net (104.44.226.241) 26.009 ms
11 ae5-0.maa30-96cbe-1a.ntwk.msn.net (104.44.226.241) 25.940 ms ae10-0.maa01-96cbe-1b.ntwk.msn.net (104.44.226.243) 27.591 ms 27.587 ms
12 ae10-0.maa01-96cbe-1b.ntwk.msn.net (104.44.226.243) 27.698 ms ae0-0.maa01-96cbe-1a.ntwk.msn.net (104.44.227.208) 28.090 ms *
13 * ae0-0.maa01-96cbe-1a.ntwk.msn.net (104.44.227.208) 27.682 ms *
14 * * *
15 * * *
16 * * *
17 * * *
18 *^C

 

Amazon India to Cloudflare India PoP

traceroute to amy.ns.cloudflare.com (173.245.58.101), 30 hops max, 60 byte packets
 1 ec2-52-66-0-130.ap-south-1.compute.amazonaws.com (52.66.0.130) [AS38895/AS16509] 13.912 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) [AS38895/AS16509] 15.167 ms 15.159 ms
 2 100.64.0.142 (100.64.0.142) [AS22773] 21.393 ms 100.64.3.72 (100.64.3.72) [AS22773] 16.098 ms 100.64.3.14 (100.64.3.14) [AS22773] 11.766 ms
 3 100.64.2.67 (100.64.2.67) [AS22773] 27.118 ms 100.64.2.131 (100.64.2.131) [AS22773] 27.097 ms 100.64.2.133 (100.64.2.133) [AS22773] 21.532 ms
 4 100.64.16.39 (100.64.16.39) [AS22773] 0.253 ms 100.64.17.193 (100.64.17.193) [AS22773] 0.256 ms 100.64.16.195 (100.64.16.195) [AS22773] 0.260 ms
 5 52.95.67.209 (52.95.67.209) [AS16509] 1.850 ms 1.846 ms 52.95.67.213 (52.95.67.213) [AS16509] 1.808 ms
 6 52.95.66.108 (52.95.66.108) [AS16509] 3.007 ms 52.95.66.174 (52.95.66.174) [AS16509] 13.312 ms 13.285 ms
 7 52.95.66.201 (52.95.66.201) [AS16509] 1.890 ms 52.95.66.135 (52.95.66.135) [AS16509] 2.571 ms 52.95.66.71 (52.95.66.71) [AS16509] 1.867 ms
 8 115.114.89.209.static-Mumbai.vsnl.net.in (115.114.89.209) [AS4755] 2.448 ms 115.114.89.189.static-Mumbai.vsnl.net.in (115.114.89.189) [AS4755] 1.984 ms 1.888 ms
 9 * * *
10 * 182.79.203.36 (182.79.203.36) [*] 2.459 ms *
11 182.79.223.54 (182.79.223.54) [*] 2.357 ms 2.368 ms 14.142.21.18.static-vsnl.net.in (14.142.21.18) [AS4755] 2.008 ms
12 amy.ns.cloudflare.com (173.245.58.101) [AS13335] 2.467 ms 182.79.203.44 (182.79.203.44) [*] 2.777 ms amy.ns.cloudflare.com (173.245.58.101) [AS13335] 2.445 ms

 

 

Some traces where traffic going from Amazon India is getting routed to Indian networks from outside:

traceroute to 125.18.237.1 (125.18.237.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 18.252 ms ec2-52-66-0-130.ap-south-1.compute.amazonaws.com (52.66.0.130) 17.868 ms ec2-52-66-0-132.ap-south-1.compute.amazonaws.com (52.66.0.132) 20.634 ms
 2 100.64.3.76 (100.64.3.76) 15.640 ms 100.64.2.202 (100.64.2.202) 13.602 ms 100.64.0.74 (100.64.0.74) 14.663 ms
 3 100.64.3.197 (100.64.3.197) 2.152 ms 100.64.2.69 (100.64.2.69) 2.128 ms 100.64.3.69 (100.64.3.69) 2.134 ms
 4 100.64.17.37 (100.64.17.37) 0.337 ms 100.64.16.37 (100.64.16.37) 0.354 ms 100.64.17.129 (100.64.17.129) 0.293 ms
 5 52.95.67.213 (52.95.67.213) 1.902 ms 1.861 ms 52.95.67.209 (52.95.67.209) 1.867 ms
 6 52.95.66.64 (52.95.66.64) 5.317 ms 52.95.66.218 (52.95.66.218) 6.883 ms 52.95.66.196 (52.95.66.196) 9.687 ms
 7 52.95.66.203 (52.95.66.203) 1.817 ms 52.95.66.69 (52.95.66.69) 1.844 ms 52.95.66.137 (52.95.66.137) 1.792 ms
 8 115.114.89.121.static-Mumbai.vsnl.net.in (115.114.89.121) 1.745 ms 115.114.89.209.static-Mumbai.vsnl.net.in (115.114.89.209) 1.799 ms 115.114.89.121.static-Mumbai.vsnl.net.in (115.114.89.121) 2.034 ms
 9 * * *
10 ix-ae-4-2.tcore1.CXR-Chennai.as6453.net (180.87.36.9) 54.804 ms * *
11 if-ae-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) 228.066 ms ix-ae-4-2.tcore1.CXR-Chennai.as6453.net (180.87.36.9) 56.587 ms 53.271 ms
12 if-ae-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) 225.894 ms 226.883 ms 227.007 ms
13 if-ae-6-2.tcore2.SVW-Singapore.as6453.net (180.87.37.14) 226.616 ms 227.081 ms if-ae-10-2.tcore2.SVW-Singapore.as6453.net (180.87.37.65) 226.699 ms
14 if-ae-7-2.tcore2.LVW-Los-Angeles.as6453.net (180.87.15.26) 227.935 ms 225.266 ms 235.497 ms
15 xe-4-1-0.edge1.SanJose3.level3.net (4.68.62.105) 247.387 ms 257.588 ms 249.384 ms
16 ae-116-3502.edge3.London15.Level3.net (4.69.167.78) 241.053 ms 241.539 ms ae-115-3501.edge3.London15.Level3.net (4.69.167.74) 242.991 ms
17 ae-116-3502.edge3.London15.Level3.net (4.69.167.78) 240.407 ms ae-225-3601.edge3.London15.Level3.net (4.69.167.90) 243.525 ms ae-226-3602.edge3.London15.Level3.net (4.69.167.94) 241.484 ms
18 BHARTI-AIRT.edge3.London15.Level3.net (212.187.139.82) 244.356 ms 244.610 ms *
19 * * *
20 * * *
21 * * *
22 *^C

 

Summary of results

Network Name Network ASN Domestic Connectivity
Tata Communications 4755 Yes*
Airtel 9498 Yes*
Idea Cellular 55644 Yes
Reliance Communucations 18101 No
Aircel 10201 No
Powergrid Telecom 132215 Yes
Railtel 24186 Yes
BSNL 9829 No
NKN 55824 No
CtrlS 18229 No
Syscon Infoway 45194 No
Intech Online 58678 Yes
Charotar Broadband 132933 Yes
Ishan Tech 45117 Yes
Webwerks 33480 No
Blazenet 17625 Yes
Spectra 10029 Yes
Excitel Broadband 133982 Yes
Google 15169 Yes
Cloudflare 13335 No
Microsoft 8075 Yes

 

Misc points:

  1. *TCL and *Airtel seem to be IP transit provider of Amazon. For certain routes going outside India traffic seem to be going via Tata VSNL AS4755.
  2. Connectivity checked for certain random pools only and not for all announced routes for a large telco network.
29 May

What makes BSNL AS9829 as most unstable ASN in the world?!

On weekend  I was looking at BGP Instability Report data. As usual (and unfortunately) BSNL tops that list. BSNL is the most unstable autonomous network in the world. In past, I have written previously about how AS9829 is the rotten IP backbone.

 

This isn’t a surprise since they keep on coming on top but I think it’s well worth a check on what exactly is causing that. So I looked into BGP tables updates published on Oregon route-views from 21st May to 27th May and pulled data specifically for AS9829. I see zero withdrawals which are very interesting. I thought there would be a lot of announcements & withdrawals as they switch transits to balance traffic.

If I plot the data, I get following chart of withdrawals against timestamp. This consists of summarised view of every 15mins and taken from 653 routing update dumps. It seems not feasible to graph data for 653 dumps, so I picked top 300. Here’s how it look like:


 

Except for few large spikes, it seems to have a relatively consistent pattern. We can see daily fresh announcements of close to 50,000 announcements.

This data gives no idea and I can’t say much by looking at it. Instead of looking at updates, I pulled last weeks RIBs and pulled AS9829 announcements. The idea here is to get map announcements to each upstream against time stamp along with announcements across various subnet masks.

Here’s total route announcement graph:

The graph above clearly shows that total routes announcements increased significantly on 23rd May at 06:00 UTC from 127664 to 129298. Thus dipped significantly at 14:00 on 26th May to 124301. So between 10:00 to 14:00 on 26th, the drop in routes as much as 4% drop clear indicating a large outage they had in their network.

Next part is to look at how they tweak their announcements to upstream.

So clearly they are announcing a large number of routes to Tata AS6453 and these are IPLC links where they are buying IP transit outside India. Some of these key spikes show a mirror among other transit giving a clear hint of circuit balancing by moving route announcement.

 

Next part is to view their announcements in terms of prefix size.


/20 as well as /22 as both seems relatively consistent except showing a dip on 26th.

 

So all I can say based on above data is following:

  1. BSNL had some issues last week. Possibly one of their upstream pipes had issues and they increased their announcements on Tata AS6453 during that time.
  2. They are an only large operator who is buying transits from as many as 9 upstream. This would result in broken capacity across at least 9 and possibly 30-40 circuits resulting in a major capacity management challenge across these upstream.
  3. They are announcing a large number of prefix sizes. /18, /20, /22, /23 and even /24s. This isn’t good practice at their large scale.
  4. They need to start peering. They are the only network of that scale who isn’t peering except with a couple of content players like Google AS15169. They need to peer aggressively inside India & follow same outside India if they actually keep on running such network. Or else even buying transit domestic only will be a better strategy.

 

Most of these problems can be fixed if BSNL aggregates it’s a number of transits (and circuits per transit) along with aggregation of routes. For a three transit scenario, they can follow /18, /20 and /22 strategy and leave /24 only for emergency cases to balance traffic.

07 Mar

Confusing traceroutes and more

And here goes my first post for 2017. The start of this year did not go well as I broke my hand in Jan and that resulted in a lot of time loss. Now I am almost recovered and in much better condition. I just attended HKNOG 4.0 at Hong Kong followed by APRICOT 2017 at Ho Chi Minh, Vietnam. an event and I enjoyed the both. Here’s my presentation from APRICOT 2017.

I recently I came across some of crazy confusing traceroutes as passed by one of my friends. I cannot share that exact traceroute on this blog post but can produce the same effect about which I am posting by doing a trace from one of large network like Telia London PoP to one of the Indian destinations via their looking glass

Example traceroute:

Router: London 
Command: traceroute inet 45.64.213.161 as-number-lookup


traceroute to 45.64.213.161 (45.64.213.161), 30 hops max, 40 byte packets
 1  ldn-bb3-link.telia.net (80.91.246.96)  3.186 ms ldn-bb2-link.telia.net (80.91.247.93)  91.337 ms ldn-bb3-link.telia.net (213.155.132.194)  0.512 ms
 2  ldn-b7-link.telia.net (62.115.114.177)  0.541 ms ldn-b7-link.telia.net (62.115.137.189)  0.600 ms ldn-b7-link.telia.net (62.115.141.151)  0.786 ms
 3  flag-ic-310275-ldn-b7.c.telia.net (62.115.47.106)  3.271 ms  0.777 ms  0.702 ms
 4  xe-0-0-1.0-pjr04.mmb004.flagtel.com (85.95.26.158) [AS  15412]  210.160 ms xe-3-0-1.0.pjr04.mmb004.flagtel.com (85.95.25.138) [AS  15412]  207.552 ms  207.699 ms
 5  * * *
 6  * * *
 7  nsg-static-222.29.72.182.airtel.in (182.72.29.222) [AS  9498]  149.335 ms  148.746 ms  148.834 ms
 8  45.64.213.161 (45.64.213.161) [AS  132933]  158.350 ms  155.053 ms  146.613 ms

 

 

Here’s trace is as London (AS1299) > London (AS15412) > Mumbai (AS15412) >>>> Somewhere in India (AS9498) > destination (AS132933)

 

So traffic enters India via Reliance and next handed off to Airtel and reaches the destination. Let’s check BGP table view of same PoP for this prefix:

Telia Carrier Looking Glass - show route protocol bgp 45.64.213.161 table inet.0

Router: London 
Command: show route protocol bgp 45.64.213.161 table inet.0



inet.0: 695650 destinations, 1553714 routes (695458 active, 1034 holddown, 554 hidden)
+ = Active Route, - = Last Active, * = Both

45.64.213.0/24     *[BGP/170] 5d 01:25:46, MED 0, localpref 200, from 62.115.128.73
                      AS path: 15412 18101 132933 I, validation-state: unverified
                      to 213.155.132.194 via ae0.0
                      to 213.155.132.196 via ae1.0
                      to 80.91.246.96 via ae15.0
                      to 80.91.246.114 via ae16.0
                      to 213.155.136.74 via ae22.0
                      to 213.155.136.76 via ae23.0
                    > to 80.91.248.217 via ae3.0
                      to 80.91.246.144 via ae4.0
                      to 80.91.247.91 via ae5.0
                      to 80.91.249.181 via ae6.0
                      to 80.91.246.146 via ae7.0
                      to 80.91.247.93 via ae8.0
                    [BGP/170] 5d 01:25:46, MED 0, localpref 200, from 213.248.64.252
                      AS path: 15412 18101 132933 I, validation-state: unverified
                      to 213.155.132.194 via ae0.0
                      to 213.155.132.196 via ae1.0
                      to 80.91.246.96 via ae15.0
                      to 80.91.246.114 via ae16.0
                      to 213.155.136.74 via ae22.0
                      to 213.155.136.76 via ae23.0
                    > to 80.91.248.217 via ae3.0
                      to 80.91.246.144 via ae4.0
                      to 80.91.247.91 via ae5.0
                      to 80.91.249.181 via ae6.0
                      to 80.91.246.146 via ae7.0
                      to 80.91.247.93 via ae8.0
                    [BGP/170] 2w2d 19:39:07, MED 0, localpref 150
                      AS path: 6453 4755 132933 I, validation-state: unverified
                    > to 62.115.9.174 via ae25.0

 

So out of both available routes both are 15412 > 18101 > 132933 direct and there are no AS9498 while Airtel (AS9498) does appear in the traceroute. 2nd last hop in the trace is 182.72.29.222 and that indeed belongs to Airtel.

If we trust routing table as well as the fact that usually Airtel and Reliance exchange domestic traffic only and typically we do not see AS15412 pushing traffic via Airtel. This means trace is wrong and it indeed is. Before we get to on why it’s wrong to let’s try to understand how exactly traceroute works.

 

Working of traceroute

The way traceroute works is by using TTL i.e Time to live on packets the tool is sending out. IP headers carry TTL to prevent them for looping forever. So for instance, if router R1 sends some traffic to router R2 and R2 is not learning that route from anywhere while has a default back to R1 then traffic will start looping between R1 and R2. IP routing prevents this by using TTL and IP packets are sent with certain TTL value and as soon as they cross a router, TTL is decreased. When TTL is zero a router is supposed to drop the traffic and not carry them any further. When a router drops traffic it is supposed to reply back with error “TTL exceeded”.

Now the way “traceroute tool” works is by sending packets with increasing TTL one after other. It sends first one with TTL 1. Router directly connected to it gets the packet. It reduces TTL (and 1 – 1 so it becomes zero) and since next TTL is now zero it just drops prefix instead of sending it further. And as a part of dropping it replies back to a system running a trace with “TTL time exceeded error” revealing it’s IP to the tool. Next, another packet will be sent with TTL 2 and it will cross 1st router & would drop on a 2nd router with “TTL time exceeded” revealing it’s IP.

 

 

Back to our problem…

Now, so that was about the working of traceroute. Now going back to the case I was discussing. Think of routing between two networks when routing is not symmetric. With asymmetric routing, I mean that source & destinations may be carried via different paths.

 

Say e.g here A is sending traffic to B via R1-R2 and B is replying back to A via R3. Now if A does a trace to B, R1 & R2 may appear fine but what source IP B uses to convey the message of TTL exceeded can confuse things. When packets reach B with TTL 1, B decrements TTL and drops them. Next to send that “TTL timeout exceeded message” B has two options:

  1. B can reply back from IP address on the interface connected to R2. Remember I am talking about B just using source IP for TTL exceeded error.  Actual reply path, of course, is via R3
  2. B can reply back from IP of address of the interface connected to R3 using the usual logic of how packets go out – use the source IP of the interface of the best path installed in the router

 

What logic B uses has it’s own advantages and disadvantages. If B follows #1 i.e sends TTL exceeded from the same interface which is connected to R2 then it will give very logical traceroute output. But if network R3 is filtering packets based on BCP38, it will just drop the traffic coming from B from R2’s IP. While if B follows #2 it won’t cause any issues with BCP38 but will confuse the traceroute replies as suddenly one hop in trace will appear from entirely another network. That is what exactly has been happening in the trace I shared above. Let’s read trace again.

 1 ldn-bb3-link.telia.net (80.91.246.96) 3.186 ms ldn-bb2-link.telia.net (80.91.247.93) 91.337 ms ldn-bb3-link.telia.net (213.155.132.194) 0.512 ms
 2 ldn-b7-link.telia.net (62.115.114.177) 0.541 ms ldn-b7-link.telia.net (62.115.137.189) 0.600 ms ldn-b7-link.telia.net (62.115.141.151) 0.786 ms
 3 flag-ic-310275-ldn-b7.c.telia.net (62.115.47.106) 3.271 ms 0.777 ms 0.702 ms
 4 xe-0-0-1.0-pjr04.mmb004.flagtel.com (85.95.26.158) [AS 15412] 210.160 ms xe-3-0-1.0.pjr04.mmb004.flagtel.com (85.95.25.138) [AS 15412] 207.552 ms 207.699 ms
 5 * * *
 6 * * *
 7 nsg-static-222.29.72.182.airtel.in (182.72.29.222) [AS 9498] 149.335 ms 148.746 ms 148.834 ms
 8 45.64.213.161 (45.64.213.161) [AS 132933] 158.350 ms 155.053 ms 146.613 ms

 

Here router right before destination i.e on hop7 is connected to Reliance & Airtel. It’s announcing the prefix covering the destination to Reliance and Reliance is bringing traffic but it’s using Airtel to send traffic out back to London router of Telia. While replying for “TTL exceeded” router 7 is using source IP of Airtel and thus we see the PTR record pointing to Airtel. This can be referred as “Random factoid” behaviour in traceroute. This comes from RFC1812 which suggests “ICMP source must be from the egress iface” and  Richard Steenbergen puts its very nicely in his presentation at NANOG here.

Checko

So that’s all about it for now!