24 Sep

BSNL-Level3 bad routing case

Quick analysis of BSNL-Level3 bad routing issue

 

I can see BSNL having pretty high latency again with most of Europe again. It seems like they are using Level3 Communications AS 3356 along with Tata-VSNL for upstream. With Level3 transit BSNL has badly screwed up reverse path causing very high latency and awful bandwidth.

 

anurag@laptop:~$ ping server7 -c 5
PING server7.anuragbhatia.com (178.238.225.247) 56(84) bytes of data.
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=1 ttl=52 time=320 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=2 ttl=52 time=320 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=3 ttl=52 time=319 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=4 ttl=52 time=327 ms
64 bytes from server7.anuragbhatia.com (178.238.225.247): icmp_req=5 ttl=52 time=320 ms
--- server7.anuragbhatia.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 319.880/321.765/327.384/2.828 ms
anurag@laptop:~$

Expected latency values here should be around 150ms. A packet should not take more then 150ms round trip between Radaur, Haryana to Munich located server.

 

Quick view at traceroute:

anurag@laptop:~$ traceroute server7
traceroute to server7 (178.238.225.247), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) 1.440 ms 1.954 ms 2.433 ms
2 117.207.48.1 (117.207.48.1) 26.071 ms 29.648 ms 32.078 ms
3 218.248.173.42 (218.248.173.42) 34.528 ms 36.011 ms 39.674 ms
4 218.248.246.130 (218.248.246.130) 70.313 ms 72.635 ms 75.355 ms 5 so-6-0-2.edge1.London1.Level3.net (212.113.12.29) 324.058 ms 350.902 ms 351.340 ms 6 ae-1-51.edge5.London1.Level3.net (4.69.139.75) 349.419 ms 348.461 ms ae-2-52.edge5.London1.Level3.net (4.69.139.107) 348.564 ms
7 Telia (4.68.111.182) 349.354 ms ldn-b5-link.telia.net (213.248.96.37) 294.946 ms 296.696 ms
8 ldn-bb1-link.telia.net (80.91.246.96) 346.667 ms ldn-bb1-link.telia.net (80.91.248.217) 301.921 ms 304.189 ms
9 prs-bb1-link.telia.net (213.155.134.40) 426.722 ms prs-bb1-link.telia.net (80.91.247.34) 315.777 ms 318.168 ms
10 ffm-bb1-link.telia.net (80.91.245.102) 345.072 ms ffm-bb1-link.telia.net (213.155.132.157) 345.609 ms ffm-bb1-link.telia.net (80.91.245.102) 346.060 ms
11 mcn-b2-link.telia.net (80.91.248.29) 347.000 ms 348.939 ms 351.277 ms
12 gigahosting-ic-138043-mcn-b2.c.telia.net (213.248.101.78) 355.053 ms 356.168 ms 324.647 ms
13 server7.anuragbhatia.com (178.238.225.247) 321.058 ms 323.318 ms 332.473 ms
anurag@laptop:~$

Clearly hop 3 is New Delhi (30ms latency), hop 4 is Mumbai (again as per latency values). Hop 5 is London Level3. Seems like BSNL used Europe-India gateway link here (a submarine cable from Mumbai to London owned by multiple providers including BSNL and Bharti Airtel along with Global Crossing which is now owned by Level3). Also, as far as I know Level3 does not has a ISP license in India (doT’s list here) and thus they cannot sell bandwidth at Mumbai. Likely BSNL is using its own ILD license in this case and thus BSNL is responsible for purchase of bandwidth in London.

Thus, as per that traceroute and fact that BSNL is one who is purchasing transit from Level3 in London, BSNL should be having BGP session in London and should be exchanging it’s routing table in turn for global routing table provided by transit. While latency jumps as soon as we hit London as per that traceroute. Clearing BSNL > Level3 path seems OK while return path on Level3 > BSNL is faulty. 

 

Using Level3’s looking glass, we can have a quick check on traceroute to my IP:

Show Level 3 (London, England) Traceroute to 117.207.48.1
1 ae-51-51.csw1.London1.Level3.net (4.69.139.88) 0 msec
ae-52-52.csw2.London1.Level3.net (4.69.139.120) 0 msec
ae-51-51.csw1.London1.Level3.net (4.69.139.88) 0 msec
2 ae-227-3603.edge3.London1.Level3.net (4.69.166.154) 0 msec
ae-117-3503.edge3.London1.Level3.net (4.69.166.138) 0 msec
ae-226-3602.edge3.London1.Level3.net (4.69.166.150) 32 msec
3 gblx-level3-50g.London1.Level3.net (4.68.110.158) 8 msec 4 msec 0 msec
4 ae6.scr4.LON3.gblx.net (67.17.106.150) [AS3549 {GBLX}] 0 msec 0 msec
ae5.scr3.LON3.gblx.net (67.17.72.22) [AS3549 {GBLX}] 4 msec
5 so5-0-0-2488M.ar1.NYC1.gblx.net (67.17.64.146) [AS3549 {GBLX}] 104 msec
so6-0-0-2488M.ar1.NYC1.gblx.net (67.17.64.154) [AS3549 {GBLX}] 68 msec 68 msec
6 BHARTIBSNL.so-7-0-0.ar1.NYC1.gblx.net (64.210.30.70) [AS3549 {GBLX}] 268 msec 268 msec 264 msec
7 218.248.255.101 [AS9829 {APNIC-AS-3-BLOCK}] 276 msec 272 msec 276 msec
8 117.207.48.1 [AS9829 {APNIC-AS-3-BLOCK}] 272 msec 280 msec 276 msec

 

Hop3 – Level3, hop4 is Gblx (which is now owned by Level3), hop 5 is Gblx New York and hop 6 is BSNL router in New York. The target BSNL ip is coming from 117.207.48.0/20. Now interesting thing here is BSNL uses Level3 + Gblx both for transit. So return path via Gblx is not an issue but the path London > New York > India is surely an issue.

 

Looking for prefix 117.207.48.0/20 in Level3 London router:

BGP routing table entry for 117.207.48.0/20
Paths: (2 available, best #1)
3549 9829
AS-path translation: { GBLX APNIC-AS-3-BLOCK }
edge3.London1 (metric 20020)
Origin IGP, metric 100000, localpref 88, valid, internal, best
Community: Europe Lclprf_86 United_Kingdom Level3_Peer London 3549:4351 3549:7000 3549:30840
Originator: edge3.London1
3549 9829
AS-path translation: { GBLX APNIC-AS-3-BLOCK }
edge3.London1 (metric 20020)
Origin IGP, metric 100000, localpref 88, valid, internal
Community: Europe Lclprf_86 United_Kingdom Level3_Peer London 3549:4351 3549:7000 3549:30840
Originator: edge3.London1

Only two paths that too via Gblx. No direct return path. Again, it is not big issue since Gblx could have a return path right within London (or somewhere else in Europe).

 

Let’s check Gblx Europe router for entry for 117.207.48.0/20

 

route-server.ams2>show ip bgp 117.207.48.0/20 long
BGP table version is 176033437, local router ID is 67.17.81.187
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, x best-external
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i117.207.48.0/20 67.16.147.121 300 0 9829 i
* i 67.16.147.121 300 0 9829 i
route-server.ams2>

 Just one path. Doing a traceroute to see the actual path (since I don’t know where that next hop is located!) 🙂

 

route-server.ams2>traceroute 117.207.48.1
Type escape sequence to abort.
Tracing the route to 117.207.48.1
1 67.16.147.121 0 msec 72 msec 4 msec
2 BHARTIBSNL.so-7-0-0.ar1.NYC1.gblx.net (64.210.30.70) 380 msec 376 msec 380 msec
3 218.248.255.101 [AS 9829] 376 msec 376 msec 376 msec
4 117.207.48.1 [AS 9829] 384 msec 384 msec 384 msec
route-server.ams2>

 

Clearly here’s the issue. BSNL again is doing selective BGP announcement of prefixes at New York only and that is why Europe to India traffic is being routed via New York. BSNL is allowing entry path into it’s network from outside India only at New York and few other selected locations which causes serious damage to latency.

 

Time for me to get back on work of routing packets! Thanks for reading. 🙂

23 Sep

Back at College | Final year | Life in US |Thoughts on Linux!

 

Finally blog post after long time! 

Last few months were busy with US trip, and roaming around.  Things are pretty much funny as usual at college. US went OK as usual – finding better infrastructure, more politeness in general but weird relationship and culture in few cases. Quite a different food in Michigan and those $8-$10 bills for each dinner! It is always interesting to see how much different US is in reality then what we always hear from people around us and movies etc. 

 

Things which look strange to me whenever I visit US:

  1. How well system works with discipline
  2. No predicable eating pattern – a Apple can be breakfast one Monday, while entire Pizza on Tue!
  3. People drink more soft drinks in general then water. It is easy to find drinks over water.
  4. Internet is way too faster then India. (Will fix ours in a while!) 🙂
  5. Quite a few people live life on paycheck with absolutely zero savings.
  6. Too much of “step” brother/sister/father/mother concept, which isn’t fun when looking from Indian eyes.
  7. McDonald food is cheap (which is relatively expensive in India!) + no vegetarian food!
  8. Size of Walmart & Best Buy itself is a surprise & mystery!
  9. Efficient & quick legal systems help regulators to great extent 
  10. How inefficient & ridiculous Indian paper oriented bureaucracy has become!
  11. How boring a place looks with less population to Indian eyes which are used to crowd all around! 🙂

 

 

Linux…

 

Finally I am back in village into my “student life“. We have some interesting courses including Linux and Broadband Communications.  Syllabus for Linux looks good but classes are going awful. Well no offence to our teachers but seriously entire design of Linux courses in B.tech degree is crap. 

The course in B.tech 7th semester under Kurukshetra University covers basics of Linux from commands to basic idea of Linux Kernel, GRUB boot loader, Linux Networking (yay!), LDAP and other integration methods with Windows and finally Linux security concepts via PGP keys, firewall, NAT etc. To be true – course pretty much covers a lot if done in right manner but seriously based on few classes I have attended so far I can clearly see there won’t be much to learn for students in general from this “yet another subject” at college.

 

Here’s the image of Linux which most of students have:

  1. It’s all CLI and that too from UNIX days 🙂
  2. Fedora and CentOS are awesome, while simple ones like Ubuntu aren’t “powerful enough” to even try!
  3. Linux is all about cramming commands with all their arguments (well no concept of man tool here!) 
  4. Linux is all for geeky servers and can’t be used on desktop
  5. Linux is without any pre-compiled debian .deb or red hat .rpm packages and one must hit head on tasteless .tar.gz for installation
  6. File system is super hard to understand!
  7. No clue on Debian based Vs Red Hat based! (strange bird names huh?)

 

Now almost all these points are wrong based on what I have seen so far. I have been using Linux from over 6 years and being working a Linux systems admin from over 4 years. Linux has way too cool GUI  with gnome and KDE. Though I personally never liked KDE much due to instability but still it is way too better then assuming Linux is all about commands and one can’t even move a file without remembering mv command! 🙂

With regard to commands, this is a core fault in University’s course design. I expect students to be getting Linux GUI only version in one semester where one can get familiar with “life outside windows”, meaning of Linux file system etc rather then just cramming commands whole day long. Also, I was myself surprised on number of arguments taught for each command – seriously way too high and mostly useless. One can always find most of less used ones via manual tool (man $command) and thus there’s no point in cramming all of them in start or even ever!

With regards to distro – for some strange reason everyone prefers Fedora or CentOS missing the fact that they run laptops on their laps and servers tops! 🙂 
Most of these hard core Linux distros are more for non-GUI based servers and surely graphics look so un-developed and scary one. One has to understand that a powerful distro on server is not really the most powerful one on Laptop too. I would love to go for CentOS on my servers but would stick with Ubuntu, Mandriva, or Mint for desktop experience.  

Also, to start with one should stick with distro which has better repository as we don’t expect new Linux users to be messing up with compiling software packages. With regards to file system – college/syllabus does puts enough effort but still this all look meaningless to people unless they could simply roam around into file system with decent GUI rather then looking at / on testing ls command. 🙂

  

With hope that we will have more Linux admins out of young B.tech students, time to end this post!

 

 

Some important life changes…

Sidenote: I am not with Cloudaccess.net anymore. It was very nice experience of setting up AS54456 from scratch along with design of anycasting DNS instances.  Wishing Cloudaccess.net good luck for time ahead.

I just started working with Packet Clearing House (PCH). Looking forward towards exciting time ahead! Interesting pages to checkout – Anycasting DNS for TLDs  & Peering information

Thanks for reading.