One interesting case to report today – F root server has quite bad connectivity in India. Last week a friend asked me for traceroutes to all root servers and here’s what I saw when I did traceroute for F root from BSNL connection:

traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) [AS8151/AS28513] 1.508 ms 2.103 ms 2.614 ms
2 117.207.48.1 (117.207.48.1) [AS9829] 27.243 ms 29.811 ms 32.483 ms
3 218.248.173.42 (218.248.173.42) [AS9829] 39.861 ms 40.320 ms 40.755 ms
4 218.248.250.142 (218.248.250.142) [AS9829] 90.778 ms 93.919 ms 95.856 ms
5 * * *
6 * * *
7 * * *
8 * * *

There’s no route! 

This is not a overnight problem. I looked at my hosted RIPE Probe #1032 data and found this:

How can an ISP like BSNL can have missing route to one of key part of core Internet infrastructure?

Quick look at F root server:

F root server has a anycasting node deployed in Chennai, Tamil Nadu, India. This server is hosted at National Internet Exchange of India (NIXI) Chennai and ISC is responsible for this root server. 

F root uses 192.5.5.241 across all anycasting instances and in India this block is being announced by Autonomous System Number 24049 which is of ISC. AS24049 does a BGP announcement for  203.119.18.0/24 at NIXI Chennai and this is “supposed to be” taken by all ISPs participating at Chennai IXP. 

When I requested my friends across India for traceroutes to F root, it was a very interesting result!

We found connectivity works from everyone  including BSNL too for people living in or near around Chennai.

Here’s a traceroute from my friend in Chennai on BSNL to F root:

traceroute to f.root-servers.net (192.5.5.241), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 3.328 ms 3.311 ms 4.616 ms
2 59.92.64.1 (59.92.64.1) 32.489 ms 36.188 ms 37.448 ms
3 218.248.255.2 (218.248.255.2) 45.312 ms 48.171 ms 51.017 ms
4 218.248.250.142 (218.248.250.142) 63.279 ms 66.086 ms 69.025 ms
5 218.100.48.142 (218.100.48.142) 71.112 ms 72.939 ms 76.107 ms
6 192.5.5.241 (192.5.5.241) 79.131 ms 44.760 ms 46.550 ms

Here we can see hop 4 is core network of BSNL while hop 5 is NIXI Chennai and hop 6 is ultimately F root server. 

Initially I got feeling that this is because of broken IGP implementation for BSNL network since their border gateway routers are holding different routing tables and they are not syncing them properly but one strange thing here – Hop 4 in this traceroute i.e 218.248.250.142 is same as last hop in 1st traceroute (done from BSNL Haryana). How come same router has no route when looked from Haryana while it works for Chennai (and nearby) users! 

This gives clue that forward path is there but return path has issues. For some reason ISC router in NIXI Chennai is not able to get return path to reply for non-Chennai region users. 

I raised this concern with Network Operations Center of ISC yesterday along with mailing lists like APNIC & RIPE Atlas. One of my friend who is expert into these issues pointed us to right direction which is NIXI routing policy. 

Reading out policy:

“ An ISP at any NIXI node must at a minimum announce all its regional routes to the NIXI router at that NIXI location. All ISPs connecting to that NIXI node are entitled to receive these routes using a single BGP session with the NIXI router. This will guarantee the exchange of regional traffic within a NIXI node. This is referred to as forced regional multi-lateral peering under the policy”

So this is what is happening:

Each operator is providing ISC with limited regional routes around Chennai area and not to it’s entire network. This was later confirmed by ISC NOC reply. This is sort of awful situation in terms of policy which is breaking India backbones badly and ISPs not able to even reach root servers instances hosted within country. Worst, I have been told that there is 10 day time for fix of this problem else ISC is prepared to pull off plug from Chennai F root node. Since it’s anycasting, once there won’t be any instance inside India injecting routes, traffic will simply start flowing towards other F root server instances in nearby countries. If this happens, it will be surely a sad thing for India since we will loose one out of three four root nameservers.

With hope that next update on this issue will be positive, time to end this post for now. Feel free to share your comments below.

(Incase your are ISP or datacenter in India – I would be happy to discuss this issue with you. You can contact me directly from here)

***Updates***

ISC Network Operations Center acted very efficiently on this issue.  Special thanks to Mr Leo Bicknell from ISC who is taking care of it. In initial phase full routing table was made available to F root server via STPI Internet transit. This fixed the problem (on test basis) and after a week of testing,  F root instance in Chennai has been switched off until ISC & NIXI get into a contract over additional bandwidth and costs.

For now Indian traffic is being routed to other anycasting instance of F root. Earlier it was California, US instance and now it is Hong Kong instance.

traceroute to f.root-servers.net. (192.5.5.241), 30 hops max, 60 byte packets
1 router.local (192.168.1.1) [AS8151/AS28513] 2.536 ms 2.799 ms 3.624 ms
2 117.200.48.1 (117.200.48.1) [AS9829] 29.778 ms 32.488 ms 34.677 ms
3 218.248.173.38 (218.248.173.38) [AS9829] 36.966 ms 43.283 ms 44.119 ms
4 59.163.207.81.static.chennai.vsnl.net.in (59.163.207.81) [AS4755] 189.973 ms 192.398 ms 195.118 ms
5 ix-4-2.tcore1.CXR-Chennai.as6453.net (180.87.36.9) [*] 199.965 ms 202.296 ms 203.396 ms
6 if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 206.915 ms if-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) [*] 166.267 ms if-5-2.tcore1.SVW-Singapore.as6453.net (180.87.12.53) [*] 160.231 ms
7 if-2-2.tcore2.SVW-Singapore.as6453.net (180.87.12.2) [*] 161.044 ms if-5-2.tcore2.SVW-Singapore.as6453.net (180.87.15.69) [*] 164.876 ms 167.178 ms
8 Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 217.994 ms Vlan1779.icore1.HK2-HongKong.as6453.net (180.87.15.38) [*] 216.922 ms Vlan1870.icore1.HK2-HongKong.as6453.net (180.87.15.61) [*] 218.664 ms
9 isc2-FE.hkix.net (202.40.161.200) [AS2687/AS4862/AS9498/AS10026/AS1221] 206.692 ms 208.920 ms isc1-FE.hkix.net (202.40.161.202) [AS2687/AS4862/AS9498/AS10026/AS1221] 211.561 ms
10 f.root-servers.net (192.5.5.241) [AS55440/AS3557/AS23708/AS8167] 247.908 ms 252.645 ms 253.544 ms

I am looking forward towards permanent fix and will update here once I get updates or find any further changes in the routing tables.

Based on my collected data and BGPlay’s routing records, here’s what’s happening. My IP is coming /20 BGP annoucement from BSNL Autonomous System 9829 - 117.207.48.0/20. Looking at BGP table records for that block from BGPlay’s routing data archive source.

On Sunday Morning 00:00am UTC, BSNL was found to be announcing 117.207.48.0/20 from AS9829 which was carried over via it’s upstream ISPs – Tata Communications (AS6453) and Reliance Globalcom (AS18101). From Tata’s AS6453 most of other Tier 1 backbones and many other small ISPs were getting routes. Similarly in case of Reliance, AS18101 was announcing blocks to it’s other network FLAG Telecom AS15412 which was further passing routes across many ISPs in the world. ISPs like Tinet, AT&T, Savvis, Seabone, Sprint etc were getting announcement via AS6453 while Level3, NTT, Hurricane Electric, Swisscom & many others were getting annoucements via Reliance-FLAG backbone. 

At this instance – connectivity to network via Reliance route looks pretty good. All traffic comes via direct path – entering Reliance’s network from nearest point & next reaching BSNL. While at the same time, unfortunately in case of other path via Tata Communications – it seems like no matter where traffic orignates from, it is always routed via US. That is even if traffic is orignating from Singapore, it is being routed to India via US.

Quick check on Tata Communications AS6453 PoPs at this instant:

Router: gin-svq-core1
Site: SG, Singapore – SVQ, EQUNIX
Command: show ip bgp 117.207.48.0/20

BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (3 available, best #2)
Multipath: eBGP
9829
nyy-mcore4. (metric 3777) from tv2-core1. (tv2-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3777) from hk2-core3. (hk2-core3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3777) from s9r-core1. (s9r-core1.)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

Router: gin-lhx-core1
Site: GB, London – LHX, TATA COMM. HARBOR EXCHANGE
Command: show ip bgp 117.207.48.0/20

BGP routing table entry for 117.207.48.0/20
Bestpath Modifiers: deterministic-med
Paths: (2 available, best #1)
Multipath: eBGP
9829
nyy-mcore4. (metric 3036) from ldn-mcore3. (ldn-mcore3.)
Origin IGP, valid, internal, best
Community:
Originator: nyy-mcore4.

9829
nyy-mcore4. (metric 3036) from l78-tcore1. (66.110.10.237)
Origin IGP, valid, internal
Community:
Originator: nyy-mcore4.

Thus we can see in both cases – Tata’s router is getting updates from nyy-mcore4 which is their router in New York city. This forces packets all way down to New York before they are routed back in Asia to BSNL. 

Next, same thing goes for next few hours. On Sunday noon time 12:55pm, we can see a path change from multiple providers like Savvis AS3561. We can see change in routing table and now between Tata AS6453 and BSNL AS9829, a new network comes in. It is AS4755 which is Tata Communications other network VSNL. Tata still uses AS4755 in India and AS6453 everywhere else.

Within next few seconds, we can see similar changes from AS293 – ESnet, 
 

Path Change from 3561 6453 9829
to 3561 6453 4755 9829

Path Change from 293 6453 9829
to 293 6453 4755 9829

Path Change from 3303 15412 18101 9829
to 3303 6453 4755 9829

Path Change from  812 6453 9829
to 812 6453 4755 9829

Path Change from 3257 6453 9829
to 3257 6453 4755 9829

Path Change from 3130 1239 6453 9829
to 3130 1239 6453 4755 9829 

Path Change from 7018 6453 9829
to 7018 6453 4755 9829 

Path Change from 1299 6453 9829
to 1299 6453 4755 9829 

Within a min, we can see half of routes are going via AS4755 rather then AS6453 directly.

Next, we see a route withdrawal – 701 6453 9829 followed by route re-annoucement 701 6453 4755 9829 and with this we see whole block being announced via AS4755 (which exists only in India as per as I know). Next on 12:58pm, we see another series of path changes all reversing back to AS6453 – AS9829 skipping AS4755 in between. On 12:50pm we can see half of routes being back with AS6453 link directly and half still with AS6453. Very soon all routes get back on direct link AS4755 goes out of picture again.

Summary of what’s happening:

1. BSNL’s network is having high latency on various routes including Singapore and Europe.
2. For specific block in testing, we can see BSNL is announcing them via Tata Communications & Reliance Globalcom.
3. Tata Communications uses two autonomous systems – AS4755 (VSNL) for Indian operations & AS6453 for everywhere else.
4. We can see routing table changes almost daily which bring AS4755 between AS9829 and AS6453 on random basis.
5. AS4755 is believed to be operated only in India and I can see whenever AS4755 is coming in picture, packets to BSNL-AS9829 are handed off directly and latency is pretty good.
6. For other times when we have AS6453 > AS9829 routing, we can see block is being annouced from AS6453 in New York.
7. Very likely problem is NOT from Tata Communications end, but rather from BSNL’s end. BSNL is constantly switching annoucing peers – AS4755 at one end in India while AS6453 router on other end in New York. Though it is hard to confirm this speculation.

Glad Airlines don’t route flights in the manner in which wrong routing goes!

Subnetting IPv6 sounds very complex but to be true – it is very easy!
All you need to do is to understand basics of IPv6 addressesing - how an address is formed and how to efficiently use CIDR notation.

Firstly how an IPv6 address looks like? (good to clear fundamentals first!) 

An IPv6 address has 8 sections seprated by coloums and each sections has carries 4 hexadecimal digits. So an IPv6 address is something like:

xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx – Each x can have a hexa decimal value i.e from 0 to 9 and a to f. Thus 16 possible values for each x. Since each each x is stored in binary i.e 0 or 1 (that is 2 possible value) – number of bits per section turns out to be 2x2x2x2 = 16bits. Thus we have now each section with 16 bits per section and 8 sections in total. This turns out to be 16 + 16 + 16 + 16 + 16 + 16 + 16 + 16 bits = 128bit. This is why an IPv6 address has 128bits.

This means total possible addresses in IPv6 space is 2^128 = 340 282 366 920 938 463 463 374 607 431 768 211 456 addresses.

Next, an important point to remember here is  - in IPv6 address clients are mostly based on /64 subnet which means first 64 bits go to network part while next 64 bits go to the host part i.e usage IPv6 addresses which are allocated to end machines.

Now getting back on main question on how to subnet IPv6?

In most of cases RIRs like ARIN/APNIC allocate a /32 IPv6 block. This means first 2 sections 16+16 bits are reserved and rest 6 sections i.e 128-32 = 96 bits are available for use. 

E.g let’s pick example of Google’s block.

Google has a allocation of 2404:6800::/32 from APNIC in Asia. 

Now this is HUGE chunk.

First let’s understand what is range of 2404:6800::/32 looks like. :: here means that zeros are skipped and thus we can fill zeros to understand block.

2404:6800::/32 means = 2404:6800:0000:0000:0000:0000:0000:0000/32 and since only first 32 bits (16 bit per section) are reserved – we have first 2 sections reversed while rest 6 sections are available and we can fill any hexa decimal value in those sections.

Thus block 2404:6800::/32 goes from 2404:6800:0000:0000:0000:0000:0000:0000 to 2404:6800:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Now that’s a huge number of address space. You can simply count it by doing 2 to the power 96 (128-32) which will be 792 281 625 142 643 375 935 439 503 36 unique possible addresses!

Breaking it down further…

1. If you have multi datacenter setup – it is very likely that you would like to use IPv6 space across multiple locations and thus doing a BGP announcement for whole /32 isn’t a very good idea. 
2. Many people on NANOG mailing list suggested me to use /48 block as it works well with BGP and most of ISPs do accept a /48 block.
3. Most of servers are allocated /64 block of IPv6 further down.

So in idea situation – you would have to break your /32 allocation into multiple /48s – which you can annouce from BGP and further /64s out of /48 for allocation per server/per client.

At this point it is likely that will think of how many such small blocks are possible out of main bigger block?

Ok – here’s the answer. You can break /32 into 65,6536 /48s. Each can represent a separate network below a BGP session. Next, you can further break /48 block into 65,536 /64s and each /64 you can allocate to a client. Thus each client will have 2^64 addresses i.e 184 467 440 737 095 516 16 addresses per client! 

Let’s break it!

Coming back on example of Google’s block - 2404:6800::/32 here to get /48s out of the block – all you need to do is to change the 3rd section. Remember as each section represents 16bits, altering 3rd section gives 16+16+16 = 48 bits. Thus possible /48s out of 2404:6800::/32 will be

2404:6800:1::/48
2404:6800:2::/48
2404:6800:3::/48 

also since it takes hexadecimal values, we can put a,b,c,d,e & f.

2404:6800:a::/48 
2404:6800:b::/48 

one can also use complete combination to fill all 4 digits i.e

2404:6800:XXXX::/48

here XXXX can take hexa decimal values of 65,536.

next, in similar manner altering 4th section gives /64s. Possible /64s out of Google’s IPv6 block:

2404:6800:1:1::/64
2404:6800:1:2::/64
2404:6800:1:3::/64

Next, each client can alter last 4 sections – and generate ton of IPv6 addresses!

E.g unique IP addresses 2404:6800:1:1::1 which is 2404:6800:1:1:0000:0000:0000:0001 

2404:6800:1b11:21dd:00ab:0030:0020:0001  or just anything! 

Quick point to remember here:

1. If you alter JUST the last i.e 8th section you can have 65,536 (2^16) IPs.
2. If characters in hexa decimal values confuse you, you can simply take last section values from 0 to 9999 i.e 10k possible IPs by just altering last section without hexa decimal.
3. Its a good idea to alter just last section and fill zeros in 5th, 6th and 7th section because 10k IPs would be sufficient per server and one can always add more later.
4. Also when filling 0 in 5th, 6th and 7th section, one can simply use double coloumn notation i.e  2404:6800:1:1:0000:0000:0000:0001 can be written as 2404:6800:1:1::1 skipping all zeros!

Well that’s all about IPv6 addressing. Hope you will find it useful!

Today I will be talking about a big problem. This is not about a routing glitch, or missing DNS entry (my usual blog posts…). It’s about one of fundamental problems in India which have ruined our society, relations & way of living.

It’s about Caste based discrimination in India. As we all know Jats demand for reservation isn’t something very old and people from Haryana did suffered quite a lot because of those protests. Overall I am just surprised from demands of Jats who are majority in Haryana, who grab over 80% of Govt. based jobs already in current system, who play a key role in choosing Govt. of Haryana, who have the Chief Minister from their caste ruling the state!
Infact Jats are considered as one of high & rich class in Haryana, and considered among ones who enjoy very good life style in India. And now they are ready to give all that pride & values just for getting so called “reservation”. Shameful.

My today’s post is not triggered by Jat reservation protest but because of painful life of my good friend who (call it unfortunately) is from higher caste but financially not very strong. I could see him running at admin block of my college from last one week. He was trying to get permission for getting free books from college library’s book bank.

Overall book bank concept was is something like – Free books for scheduled and backward class students (regardless of fact if or if not they can afford purchasing own books). Luckly, my friend (who is from upper caste) got permission to get free books from registrar sir on ground that he’s financially weak and cannot afford purchasing own books. I was quite happy to see college is supporting these students rather then sticking to caste based thumb rule. 

But story does not ends here. There’s another student in our class who is from backward class. (I would rather call him of backward mindset rather then caste). He came to knew that my friend (who’s from upper class) got permission to get free books. He got so angry that he forcefully grabbed permission letter from my friend and tried best to keep him away from library unless he himself applied for free book offer. His action is not justified even if he was yet another financially weak student but irony – he’s one of richest guy in class (based on expenditure). Overall cost of books for this semester would be around 1000Rs ($20 USD). Is that really very expensive for backward class (again call it mindset) student? No.

The backward class guy owns phone costing over $300, his laptop costs over $1200, external hard disk of over $100 (for storing precious porn!) and his monthly pocket money is over $50 already. This is an absurd & shameless use of system. This is why whole concept of caste based reservation is meaningless and completely stupid idea. And I am sad because I can see young generation is nourishing it well in worst possible manner.

Whole idea of reservation was essentially made to end the caste based “discrimination“, while it started yet another discrimination just from other side of caste system. Thus backward class boy gets reservation always (including the one he got while entering this college which he didn’t deserved other wise), he gets free books, next reservation in higher studies and worst reservation even at job level and my good friend who as I said unfortunately is from upper class has not much from our society except general hard working life and capability to compete against these idiots. Reservation system is probably only system in India where money doesn’t matter!    

Sad to see such things happening at bottom of Pyramid – that is level where we live. This further reminds me of my plans to join politics at some later stage in life (certainly when I can “afford it”).

With hope that we will see a better system, less prone to such abuse – time for me to end this post with this nice song from movie “Peepli live“

Thought to put a quick blog post on the issue.

So what’s exactly wrong? 

To understand what’s wrong, let’s understand how DNS works at core level. 

DNS relies on a hierarchy model with . (dot) on top which is Root and TLD i.e Top Level Domains below Root, which further  follow 2nd level domains which are popularly domain names we use.

So e.g mail.google.com is actually like

.
com.
google.com. 
mail.google.com.

First 3 are real DNS zones with own delegation. Let’s see their DNS servers using dig:

anurag@laptop:~$ dig . ns +short
j.root-servers.net.
b.root-servers.net.
c.root-servers.net.
a.root-servers.net.
l.root-servers.net.
g.root-servers.net.
e.root-servers.net.
k.root-servers.net.
f.root-servers.net.
m.root-servers.net.
d.root-servers.net.
h.root-servers.net.
i.root-servers.net.

Next, com.

anurag@laptop:~$ dig com. ns +short
l.gtld-servers.net.
f.gtld-servers.net.
g.gtld-servers.net.
j.gtld-servers.net.
i.gtld-servers.net.
a.gtld-servers.net.
h.gtld-servers.net.
k.gtld-servers.net.
m.gtld-servers.net.
c.gtld-servers.net.
e.gtld-servers.net.
d.gtld-servers.net.
b.gtld-servers.net.

Next, google.com.

anurag@laptop:~$ dig google.com. ns +short
ns2.google.com.
ns3.google.com.
ns4.google.com.
ns1.google.com.

So here dot was the “root zone” which is on top of hierarchy, next com is Top Level Domain, just like net, org, in, us etc. Next, google.com. is 2nd level domain. Nameservers which hold data for google.com domain name sit on gTLD servers of com while root holds ALL dns servers of all Top level domains. So root knows who knows about com/net/org/biz/asia/in/se/us etc. 

There are 13 root servers in world theoritically but actual number is over 100 since they are using anycasting very much and have nodes across multiple places. You can read more on official site of Root Servers along with their location map here.

That was the fundamental part. Coming back on main point, what’s missing in India?

We have 4 root servers deployed at Delhi, Mumbai & Chennai which seems like decent number but there are NO gTLD servers at all. Thus India relies on external world for resolving gTLD domains like com/net/ org. This is real problem. If you are from India, I would suggest you to take traceroutes to each of gTLD servers i.e

l.gtld-servers.net.
f.gtld-servers.net.
g.gtld-servers.net.
j.gtld-servers.net.
i.gtld-servers.net.
a.gtld-servers.net.
h.gtld-servers.net.
k.gtld-servers.net.
m.gtld-servers.net.
c.gtld-servers.net.
e.gtld-servers.net.
d.gtld-servers.net.
b.gtld-servers.net.

and pass me on directly on email or via comments on the page.

Here is my original post at NANOG mailing list.

Hello readers!

I am blogging after quite a few days. Spent bit of time at events like APRICOT 2012 – New Delhi, followed by (unexpected) college start (on time) with heavy fine!

Overall time is moving fast. Last week TeamARIN posted a guest blog post from me on DNS in case of IPv6. Apart from that I was excited with small article from RIPE NCC about K-root server connectivity issue as reported by me on mailing list. I have been hosting a RIPE NCC Probe for Atlas Project at home and its overall going quite good. It’s giving really very interesting data about our connectivity with root nameservers, response time, latency etc. I will spend some time on analysis of latest data and will post it here.

That’s all from me from now. Overall exciting times ahead!

Few days back I mentioned how reverse DNS setup of Airtel was incorrect. Sad to say it has not been fixed yet.

In meanwhile I was looking at domain name – airtel.in the main domain which runs website for Bharti Airtel’s Indian operations. I am little surprised to find that DNS server of airtel.in are failing randomly! 

Problem:

airtel.in uses 4 DNS servers from Mantra Online – a small ISP which Bharti took over years back. Here are the DNS servers used by domain name:

aaadel.mantraonline.com.
dnsbom.mantraonline.com.
dnsdel.mantraonline.com.
dnsblr.mantraonline.com.

Now interesting part here is that out of these 4, only 1 behaves normally. 

DNS server – dnsblr.mantraonline.com. seems working fine but rest all are rejecting queries “randomly” which is interesting. I have mostly seen DNS servers being up or down. This is probably first case when I can see DNS servers failing in random fashion.

Let’s query rest 3 DNS servers one by one:

anurag@laptop:~$ dig @aaadel.mantraonline.com airtel.in ns

; <<>> DiG 9.7.1-P2 <<>> @aaadel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 63903
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;airtel.in. IN NS

;; Query time: 81 msec
;; SERVER: 202.56.230.6#53(202.56.230.6)
;; WHEN: Mon Feb 6 01:25:20 2012
;; MSG SIZE rcvd: 27

In another 5 random tries, here’s what I get:

anurag@laptop:~$ dig @aaadel.mantraonline.com airtel.in ns

; <<>> DiG 9.7.1-P2 <<>> @aaadel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2044
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;airtel.in. IN NS

;; ANSWER SECTION:
airtel.in. 86400 IN NS dnsblr.mantraonline.com.
airtel.in. 86400 IN NS dnsdel.mantraonline.com.
airtel.in. 86400 IN NS aaadel.mantraonline.com.
airtel.in. 86400 IN NS dnsbom.mantraonline.com.

;; ADDITIONAL SECTION:
aaadel.mantraonline.com. 86400 IN A 202.56.230.6
dnsblr.mantraonline.com. 86400 IN A 202.56.250.5
dnsbom.mantraonline.com. 86400 IN A 202.56.240.5
dnsdel.mantraonline.com. 86400 IN A 202.56.230.5

;; Query time: 87 msec
;; SERVER: 202.56.230.6#53(202.56.230.6)
;; WHEN: Mon Feb 6 01:26:05 2012
;; MSG SIZE rcvd: 191

This time it worked. Pretty crazy. Same applies on other 2 DNS servers too:

anurag@laptop:~$ dig @dnsbom.mantraonline.com airtel.in ns

; <<>> DiG 9.7.1-P2 <<>> @dnsbom.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 29601
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;airtel.in. IN NS

;; Query time: 82 msec
;; SERVER: 202.56.240.5#53(202.56.240.5)
;; WHEN: Mon Feb 6 01:28:21 2012
;; MSG SIZE rcvd: 27

anurag@laptop:~$ dig @dnsdel.mantraonline.com airtel.in ns

; <<>> DiG 9.7.1-P2 <<>> @dnsdel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 34334
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;airtel.in. IN NS

;; Query time: 86 msec
;; SERVER: 202.56.230.5#53(202.56.230.5)
;; WHEN: Mon Feb 6 01:27:42 2012
;; MSG SIZE rcvd: 27

Pretty crazy case. Something is wrong at DNS servers itself – not sure what’s logic of rejecting queries randomly. But anyways – http://www.airtel.in will always open since 1/4 DNS server seems working normal. If that’s the case then Airtel still won’t be loosing much of traffic but unfortunately case is further complex.

Another problem… 

Remember that 4 DNS servers mentioned here are the ones which are NS records at “airtel.in” zone at delegated servers. In other terms these are just servers which host and have NS entries for the zone but root nameservers of in-registry hold only 2 DNS servers in total which host zone. A quick whois check reveals that airtel.in is using:

Name Server:AAADEL.MANTRAONLINE.COM
Name Server:DNSDEL.MANTRAONLINE.COM

and thus missing the only server which gives consistent results. Incoming traffic never hits other 2 DNS servers which are just mentioned in NS.

Poor & bad DNS setup!

With hope that you hit right server this month rather then dead servers before reaching the working one, time for me to say Good night!

Sify is one of really interesting companies. One time pioneer of Indian internet market via chain of cyber cafes. Good old days. Present situation of Sify in consumer market is not significant. Latest earing figures clearly state company is moving towards enterprise segment.

Company is quite aggressive in enterprise segment offerings specially datacenters & corporate leased lines.
Is consumer market really over for Sify or there’s still some hope?

Well, consumer broadband market isn’t really over! Infact this is the main market which is yet to explode in India!

These are the factors which went against Sify:

1. Company over estimated growth via Cybercafe chains. Cheap computers killed market of cyber cafe itself!
2. Company failed to capture high end consumer segment and lost it to DSL players like BSNL/MTNL & Airtel. Quality of services was way too low (due to franchise model) adopted for distribution via last mile cable operators.
3. Low end users were stolen by cheap wireless internet via 2G (and now 3G).
4. Company tried using pointed wireless links when technology was very new & yet to become advance with MIMO & beam forming.
5. Indian regulation which prohibits VoIP-PSTN connectivity again put them on back seat against main telcos like BSNL & Airtel.

Despite of these issues, I still see a huge potential for market which Sify tried to capture. Taking example of this village Radaur (where I am staying) – I can see over 20 connections to small computer shops, cell phone shops etc from Sify broadband. This is very interesting as a private broadband player is in village besides Govt. telecom player BSNL. Infact I can see lot of small shops giving priority to Sify over BSNL. I tried asking them for reason and here’s list of those:

1. Fast installation 1-2days Vs 30+ days for BSNL.
2. Fast fix in case of downtime – in few hours Vs 2-3 days for BSNL.
3. No issues of breaking of cable (as in case of BSNL)

Well, these all issues are quite true. Most of these shops have subscribed to 256Kbps to 512Kbps plans and they work pretty well (as per my tests). Here Sify is using a “point to multi-point wireless network” to reach last mile. They run last mile over 60 degree sector antenna located in front of Radaur bus stand pointed towards village. Here’s the picture:

and here’s one of end user CPE antenna’s pointing towards it:

Unfortunately 1st picture isn’t clear.
It has small antenna with two parabolic antenna’s on top – one towards Yamunanagar city & other towards Ladwa village. It uses high end frequency band of 5.8Ghz which has over 40Mhz of bandwidth in free WiFi spectrum for tower backhaul. This sort of backhual works pretty well since one can get over 100Mbps for 40-50Km of distances (in this case Yamunanagar district is 20Km off from here) where they feed core network from leased fibers and run a MPLS on top of it. Though backhual capacity of such towers is quite high and for small village (with less users) – even last mile shared sector antenna too isn’t a problem. What surprises me is that – still at the end of day Sify offers similar or worst plans as compared to BSNL. Reason remains bit of backbone bandwidth costs, high amount of commission taken by middle re-sellers etc. Sify offcourse failing here to realize big market and capturing it with good marketing followed by massive deployment. At this point low end market will be again grabbed by cell phone players with 2G/3G while high end will sooner or later go to BSNL or other telcos (if last mile unbundle happens) since DSL is way more scalable then this technology apart from fact that BSNL won’t have any issue in backbone (when they have 104 pair fiber landing in village exchange!). Unless last mile unbundle happens, there’s hardly any technology which can provide wired broadband equivalent speeds for this network.

With hope for good future of broadband in India, time for me to wash my clothes!

Issue was reported by Kevin, Senior Engineer at Altus Communications (AS11325). Problem was that SBJ Media LLC (AS33611) was making a /24 block announcement for specific slices of Altus –  208.110.48.0/20, 63.246.112.0/20, and 68.66.112.0/20 which are allocated to Altus Communications (as per ARIN whois).

Good news for now is problem seems on it’s way to fix, and route servers of At&t and Hurricane Electric are showing right path for /24 blocks.Just now Kevin updated NANOG saying: 

I hope none of you ever get hijacked by a spammer housed at Phoenix NAP.
We’re still not out of the woods, announcing /24s and working with upper
tier carriers to filter out our lists. However, I just got this response from Phoenix NAP and found it funny. The “thief” is a former customer,whom we terminated their agreement with. They then forged an LOA, submitted it to CWIE.net and Phoenix NAP and resumed using space above and beyond their terminated agreement.

This one is very interesting case and shows even today there’s no guarantee of correct routing on the Internet. So many autonomous systems out there but still at the end of day routing somehow works! 

What an ISP can do in such cases? (what I myself learned from looking at such cases so far):

1. Small chunks like /24 are given more priority over /20, thus if someone hijacks /24 out of your /20 block then you can (should) also start announcing /24 to make sure hijacker does not get any additional benefit by announcing small specific route.
2. Pick out upstream ISP’s of attacker’s autonomous system & eventually get announced prefixes filtered out at the source itself.
3. Pick your upstream ISP’s and eventually request them for prefix filtering. 

This whole incidence reminds me of YouTube blackout in 2008 by Pakistan Telecom. Other then prefix filtering by big ISP’s one can’t really do much if such wrong announcement continues.

With hope that your ISP’s network is not “stealing” others IP’s time for me to go out for morning walk in village!

Special thanks to John Schneider from Iowa Network Services for his inputs & answering my questions!

Here we start!

Testing routing to European networks from Tata Communications Mumbai nodes.
Test IP of my Germany based server: 93.104.209.174 

Checking routing from both Autonomous Systems of Tata Communications – AS4755 (VSNL) and AS6453 (TeleGlobe).

Click to expand ping result

326ms latency is way too high for route between India and Europe.

Next, checking from other Autonomous System AS6453:

Click to expand ping result

132ms – pretty good something what we can expect for this route.

So what’s really wrong? Why VSNL-AS4755 has high latency while AS6453 low – even when AS4755 routes (most of) international traffic via AS6453 which is very popular system (and one of tier 1 networks). 

Let’s observe traceroute from both Mumbai PoPs of each system

Click to expand traceroute from AS4755

Comparing with results from other ASN

Click to expand traceroute from AS6453

We can see beyond hop 4 in AS4755 traceroute, path is pretty much same for AS6453. So something is wrong within 1st 4 hops of AS4755.

Looking at initial part again:

1 59.163.16.14 [MPLS: Label 8237 Exp 0] 200 msec
59.163.55.150 [MPLS: Label 2188 Exp 0] 200 msec
121.240.12.110 [MPLS: Label 2188 Exp 0] 200 msec
2 172.31.33.125 200 msec
172.31.33.201 400 msec 4 msec
3 180.87.39.25 200 msec 204 msec 0 msec
4 80.231.130.5 [AS 6453] 404 msec 260 msec 408 msec

Taking lowest (i.e best) possible values on hop 3 = 0 ms, hop 4 = 260ms. Something is wrong here. Since we can clearly see it’s a jump between two Autonomous Systems (though belonging to same operator), looking at reverse path from AS6453 to AS4755 test IP - 59.163.16.14 from 59.163.16.0/22 announcement of AS4755 in Mumbai.

Tracing return paths from Mumbai & London PoP’s of AS6453:

Click to expand traceroute

Pretty good – Mumbai AS6453 to Mumbai AS4755 – 4ms latency – fine.

Next, looking at route from London AS6453 to Mumbai AS4755 router:

Click to expand traceroute

And here we go – London to Mumbai trip via New York-Los Angles!

Summary

Forward path from Mumbai to Germany based server is like:

Mumbai (AS4755) – Mumbai (AS6453) – London (AS6453) – further to Level3 (AS3356) which carries it to destination network

Return path is like:

Server in Germany – routed to London via Level3 – handed over to Tata (AS6453) in London – New Yark (AS6453) – Los Angles (AS6453) – Mumbai (AS4755). 

This broken reverse path is screwing up connectivity of VSNL backbone from Europe since packets are returning from US with very high latency.

What exactly is causing this issue?

Well, problem seems because of London PoP’s (infact many others in Europe) of Tata AS6453 which are not having proper entry in BGP routing table for blocks holded by VSNL-AS4755. 

E.g looking at BGP table for VSNL test IP 59.163.16.14 from London Telehouse Tata router:

Click  to expand BGP output

Clearly – only route is to Los Angles.

Quick check on Los Angles BGP table:

 Click to expand BGP output

Here we can see entry for route to Mumbai. 

So is Tata Communications AS6453 has issues on routing tables within own network?

Well, not exactly. If we check route from London AS6453 to Mumbai AS6453:

Click to expand traceroute result

Pretty much good. So whole problem is between routing tables for AS4755 which used to be VSNL – Govt, owned monopoly in India and AS6453 which used to TeleGlobe – both of which now belongs to Tata Communications.

Both of these Autonomous Systems play an important role since AS6453 is one of big tier 1 backbone network, while AS4755 has significant presence in India and most of domestic ISP’s purchase bandwidth from Tata Communications via AS4755 backbone. The extent of brokeness can be seen from a end user BSNL connection – e.g a traceroute to Mumbai based PoP of AS6453 is goes like:

Click to expand traceroute

With hope that Tata Communications will eventually fix these broken backbones, I get back to my Java exam preparation!

Disclaimer: Analysis done in this post is purely based on facts found from publically available information. I have no intention to harm Tata Communications from this. If anyone finds anything incorrect in this post, please feel free to contact me, and I would be happy to fix it.