06 Feb

airtel.in – bad DNS setup

 

Few days back I mentioned how reverse DNS setup of Airtel was incorrect. Sad to say it has not been fixed yet.
In meanwhile I was looking at domain name – airtel.in the main domain which runs website for Bharti Airtel’s Indian operations. I am little surprised to find that DNS server of airtel.in are failing randomly!
 

Problem:

airtel.in uses 4 DNS servers from Mantra Online – a small ISP which Bharti took over years back. Here are the DNS servers used by domain name:
aaadel.mantraonline.com.
dnsbom.mantraonline.com.
dnsdel.mantraonline.com.
dnsblr.mantraonline.com.
 
Now interesting part here is that out of these 4, only 1 behaves normally.
DNS server – dnsblr.mantraonline.com. seems working fine but rest all are rejecting queries “randomly” which is interesting. I have mostly seen DNS servers being up or down. This is probably first case when I can see DNS servers failing in random fashion.

Let’s query rest 3 DNS servers one by one:
anurag@laptop:~$ dig @aaadel.mantraonline.com airtel.in ns
; <<>> DiG 9.7.1-P2 <<>> @aaadel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 63903
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;airtel.in. IN NS
;; Query time: 81 msec
;; SERVER: 202.56.230.6#53(202.56.230.6)
;; WHEN: Mon Feb 6 01:25:20 2012
;; MSG SIZE rcvd: 27
 
 
In another 5 random tries, here’s what I get:
anurag@laptop:~$ dig @aaadel.mantraonline.com airtel.in ns
; <<>> DiG 9.7.1-P2 <<>> @aaadel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2044
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;airtel.in. IN NS
;; ANSWER SECTION:
airtel.in. 86400 IN NS dnsblr.mantraonline.com.
airtel.in. 86400 IN NS dnsdel.mantraonline.com.
airtel.in. 86400 IN NS aaadel.mantraonline.com.
airtel.in. 86400 IN NS dnsbom.mantraonline.com.
;; ADDITIONAL SECTION:
aaadel.mantraonline.com. 86400 IN A 202.56.230.6
dnsblr.mantraonline.com. 86400 IN A 202.56.250.5
dnsbom.mantraonline.com. 86400 IN A 202.56.240.5
dnsdel.mantraonline.com. 86400 IN A 202.56.230.5
;; Query time: 87 msec
;; SERVER: 202.56.230.6#53(202.56.230.6)
;; WHEN: Mon Feb 6 01:26:05 2012
;; MSG SIZE rcvd: 191
 
This time it worked. Pretty crazy. Same applies on other 2 DNS servers too:
anurag@laptop:~$ dig @dnsbom.mantraonline.com airtel.in ns
; <<>> DiG 9.7.1-P2 <<>> @dnsbom.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 29601
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;airtel.in. IN NS
;; Query time: 82 msec
;; SERVER: 202.56.240.5#53(202.56.240.5)
;; WHEN: Mon Feb 6 01:28:21 2012
;; MSG SIZE rcvd: 27
 
anurag@laptop:~$ dig @dnsdel.mantraonline.com airtel.in ns
; <<>> DiG 9.7.1-P2 <<>> @dnsdel.mantraonline.com airtel.in ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 34334
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;airtel.in. IN NS
;; Query time: 86 msec
;; SERVER: 202.56.230.5#53(202.56.230.5)
;; WHEN: Mon Feb 6 01:27:42 2012
;; MSG SIZE rcvd: 27
 
Pretty crazy case. Something is wrong at DNS servers itself – not sure what’s logic of rejecting queries randomly. But anyways – http://www.airtel.in will always open since 1/4 DNS server seems working normal. If that’s the case then Airtel still won’t be loosing much of traffic but unfortunately case is further complex.
 

Another problem…

Remember that 4 DNS servers mentioned here are the ones which are NS records at “airtel.in” zone at delegated servers. In other terms these are just servers which host and have NS entries for the zone but root nameservers of in-registry hold only 2 DNS servers in total which host zone. A quick whois check reveals that airtel.in is using:
Name Server:AAADEL.MANTRAONLINE.COM
Name Server:DNSDEL.MANTRAONLINE.COM
and thus missing the only server which gives consistent results. Incoming traffic never hits other 2 DNS servers which are just mentioned in NS.
Poor & bad DNS setup!
With hope that you hit right server this month rather then dead servers before reaching the working one, time for me to say Good night! 🙂

01 Feb

Sify broadband in rural areas

 

Sify is one of really interesting companies. One time pioneer of Indian internet market via chain of cyber cafes. Good old days. Present situation of Sify in consumer market is not significant. Latest earing figures clearly state company is moving towards enterprise segment.

Company is quite aggressive in enterprise segment offerings specially datacenters & corporate leased lines.
Is consumer market really over for Sify or there’s still some hope?

 

Well, consumer broadband market isn’t really over! Infact this is the main market which is yet to explode in India!

These are the factors which went against Sify:

  1. Company over estimated growth via Cybercafe chains. Cheap computers killed market of cyber cafe itself!
  2. Company failed to capture high end consumer segment and lost it to DSL players like BSNL/MTNL & Airtel. Quality of services was way too low (due to franchise model) adopted for distribution via last mile cable operators.
  3. Low end users were stolen by cheap wireless internet via 2G (and now 3G).
  4. Company tried using pointed wireless links when technology was very new & yet to become advance with MIMO & beam forming.
  5. Indian regulation which prohibits VoIP-PSTN connectivity again put them on back seat against main telcos like BSNL & Airtel.

 

Despite of these issues, I still see a huge potential for market which Sify tried to capture. Taking example of this village Radaur (where I am staying) – I can see over 20 connections to small computer shops, cell phone shops etc from Sify broadband. This is very interesting as a private broadband player is in village besides Govt. telecom player BSNL. Infact I can see lot of small shops giving priority to Sify over BSNL. I tried asking them for reason and here’s list of those:

  1. Fast installation 1-2days Vs 30+ days for BSNL.
  2. Fast fix in case of downtime – in few hours Vs 2-3 days for BSNL.
  3. No issues of breaking of cable (as in case of BSNL)

 

Well, these all issues are quite true. Most of these shops have subscribed to 256Kbps to 512Kbps plans and they work pretty well (as per my tests). Here Sify is using a “point to multi-point wireless network” to reach last mile. They run last mile over 60 degree sector antenna located in front of Radaur bus stand pointed towards village. Here’s the picture:

 

 

and here’s one of end user CPE antenna’s pointing towards it:

 

 

 

Unfortunately 1st picture isn’t clear.
It has small antenna with two parabolic antenna’s on top – one towards Yamunanagar city & other towards Ladwa village. It uses high end frequency band of 5.8Ghz which has over 40Mhz of bandwidth in free WiFi spectrum for tower backhaul. This sort of backhual works pretty well since one can get over 100Mbps for 40-50Km of distances (in this case Yamunanagar district is 20Km off from here) where they feed core network from leased fibers and run a MPLS on top of it. Though backhual capacity of such towers is quite high and for small village (with less users) – even last mile shared sector antenna too isn’t a problem. What surprises me is that – still at the end of day Sify offers similar or worst plans as compared to BSNL. Reason remains bit of backbone bandwidth costs, high amount of commission taken by middle re-sellers etc. Sify offcourse failing here to realize big market and capturing it with good marketing followed by massive deployment. At this point low end market will be again grabbed by cell phone players with 2G/3G while high end will sooner or later go to BSNL or other telcos (if last mile unbundle happens) since DSL is way more scalable then this technology apart from fact that BSNL won’t have any issue in backbone (when they have 104 pair fiber landing in village exchange!). Unless last mile unbundle happens, there’s hardly any technology which can provide wired broadband equivalent speeds for this network.

 

With hope for good future of broadband in India, time for me to wash my clothes! 😉

01 Feb

Network hijacking: Wrong BGP announcements screwing up traffic

Yesterday I came across a very interesting case of network hijacking of an ISP from wrong BGP announcements by another network. This issue was reported to NANOG mailing list. 

Issue was reported by Kevin, Senior Engineer at Altus Communications (AS11325). Problem was that SBJ Media LLC (AS33611) was making a /24 block announcement for specific slices of Altus –  208.110.48.0/2063.246.112.0/20, and 68.66.112.0/20 which are allocated to Altus Communications (as per ARIN whois).

Good news for now is problem seems on it’s way to fix, and route servers of At&t and Hurricane Electric are showing right path for /24 blocks.Just now Kevin updated NANOG saying: 

I hope none of you ever get hijacked by a spammer housed at Phoenix NAP. 🙂
We’re still not out of the woods, announcing /24s and working with upper
tier carriers to filter out our lists. However, I just got this response from Phoenix NAP and found it funny. The “thief” is a former customer,whom we terminated their agreement with. They then forged an LOA, submitted it to CWIE.net and Phoenix NAP and resumed using space above and beyond their terminated agreement.

 

This one is very interesting case and shows even today there’s no guarantee of correct routing on the Internet. So many autonomous systems out there but still at the end of day routing somehow works! 

 

What an ISP can do in such cases? (what I myself learned from looking at such cases so far):

  1. Small chunks like /24 are given more priority over /20, thus if someone hijacks /24 out of your /20 block then you can (should) also start announcing /24 to make sure hijacker does not get any additional benefit by announcing small specific route.
  2. Pick out upstream ISP’s of attacker’s autonomous system & eventually get announced prefixes filtered out at the source itself.
  3. Pick your upstream ISP’s and eventually request them for prefix filtering. 

 

This whole incidence reminds me of YouTube blackout in 2008 by Pakistan Telecom. Other then prefix filtering by big ISP’s one can’t really do much if such wrong announcement continues.

 

 

With hope that your ISP’s network is not “stealing” others IP’s time for me to go out for morning walk in village!

Special thanks to John Schneider from Iowa Network Services for his inputs & answering my questions! 🙂