Isp-Column

Today Google (AS15169) seem to be facing issues in their Indian network due to tropical cyclone Vardah. Their traffic at PoPs in Mumbai dipped considerably for a number of ISPs. My guess is that it’s likely because of outage in a large Govt. operator’s network who has overhead fibres along with utility lines.

It’s very important to note that “Google PoP” faced the issue and it’s no where close to saying that Google services went down. Google has a large network across the globe where they peer with networks. If one segment of this network goes down, traffic is re-routed via other parts and as per design even if the network goes down completely in say Mumbai or Chennai, services should stay live. While in real practice considerable degradation occurs because most of the Indian networks get a very large amount of traffic from Google and usually do not have that much extra capacity on their IP transit links, resulting in choking of transits during issues on their PNI with Google.   This shows how traffic of an ISP connected to Google in Mumbai dipped during peak time around at 4pm on Monday 12th Dec (IST) and went to zero little before midnight. I triggered a trace to aspmx.l.google.com. which is outside India from RIPE atlas probes in India and in general routing to that goes via Google’s backbone. Cluster with hostname aspmx.l.google.com (and few others) carry the Gmail/Google Apps traffic and it’s published by Google Apps users in their domain’s MX records.

One of frequent email and contact form message I get my blog is about available content networks in India and where one can peer. There are certain content networks in India and of course most of the content networks have open peering policy and are usually happy with direct inter-connection (we call as “peering”) with the ISP networks (often referred to as “eyeball networks”). Some of these networks have a backbone which connects back to their key datacenter locations on their own circuits via Singapore/Europe, some other have simply placed their caching server where cache fill happens over IP transit. Based on publically known information across community and of course peeringdb, following content players are available in India and known to be open for peering:

An excellent presentation by James Quinn from Facebook on “Being Open How Facebook Got Its Edge” at NANOG68. YouTube link here and video is embedded in the post below.

Some key points mentioned by James:

1. BGP routing is inefficient as scale grows especially around distributing traffic. They can get a lot of traffic concentrated to a specific PoP apart from the fact that BGP best AS_PATH can simply be an inefficient low AS_PATH based path.
2. Facebook comes with a cool idea of “evolving beyond BGP with BGP” where they use BGP concepts to beat some of the BGP-related problems.
3. He also points to fact that IPv6 has much larger address space and huge summarization can result in egress problems for them. A single route announcement can just have almost entire network behind it!
4. Traffic management is based on local and a global controller. Local controller picks efficient routes, injects them via BGP and takes care of traffic balancing within a given PoP/city, balancing traffic across local circuits. On the other hand, Global PoP is based on DNS logic and helps in moving traffic across cities.

It’s wonderful to see that Facebook is solving the performance and load related challenges using fundamental blocks like BGP (local controller) and DNS (global controller). :)

Bangladesh’s .bd ccTLD faced another outage. As I mentioned in one of the previous posts - .bd domain seems to be primarily on BTCL (AS17494).  Zone delegation of .bd is still pending with PCH and while PCH is mentioned in NS records of the authoritative DNS servers but delegation is pending in the root DNS servers as per reply from Kabindra from PCH on the bdNOG mailing list during the last outage.

So it seems like apart from voice interconnect issues, Jio is also facing routing issues on the backbone. I ran a trace to one of IP’s on Jio network allocated to end customer - 169.149.212.122. I ran trace from all Indian RIPE Atlas probes measurement here. There seem quite a few RIPE Atlas probes which are giving latency on 150ms + range. Seems like they are downstream or downstream of downstream of Tata Comm’s AS4755 and routing is happening via Singapore!    

Day before yesterday i.e on 18th August 2016 Bangladesh’s TLD .bd went had an outage. It was originally reported by Jasim Alam on bdNOG mailing list.

dig btcl.com.bd @8.8.8.8
; <<>> DiG 9.10.4-P2 <<>> btcl.com.bd @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 8114
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;btcl.com.bd.                   IN      A
;; Query time: 76 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Aug 18 14:24:25 Bangladesh Standard Time 2016
;; MSG SIZE  rcvd: 40

His message shows that DNS resolution of BTCL (Bangladesh Telecommunications Company Ltd) was failing. Later Alok Das that it was the power problem resulting in outage. Let’s look ask one of 13 root DNS server about NS records on who has the delegation for .bd.

Just noticed that Reliance Jio (AS55836) seems to be originating a /16 which is for Charter Communications (AS20115) - 47.35.0.0/16.  

route-views>sh ip bgp 47.35.0.0/16 long | exclude 20115
BGP table version is 18764390, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
     Network          Next Hop            Metric LocPrf Weight Path
 *   47.35.0.0/16     195.208.112.161                        0 3277 3267 174 64049 55836 i
 *                    217.192.89.50                          0 3303 6762 64049 55836 i
 *                    212.66.96.126                          0 20912 1267 64049 55836 i
 *                    162.243.188.2                          0 393406 6453 6762 64049 55836 i
 *                    192.241.164.4                          0 62567 2914 174 64049 55836 i
 *                    129.250.0.11          1007             0 2914 174 64049 55836 i
 *                    104.192.216.1                          0 46450 174 64049 55836 i
 *                    202.93.8.242                           0 24441 3491 3491 174 64049 55836 i
 *                    66.59.190.221                          0 6539 577 6762 64049 55836 i
 *                    144.228.241.130         80             0 1239 174 64049 55836 i
 *                    207.172.6.20             0             0 6079 3356 174 64049 55836 i
 *                    203.62.252.83                          0 1221 4637 174 64049 55836 i
 *                    93.104.209.174                         0 58901 51167 3356 6762 64049 55836 i
     Network          Next Hop            Metric LocPrf Weight Path
 *                    162.250.137.254                        0 4901 174 64049 55836 i
 *                    4.69.184.193             0             0 3356 174 64049 55836 i
 *                    208.51.134.254           1             0 3549 3356 174 64049 55836 i
 *                    89.149.178.10           10             0 3257 174 64049 55836 i
 *                    66.110.0.86                            0 6453 6762 64049 55836 i
 *                    134.222.87.1           650             0 286 174 64049 55836 i
 *                    95.85.0.2                              0 200130 6453 174 64049 55836 i
 *                    12.0.1.63                              0 7018 174 64049 55836 i
 *                    173.205.57.234                         0 53364 3257 174 64049 55836 i
 *                    206.24.210.80                          0 3561 174 64049 55836 i
 *                    5.101.110.2                            0 202018 2914 174 64049 55836 i
 *                    207.172.6.1              0             0 6079 3356 174 64049 55836 i
 *                    154.11.98.225            0             0 852 174 64049 55836 i
 *                    194.85.40.15                           0 3267 174 64049 55836 i
 *                    208.74.64.40                           0 19214 174 64049 55836 i
 *                    209.124.176.223                        0 101 101 174 64049 55836 i
 *                    66.185.128.48            6             0 1668 174 64049 55836 i
 *                    203.181.248.168                        0 7660 2516 6762 64049 55836 i
 *                    202.232.0.2                            0 2497 701 6762 64049 55836 i
 *                    103.247.3.45                           0 58511 64049 55836 i
 *                    193.0.0.56                             0 3333 1103 64049 55836 i
     Network          Next Hop            Metric LocPrf Weight Path
 *                    80.241.176.31                          0 20771 47872 64049 55836 i
 *>                   216.218.252.164                        0 6939 64049 55836 i
 *                    132.198.255.253                        0 1351 174 64049 55836 i
 *                    103.255.249.22                         0 58443 45177 64049 55836 i
 *                    114.31.199.1                           0 4826 174 64049 55836 i
route-views>

This shows Reliance Jio’s ASN AS55836 announcing 47.35.0.0/16. Charter Communications (AS20115) is originating multiple of /18s out of the same pool.  

It’s great to see Amazon announcement two days back about launch of their region in Mumbai. In past I was quite happy to see their Cloudfront CDN PoPs in Mumbai & Chennai (blog post here). Now it’s just great to see a full AWS region out of Mumbai. :) Though it’s going to eat most of important customers from the smaller players still it’s good for industry as industry is too big and we need more & more of such large Cloud players in India to bring more and more content hosting in India.

I just presented data on Performance of Root DNS anycast in UK & Ireland at INEX meeting.

Last month India & Bangladesh went into an agreement for power and bandwidth. India stated export of an additional 100MW of power to Bangladesh while Bangladesh started a 10Gbps link to Indian state of Tripura. (News article on this here)

Tripura is an Indian state having its boundaries with Bangladesh as you can see in above map. Coming to routing side of things setup is that BSNL (AS9829) is buying IP transit from Bangladesh Submarine Cable Co. Ltd (BSCCL) at $1.2 million / year. This means a cost of around $10/Mbps/month or 662Rs/Mbps/month. It’s hard to say if it’s good or bad since other link from BSNL is via it’s other links. But yes it’s good to see a layer 3 connectivity in terms of IP transit relationship rather then leasing dark fiber or L1 waves as they would have caused bit inefficient routing in the area. In order to do this BSNL has setup a “gateway node” at Agartala. I think it would be pretty much a node with approvals under ILD from doT and extremely likely a LIM device for lawful interception.   Months before it actually came up, Dyn research tweeted about this visible routing relationship.