06 Oct

K root server – Noida anycast and updates

K root in Noida seems to be not getting enough traffic from quite sometime and connectivity does seems bit broken. This is a blog post following up to Dyn’s excellent and detailed post about how TIC leaked the world famous 193.0.14.0/24 address space used by AS25152. It was good to read this post from RIPE NCC written by my friend Emile (and thanks to him for crediting me to signal about traffic hitting outside!)
 

The route leak…

TIC AS48159 was supposed to keep the route within it’s IGP but it leaked it to Omantel AS8529 – a large International backbone which propagated route leak further to global table. It was mistake at by both players primarily by TIC for leaking route.
 
If we look at IPv4 route propagation graph of Omatel AS8529 on Hurricane Electric BGP tool kit, it shows two import ASNs:
 
Omantel IPv4 routing
 
 
This has AS9498 (Bharti Airtel) and AS6453 (Tata Communications). Both of these are extremely important networks and two of large International and domestic IP transit providers in India. Very likely Omantel is customer of Bharti Airtel and if we look at IRR record of Airtel as published in their peeringdb record: AS9498:AS-BHARTI-IN
 

Anurags-MacBook-Pro:~ anurag$ whois -h whois.apnic.net AS9498:AS-BHARTI-IN |grep -w AS8529
members: AS38476,AS45219,AS45264,AS45283,AS45514,AS45451,AS37662,AS45491,AS7642,AS45517,AS45514:AS-TELEMEDIA-SMB,AS45609,AS38740,As131210,AS45335,AS23937,AS132045,AS8529,AS132486,AS8164,AS133967,AS37048
Anurags-MacBook-Pro:~ anurag$

 
This also confirms the same. Airtel did picked this route and since it was a customer route, it had a higher local preference then the peering route Airtel learnt from NIXI Noida peering with  K root. For now route leak fixed and Airtel seems to be having good routing with K root anycast instance in Noida.
 

Current status

From Tata Communications – it’s yet not picking announcement of K root anycast instance from Noida since their peering session at NIXI Noida has been down from long time. NIXI moved over from STPI to Netmagic Sector 63 Noida in August (see heavy drop of traffic in NIXI Noida graphs here). From that time onwards Tata’s domestic backbone AS4755’s peering session seems down.

NIXI Looking Glass - show ip bgp summary
Router: NIXI Delhi (Noida)
Command: show ip bgp summary
BGP router identifier 218.100.48.1, local AS number 24029
BGP table version is 541676, main routing table version 541676
10616 network entries using 1528704 bytes of memory
13657 path entries using 1092560 bytes of memory
1546/1197 BGP path/bestpath attribute entries using 210256 bytes of memory
1275 BGP AS-PATH entries using 40472 bytes of memory
566 BGP community entries using 22196 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2894188 total bytes of memory
BGP activity 523875/512278 prefixes, 1016379/1001610 paths, scan interval 60 secs
Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
  218.100.48.6    4        25152   35502  102431   541675    0    0 3w3d              1
  218.100.48.10   4        10029    8285   15774   541675    0    0 2d16h           194
  218.100.48.12   4         9583    4750    9899   541675    0    0 2d16h          1969
  218.100.48.13   4        17439  109297  191050   541675    0    0 9w5d             32
  218.100.48.15   4         9829     713    2669   541675    0    0 11:04:52        857
  218.100.48.17   4        17426    1205    3995   541675    0    0 19:57:29         17
  218.100.48.20   4         9498  190999  159646   541675    0    0 3w3d           7254
  218.100.48.21   4         4637   63761  141723   541675    0    0 6w2d              5
  218.100.48.23   4        63829   30808   80566   541675    0    0 2w5d              5
  218.100.48.25   4        17754   20071   50107   541675    0    0 1w5d            102
  218.100.48.26   4        18101   14641   29277   541675    0    0 5d00h           190
  218.100.48.27   4        17488   22887   58026   541675    0    0 2w0d            354
  218.100.48.28   4        55410   58592  107852   541675    0    0 2w2d           2637
  218.100.48.29   4        10201       0       0        1    0    0 2d08h    Active
  218.100.48.31   4        55836    9164   23591   541675    0    0 6d08h             7
  218.100.48.34   4        45528   38354  107593   541675    0    0 3w5d             18
  218.100.48.36   4       132215   27000   56646   541675    0    0 1w2d             15
  218.100.48.40   4       132453       0       0        1    0    0 2d07h    Idle

 
As per NIXI’s connected parties page, Tata Comm’s IP is 218.100.48.30. From NIXI’s looking glass there seems to no peer on that IP !

NIXI Looking Glass - show ip bgp neighbors 218.100.48.30 routes
Router: NIXI Delhi (Noida)
Command: show ip bgp neighbors 218.100.48.30 routes
% No such neighbor or address family

 
Hence for now Tata Comm isn’t getting route at all from Noida instance and that explains reason for bad outbound path.
 
Example of trace from Tata Comm to K root:

## AS4755/TATACOMM-AS - TATA Communications formerly VSNL is Leading ISP (2.7% of browser users in IN)
#prb:15840 dst:193.0.14.129
1 () 192.168.34.1 [0.344, 0.426, 17.445]
2 err:{u'x': u'*'}
3 (AS4755) 115.114.137.158.static-pune.vsnl.net.in [2.73, 2.916, 2.921] |Pune,Maharashtra,IN|
4 () 172.29.250.33 [5.659, 5.789, 6.274]
5 (AS6453) ix-0-100.tcore1.mlv-mumbai.as6453.net [5.143, 5.168, 5.755]
6 (AS6453) if-9-5.tcore1.wyn-marseille.as6453.net [125.474, 125.554, 125.596] |Marseille,Provence-Alpes-C?te d'Azur,FR|
7 (AS6453) if-2-2.tcore2.wyn-marseille.as6453.net [125.723, 125.739, 126.525] |Marseille,Provence-Alpes-C?te d'Azur,FR|
8 (AS6453) if-7-2.tcore2.fnm-frankfurt.as6453.net [126.535, 126.788, 127.22]
9 (AS6453) if-12-2.tcore1.fnm-frankfurt.as6453.net [125.75, 125.828, 125.871]
10 (AS6453) 195.219.156.146 [262.957, 265.3, 266.39]
11 (AS20485) spb03.transtelecom.net [297.919, 297.954, 302.452] |Saint-Petersburg,St.-Petersburg,RU|
12 (AS20485) selectel-gw.transtelecom.net [288.789, 296.574, 298.442]
13 (AS25152) k.root-servers.net [296.981, 297.042, 297.118]

 
even same stays for its downstream customers who have outbound via TCL:

## AS45528/TDN - Tikona Digital Networks Pvt Ltd. (1.4% of browser users in IN)
#prb:22793 dst:193.0.14.129
1 () 10.135.150.254 [0.521, 0.539, 0.814]
2 (AS45528) 1.22.55.185 [5.774, 7.721, 8.195]
3 (AS4755) 115.113.133.125.static-mumbai.vsnl.net.in [7.282, 14.754, 48.013] |Mumbai,Maharashtra,IN|
4 (AS6453) if-2-590.tcore2.l78-london.as6453.net [121.089, 122.755, 124.416] |London,England,GB|
5 (AS6453) if-2-2.tcore1.l78-london.as6453.net [121.828, 122.077, 123.869] |London,England,GB|
6 (AS6453) if-17-2.tcore1.ldn-london.as6453.net [120.716, 122.008, 122.768] |London,England,GB|
7 (AS6453) 195.219.83.10 [122.039, 123.532, 125.424]
8 (AS8468) te2-2.interxion.core.enta.net [125.262, 126.587, 127.04]
9 (AS8468) 188-39-11-66.static.enta.net [122.424, 123.028, 123.163]
10 (AS5459) ge0-1-101.tr1.linx.net [121.656, 124.826, 125.182] |London,England,GB|
11 (AS5459) fe3-0.tr4.linx.net [120.654, 120.721, 138.858] |London,England,GB|
12 (AS5459) g00.router.linx.k.ripe.net [123.306, 123.536, 125.486] |London,England,GB|
13 (AS25152) k.root-servers.net [121.285, 122.653, 122.942]

 
 
Another issue which is causing serious trouble around K root is the fact that it appears to be broken IP transit pipe of K root Noida. Due to the way NIXI works, K root must have a IP transit pipe. I pointed long back about broken connectivity of root DNS servers due return path problems. After that both K root and i root got transit but seems like after NIXI moved over, IP transit has been broken for current setup in Netmagic.
 
Why “local node” of root server needs IP transit?
It needs transit because:

    1. NIXI has a weird pricing of “x-y” where requester pays and this leads to a quite high settlement amount for a network which has a high inbound traffic (eyeball network) – even few x times than that of transit! (paying 5Rs/GB!). This leads to scenario where networks do “partial prefix announcement” to keep their traffic balanced (or slightly in outbound direction) to avoid high settlement cost. Hence most of such eyeball networks announce their regional routes but avoid all routes while they still do learn K root’s route and inject in their IGP.This leads in case where K root’s 193.0.14.0/24 is leant by networks in West and South India and hence there’s a forward path from customers >>> K root Noida node. Now since these networks aren’t announcing their West or South Indian routes at NIXI Noida, there’s no return path for packets. Thus for root DNS to stay operationally stable (which they should since they are critical) they must have transit / default route to return packets as last resort to IP’s which aren’t visible via peering.
    2. Similar case of some other random leaked routes. E.g if a large ISP decided to learn K root route and announce to customer’s table thus leading to Customer > Large network > K root Noida path while not announcing that customer’s route at NIXI resulting in no return path.

 
 
So in short – It does needs transit but just for outbound traffic, not for announcing routes on the transit.
I have informed of broken connectivity issue to RIPE NCC and their team is actively working on the fix. Hopefully it would be fixed very soon!
 
With hope that your DNS is not getting resolved from other side of world, good night! 🙂
 
Disclaimer: As usual – thoughts & comments are completely personal.

03 Oct

Airtel 3G running CGNAT

Yesterday I was driving and radio was pretty boring. Next, I connected cell phone to car’s stereo (I use a PT-750 to wirelessly connected my devices to car’s audio system). Next I tuned into Gaana.com app and experience was overall good. The way whole setup was working itself is a wonder – wireless profiles keeping layer 3 link (IP address of device) consistent and handovers happening on layer 1. On top of that a while world of backbone routing across AS9498 backbone the hosting provider’s network of the app.
 
Now an interesting thing in this setup was the IP allocations. I that IP allocated by Airtel was 100.92.215.253.
My_Phone_Connectivity
 
 

Is that an Airtel allocated IP range?

 
Let’s see whois data on it:
 

NetRange: 100.64.0.0 - 100.127.255.255
CIDR: 100.64.0.0/10
OriginAS:
NetName: SHARED-ADDRESS-SPACE-RFCTBD-IANA-RESERVED
NetHandle: NET-100-64-0-0-1
Parent: NET-100-0-0-0-0
NetType: IANA Special Use
Comment: This block is used as Shared Address Space. Traffic from these addresses does not come from IANA. IANA has simply reserved these numbers in its database and does not use or operate them. We are not the source of activity you may see on logs or in e-mail records. Please refer to http://www.iana.org/abuse/
Comment:
Comment: Shared Address Space can only be used in Service Provider networks or on routing equipment that is able to do address translation across router interfaces when addresses are identical on two different interfaces.
Comment:
Comment: This block was assigned by the IETF in the Best Current Practice document,
Comment: RFC 6598 which can be found at:
Comment: http://tools.ietf.org/html/rfc6598
RegDate: 2012-03-13
Updated: 2012-04-23
Ref: http://whois.arin.net/rest/net/NET-100-64-0-0-1

 
 
 
The IP is part of 100.64.0.0/10 which is a well known pool for CGNAT or Carrier Grade NAT. Checkout wikipedia’s small into to CGNAT here. Basically Airtel is out of publically unique IP address pools and hence doing NAT at carrier level. This is something very common across 3G provider’s in India where they are getting a demand of high growth and “always on” connectivity where end users just grab an IP address and keep it for long time and carriers can’t re-use it anywhere else in network.
 

Why use 100.64.0.0/10 ?

This is because other private pools from RFC1918 address space are already in use by lot of home and business networks for NATing inside a home or organization network. If carriers also use the same, it will cause a major conflict and routing will just fail. Imagine using 192.168.1.0/24 on your home router and then getting a WAN IP of 192.168.1.100 from your upstream. It will just not work. Thus a pool 100.64.0.0/10 is just like other private IP’s but simply not used by CPE vendors as default pool for NATing. Further more 100.64.0.0/10 is supposed to stay within an organization and not to be announced/leaked to any peer. It’s a one-to-many NAT and multiple IP’s in pool 100.64.0.0/10 have a single public IP as source address.
 
Let’s check what is my public IP on same 3G connection from bgp.he.net.
Airtel-Link_Public_IP
 
 
 
So my public IP at that instant was 223.225.243.83. This public IP has many such private IP’s behind it. It is part of 223.225.240.0/20 pool announced by Airtel AS9498 in global routing table. Though technically 100.64.0.0/10 is supposed to stay within a network and not hit global table at all but just like other routing issues, it’s very common to see this pool in global table. At the time of this blog post I see that BELTelecom in Belarus (AS6697) is leaking 100.92.0.0/16 in the global routing table. Route has very limited visibility but does seems visible at Oregon route-views. Russian provider MegaFon AS31133 seems to be transiting it.

route-views.routeviews.org> show ip bgp 100.92.0.0/16 long
BGP table version is 0, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale, R Removed
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
* 100.92.0.0/16 95.85.0.2 0 200130 31133 6697 i
* 217.75.96.60 0 0 16150 31133 6697 i
* 5.101.110.2 0 202018 31133 6697 i
* 194.85.40.15 0 3267 31133 31133 6697 i
* 194.85.102.33 0 3277 3267 31133 31133 6697 i
*> 164.128.32.11 0 3303 31133 6697 i
Total number of prefixes 1
route-views.routeviews.org>

 
Next time when you are streaming music over 3G, think about all nuts and bolts running in background to keep it going. 😉