06 Aug

AS Number hijacking due to misconfiguration

 

This Sunday I was looking at global routing table dump and found AS1 announcing some very weird prefixes.

 

AS1 i.e Autonomous System Number 1 belongs to Level3 but as far as I know they are not actively using it. They use AS3356 globally (along with Global Crossing’s AS3549). I noticed quite a few prefixes of a Brazil based telecom provider – Netvip Telecomunicaes being announced by AS1. 

 

Some of entries in global routing table belonging to AS1 (as picked from BGP table dump of route-views archive):

Anurags-MacBook-Pro:Downloads anurag$ grep -w ‘1 i’ oix-full-snapshot-latest.dat|cut -f 3 -d ‘ ‘ |sort -u

177.185.100.0/23
177.185.100.0/24
177.185.101.0/24
177.185.102.0/23
177.185.102.0/24
177.185.103.0/24
177.185.104.0/23
177.185.104.0/24
177.185.105.0/24
177.185.106.0/23
177.185.106.0/24
177.185.107.0/24
177.185.108.0/23
177.185.108.0/24
177.185.109.0/24
177.185.110.0/23
177.185.110.0/24
177.185.111.0/24
177.185.96.0/23
177.185.96.0/24
177.185.97.0/24
177.185.98.0/23
177.185.98.0/24
177.185.99.0/24
186.251.240.0/21
186.65.112.0/20
190.185.108.0/22
4.31.236.64/29
4.34.12.0/24
4.34.13.0/24
94.31.44.0/24
Anurags-MacBook-Pro:Downloads anurag$

 

So there are quite a few prefixes belonging to different network providers being originated by AS1. Only 4.34.12.0/24 and 4.34.13.0/24 seem to be with Level3. Red ones here 188.185.xx.0/24 all belong to Netvip. This appeared very strange to me as why Level3 would let anyone to use AS1 and announce their own prefix? Could it be a hijacked ASN i.e someone using AS1 without having any specific relation to Level3? My past experience tells that if there’s a chance of hijacked ASN then easiest way out is to observe AS path and find who is providing upstream to that ASN.

 

Asking bgp table dump on what it knows about that prefix:

 

Anurags-MacBook-Pro:Downloads anurag$ grep -w 177.185.100.0/24 oix-full-snapshot-latest.dat
* 177.185.100.0/24 85.114.0.217 0 0 0 8492 9002 16735 52931 1 i
* 177.185.100.0/24 213.144.128.203 1 0 0 13030 16735 52931 1 i
* 177.185.100.0/24 66.185.128.1 556 0 0 1668 6762 26615 28309 52931 i
* 177.185.100.0/24 208.51.134.246 14233 0 0 3549 16735 52931 1 i
* 177.185.100.0/24 206.24.210.102 0 0 0 3561 6762 26615 28309 52931 i
* 177.185.100.0/24 67.17.82.114 14023 0 0 3549 16735 52931 1 i
* 177.185.100.0/24 134.222.87.1 0 0 0 286 6762 26615 28309 52931 i
* 177.185.100.0/24 157.130.10.233 0 0 0 701 3549 16735 52931 1 i
* 177.185.100.0/24 203.62.252.186 0 0 0 1221 4637 3549 16735 52931 1 i
* 177.185.100.0/24 198.129.33.85 0 0 0 293 16735 52931 1 i
* 177.185.100.0/24 216.18.31.102 0 0 0 6539 577 3549 16735 52931 1 i
* 177.185.100.0/24 154.11.11.113 0 0 0 852 2914 3549 16735 52931 1 i
* 177.185.100.0/24 137.164.16.84 0 0 0 2152 3356 3549 16735 52931 1 i
* 177.185.100.0/24 89.149.178.10 10 0 0 3257 3549 16735 52931 1 i
* 177.185.100.0/24 154.11.98.225 0 0 0 852 2914 3549 16735 52931 1 i
* 177.185.100.0/24 194.153.0.253 1015 0 0 5413 1299 3549 16735 52931 1 i
* 177.185.100.0/24 129.250.0.11 6 0 0 2914 3549 16735 52931 1 i
* 177.185.100.0/24 96.4.0.55 0 0 0 11686 11164 3549 16735 52931 1 i
* 177.185.100.0/24 195.22.216.188 100 0 0 6762 26615 28309 52931 i
* 177.185.100.0/24 216.218.252.164 0 0 0 6939 16735 52931 1 i
* 177.185.100.0/24 168.209.255.23 0 0 0 3741 2914 3549 16735 52931 1 i
* 177.185.100.0/24 144.228.241.130 0 0 0 1239 6762 26615 28309 52931 i
* 177.185.100.0/24 4.69.184.193 0 0 0 3356 3549 16735 52931 1 i
* 177.185.100.0/24 91.209.102.1 0 0 0 39756 3257 3549 16735 52931 1 i
* 177.185.100.0/24 80.91.255.62 0 0 0 1299 3549 16735 52931 1 i
* 177.185.100.0/24 12.0.1.63 0 0 0 7018 6762 26615 28309 52931 i
* 177.185.100.0/24 203.181.248.168 0 0 0 7660 2516 6762 26615 28309 52931 i
* 177.185.100.0/24 202.232.0.3 0 0 0 2497 3356 3549 16735 52931 1 i
* 177.185.100.0/24 147.28.7.1 0 0 0 3130 2914 3549 16735 52931 1 i
* 177.185.100.0/24 147.28.7.2 0 0 0 3130 1239 6762 26615 28309 52931 i

 

This is interesting. We can see two ASNs are originating this prefix – AS1 (as we know already) and AS52931. The fun fact is that wherever there’s AS1, the next ASN in AS path is AS52931 i.e AS1 for such prefixes is sitting below AS52931 which on other side is originating same prefix. Further AS52931 has upstream from AS28309 & AS16735. It seems like AS1 is coming only for routes which have AS16735 as upstream while for other case it’s direct announcement by AS52931. This gave me an interesting clue which was later verified by replies to my post on NANOG mailing list. 

 

Basically AS52931 – Netvip did not hijack AS1 intentionally but rather it was a case of mis-configured prepending. Netvip has two upstreams and was trying to prepend one of them (AS16735). In prepending networks simply repeat their own AS few times to increase AS-PATH which makes a route less preferred. 

 

Ideally what the needed is a AS path like this:

XXX XXX XXX 28309 52931 i – Preferred via AS28309 transit

XXX XXX XXX 16735 52931 52931 i – Not-preferred via AS16735 transit. 

 

Instead of putting their own ASN once in route map, they put “number 1” in the prepend which brought AS1 in global table for this prefix. I tried looking around and saw some funny prefixes from AS2, AS3, AS4 etc. 

 

Anurags-MacBook-Pro:Downloads anurag$ grep -w ‘2 i’ oix-full-snapshot-latest.dat|cut -f 3 -d ‘ ‘ |sort -u
128.4.0.0/16
177.129.161.0/24
31.192.64.0/19

 

Last prefix 31.192.64.0/19 does not belongs to AS2 (which is with UDEL-DCN – University of Delaware). 

 

route-views>sh ip bgp 31.192.64.0/19 long
BGP table version is 4043628875, local router ID is 128.223.51.103
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete

Network Next Hop Metric LocPrf Weight Path
* 31.192.64.0/19 208.74.64.40 0 19214 25973 6830 3.1190 i
* 194.85.102.33 0 3277 3267 9002 6830 3.1190 i
* 4.69.184.193 0 0 3356 6830 3.1190 i
* 154.11.98.225 0 0 852 174 12570 3.1190 2 i
* 207.172.6.20 0 0 6079 6830 3.1190 i
* 193.0.0.56 0 3333 6830 3.1190 i
* 154.11.11.113 0 0 852 174 12570 3.1190 2 i
* 69.31.111.244 0 0 4436 6830 3.1190 i
* 194.85.40.15 0 3267 9002 6830 3.1190 i
* 66.59.190.221 0 6539 577 6830 3.1190 i
* 209.124.176.223 0 101 101 3356 6830 3.1190 i
* 128.223.253.10 0 3582 3701 3356 6830 3.1190 i
* 207.172.6.1 0 0 6079 6830 3.1190 i
* 157.130.10.233 0 701 1299 6830 3.1190 i
* 134.222.87.1 0 286 6830 3.1190 i
* 66.185.128.48 547 0 1668 6830 3.1190 i
* 202.249.2.86 0 7500 2497 6830 3.1190 i
* 216.218.252.164 0 6939 6830 3.1190 i
* 207.46.32.34 0 8075 6830 3.1190 i
* 144.228.241.130 0 1239 3257 8928 12570 3.1190 2 i
* 114.31.199.1 0 0 4826 6939 6830 3.1190 i
* 208.51.134.254 1 0 3549 3356 6830 3.1190 i
* 129.250.0.11 384 0 2914 8928 12570 3.1190 2 i
* 217.75.96.60 0 0 16150 6830 3.1190 i
* 195.66.232.239 0 5459 6830 3.1190 i
*> 164.128.32.11 0 3303 6830 3.1190 i
* 202.232.0.2 0 2497 6830 3.1190 i
* 203.62.252.186 0 1221 4637 6830 3.1190 i
* 203.181.248.168 0 7660 2516 3257 8928 12570 3.1190 2 i
* 66.110.0.86 0 6453 3356 6830 3.1190 i
* 89.149.178.10 10 0 3257 8928 12570 3.1190 2 i
* 12.0.1.63 0 7018 1299 6830 3.1190 i
* 206.24.210.102 0 3561 3356 6830 3.1190 i
route-views>

 

 

This seems even more interesting because of doted ASN. 🙂 

3.1190 means AS197798 as per dot conversion following RFC 5396. So we have AS197798 as well as AS2 sitting below AS197798 announcing that prefix – hence another misconfigured prepend case. (Nice tool by Sprint for dot.ASN conversion)

 

Regarding original case of AS1, I observed that yesterday  at 18:44:13 RIPE NCC route collectors noticed change in BGP announcements changes for this. 
One of route change noticed by Tinet AS3257 as route 3257 3549 16735 52931 1 was changed to 3257 3549 16735 52931. By 21:37:59 GMT, Netvip pulled off all routes from that mis-configured prepend. 

 

With hope that you won’t hijack an ASN while prepending, time for me to end this blog post and get back to work!

 

Note: 

I missed to thank Doug Madory from Renesys for his detailed explanation & Stephen Wilcox from IX Reach for giving clue about prepending in my original post. 

30 Jun

Private IPs in Public routing

Sometimes we see interesting IP’s in traceroute & they confuse lot of people.

I have seen this topic in discussion twice on NANOG and once on Linux Delhi user group. 

 

OK – let’s pick an example: 

anurag:~ anurag$ traceroute 71.89.140.11
traceroute to 71.89.140.11 (71.89.140.11), 64 hops max, 52 byte packets
1 router (10.10.0.1) 1.176 ms 0.993 ms 0.941 ms
2 117.220.160.1 (117.220.160.1) 20.626 ms 29.101 ms 19.216 ms
3 218.248.169.122 (218.248.169.122) 23.983 ms 43.850 ms 45.057 ms
4 115.114.89.21.static-mumbai.vsnl.net.in (115.114.89.21) 118.094 ms 81.447 ms 66.838 ms
5 172.31.16.193 (172.31.16.193) 115.979 ms 90.947 ms 90.491 ms
6 ix-4-2.tcore1.cxr-chennai.as6453.net (180.87.36.9) 95.778 ms 98.601 ms 98.920 ms
7 if-5-2.tcore1.svw-singapore.as6453.net (180.87.12.53) 321.174 ms
if-3-3.tcore2.cxr-chennai.as6453.net (180.87.36.6) 331.386 ms 326.671 ms
8 if-6-2.tcore2.svw-singapore.as6453.net (180.87.37.14) 317.442 ms
if-2-2.tcore2.svw-singapore.as6453.net (180.87.12.2) 334.647 ms 339.289 ms
9 if-7-2.tcore2.lvw-losangeles.as6453.net (180.87.15.26) 318.003 ms 328.334 ms 309.234 ms
10 if-2-2.tcore1.lvw-losangeles.as6453.net (66.110.59.1) 306.500 ms 326.194 ms 341.537 ms
11 66.110.59.66 (66.110.59.66) 315.431 ms 330.417 ms 308.372 ms
12 dls-bb1-link.telia.net (213.155.136.40) 354.768 ms 344.360 ms 357.050 ms
13 chi-bb1-link.telia.net (80.91.248.208) 352.479 ms 358.751 ms 359.987 ms
14 cco-ic-156108-chi-bb1.c.telia.net (213.248.89.46) 367.467 ms 370.482 ms 377.280 ms
15 bbr01aldlmi-bue-4.aldl.mi.charter.com (96.34.0.98) 387.269 ms 385.362 ms 365.694 ms
16 crr02aldlmi-bue-2.aldl.mi.charter.com (96.34.2.11) 375.275 ms 375.356 ms 371.621 ms
17 dtr02grhvmi-tge-0-1-0-0.grhv.mi.charter.com (96.34.34.83) 383.539 ms 371.817 ms 383.804 ms
18 dtr02whthmi-tge-0-1-0-0.whth.mi.charter.com (96.34.34.85) 384.400 ms 391.197 ms 393.340 ms
19 dtr02ldngmi-tge-0-1-0-0.ldng.mi.charter.com (96.34.34.87) 371.192 ms 375.679 ms 378.457 ms
20 acr01mnplmi-tge-0-0-0-3.mnpl.mi.charter.com (96.34.40.75) 364.824 ms 385.534 ms 374.401 ms
21 * *^C
anurag:~ anurag$

 

 

Let’s try pinging IP on 14th hop (which is with a major backbone Telia) – 213.248.89.46

anurag:~ anurag$ ping -c 5 213.248.89.46
PING 213.248.89.46 (213.248.89.46): 56 data bytes
64 bytes from 213.248.89.46: icmp_seq=0 ttl=240 time=517.305 ms
64 bytes from 213.248.89.46: icmp_seq=1 ttl=240 time=329.230 ms
64 bytes from 213.248.89.46: icmp_seq=2 ttl=240 time=324.397 ms
64 bytes from 213.248.89.46: icmp_seq=3 ttl=240 time=331.474 ms
64 bytes from 213.248.89.46: icmp_seq=4 ttl=240 time=326.409 ms

— 213.248.89.46 ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 324.397/365.763/517.305/75.809 ms
anurag:~ anurag$

  

Works fine! 

 

Game begins here…

 

Next, let’s try pinging hop 15th IP which is with a major cable company Charter operating in US East – 96.34.0.98

PING 96.34.0.98 (96.34.0.98): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

— 96.34.0.98 ping statistics —
5 packets transmitted, 0 packets received, 100.0% packet loss
anurag:~ anurag$

  

So we see some nice timeouts. This confuses lot of people as we can’t have a firewall blocking ICMP packets here since we did had ICMP based traceroute with ICMP replies from 15th hop in last trace.

 

Let’s try to do a trace to this IP to see where exactly is connection breaking.

anurag:~ anurag$ traceroute -a 96.34.0.98
traceroute to 96.34.0.98 (96.34.0.98), 64 hops max, 52 byte packets
1 [AS65534] router (10.10.0.1) 1.661 ms 0.887 ms 0.934 ms
2 [AS9829] 117.220.160.1 (117.220.160.1) 18.867 ms 31.898 ms 20.931 ms
3 [AS9829] 218.248.169.118 (218.248.169.118) 43.427 ms 22.327 ms 34.790 ms
4 [AS4755] 115.114.89.17.static-mumbai.vsnl.net.in (115.114.89.17) 78.673 ms 79.056 ms 70.441 ms
5 * * *
6 * * *
7 * * *
8 * * *
^C
anurag:~ anurag$

 

(Surprising?) Well as we see – we can’t go beyond Tata-VSNL AS4755 border router in Mumbai. Why? Let’s ask it’s neighbor upstream router Tata AS6453. Checking route for IP 96.34.0.98 in Tata AS6453 routing table:

 

show ip bgp 96.34.0.98

Router: gin-mlv-core1
Site: IN, Mumbai, MLV
Command: show ip bgp 96.34.0.98

% Network not in table 

 

This situation is the one this blog post is about! 🙂

What’s bit confusing here is the fact that we are able to reach a destination IP say 71.89.140.11 as taken in this example and middle routers just seem normal but if we try to explicitly reach these middle routers then we don’t see a route. 

 

Why we see no route?

Because there’s just no route. These prefixes are not announced in global routing table via BGP. 

 

So technically no one is announcing any subnet in global IPv4 table which covers address space for 96.34.0.98.

 

Here’s another major backbone router in US:

route-server>
route-server> sh bgp ipv4 unicast 96.34.0.98
% Network not in table
route-server>

 

Did someone missed to announce a prefix? 

Well, answer is NO!
Everything is just fine in such setup. Basically many providers like Charter (and many ISPs) do not announce address space allocated to their backbone routers which are middle in chain to avoid possibility of packet flooding and possibly some other attacks.

 

Then how we are getting ICMP replies during initial trace to destination IP?

We get ICMP replies because we just followed chain, and in chain last router before Charter was Telia which is announcing its address space normally and we are able to reach it. Now that specific Telia router is having a BGP session with Charter router (since Charter is their downstream customer network) and that Telia router has multiple broadcast domains. Including the one which takes us to it 213.248.89.46 (coming from BGP announcement for 213.248.64.0/18 from AS1299). The other possible broadcast domain it has is /30 which is used for BGP session with Charter. /30 = 4IP’s. One goes for Telia router, other goes to Charter router, third one becomes broadcast IP and last one lies useless due to Maths. 😉

Hence that specific Telia router has routing table of Charter and knows from which “Physical interface” is the “next hop” to that Charter router and so does and next, next and next till we reach destination router (which is always on a well advertised address space).  The same logic pretty much applies on RFC 1918 based private address space too. Like 10.0.0.0/8 or 192.168.0.0/24 etc. 

Now as soon as one knows this chain – one can always add static routes in routing table and flood those routers (taking off the reason for not announcing address space). For IXP’s this part is also important – since lot of them use a shared peering VLAN which stays on single broadcast subnet often a /23 or /24. Will discuss more about IX prefix and announcement impacts in my future posts.

 

So that’s all about it. Have a good week ahead! 🙂