30 Jun

Private IPs in Public routing

Sometimes we see interesting IP’s in traceroute & they confuse lot of people.

I have seen this topic in discussion twice on NANOG and once on Linux Delhi user group. 

 

OK – let’s pick an example: 

anurag:~ anurag$ traceroute 71.89.140.11
traceroute to 71.89.140.11 (71.89.140.11), 64 hops max, 52 byte packets
1 router (10.10.0.1) 1.176 ms 0.993 ms 0.941 ms
2 117.220.160.1 (117.220.160.1) 20.626 ms 29.101 ms 19.216 ms
3 218.248.169.122 (218.248.169.122) 23.983 ms 43.850 ms 45.057 ms
4 115.114.89.21.static-mumbai.vsnl.net.in (115.114.89.21) 118.094 ms 81.447 ms 66.838 ms
5 172.31.16.193 (172.31.16.193) 115.979 ms 90.947 ms 90.491 ms
6 ix-4-2.tcore1.cxr-chennai.as6453.net (180.87.36.9) 95.778 ms 98.601 ms 98.920 ms
7 if-5-2.tcore1.svw-singapore.as6453.net (180.87.12.53) 321.174 ms
if-3-3.tcore2.cxr-chennai.as6453.net (180.87.36.6) 331.386 ms 326.671 ms
8 if-6-2.tcore2.svw-singapore.as6453.net (180.87.37.14) 317.442 ms
if-2-2.tcore2.svw-singapore.as6453.net (180.87.12.2) 334.647 ms 339.289 ms
9 if-7-2.tcore2.lvw-losangeles.as6453.net (180.87.15.26) 318.003 ms 328.334 ms 309.234 ms
10 if-2-2.tcore1.lvw-losangeles.as6453.net (66.110.59.1) 306.500 ms 326.194 ms 341.537 ms
11 66.110.59.66 (66.110.59.66) 315.431 ms 330.417 ms 308.372 ms
12 dls-bb1-link.telia.net (213.155.136.40) 354.768 ms 344.360 ms 357.050 ms
13 chi-bb1-link.telia.net (80.91.248.208) 352.479 ms 358.751 ms 359.987 ms
14 cco-ic-156108-chi-bb1.c.telia.net (213.248.89.46) 367.467 ms 370.482 ms 377.280 ms
15 bbr01aldlmi-bue-4.aldl.mi.charter.com (96.34.0.98) 387.269 ms 385.362 ms 365.694 ms
16 crr02aldlmi-bue-2.aldl.mi.charter.com (96.34.2.11) 375.275 ms 375.356 ms 371.621 ms
17 dtr02grhvmi-tge-0-1-0-0.grhv.mi.charter.com (96.34.34.83) 383.539 ms 371.817 ms 383.804 ms
18 dtr02whthmi-tge-0-1-0-0.whth.mi.charter.com (96.34.34.85) 384.400 ms 391.197 ms 393.340 ms
19 dtr02ldngmi-tge-0-1-0-0.ldng.mi.charter.com (96.34.34.87) 371.192 ms 375.679 ms 378.457 ms
20 acr01mnplmi-tge-0-0-0-3.mnpl.mi.charter.com (96.34.40.75) 364.824 ms 385.534 ms 374.401 ms
21 * *^C
anurag:~ anurag$

 

 

Let’s try pinging IP on 14th hop (which is with a major backbone Telia) – 213.248.89.46

anurag:~ anurag$ ping -c 5 213.248.89.46
PING 213.248.89.46 (213.248.89.46): 56 data bytes
64 bytes from 213.248.89.46: icmp_seq=0 ttl=240 time=517.305 ms
64 bytes from 213.248.89.46: icmp_seq=1 ttl=240 time=329.230 ms
64 bytes from 213.248.89.46: icmp_seq=2 ttl=240 time=324.397 ms
64 bytes from 213.248.89.46: icmp_seq=3 ttl=240 time=331.474 ms
64 bytes from 213.248.89.46: icmp_seq=4 ttl=240 time=326.409 ms

— 213.248.89.46 ping statistics —
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 324.397/365.763/517.305/75.809 ms
anurag:~ anurag$

  

Works fine! 

 

Game begins here…

 

Next, let’s try pinging hop 15th IP which is with a major cable company Charter operating in US East – 96.34.0.98

PING 96.34.0.98 (96.34.0.98): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

— 96.34.0.98 ping statistics —
5 packets transmitted, 0 packets received, 100.0% packet loss
anurag:~ anurag$

  

So we see some nice timeouts. This confuses lot of people as we can’t have a firewall blocking ICMP packets here since we did had ICMP based traceroute with ICMP replies from 15th hop in last trace.

 

Let’s try to do a trace to this IP to see where exactly is connection breaking.

anurag:~ anurag$ traceroute -a 96.34.0.98
traceroute to 96.34.0.98 (96.34.0.98), 64 hops max, 52 byte packets
1 [AS65534] router (10.10.0.1) 1.661 ms 0.887 ms 0.934 ms
2 [AS9829] 117.220.160.1 (117.220.160.1) 18.867 ms 31.898 ms 20.931 ms
3 [AS9829] 218.248.169.118 (218.248.169.118) 43.427 ms 22.327 ms 34.790 ms
4 [AS4755] 115.114.89.17.static-mumbai.vsnl.net.in (115.114.89.17) 78.673 ms 79.056 ms 70.441 ms
5 * * *
6 * * *
7 * * *
8 * * *
^C
anurag:~ anurag$

 

(Surprising?) Well as we see – we can’t go beyond Tata-VSNL AS4755 border router in Mumbai. Why? Let’s ask it’s neighbor upstream router Tata AS6453. Checking route for IP 96.34.0.98 in Tata AS6453 routing table:

 

show ip bgp 96.34.0.98

Router: gin-mlv-core1
Site: IN, Mumbai, MLV
Command: show ip bgp 96.34.0.98

% Network not in table 

 

This situation is the one this blog post is about! 🙂

What’s bit confusing here is the fact that we are able to reach a destination IP say 71.89.140.11 as taken in this example and middle routers just seem normal but if we try to explicitly reach these middle routers then we don’t see a route. 

 

Why we see no route?

Because there’s just no route. These prefixes are not announced in global routing table via BGP. 

 

So technically no one is announcing any subnet in global IPv4 table which covers address space for 96.34.0.98.

 

Here’s another major backbone router in US:

route-server>
route-server> sh bgp ipv4 unicast 96.34.0.98
% Network not in table
route-server>

 

Did someone missed to announce a prefix? 

Well, answer is NO!
Everything is just fine in such setup. Basically many providers like Charter (and many ISPs) do not announce address space allocated to their backbone routers which are middle in chain to avoid possibility of packet flooding and possibly some other attacks.

 

Then how we are getting ICMP replies during initial trace to destination IP?

We get ICMP replies because we just followed chain, and in chain last router before Charter was Telia which is announcing its address space normally and we are able to reach it. Now that specific Telia router is having a BGP session with Charter router (since Charter is their downstream customer network) and that Telia router has multiple broadcast domains. Including the one which takes us to it 213.248.89.46 (coming from BGP announcement for 213.248.64.0/18 from AS1299). The other possible broadcast domain it has is /30 which is used for BGP session with Charter. /30 = 4IP’s. One goes for Telia router, other goes to Charter router, third one becomes broadcast IP and last one lies useless due to Maths. 😉

Hence that specific Telia router has routing table of Charter and knows from which “Physical interface” is the “next hop” to that Charter router and so does and next, next and next till we reach destination router (which is always on a well advertised address space).  The same logic pretty much applies on RFC 1918 based private address space too. Like 10.0.0.0/8 or 192.168.0.0/24 etc. 

Now as soon as one knows this chain – one can always add static routes in routing table and flood those routers (taking off the reason for not announcing address space). For IXP’s this part is also important – since lot of them use a shared peering VLAN which stays on single broadcast subnet often a /23 or /24. Will discuss more about IX prefix and announcement impacts in my future posts.

 

So that’s all about it. Have a good week ahead! 🙂

15 May

Do connected interface ping?

And an interesting day full of bit frustrating drama. Today was “External viva” for Major Project at college. It went good with external teacher but “internal ones” tend to cause un-necessary issues. Quite a few people put personal egos and frustration on top priority to an extent that they violate their own points for which they are arguing. They go completely unethical in way they deal with world.

 

 

I am saying this with full responsibility for couple of teachers from my college who have completely lost some “fundamentals of life” as taught in childhood to most of us. Some key principles like staying cool & calm, being humble, making best possible use of time and just being good with everyone. In last 4 years they haven’t learn how to give respect & talk with sense and they expect students to be learning “technology” from them? What an absurd!

 

Anyways not much I can do about it. My own external viva went fine and that’s good enough to stay happy. 🙂
All this reminds of an old amazing poem by Former Prime Minister of India (find it embedded below):

 

 

 

Today’s post…

Personal frustration aside – An interesting topic for today. One of my friend came up with an interesting question. Are all connected routers supposed to be pinging?

It’s simply about two routers – say router A and router B which whose interfaces are connected by a cable. Should they ping? If yes then under what circumstances?

 

Answer is …..mmm…let’s first dive in a little before I gave plain answer.

 

 

Ping/ICMP

Ping works on ICMP protocol and that comes on layer 3 on OSI model.

ICMP simply sends a packet (in a form of “Hello there!”) to destination host machine and if everything is OK then host machine is expected to be replying back with “Hi!” to the source. Time taken in this is counted in round trips (because a non-round trip based counting is logically not possible). This provides with connectivity confirmation + roundtrip latency in the connection.

 

So what are conditions required for ICMP to work?

 

Going into fundamentals of TCP/IP networking, a packet can route only when it knows answer to three simple questions:

  1. Who I am?
  2. Where I need to go?
  3. How I will get there?

 

 

Answer to “Who I am?” comes from the unique IP address (on broadcast domain), “Where I need to go” comes from the user himself who is pinging destination machine IP and “How I will get there?” comes from routing table.

 

So #1 and #3 are important here.  There’s not really BGP, or any IGP protocol when two routers are connected directing on same broadcast (layer 2) and this is where other low level protocol ARP comes into picture. ARP is used in creating a simple table which keeps a reference between Mac address, IP address and interface.

 

How does ARP work?

ARP simply works using “broadcast address” we use while configuring IP address on a interface. E.g if I am putting IP as 10.0.0.1 coming from /29 subnet (or call it 255.255.255.248), it assumes last IP to be “broadcast IP”. A /29 here means 2^32-29 = 8 IPs. Starting from 10.0.0.1, it goes till 10.0.0.7. So last IP = 10.0.0.7 is broadcast IP. This is used by all machines under same broadcast domain to “announce/advertise” their IP address from their Mac address.

 

Let’s play around and connect three routers to a switch (same layer 2 broadcast, single VLAN).

We have router A, router B and router C.

I am putting A and B on 10.0.0.1 and 10.0.0.2 under same subnet (a /29) while router C will be on 10.0.0.100 coming from /24 subnet (for fun!).

 

Screen Shot 2013-05-15 at 6.51.34 PM

 

 

Router>
Router>en
Router>enable
Router#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#host
Router(config)#hostname A
A(config)#in
A(config)#interface F
A(config)#interface FastEthernet0/0

A(config-if)#ip add 10.0.0.1 255.255.255.248
A(config-if)#no shut
A(config-if)#no shutdown
A(config-if)#end
A#
00:11:44: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
00:11:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
00:11:46: %SYS-5-CONFIG_I: Configured from console by console
Building configuration…
[OK]
A#
A#

 

OK – let’s go on B:

 

Router>en
Router>enable
Router#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#host
Router(config)#hostname B
B(config)#int
B(config)#interface F
B(config)#interface FastEthernet0/0
B(config-if)#ip add 10.0.0.2 255.255.255.248
B(config-if)#no shutdown
B(config-if)#end
B#
00:16:06: %SYS-5-CONFIG_I: Configured from console by console
B#
B#
00:16:06: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
00:16:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
B#
B#write
Building configuration…
[OK]
B#

 

Now comes the “different one” i.e router C:

Router>
Router>en
Router>enable
Router#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#rou
Router(config)#host
Router(config)#hostname C
C(config)#int
C(config)#interface F
C(config)#interface FastEthernet0/0
C(config-if)#ip add 10.0.0.100 255.255.255.0
C(config-if)#no shu
C(config-if)#no shutdown
C(config-if)#write
00:01:38: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
00:01:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state
C(config-if)#
C(config-if)#end
C#wr
00:01:44: %SYS-5-CONFIG_I: Configured from console by consoleite
Building configuration…
[OK]
C#

 

 

OK – now we have three routers with two on same /29 subnet and third one on a /24 subnet. Let’s try to ping:

A#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms

 

Worked. OK – let’s try C now:

 

A#ping 10.0.0.100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.100, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
A#

 

Failed.

 

Let’s look at routing table on A:

A#sh ip route connected
10.0.0.0/29 is subnetted, 1 subnets
C 10.0.0.0 is directly connected, FastEthernet0/0
A#

 

OK – so only one entry for 10.0.0.0/29 which is connected directly to Ethernet interface 0/0.

 

If we look at ARP table, we get:

A#
A#sh ip arp
A#sh ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.0.0.2 12 ca03.35dc.0008 ARPA FastEthernet0/0
Internet 10.0.0.1 – ca02.35dc.0008 ARPA FastEthernet0/0
A#

 

This answers the question completely. 🙂

We have only A and B it ARP table of A. Both B and C are connected to same switch, same VLAN but C is not “visible” to A because C is on /24 subnet and that means last IP of 10.0.0.0/24 i.e 10.0.0.255 is acting as broadcast. C is sending it’s live updates for ARP on 10.0.0.255 while A and B are doing that on broadcast IP 10.0.0.7 (last IP from 10.0.0.0/29) and are not “hearing” on 10.0.0.255. Hence A and B are “hearing” on same broadcast and are updating ARP table while C even connected on same switch, same layer 2 is not connected on layer 3 and thus ICMP ping does not work.

 

It pretty much about those three fundamental questions. Again, third question was “How I will get there?” needs an answer from routing table. So let’s go ahead and tell A about lonely router C. 🙂

 

A(config)#ip route 10.0.0.0 255.255.255.0 F
A(config)#ip route 10.0.0.0 255.255.255.0 FastEthernet0/0
A(config)#end
A#writ
00:38:50: %SYS-5-CONFIG_I: Configured from console by consolee
Building configuration…
[OK]
A#
A#

 

Now there’s a static route entry on A which tells where it for subnet 10.0.0.0/24 and IP 10.0.0.100 belongs to that.

Let’s try pinging again:

A#ping 10.0.0.100

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms
A#

 

And A can now ping C.  What about C?

C#ping 10.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms

 

C#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
C#

 

So now C is able to reach A but not B because again B doesn’t knows where is C. If we look at ARP table of C now:

C#
C#sh ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.0.0.1 3 ca02.35dc.0008 ARPA FastEthernet0/0
Internet 10.0.0.100 – ca06.35dc.0008 ARPA FastEthernet0/0
C#

 

So 10.0.0.2 i.e B is still missing. Let’s tell B to “default” all traffic via A.

 

B>
B>en
B>enable
B#conf t
Enter configuration commands, one per line. End with CNTL/Z.
B(config)#ip ro
B(config)#ip route
B(config)#ip route 0.0.0.0 0.0.0.0 10.0.0.1
B(config)#end
B#write
Building configuration…
[OK]
B#

 

Checking again on B now:

 

C#ping 10.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms
C#

 

 

So that’s about it.

 

Summary

Ping works on ICMP and it is on layer 3. For it to work, layer 2 should be connected along with working logical layer 3 with entries in routing table. Unless two directly connected machines are on same subnet they won’t ping because different subnet will get them on different broadcast IP and thus different ARP tables all together.

And don’t miss that lot of firewalls block ICMP partially as well as completely because it tends to overload routers in processing those packets rather then doing normal switching operation.

 

Time for me to get back to work!

 

Note: My comments on college teachers are specifically for two teachers (and everyone around me knows whom I am referring too) and one should not make a general impression out from those. There are some very good teachers in dept. as well.