Just came across this brilliant talk by my friend Bert Hubert. It covers so nicely about the mad rush to just outsource everything and how innovation is lost. While he mentioned names of EU telcos in examples, unfortunately situation isn’t that different in this side of world either. Operator in South Asia also very much suffer with this problem.
Slides of this presentation are here.
I thought about this long back – “Who pays to whom in case of internet bandwidth?” I have been working in this domain from sometime and so far I have learnt that it’s really complex. I will try to put a series of blog post to give some thoughts on this subject.
Firstly we have to understand that when we talk about “bandwidth price” it’s often layer 3 bandwidth which you buy in form of capacity over ethernet GigE, Ten-GigE and so on (or STMs if you are in India). As we know from back school class in networking – layer 3 works over layer 2 and so to deliver “bandwidth” on layer 3, one needs layer 2 physical circuit. Price paid by companies on layer 2 Vs layer 3 varies significantly based on their location, type of business, their target goal etc. E.g a content heavy company like Google pays hell lot of money on layer 2 circuits while it is strongly believed among networking community that Google is a tier 1 network and hence a “transit free” zone and they do not pay any amount on layer 3. In general the trend is pretty much as big networks have larger network footprint and connected “PoPs” over layer 2 (leading to a higher layer 2 bill) while relatively lower layer 3 bill while small networks depend significantly just on transit bandwidth (in form of layer3) and have very low layer 2 footprint.
What’s more interesting here is the trend on “Who pays for the layer 2?”
Well it has relatively less clear trend and entire setup is based on fact that internet started in US and a significant content was hosted in US for a long time. Now overall that’s changing fast due to ways web is evolving but still if you consider traffic is flowing to/from Asia, you will often find that it’s on Asian telco’s layer 2 and not the Western telcos. E.g traffic from New York to Mumbai say between AT&T in New York to Tata Comm in Mumbai will be handed over to Tata’s PoP in New York and they will carry it all way down to Mumbai over their layer 2. While for packets travelling from Mumbai to New York, they will be carried over Tata’s layer 2 (yet again!) till New York and will be handed off to AT&T in New York. Now considering fact that Tata and AT&T both are Tier 1 settlement free ISPs, why it’s that only Tata pays? Same applies on relation between say NTT & Level3. And same even applies for traffic between small regional networks which eventually go via big tier 2 networks like say traffic flow between Level3 and Airtel etc (Airtel pays for layer 2).
Here’s list of reasons I can think off and have found so far in my numerous discussions with people around in industry:
Will share more on this subject in upcoming blog posts.
Sidenote: Visited Amsterdam Botanic Garden today and ending this post with some of pics from there. 🙂
Apart from that I also saw snowfall at Klagenfurt, Austria which was very nice experience. Here’s how it looked like:
Some free time here in Europe and thus time for another quick blog post & to take my mind away from depressing people!
One of impressive features of major European networks is support for BGP communities. In India it’s almost non-existent. Setting it up isn’t hard technically but from capacity management side, Indian ISPs are somewhat shy in setting it up.
Let’s put a case where we have a Customer router (R1 with AS1), upstream of customer (R2 with AS2), upstream of upstream (R3 with AS3), peer of upstream (R4 with router4). Let’s try to setup communities so that customer at AS1 can control his BGP announcements and announce some prefixes to AS3 and some to AS4 selectively to control inbound traffic flow.
All of them are peering with basic simple BGP session. AS1 is announcing 8.8.8.0/24 and 9.9.9.0/24 to R2 and wishes to announce 8.8.8.0/24 to R3 and R4 while 9.9.9.0/24 just to R4.
Now this selective announcement thing will be done at R2 but triggered by R1 based on community tags.
Here provider R2 will provide say following community strings:
3000 – for announcement to R3 only
4000 – for announcement to R4 only
If route is not tagged with any community, it will be announced to both (default behavior of BGP/upstream).
Before putting any community here’s what we can see on all routers:
R1#sh ip bgp
BGP table version is 3, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 0.0.0.0 0 32768 i
*> 9.9.9.0/24 0.0.0.0 0 32768 i
R1#R2#sh ip bgp
BGP table version is 26, local router ID is 1.1.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 1.1.1.1 0 0 1 i
*> 9.9.9.0/24 1.1.1.1 0 0 1 i
R2#
R3>sh ip bgp
BGP table version is 15, local router ID is 1.1.1.6
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 1.1.1.5 0 2 1 i
*> 9.9.9.0/24 1.1.1.5 0 2 1 i
R3>
R4>sh ip bgp
BGP table version is 13, local router ID is 1.1.1.10
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 1.1.1.9 0 2 1 i
*> 9.9.9.0/24 1.1.1.9 0 2 1 i
R4>
So basic interface conf and BGP conf seems all good. Now setting up community rules on R2 for announcement:
Now here’s simple logic I will put on R1 to tag routes:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ip pre
R1(config)#ip prefix-list ?
WORD Name of a prefix list
sequence-number Include/exclude sequence numbers in NVGENR1(config)#ip prefix-list prefix-list1 ?
R1(config)#ip prefix-list prefix-list1 permit 8.8.8.0/24R1(config)#ip prefix-list prefix-list2 permit 9.9.9.0/24
R1(config)#R1(config-route-map)#match ip address prefix-list prefix-list1
R1(config-route-map)#set community 3000
R1(config-route-map)#exit
R1(config)#route-map rmap permit 20
R1(config-route-map)#match ip address prefix-list prefix-list2
R1(config-route-map)#set community 4000
R1(config-route-map)#exit
R1(config)#R1(config)#router bgp 1
R1(config-router)#neighbor 1.1.1.2 send-community
R1(config-router)#neighbor 1.1.1.2 route-map rmap out
R1(config-router)#end
R1#wr
Building configuration…
[OK]
R1#
*Nov 8 19:04:21.810: %SYS-5-CONFIG_I: Configured from console by console
R1#clear bgp all 2
R1#
*Nov 8 19:04:28.898: %BGP-5-ADJCHANGE: neighbor 1.1.1.2 Down User reset
*Nov 8 19:04:29.486: %BGP-5-ADJCHANGE: neighbor 1.1.1.2 Up
Now let’s check R2 on what it is getting:
R2#sh ip bgp 8.8.8.0
BGP routing table entry for 8.8.8.0/24, version 56
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1 2
1
1.1.1.1 from 1.1.1.1 (1.1.1.1)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 3000
R2#sh ip bgp 9.9.9.0
BGP routing table entry for 9.9.9.0/24, version 55
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1
1
1.1.1.1 from 1.1.1.1 (1.1.1.1)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 4000
R2#
All good! 🙂
So now R2 is getting communities. Next logical step is setup of R2 to announce prefixes with community 3000 to R3 and 4000 to R4.
Next logical steps:
Here we go!
R2#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R2(config)#ip community-list 1 permit 3000
R2(config)#ip community-list 2 permit 4000
R2(config)#R2(config)#route-map rmap1 permit 10
R2(config-route-map)#match community 1
R2(config-route-map)#exit
R2(config)#route-map rmap2 permit 10
R2(config-route-map)#match community 2
R2(config-route-map)#exit
R2(config)#R2(config)#router bgp 2
R2(config-router)#neighbor 1.1.1.6 route-map rmap1 out
R2(config-router)#neighbor 1.1.1.10 route-map rmap2 out
R2(config-router)#end
R2#wr
Building configuration…
[OK]
R2#c
*Nov 8 19:16:44.394: %SYS-5-CONFIG_I: Configured from console by consol
R2#
R2#
R2#clear bgp all 3
R2#clear bgp all 4
R2#
Checking BGP announcements to each peer now:
R2#sh ip bgp neighbors 1.1.1.6 advertised-routes
BGP table version is 56, local router ID is 1.1.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 1.1.1.1 0 0 1 iTotal number of prefixes 1
R2#sh ip bgp neighbors 1.1.1.10 advertised-routes
BGP table version is 56, local router ID is 1.1.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 9.9.9.0/24 1.1.1.1 0 0 1 iTotal number of prefixes 1
R2#
And cross checking on each R3 and R4:
R3>sh ip bgp
BGP table version is 34, local router ID is 1.1.1.6
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 8.8.8.0/24 1.1.1.5 0 2 1 i
R3>
Only 8.8.8.0/24 is visible while on R3. Similarly on R4:
R4>sh ip bgp
BGP table version is 44, local router ID is 1.1.1.10
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 9.9.9.0/24 1.1.1.9 0 2 1 i
R4>
You can find config of each R1, R2, R3 and R4 for reference. Also checkout One Step Consulting page with BGP communities used by some major networks.
Time to end this blog post with a nice motivating poem from Former Prime Minister of India (sung in Hindi by popular Indian singer).