Google's routing issues because of an Indonesian ISP

Yesterday it was reported across networking community that Google’s prefixes were having issue due to an Indonesian ISP Moratel AS23947.

Quick analysis

From data logged by routeviews it seems like it wasn’t exactly a prefix hijack. AS23947 did not originated prefixes but rather had a route leak leading to path leak of AS23947 > AS15169. 

Here’s a view of global routing table for Google’s prefix 216.239.32.0/24 at 15:57 GMT on 4th Nov:

Next at 02:07:27 GMT on 6th Nov morning, a route change is logged.

FROM 4436 15169
TO 4436 3491 23947 15169

This path change is observed by routeviews only for one of its participating networks - nLayer AS4436. Rest participating networks seem not having any change. By 02:07:31 i.e within 4 seconds entire route goes via AS23947 for this specific network (and likely few more). At 02:35:06 i.e after 28 mins of this route leak, it is withdrawn and we can see within next few seconds direct route is preferred again. 

The impact was only on very limited part of internet because of fact that Google peers with lot of big networks directly and thus a short path is preferred  E.g Comcast in US will ignore this because Comcast AS36732 > Google AS15169 is short AS path as compared to Comcast > Someone else > Google. 

In above specific part - nLayer preferred long path likely because of its relation with Google. Most of networks prefer Customer routes (in which they get paid) over settlement free peering routes (in which they don’t get anything) over transit routes (in which they have to pay). The Indonesian ISP Moratel seems to have transit from few major players including PCCW Global which was likely running an unfiltered BGP session with its client. Thus largest impact came on PCCW Global Network which is Hong Kong based and fairly large in Asia. 

These glitches remind network engineers to careful configure router to avoid screwups! :)