22 Dec

DNS hack of Google, Facebook & more sites in .bd

Yesterday Google’s Bangladeshi website google.com.bd was hacked and this happened via DNS. It was reported on the bdNOG mailing list at morning in a thread started by Mr Omar Ali where he shared this screenshot:

This clearly shows how authoritative DNS for “com.bd.” (which is same as bd. btw) was poisoned and was reflecting attackers authoritative DNS. Later Mr Farhad Ahmed posted a screenshot of google.com.bd showing hackers page:

Later Mr Sumon Ahmed mentioned that it happened because web frontend of .bd was compromised. This was an interesting hijack as attacker attacked the key infrastructure of the registry instead of Google or Facebook servers. It’s also a warm reminder of the way DNS depends on the hierarchal structure by design and at this stage, we need to focus on DNSSEC to add on the security to the current system.
Lately .bd domain faced issues multiple time this year. I hope it will have a good stable time in the upcoming year. In terms of stability it is being backed by PCH’s anycast infrastructure but PCH’s DNS servers are just published in NS records of it’s existing auth servers, but not on the parent zone (which is root zone). Thus the point of failure remains and is yet to be fixed.

dig @dns.bd. bd. ns +short
dig @i.root-servers.net. bd. ns
; <<>> DiG 9.8.3-P1 <<>> @i.root-servers.net. bd. ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54130
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;bd.				IN	NS
bd.			172800	IN	NS	surma.btcl.net.bd.
bd.			172800	IN	NS	jamuna.btcl.net.bd.
bd.			172800	IN	NS	dns.bd.


13 Dec

Issues with Google's network in India due to Vardah

Today Google (AS15169) seem to be facing issues in their Indian network due to tropical cyclone Vardah. Their traffic at PoPs in Mumbai dipped considerably for a number of ISPs. My guess is that it’s likely because of outage in a large Govt. operator’s network who has overhead fibres along with utility lines.
It’s very important to note that “Google PoP” faced the issue and it’s no where close to saying that Google services went down. Google has a large network across the globe where they peer with networks. If one segment of this network goes down, traffic is re-routed via other parts and as per design even if the network goes down completely in say Mumbai or Chennai, services should stay live. While in real practice considerable degradation occurs because most of the Indian networks get a very large amount of traffic from Google and usually do not have that much extra capacity on their IP transit links, resulting in choking of transits during issues on their PNI with Google.

This shows how traffic of an ISP connected to Google in Mumbai dipped during peak time around at 4pm on Monday 12th Dec (IST) and went to zero little before midnight. I triggered a trace to aspmx.l.google.com. which is outside India from RIPE atlas probes in India and in general routing to that goes via Google’s backbone.
Cluster with hostname aspmx.l.google.com (and few others) carry the Gmail/Google Apps traffic and it’s published by Google Apps users in their domain’s MX records. Measurement results here – https://atlas.ripe.net/measurements/6959474/#!probes

Cases, where latency is less than 200ms, are actually failures. Some examples of failed traces:
From AS4755: 

from AS9430:

from AS9829:

Clearly, a lot of re-routing was happening and many times it was routers which took a while to re-route traffic across a down path due to BGP convergence.
With the hope that your YouTube streams run fine while you share them on Gmail, time for me to get back to work! 🙂