Isp-Column

Just back from APRICOT 2018. As I mentioned in my previous blog post, APNIC had its first Hackathon and it was fun (blog post of APNIC here). There was one project on the ranking of CDNs using RIPE Atlas data. To achieve this team was trying to find strings/hostnames which they can trace to and figure out nearby CDN. As part of that, I suggested them to look into www.facebook.com and carefully noting the sources from where elements get loaded. It’s quite common that Facebook.com (or Google.com for the logic) would be hosted on some server at a large PoP while FNA (or GGC) would serve only specific static content out of it. FNA, of course, sits on the IPs of the ISP hosting it. So in the source list, we found scontent.fktm1-1.fna.fbcdn.net and that gives an idea that FNA strings are around logic: scontent.fxxx1-1.fna.fbcdn.net where xxx is the airport code. 1-1 means 1st PoP in 1st ISP over there probably (strong guess!). If there are more FNA nodes in a given area, the number goes further up. The team used it and for now, the project is over. But while I was on the way back to India, I thought that this is very interesting data if we pull the full picture by querying all possible IATA airport codes with a logic. This logic can be used for two things:

Writing this post from my hotel room in Kathmandu. I found that many of the servers appear to be DNS resolvers which is unusual.
Have a look at these weird DNS replies:

dig @anuragbhatia.com . ns +short
a.root-servers.net.
b.root-servers.net.
c.root-servers.net.
d.root-servers.net.
e.root-servers.net.
f.root-servers.net.
g.root-servers.net.
h.root-servers.net.
i.root-servers.net.
j.root-servers.net.
k.root-servers.net.
l.root-servers.net.
m.root-servers.net.

dig @google.com . ns +short
b.root-servers.net.
c.root-servers.net.
d.root-servers.net.
e.root-servers.net.
f.root-servers.net.
g.root-servers.net.
h.root-servers.net.
i.root-servers.net.
j.root-servers.net.
k.root-servers.net.
l.root-servers.net.
m.root-servers.net.
a.root-servers.net.

This seems unusual and is the result of basically port 53 DNS hijack. Let’s try to verify it using popular “whoami.akamai.net” query.

APNIC and RIPE NCC are doing a hackathon at APRICOT 2018. It just started today with some light interaction with various participating members yesterday. The theme of the hackathon is around IPv6. Many cool projects were suggested yesterday and teams started working today on certain shortlisted projects like:

1. A tool for ranking CDNs - A tool based on RIPE Atlas data to rank CDNs based on latency across different regions.
2. An IPv6 fun word game - Where anyone with a member account can suggest a word, and compete with other members who share more IPv6 addresses. It may include things like showcasing creative use of hexadecimal strings in an IPv6 address like Facebook popularly does face:b00c in their IPv6 pools.
3. IPv4 and IPv6 network security  - Study of attacks and overall security in IPv6. It would involve study and possibly a report on various attack vectors in the IPv6 domain.
4. A countrywide report on IPv6 deployment - I have yet to see how it is different from existing other reports.
5. IPv6 tunnel detection - Figuring out where tunnels used and figuring out the IPv4 address of those endpoints via a javascript plugin and possibly comparing IPv4 Vs IPv6 performance.

Let’s see how things go in next 12hrs. Super fun. Things should show up on Github in next few hours. :)

And here goes first blog post of 2018. Last few months went busy with some major changes in personal life. :) I looked into Amazon’s India connectivity with various ASNs tonight. Here’s how it looks like. (Note: Jump to bottom most to skip traces and look at the summary data).  

 

Traceroutes

Amazon India to Vodafone India

traceroute to 118.185.107.1 (118.185.107.1), 30 hops max, 60 byte packets
 1 ec2-52-66-0-128.ap-south-1.compute.amazonaws.com (52.66.0.128) 21.861 ms ec2-52-66-0-134.ap-south-1.compute.amazonaws.com (52.66.0.134) 19.244 ms 19.233 ms
 2 100.64.2.200 (100.64.2.200) 14.789 ms 100.64.0.200 (100.64.0.200) 20.731 ms 100.64.3.12 (100.64.3.12) 13.187 ms
 3 100.64.0.193 (100.64.0.193) 14.418 ms 100.64.3.69 (100.64.3.69) 15.469 ms 100.64.3.67 (100.64.3.67) 15.946 ms
 4 100.64.16.67 (100.64.16.67) 0.343 ms 100.64.17.165 (100.64.17.165) 0.312 ms 100.64.17.199 (100.64.17.199) 0.313 ms
 5 52.95.67.213 (52.95.67.213) 1.942 ms 52.95.67.209 (52.95.67.209) 1.967 ms 52.95.67.213 (52.95.67.213) 1.935 ms
 6 52.95.66.218 (52.95.66.218) 4.998 ms 4.694 ms 52.95.66.130 (52.95.66.130) 4.650 ms
 7 52.95.66.67 (52.95.66.67) 1.752 ms 52.95.66.89 (52.95.66.89) 1.850 ms 1.806 ms
 **8 52.95.217.183 (52.95.217.183) 3.111 ms 3.102 ms 3.088 ms <- Amazon India**
 **9 182.19.106.204 (182.19.106.204) 3.426 ms 4.547 ms 4.537 ms <- Vodafone India**
10 118.185.107.1 (118.185.107.1) 2.035 ms 2.059 ms 2.039 ms

 

A few weeks back an Indian ISP contacted me via a contact form on my blog. That ISP has been struggling with a targetted DDoS attack. For the reason of privacy as well as the stability of their network, I will not put their name or AS number. The attack on that ISP was much higher than their bandwidth levels. Their upstream did not really share the volume of attack but I could tell from the screenshots they shared was that it was distributed volumetric attack choking their upstream bandwidth. I suggested that ISP get the blackholing option from his upstream (preferred way) or buy a cheap server/VM somewhere outside India with BGP (and BGP blackholing) and manually blackhole traffic when the attack comes. They were able to get blackholing enabled from their upstream and it did work. They started blackholing traffic and it helped to manually drop traffic going towards the IPs which were being attacked. It’s important to have BGP blackholing because it helps a network to signal upstream ISP about the pools which are under attack and to drop traffic towards them. ISPs further signal the same to their upstream and larger networks typically drop that traffic on all their edge routers i.e closer to the entry fo attack.   Next, the problem which hit that ISP was that it was a pain to manually find IPs which were under attack and quickly drop them. I suggested them to try fastnetmon. I heard of fastnetmon from the presentation of Job Snijders at NLNOG (slides here & video here). Fastnetmon is developed and maintained by Pavel Odintsov (who works at Cloudflare AS13335).