DNS Query Duplication

In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss a traffic behaviour in the DNS which Geoff has noticed in the labs advertising based experimental data capture. Virtually every DNS query Geoff sees, he sees twice (or more). For a cohort of about 150 million unique DNS labels on a given day, The Labs system is collecting 270 million incoming DNS queries. Thats a lot of duplication. What's going on? The advertising data collection depends on issuing unique DNS queries, which in turn generate unique web page serves. This allows measurement of internet-wide behaviour on about 30 million browsers, games, devices every day. Not to say that these names and web URLs are not routinely seen by more than one entity, intermediary systems such as caches and proxies as well as re-visiting old open website tabs on a browser can cause this. The point is that after the "first" fetch, the subsequent fetches can usually be held to be re-presentations of the same experiment and can therefore often be discarded (in the case of the web). But, for the DNS which has always had an element of unreliable transport, and which in turn invites measurement of features like DNSSEC which cause SERVFAIL messages, and demand repeated attempts to "find the DNS name-to-address mapping, re-fetching is itself something under test. How many resolvers lie behind a given users systems? How many kinds of resolver (DNSSEC enabled, or not) does the user depend on? Finding that almost all DNS queries are repeated, and it turns out repeated very quickly invited Geoff to have a look at what's going on "under the covers". There are some patterns behind what's being seen which Geoff explored in the APNIC Labs Blog.