Hey Cortana, how do I do a crime?

We are all MUID=1250DBD27B4260BC32E2D5E67F4263BF on this blessed day

Last week I was surprised to hear Microsoft offers free VM images of Windows 10, the most recent edition of their cross-platform Minecraft runtime. In fact they've been doing this since 2017, in an effort to make Windows a more attractive target for developers:

Microsoft is providing a free virtual machine that comes preloaded with Windows 10 Enterprise, Visual Studio 2017, and various utilities in order to promote the development of Universal Windows Platform apps.

[…]

When downloading the development environment, you can choose either a VMware, VirtualBox, Hyper-V, or Parallels virtual machine depending on what virtual machine software you use. Each of these images are about 17-20GB when extracted from the downloaded archive and include almost everything you need to develop Universal Windows Platform apps.

Technically it's a limited-time trial – activation costs $200 – but other than that it has most[1] of the functionality of a physical Windows machine. You can compile things, run shell scripts, and it even comes pre-provisioned with a Bing session so all of your web searches get recorded in 𝓉𝒽𝑒 π’Έπ“π‘œπ“Šπ’Ή.

Except – and this is really important – it's not your web searches. Remember, VM image! All the downloads have the same Bing session ID, and therefore the search history is shared among all instantiations of that image. When I took these screenshots the history had about nine thousand recent entries; I haven't tested whether the different formats get their own sessions, or how often the history is cleared. Scrolling down too far would break the page DOM, but search worked and you don't have to dig too deep to find good stuff.

Wait, good stuff? Isn't this a developer tools image? How much fun are people going to have searching for msvc compiler command or how to install tensorflow?

Well, sorta. There is a cohort of people who work in computer security and/or read a lot of grim cyberpunk, and they do some (or most! or all!) of their web browsing from ephemeral VMs in a hardened hypervisor. There is another cohort of people who once attended DerbyCon and/or read a lot of Charles Stross, and they apparently do some of their web browsing in a trial-period Windows VM downloaded fresh from the open internet.

This screenshot is how I found out about the whole thing and every time I look at it I just can't keep writing. It's too funny. Some schmuck asked Bing how to commit money laundering and then – ten minutes later! – decided the search results were unsatisfactory due to a typo, and tried again!

By any other CNAME

When an engineer joins Google, they are issued a workstation – a physical computer in tower[2] form-factor that sits under their desk. Workstations have names, which the engineer gets to choose, and the fully-qualified hostname consists of this name plus an office-specific suffix. Workstations in Mountain View are in .mtv.corp.google.com, workstations in New York are in .nyc.corp.google.com, and so on – this is all tracked in various databases and synced to DNS. Intranet services that weren't office-specific, like the go/ URL shortener, were on .corp.google.com directly.

For convenience, Google uses a DNS feature named the DNS search path to let users reference workstations by short names. If I am sitting next to you, and I want to SSH into your workstation[3], I can type ssh yourbox instead of ssh yourbox.mtv.corp.google.com and it'll work. This wasn't only for workstation names, you could also use it for any sort of .corp.google.com name. In a browser you could type http://go/somelink and it would resolve to http://go.corp.google.com/somelink. In a PAM module you could #define LDAP_ADDRESS "ldap" and it would direct queries to ldap.corp.google.com. The halls rang with the sound of protobuf engineering, and all was at peace.

Some of you have noticed the problem.

One day I arrived at the office and discovered that I couldn't unlock my screen. This wasn't especially alarming, because nobody else in the building could log in and network outages happen sometimes. But then news started trickling in over working comms[4] that this wasn't a network outage. A few early risers had working desktop sessions, and the network was fine – only attempts to log in, SSH, or sudo were hanging.

Also, it was affecting the entire Mountain View campus.

The usual debugging process was followed with unusual haste and the issue was narrowed down to DNS. One of the new hires starting that day had requested their workstation be named ldap, per their initials, and as soon as that hostname hit the network it hijacked every LDAP client that had been configured to talk to "ldap". Unfortunately that was a big 'every' because (1) if the wrong value is easier to type then it outcompetes the correct value, and (2) the chances of a misconfiguration being discovered are the inverse of how often it happens. So pretty much everything was broken.

This story has a happy ending because Google does regular disaster recovery tests. The tests are always something outlandish, like aliens have invaded and all contact with California has been lost, and everyone has a good laugh around the coffee robot. The recovery procedure for total DNS outage involves taking a laptop into the panic room, a locked room with a direct connection to the Prod network. This was done, the owners of the machine inventory were able to delete the bad record, and new safety checks were installed around the important hostnames.


  1. The most important Windows software of all time, Space Cadet Pinball, remains absent.

  2. There was a brief experiment involving decommissioned Warp 19s, which were Google-designed rackmount machines with infamously sharp edges and the sound profile of a motorrad.

  3. This is not as alarming as it sounds; workstations at Google are (were?) fairly untrusted, and it was common to SSH into a coworker's machine if your own was overloaded or doing software updates or whatever.

  4. Engineers almost universally used IRC for quick conversations, team chatter, and coordinating incident response. Although Slack has some good points, I find myself missing good ol' port 6697 every time loading a channel makes my fan spin up.