TECH

When a network intel provider’s domain serves fraudulent content, something is wrong

ThousandEyes, a San Francisco-based network intelligence service, helps customers monitor all kinds of mission-critical things, from border gateway protocol leaks to DNS performance. But over the past week or so, the company has struggled with its own networking blunder that allowed scammers to host hundreds of thousands of fraudulent documents on its very own domain.
As the screenshot above shows, vps4-atl1.ag0.thousandeyes.com was hosting PDFs promoting screenplays, books, and how-to guides. By being available on a subdomain of a legitimate network intelligence company, the content was designed to manipulate Google search results in a way that tricked people into clicking on questionable links. Google searches suggest that the documents were hosted on the subdomain since the beginning of the month, before being removed on Tuesday, as this story was being reported.
To park their content, the scammers took advantage of a lapse in the management of the ThousandEyes.com domain. An entry in the domain’s authoritative name servers pointed to the IP address 74.207.229.178. The IP address belongs to Web host Linode. ThousandEyes used the IP in the past, but at some point it stopped doing so. ThousandEyes admins, however, failed to remove the DNS entry from the name servers. The scammers then noticed the lapse, obtained the same IP address from Linode, and used it to host the scammy documents.
IThis was a stale DNS record from decommissioned infrastructure that was pointing to an IP address that we no longer use. The hosting provider re-used that IP address, and someone hosted those PDFs using that IP address. Since the stale DNS record was still pointing at that IP address, it appeared to be part of our domain. There was no compromise of our hosting, DNS, website or systems and no exposure of any of our or customer data. As soon as we saw the problem, our operations team was able to fix it and reported the issue to the hosting provider.n an emailed statement, ThousandEyes spokesman Ben Stricker wrote:
This was a stale DNS record from decommissioned infrastructure that was pointing to an IP address that we no longer use. The hosting provider re-used that IP address, and someone hosted those PDFs using that IP address. Since the stale DNS record was still pointing at that IP address, it appeared to be part of our domain. There was no compromise of our hosting, DNS, website or systems and no exposure of any of our or customer data. As soon as we saw the problem, our operations team was able to fix it and reported the issue to the hosting provider.
In a later email, ThousandEyes spokeswoman Susan O’Brien wrote: “Just to be clear, the abandoned DNS record was pointing to the IP address where the content was hosted. ThousandEyes was not hosting the content on any of its infrastructure.”
This is technically true. But it’s also true that ThousandEyes’ infrastructure was linked to a fraudulent IP address and that the lapse went undetected for about two weeks. That’s a serious error for a service that’s supposed to provide real-time intelligence that helps customers identify issues that impact user experience. Until late Wednesday night or early Thursday morning, the scammer’s server remained up and running on Linode. 
But again, once ThousandEyes admins removed the DNS entry from name servers on Tuesday, the content was no longer tied to the company’s domain.
The episode shows just how easy it is for just about anyone to make serious mistakes. While PDFs designed to manipulate search results are relatively benign, the people exploiting ThousandEyes’ security lapse could have used it to host much more nefarious things, such as links to malicious downloads. ThousandEyes’ role as a trusted network monitor would have provided the perfect cover.
Update: The subheadline "ThousandEyes falls prey to the type of DNS goof it helps customers avoid" has been changed at the request of ThousandEyes. "This is not the case, as we don’t perform security or content audits that would uncover these sorts of issues, nor do we have anything to do with configuration management or automation that would help avoid this sort of issue," officials wrote in an email. "Our service provides real-time network visibility that helps companies identify issues that impact application performance and user experience."
ThousandEyes Product Marketing VP Alex Henthorn-Iwane told Ars that while some of the company's DNS monitoring services detect when the IP address for DNS records have been maliciously changed—which arguably is a service related to security—the DNS monitoring doesn't detect when records improperly point to decommissioned IP addresses, which was the lapse reported in this post. ThousandEyes also objected to the characterization that it is a security company or that it provides security services. This post has been updated to remove those characterizations.


Dan Goodin