Death of the Dark Web? DARPA's Memex search engine allows Tor tracking

A search engine more powerful than Google has been developed by the US Defence Advanced Research Projects Agency (DARPA), capable of finding results within dark web networks such as Tor.

The Memex project was ostensibly developed for uncovering sex-trafficking rings, however the platform can be used by law enforcement agencies to uncover all kinds of illegal activity taking place on the dark web, leading to concerns surrounding internet privacy.

Thousands of sites that feature on dark web browsers like Tor and I2P can be scraped and indexed by Memex, as well as the millions of web pages ignored by popular search engines like Google and Bing on the so-called Deep Web.

Websites operating on the dark web, such as the former Silk Road black marketplace, purport to offer anonymity to their users through a form of encryption known as Onion Routing.

While users' identities and IP addresses will still not be revealed through Memex results, the use of an automated process to analyse content could uncover patterns and relationships that could potentially be used by law enforcement agencies to track and trace dark web users.

"We're envisioning a new paradigm for search that would tailor content, search results, and interface tools to individual users and specific subject areas, and not the other way round," said DARPA program manager Chris White.

"By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualise access to information. Ease of use for non-programmers is essential."

Memex achieves this by addressing the one-size-fits-all approach taken by mainstream search engines, which list results based on consumer advertising and ranking.

'The most intense surveillance state the world has literally ever seen'

The search engine is initially being used by the US Department of Defence to fight human trafficking and DARPA has stated on its website that the project's objectives do not involve deanonymising the dark web.

The statement reads: "The program is specifically not interested in proposals for the following: attributing anonymous services, deanonymising or attributing identity to servers or IP addresses, or accessing information not intended to be publicly available."

Despite this, White has revealed that Memex has been used to improve estimates on the number of services there are operating on the dark web.

"The best estimates there are, at any given time, between 30,000 and 40,000 hidden service Onion sites that have content on them that one could index," White told 60 Minutes earlier this month.

Internet freedom advocates have raised concerns based on the fact that DARPA has revealed very few details about how Memex actually works, which partners are involved and what projects beyond combating human trafficking are underway.

"What does it tell about a person, a group of people, or a program, when they are secretive and operate in the shadows?" author Cassius Methyl said in a post to Anti Media. "Why would a body of people doing benevolent work have to do that?

"I think keeping up with projects underway by DARPA is of critical importance. This is where the most outrageous and powerful weapons of war are being developed.

"These technologies carry the potential for the most intense surveillance/ police state that the world has literally ever seen."