In Contempt of Bulk Surveillance: It’s Too Easy

By Nicholas Weaver
Wednesday, September 16, 2015, 11:30 AM

The Intelligence Community has a concept, NOBUS, or “Nobody but Us”, to describe unique capabilities they possess which our adversaries can’t employ against us. I may defend the effectiveness of bulk surveillance and attack, but these tools are anything but NOBUS.

About a year and a half ago, mostly for my own entertainment, I started a small hobby project. I previously argued in a talk that the primary NSA Digital Network Intelligence flow was conceptually straightforward, a blend of Network Intrusion Detection (NIDS), big-data analytics, packet injection, and malcode. Yet this was at the time an academic pontification, without a system to back it up; there was some doubt in the audience. 

So I got out my credit card, bought a small computer, a network tap, and some zip-ties, and got to work. The goal was “bulk-surveillance in miniature”, a system implementing the primary NSA capabilities on 100 Mbps networks, including easily searchable bulk recording, user identification, cookie tracking, packet-injection attacks, and a web interface.


I used the Bro Network Security Monitor, a very powerful open-source NIDS. Bulk recording in Bro, outputting all traffic into separate files, is a built-in function accessed with a single line of code. Since I wanted to quickly search the bulk record, reassembling traffic offers a significant performance advantage.

Bro already parses HTTP headers, extracts a huge amount of content-derived metadata, and provides an interface for accessing file content transferred over the wire, so it was a simple matter to write code in Bro’s scripting language to extract usernames when a browser requests one of several sites (Yahoo!, ArsTechnica, Amazon) which display the username in the returned web page. A similar small amount of code handled the cookie tracking that the NSA likely performs.   

With the addition of a web interface, this provides the key operations for surveillance: it identifies users, links their cookies, and offers a full search interface for both the metadata and the full content.

Finally, packet injection simply required calling Bro’s own packet injection utility, redirecting a target’s JavaScript request to an alternate URL. The resulting JavaScript causes the target’s web browser to replace all images in a page with pictures of a pony. It took me roughly 50 hours to assemble the system and the “under the hood” data collection required less than 600 lines of Bro code. The data collection scales both up and down almost linearly with cost: $850 for a 100 Mbps system like the one I built, $5000 for a 1 Gbps installation, and larger systems can use a load balancer and a cluster, as it is well known how to construct large parallel NIDS

It is slightly more complicated to install taps on a network backbone, as backbone taps might only see one side of the communication (such as either the page request or the server’s reply, not both). Bro currently requires seeing both halves, in order to match requests to responses. But in building the interface to search captured traffic, I needed to write a standalone parser for HTTP replies to search captured data for compressed pages. During a boring plane flight, I implemented a “single-sided” parser for HTTP traffic.

Someone building backbone-level surveillance system could start with Lockheed Martin’s Vortex, a free “near-real-time” tool designed as a building block for network surveillance, and then add custom-written parsers to handle single-sided traffic. It’s more work (since Vortex lacks Bro’s protocol parsers), but solves the problems involved in deploying these techniques on a backbone link. 

Or, if a country prefers, they can always buy an “EAGLE GLINT” turnkey system for “massive” intelligence collection from Nexa Technologies (formerly a part of Amesys). Nexa’s system does not include a packet injector, but that can be purchased separately from Hacking Team.

Surveillance systems allow foreign intelligence to identify traffic belonging to Americans when they travel in the foreign country, an ability to track their movements across the network, and offers selective “attack-by-name” system exploitation. Foreign travelers need to act like all their traffic is being monitored.

These techniques also enable directly attacking US computers. A typical web browser actually fetches advertisements from all over the world. If a foreign intelligence service can determine a target’s IP address, the same attack technologies can replace an advertisement with malicious content whenever the target’s browser fetches an advertisement from the malicious country. We should be particularly concerned with France (sorry, “Country B”), which has both a privileged network position (sufficient to enable “attack-by-name” against UK targets) and a history of economic espionage.


But in many ways, I’m more scared by the small systems. Although I have yet to port my demonstration system to one of the many small off-the-shelf ARM/WiFi embedded systems, many of these devices appear to have sufficient computational resources to perform “bulk surveillance in miniature”, such as this Olimex combination with a 1 GHz Cortex A8 processor, 512MB of RAM, an SD-card, and a WiFi board for just 41.

The small size of such embedded systems suggest that they could easily be built into a plug-in air freshener or any one of a hundred different disguises, sit on an unencrypted wireless network, and perform a full surveillance and attack flow. Further complicating matters is the deniable nature (since its off-the-shelf hardware) and that a careful designer could use encryption to make the device wipe its code when unplugged.

So any foreign intelligence agency could install surveillance devices in every downtown DC Starbucks, use bulk surveillance to identify all the network visitors and, for any visitor who meets their criteria, directly inject exploits into their web browsing. DC hotels are similarly vulnerable to slightly larger installations such as my demonstration box. Witnessing an open communication closet across from my room at the Mayflower during a DHS PI meeting gave me a distinctly uncomfortable feeling.

We need to act like every open wireless network or hotel in the Washington area is potentially compromised.  And with the low cost of such installation, it doesn’t even need to remain the realm of foreign intelligence services.  How much money could criminals make with such systems?

At this point, it doesn’t matter if the NSA disappeared tomorrow.  The precedents are now well established. After all, if the US can target NATO allies with bulk surveillance and attack-by-name, who can’t do the same to us?  And I personally believe the US has more to lose than we have to gain. 

The only robust defense against Internet surveillance is universal encryption, as cleartext traffic represents not just an information leakage but an exploitation vector.  Because what is the opposite of NOBUS?  How about a homework assignment.