How The FBI ‘Dropped The Ball’ on Nidal Hasan, by Sean Gallagher – Article and PDF’s

On November 5, 2009, an Army psychiatrist stationed at Fort Hood, Texas shot and killed 12 fellow soldiers and a civilian Defense Department employee while wounding 29 others. US Army Major Nidal Malik Hasan, the American-born son of two Palestinian immigrants, reportedly shouted “Allahu Akbar!”—“God is great!”—before launching his 10-minute shooting rampage at the Soldier Readiness Center. The shooting—the worst ever on an American military base—occurred as Hasan was facing imminent deployment to Afghanistan. A civilian police officer shot Hasan and placed him under arrest.

In the investigation that followed, the FBI and Defense Department investigators found that Hasan had been communicating with Anwar al-Aulaqi (sometimes spelled “al-Alwaki”), an American radical Islamic cleric living in Yemen. In the process of reviewing the evidence, investigators found that the FBI’s Joint Terrorism Task Forces in San Diego and Washington, DC had been aware of Hasan’s interactions with Aulaqi for over 11 months before the attack. Yet Hasan had never even been interviewed about his connection with the imam who would later be tied to “underwear bomber” Umar Farouk Abdulmutallab and to attempts to bomb US bound cargo planes with explosives packed in laser printer cartridges. (al-Aulaqi would later be killed by a US drone strike in Yemen.)

As federal officials looked into whether they had somehow missed leads that might have prevented the shooting, they found that the information technology at the heart of the FBI’s efforts to prevent terrorist attacks was fractured, overburdened, and running on aging and underpowered hardware.

Two weeks ago—coincidentally, just hours before another gunman would kill 12 and wound many more in Aurora, Colorado—an FBI independent commission led by former FBI director and federal judge William H. Webster filed its final report (see below) on the FBI’s performance leading up to the Fort Hood shooting. That report found no evidence the FBI’s data should would have set off alarms that Hasan was planning to kill fellow soldiers; he received no explicit instructions from al-Aulaqi and never mentioned his plans. But the report strongly implies that FBI IT systems and the bureau’s poor state of information sharing with other agencies played a role in the failure to take a harder look at Hasan.

Much has been made of government’s power to survey citizens using technologies such as packet capture and deep packet inspection. Even used in a limited fashion, these technologies can gather massive amounts of data on the online behaviors of individuals, and when taken together they can create an electronic profile of people’s lives. That potential—and concerns about its abuse—have driven privacy advocates to push for the repeal or alteration of laws such as the PATRIOT Act.

At the same time, US law enforcement and intelligence agencies have struggled over the past decade to take all of this information and put it to use. The poor search capabilities of the FBI’s software, inadequate user training, and the fragmented nature of the organization’s intelligence databases all meant there was no way for anyone involved in the investigation to have a complete picture of what was going on with Hasan.

While much has changed since November of 2009, the FBI’s intelligence analysis and sharing systems remain a work in progress at best—and there’s no telling what other potential threats may have gone unnoticed.

Packet captured

Hasan first drew the interest of the Joint Terrorism Task Force in the FBI’s San Diego field office back in December of 2008, while he was a captain assigned to Walter Reed Army Medical Center and he attempted to make contact with Aulaqi via a message form on Aulaqi’s personal website.

The San Diego JTFF—a team made up of FBI agents and analysts, along with officers from the Defense Criminal Investigative Service (DCIS) and Navy Criminal Investigative Service (NCIS)— had been investigating Aulaqi since the late 1990s, when he was an increasingly radical imam at a San Diego mosque. As part of that investigation, the FBI monitored his electronic communications under a secret warrant, intercepting traffic to his personal webpage, his e-mails sent to a Yahoo webmail account, and his instant messages.

While the tools to do this could be primitive at the time, they did work. Back in 1997, the FBI began intercepting e-mails and other network traffic with the custom tool “Carnivore” (later given the bland name “DCS-1000″ after copious criticism), a Windows-based packet sniffer that could capture specific types of communications as part of warranted surveillance. (In 2005, the FBI dropped its bespoke sniffer and switched to commercial deep packet inspection technology, which by then offered better features and performance.)

So when Hasan visited Aulaqi’s website in 2008 and used its “Contact the Sheik” page to send Aulaqi a message, he identified himself with his name and his own AOL e-mail address. The FBI’s surveillance software scooped it up and noted Hasan’s IP address, which resolved to Reston, Virginia:

Hasan’s message then entered an FBI database called the Data Warehousing Service (DWS), a database originally designed when the FBI was still using Carnivore. The FBI’s Special Technologies and Applications section had designed DWS in 2001, but it was misnamed. DWS wasn’t a data warehouse per se but instead was designed as a transactional database for storing intercepted communications captured in criminal investigations—not for doing analysis on large data sets.

DWS wasn’t the only system used for handling surveillance data, though. In 2002, the Bureau had launched the Electronic Surveillance (ELSUR) Data Management System (EDMS), a separate system for handling foreign intelligence surveillance. The goal of EDMS was to help language analysts translate and annotate electronic content ranging from audio (collected from wiretaps and telephone monitoring) to intercepted e-mail and seized electronic media.

At the time that Hasan’s message ended up in the hands of the San Diego JTTF, the two systems were in turmoil. In February 2009, the systems were merged under a single user interface called DWS-EDMS as part of an effort to improve and consolidate surveillance data access—combining both criminal investigations and intelligence activities.

But with the Global War on Terror in full swing, DWS and EDMS had hit the wall—neither was really intended to handle the volume of surveillance that began rolling in to support counterterrorism investigations.

A “crushing volume” of data

Both DWS and EDMS had been running beyond their intended capacity for years. The systems lacked enough disk storage, had no disaster recovery capabilities, offered inadequate data security to share data outside the FBI, and had a severe shortage of server horsepower. And the FBI knew it. In an August 2006 report justifying fiscal year 2008 budget requests to the Office of Management and Budget, FBI officials wrote:

While providing significant tactical value, EDMS cannot continue to support the FBI’s counterintelligence and counterterrorism mission objectives as it currently exists due to the increase in data collection volume and user base. Since Oct 2004, EDMS experienced a 300 percent increase in average users per month. Over the past three years, the volume of ELSUR collections has grown over 62 percent for audio wiretaps and over 3,034 percent for digital collections (e.g., e-mail, seized media). The current system is unable to scale and meet these growing demands. Due to the increased burden, the ability to share ELSUR data and collaborate efficiently with other authorized federal, state, local law enforcement, and federal intelligence agencies will no longer be feasible unless the proposed enhancements are implemented.

Back in June 2007, DWS and EDMS had 1,600 users within the FBI. The two systems handled over 70 million “products” (e-mails, chat sessions, audio, and attached files), and tracked 16,500 e-mail, instant messaging, and Web accounts between them. After combing the systems in 2008, things just got worse—the systems exceeded 3,000 users, 350 million tracked “products,” and 50,000 tracked accounts by mid-2009.

And the integration had been a bit “iffy.” A new graphic user interface was developed to make the joined systems simpler to access, but the older system (renamed “DWS-EDMS Classic”) remained in use in many field offices. Because of the underlying structure of the DWS database and the limited functionality of the user interface, finding e-mails of interest was a little like checking the world’s biggest e-mail inbox with Microsoft Outlook. Actually, it was a lot like that. The GUI, according to the Webster report, was based on Outlook.

When a user logged into DWS-EDMS, the “home page” of the application displayed system-wide announcements and a list of the user’s active cases. From there, the application’s main screen displayed headers for the “products” associated with a case in a column similar to an Outlook inbox display; selected documents opened in a panel to the right. A filtering tool allowed the user to filter content displayed in the “inbox.”

That inbox contained what the FBI San Diego field office analyst working on the Aulaqi case called a “crushing volume” of information. Between the first message sent by Hasan to Aulaqi (December 2008) and the last (June 2009), the agent and analyst assigned to the case reviewed 7,143 documents—between 65 and 70 on an average day, with as many as 132 documents on peak days. Getting through that volume of data consumed astounding amounts of time. The analyst spent about 40 percent of his total time reviewing documents for the Aulaqi investigation, while the agent assigned to the case spent about three hours each day reviewing documents.

Much of that time was spent simply trying to get the data out of the system. According to the Webster report, the search tools in DWS-EDMS “were not designed for and do not provide effective assistance for the review and management of massive collections of information, like the collection in the Aulaqi investigation.” Because of the way the underlying database was designed, the DWS-EDMS search capabilities were crippled at best.

DWS-EDMS could perform Boolean searches of document text, searches based on “participant” (the specific e-mail addresses being sought), a keyword search, and a limited full-text search capability. Depending on the search strategy used, results could vary widely. For example, the Webster commission found that a full-text search using Hasan’s AOL e-mail account only retrieved half of the messages in the system (while the “participant” search brought up all the e-mails from his account). Keyword searches didn’t include synonyms or variations, only returning documents with an exact match. And DWS-EDMS lacked any cross-investigation full-text search—if it had one, a search on Nidal Hasan’s name would have brought up an e-mail captured in an unrelated investigation that gave Hasan’s military e-mail address and tied him to the Walter Reed Army Medical Center.

It got worse. The DWS-EDMS “Classic” interface had no way to track specific e-mail account activity within cases. “A new message could be linked with an earlier message only through memory, notes, or by actively searching the system,” the Webster commission found.

And prior to the Fort Hood shootings, while the “new” DWS-EDMS system allowed for users to track “favorite” cases and specific surveillance products (and to copy content they found to the equivalent of an Outlook “shared folder”), the system had no way to automatically link e-mail addresses and other types of data together. So if a person of interest had two different e-mail addresses, users had to conduct separate searches.

This drove them to dubious tracking systems of their own. The analyst on the Aulaqi case tracked e-mail addresses of interest in a separate Excel spreadsheet; the agent in charge of the case relied on written notes—and his own memory.

Who needs disaster recovery?

While the upgrades to DWS-EDMS did give it a friendlier face, the back-end databases for the system were extended far beyond what the infrastructure could support, by any normal definition of an “enterprise system.” Most obviously, no disaster recovery system existed. That’s a problem shared by other FBI databases, such as the Office of General Counsel’s National Security Letter database—which, due to database crashes, became corrupted and in 2007 could not even give FBI Inspector General auditors an accurate count of exactly how many NSLs the bureau had sent. Even today, DWS-EDMS lacks any backup or high-availability capabilities, and it still runs on antiquated hardware. The Webster commission reported:

The lack of a modern hardware infrastructure has two major implications. First, the relatively aged server configuration for DWS-EDMS and its ever-increasing data storage demands, coupled with ever-increasing use, creates slowdowns that we witnessed repeatedly in our hands-on use of the system. An agent in the field with considerable DWS-EDMS experience reported that the slowdowns deterred searching the system. Second, DWS-EDMSs lacks a “live” or “failover” emergency backup.

The DWS-EDMS system had other significant problems that went beyond its engineering: nobody was ever actually trained on how to use it, and many people who could have benefitted from it didn’t have access. Despite the massive growth in its user base, the combined system remained available almost entirely to FBI agents and analysts. Only a few members of joint terrorism task forces from other investigative services—including the NCIS and DCIS—even knew it existed. And DWS-EDMS existed as an island separate from the other investigative tools used by the FBI and its Joint Terrorism Task Force teams.

So when Hasan’s message to Aulaqi appeared in the system in December 2008, there was only one way for the agent in charge of the case to share it with his non-FBI JTTF colleagues: he e-mailed it to them.

Communication breakdown

Because Hasan mentioned the military, the FBI agent on the Aulaqi case e-mailed the text of Hasan’s message to members of the San Diego JTTF from NCIS and DCIS.

“Here’s another e-mail sent to Aulaqi by a guy who appears to be interested in the military,” wrote the agent. “The header information suggests that his name is ‘Nidal Hasan,’ but that might not be true… Can we check to see if this guy is a military member? Also, I would like your input, from the military standpoint, on whether or not this should be disseminated further.”

An initial check didn’t find Hasan in the Defense Department’s personnel database. However, after another message from Hasan got picked up on New Year’s Day 2009, a DCIS analyst found Hasan in the Defense Employee Interactive Data System (DEIDS) and passed a printout of his information to the investigation team. The database identified Hasan as a “commissioned officer”—but because “commissioned” was abbreviated as “comm.”, agents were concerned that he was a communications officer and would have access to Information Intelligence Reports (IIRs).

As a result, the data on Hasan wasn’t shared with the Army. Instead, it was forwarded to the Washington, DC field office of the FBI and the lead was flagged to FBI headquarters:

This one is for WFO (Washington Field Office). The individual is likely an Army communications officer stationed at Walter Reed. I would recommend that this not be disseminated as an IIR, since he may have access to message traffic. If this needs to get to the military, WFO might have to do it internally.

The lead was sent through the FBI’s Automated Case Support Electronic Case File system as an “electronic communication”—the FBI’s digital equivalent of an official memo. It contained the text of Hasan’s two messages intercepted thus far, basic information about San Diego’s ongoing investigation of Aulaqi, and Hasan’s home address, phone number, and the misconstrued information about his military assignment.

“While e-mail contact with Aulaqi does not necessarily indicate participation in terrorist-related matters, Aulaqi’s reputation, background, and anti-US sentiments are well known,” the message concluded. “Although the content of these messages was not overtly nefarious, this type of contact with Aulaqi would be of concern if the writer is actually the individual identified above.”

But in being handed over to the Washington field office, the Hasan investigation lost its connection to the e-mail intelligence being gathered by the San Diego JTTF. The fragmented nature of the FBI’s information systems would keep Washington investigators from having a complete picture of Hasan’s continued communications with Aulaqi when they finally picked up the lead.

Since Hasan wasn’t seen as key to the ongoing Aulaqi investigation, the San Diego field office agent in charge of the case didn’t plan on following up with his colleagues in DC. The lead went untouched in DC for two months—possibly because the office was handling issues concerning the inauguration of President Obama. It would be May before the lead was assigned to a Washington-based DCIS agent to assess. By then, Hasan had been promoted by the Army from captain to major.

The silo-riddled world of the FBI “standard workstation”

In the course of daily work, FBI field office agents access up to 12 different IT systems from their Windows-based workstations. Each comes with a separate login, and each contains its own isolated slice of data. Some systems are served up via Web interfaces on the fbinet.fbi intranet, while others use client-server connections or even legacy disk-share based database connections.

There’s a good deal of overlap between these systems. The Investigative Data Warehouse (IDW) system is the only one that pulls together pieces from multiple sources, but that data only comes from selected data exports and is not linked to live data. The Data Loading and Analysis System (DaLAS) contains a set of data that overlaps with the DWS-EDMS system in a Web-searchable repository, in addition to data from intelligence agencies—but it’s not connected to DWS-EDMS.

In 2009, there was no way to perform federated searches across these various data stores. The Data Integration and Virtualization system project, deployed in October 2010, partially fixed the problem. It is a one-login “Google-like interface” intended to perform federated searches based on a user’s access levels across each of the FBI’s data repositories. DIVS indexes content across 50 FBI and external databases; however, to access the data within each repository, DIVS launches the application that the data is stored in. That means that users still have to log in to those databases with separate credentials and deal with the idiosyncrasies of each database’s native interface.

The Automated Case Support system itself is created from multiple legacy systems, including IBM 3270 terminal applications, the ADABAS database, and applications written in the Natural programing language. The FBI is currently migrating from ACS to the Web-based Sentinel system, the follow-up to the bureau’s Virtual Case File system project (mercifully killed in 2004 after four years of mismanagement and weak oversight). However, the legacy ACS system remains in place with the addition of PC-based components—including some that were based on DOS—while Sentinel remains in beta. Originally scheduled for delivery in 2009, Sentinel was at about 98 percent functionality as of May 2012.

As a result of the bureau’s straddling old and new systems and the lack of complete integration, much of the information sharing done between FBI field offices and between the FBI and other agencies remains a manual process. While the Guardian and eGuardian systems have provided some connections between the FBI and outside agencies (specifically on counter-terrorism efforts), the process of creating a Guardian “event” from information found within other systems remains a manual one. And because of concerns about security controls on shared data, information often doesn’t get shared at all because there’s not enough fine-grained control over its dissemination.

The Webster commission cited the cluttered FBI enterprise architecture as a key problem in its recommendations. “The historical evolution of the FBI databases as discrete platforms has impeded the FBI ability to access, search, organize, and manage electronically stored information,” the commission reported.

Missed messages

Six more messages were sent by Hasan to Aulaqi while the lead sent by San Diego agents to Washington went unattended. Because DWS-EDMS lacked any way of flagging messages within the system as being related to an existing lead, the Washington office would have had to actively search for the messages to find out about them. The analyst in San Diego continued to track correspondence to Aulaqi from Hasan, including details of a scholarship contest Hasan was trying to run at his mosque: “A $5,000.00 scholarship prize is being awarded for the best essay/piece entitled ‘Why is Anwar Al Awlaki a great activist and leader.’” In all, Hasan wrote 18 messages to Aulaqi prior to the Fort Hood shooting.

Based on the e-mails and information from Hasan’s electronic service record data—which indicated that Hasan had done research on radical Islamic beliefs as part of his work at Walter Reed—the Washington field office declined to follow up further on the lead. The central DOD copy of Hasan’s service record, however, lacked information about counseling he had received about poor work performance—information stored only in Hasan’s local record at Walter Reed.

After a back-and-forth in June vie e-mail and phone between agents in San Diego and DC over the case, no further action was taken regarding Hasan until after the shooting. On July 15, 2009, Hasan was transferred to Fort Hood’s Darnall Army Medical Center in preparation for deployment to Afghanistan. On July 31, he bought a handgun from a Guns Galore in Killeen, Texas.

Findings of fact

The Webster commission found that none of the data obtained from Hasan’s e-mail communications—nor any of the other communications by him on other e-mail addresses recovered by forensics investigation of his computer after his arrest—indicated he was preparing to act violently. Hasan did subscribe to get site updates from Aulaqi’s site via Google Feedburner—including a January 2009 PDF file entitled “44 Ways of Supporting Jihad” and a July 14 e-mail entitled “Fighting Government Armies in the Muslim World.”

If the information had been shared with Army officials early on, or if the Washington Joint Terrorism Task Force had interviewed Hasan and his superiors, both the military and the JTTF would have gained a much different picture of him—one likely to have led to the revocation of his Secret security clearance and a reconsideration of orders deploying him to Afghanistan.

Since the Fort Hood shooting, the FBI has modified DWS-EDMS so that it alerts users automatically to links between e-mails based on data within them. But this feature would have had limited benefit in the Hasan investigation. After all, seven of the 18 messages he sent were through Aulaqi’s website, not through e-mail.

Another new feature added in May 2010 allows users to flag content for review as part of an investigation, regardless of which case collection they’re located in. An upgrade to the software in September of 2011 vastly improved the full-text search capabilities of DWS-EDMS.

DWS-EDMS training has also become mandatory for its users. Joint Terrorism Task Force members now get hands-on training with all of the FBI’s databases, including DWS-EDMS, and anyone assigned to a task force team is required to complete the database training within six months of being given access (not all JTTF members get access to the databases).

Maximum stress

But while the software has been improved and users are now at least nominally trained on the systems, the changes have come without a major upgrade to the system’s hardware. The September 2011 upgrade to the system included a new generation of hardware, but at the beginning of the investigation by the Webster commission, ”some hardware components of DWS-EDMS were eight years old.”

The current hardware for DWS-EDMS “is operating under maximum stress,” the commission reported. “As a result, the responsiveness of the DWS-EDMS database to search queries is remarkably slow. Our test searches produced wait times for results that took twenty seconds and longer, and occasionally timed out (i.e. failed because of the time consumed by the search).”

The commission recommended a complete overhaul of the FBI’s software and hardware infrastructure, including expediting existing enterprise data management projects and expanding the DIVS system as a bridge to a fully aggregated search and retrieval database. The current dependency of DIVS on the individual systems in the FBI architecture “underscores the importance of the individual systems that house the FBI’s primary databases and the need to assure that those systems are robust, reliable, and sustainable,” the commission reported. “DIVS is only as good as the databases it indexes and searches. The addition of its cross-database search capability should not cause the FBI to lose focus on DWS-EDMS, whose functionality cannot be replicated or replaced by DIVS.”

And that means that the FBI needs to acquire not just additional hardware, but also build an integration and development version of the system to test software releases and to act as a failover in cases of data center disaster.

The sort of endemic problems pointed to by the Webster commission won’t surprise people familiar with how government IT has been run over the past decade. The IRS, for example, didn’t have a full internal testing environment for its e-File system until 2010. Under the Obama administration, the Office of Management and Budget and the Federal CIO’s office have been pushing agencies for IT changes to reduce such problems—but there’s a long way to go.

Given the relatively small size of the FBI and its budget—and the continuing pressures to do more with less in government IT—just how (or if) the FBI will address the Webster commission’s IT recommendations remains unclear. This is particularly true in the context of the FBI’s past IT record.

Such an upgrade would be expensive, and the FBI and its joint terrorism efforts have certainly rung up some successes in the last decade, even with limited tools. But the cost of not fixing the FBI’s database problems is, as Donald Rumsfeld might have put it, an “unknown unknown.”

FINAL REPORT of the WILLIAM H. WEBSTER COMMISSION on The Federal Bureau of Investigation, Counterterrorism Intelligence, and the Events at Fort Hood, Texas, on November 5, 2009:

FBI 1

FBI 2

FBI 3