As a result, the agency can now track every instance of every individual accessing what is in some cases a single word or name in a file. This includes when it arrived, who can access it, who did access it, downloaded it, copied it, printed it, forwarded it, modified it, or deleted it. In addition, if the data has legal requirements, such as it must be purged at five years, it will automatically pop up and tell NSA IT staff that it is ready to be purged.
"All of this I can do in the cloud but -- in many cases -- it cannot be done in the legacy systems, many of which were created before such advanced data provenance technology existed." Had this ability all been available at the time, it is unlikely that U.S. solider Bradley Manning would have succeeded in obtaining classified documents in 2010.
Adapting to the changes
Anderson described the move to a cloud-based architecture as a major change at multiple levels. "Historically, a purpose-built database was needed to make use of individual data sets, forcing analysts to access many different databases and information repositories to do their job. Questions that spanned more than one data set had to be pulled together manually by the analyst. By putting all of the data into the cloud, analysts and analytic tools only need to interface with one system. Additionally, the granularity of control we get from tagging each piece of data makes it possible to bring data together that previously required separate databases to provide the necessary protections."
Early on in the process, a problem arose: "Our analyst community came to us and said, 'Here are the applications we use in our legacy relational databases. We just want to port them to the cloud'. Well, we found that simply porting programs and data from legacy systems straight to the cloud doesn't always work. And, even if those applications would function properly in the new environment, they would likely underutilize the potential benefits because the cloud works differently than relational databases."
As the agency's cloud matured to the point of being a useable asset, Anderson noticed that analysts and developers -- who work together to solve intelligence tasks -- had a tendency to stick with their legacy systems. To get staff invested in the new system, the IT group created "Future Architecture Transition Tuesday" or "FAT Tuesday."
Every other Tuesday, they would take as many as 150 analysts-developer teams and tell them: "Today, you cannot use your legacy tools or repositories; you have to work exclusively in the cloud". To support them as they struggled to accomplish their mission in the new environment, Floor Walkers, experienced analyst mentors, observed them as they worked and interceded as necessary.
Sign up for CIO Asia eNewsletters.