Photo: Mark Bentkower, Commvault Systems.
Dark data, a relatively new term to the industry, is defined by Gartner "as the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetising)."
Almost all organisations have volumes of dark data stored away in dusty vaults and off-site storage facilities, historically unaccounted for, unmanaged, and undervalued. However, modern day information sharing with analytics is bringing the reign of dark data into the spotlight; mobility trends mean that users can create and share at will through a range of devices, including smart phones, laptops and tablets.
Many organisations are just beginning to realise that they lack both the policy and technology needed to efficiently manage data outside of the corporate data centre. Additionally, the growth of data—and big data especially—is causing enterprise to finally look to address the issue of dark data, if only to curb mounting storage costs. This is especially true in Singapore, where 10 percent of companies are expecting data growth of more than 10 percent in 2014, which is double the average for Asia Pacific on the whole.
Stepping into the light
Tackling dark data can be intimidating, even to the most accomplished of CIOs. Organisations have very little awareness of the location, volume, composition, ownership, risk, and business value of their unstructured data. Based on the complexity associated with managing dark data, Gartner recommends that "organisations should review the scope of their unstructured data problems by using file analysis (FA) tools to understand where dark unstructured data resides and who has access to it."
FA differs from traditional storage reporting tools because the technology doesn't just report on simple file attributes, but can also provide critical contextual information; with the ability to analyse, index, search, track, report on file megadata and even content.
FA tools applied to dark data provide business value in a number of ways, one of which is by helping organisations reduce risk. By identifying which files reside where, and who has access to them, FA tools introduce an element of control. They can also help organisations make more informed decisions around prioritising their unstructured data management needs for classification and information governance, providing insights in setting retention policies for data movement.
Many FA tools also offer reporting capabilities that help define these retention policies; according to Gartner, "the value of reports in FA tools is that they can be used to determine policy and strategy in areas such as access, retention and location."
IT administrators often struggle with having little to no insight into what data is being created; limited control over how it is being stored; and almost no understanding of its business value. When it comes to information lifecycle governance, more often than not, organisations choose to lean on cold storage tape vaults to keep every scrap of data due to a paralysing fear that they may throw away something of value.
Sign up for CIO Asia eNewsletters.