A dark day for data?

On average 52 percent^[2] of all data stored by organisations worldwide is ‘dark’ as those responsible for managing it don’t have any idea about its content or value. Much has been said about the financial cost of dark data but the environmental cost has, so far, often been overlooked.  Analysts predict that the amount of data that the world will be storing will grow from 33ZB in 2018 to 175ZB by 2025^[3]. This implies that, unless people change their habits, there will be 91ZB of dark data in five years’ time – over four times the volume we have today, with all the energy associated with powering the infrastructure in which it lives. 

Phil Brace, Chief Sustainability Officer and Executive Vice President, Appliances and Software-defined Storage, at Veritas Technologies explains:

“Around the world, individuals and companies are working to reduce their carbon footprints, but dark data doesn’t often feature on people’s action lists. However, dark data is producing more carbon dioxide than 80 different countries do individually, so it’s clear that this is an issue that everyone needs to start taking really seriously. Filtering dark data, and deleting the information that’s not needed, should become a moral imperative for businesses everywhere.

The IT industry has to get ahead of the challenge, since data volumes are getting bigger every year. We’re predicting huge growth in the amount of data being created by IoT devices, and this is what industry analysts suggest will comprise the lions’ share of the 175ZB of data we’re expecting by 2025. Businesses need to understand this type of data, and the storage policies around it, so that we don’t see emissions spiral.  But, we can all play a role in this individually too.  Nearly every one of us stores data that we’ll never access again, simply because cloud storage is so cheap and available to us – thousands of videos and photos that we’ll never look at, or emails that we’ll never read – and there are hundreds of millions of people doing this.  Businesses and consumers everywhere need to learn how to manage their data for the sake of the planet.”

Veritas has defined best practices that will enable organisations worldwide not only to delete data waste with confidence but also help them to reduce cost and strengthen their compliance:

• Identify all data stores and gain overview: Data Mapping and Data Discovery are the first steps in understanding how information flows through an organisation. Gaining visibility and insight into where data and sensitive information is being stored, who has access to it and how long it is being retained is a critical first step in pursuit of dark data and the key foundation to build from.
• Illuminate dark data:  a proactive Data Management approach allows organisations to gain visibility into their data, storage and backup infrastructure, so they can take control of data associated risks and make well educated decisions which data can be deleted with confidence.
• Automate the discovery and data insight routines: to keep pace with the data explosion, companies should automate the analytics, tracking, and reporting necessary to deliver organizational accountability for dark data, file use and security. Companies might need to handle petabytes of data and billions of files, so their Data Insight approach should integrate with archiving, backup and security solutions to prevent data loss and ensure policy-based data retention.
• Minimise and place controls around Data: Data minimisation and purpose limitation ensure organisations reduce the amount of data being stored and establish what is retained is directly related to the purpose in which it was collected. Classification, flexible retention and compliant policy engines allow confident deletion of non-relevant information providing a cornerstone of any dark data project and companywide compliance.
• Monitor to ensure continual adherence to compliance standards: compliance rules like GDPR introduce a duty on all organisations to report certain types of data breaches to the relevant supervisory authority, and in some cases to the individuals affected. Organisations must evaluate their ability to monitor breach activity and quickly trigger reporting procedures to ensure compliance.

^[1] Numbers calculated by mapping industry data on power consumption from data storage, industry data on emissions from data centres and Veritas Databerg Research

^[2] Veritas Databerg Research

^[3] The Digitization of the World, From Edge to Core, IDC, November 2018