Is Your Organization Prepared to Manage Dark Data?
The Business Value of Mining Dark Data in Azure Infrastructure
As organizations accelerate the pace of digital transformations, most are moving to the cloud. In 2019, 91% of organizations had at least one cloud service. But, 98% of organizations still maintain on-premises servers, often on legacy infrastructure and systems. At the same time, moving to the cloud is a given for organizations wanting to take advantage of new tools, dashboards, and data management. The global pandemic has created a prime opportunity for many to make that shift. That also means shifting data from old infrastructure to new. For most, it means analyzing, processing, and dealing with massive quantities of “Dark data”.
Most importantly, that dark data is considerable. In 2019, Satya Nadella discussed Microsoft’s shift towards a new, future-friendly Microsoft Azure. In it, he explained that 90% of all data had been created in the last 2 years. Yet, more than 73% of total data had not yet been analyzed. This includes data collected from customers as well as that generated by knowledge workers with EUC (End-user computing, such as MSFT Office, email, and a host of other applications. As a result, the process of big data creation has only accelerated and (unfortunately) more dark data exists now than ever before.
As organizations make the shift to the cloud, move away from legacy infrastructure and towards microservices with Azure, now is the time to unpack dark data.
Satya Nadella discusses Microsoft’s shift towards a new, future-friendly Microsoft Azure
Dealing with (Dark) Data
The specter of dark data has haunted large organizations for more than a decade. The simple fact of having websites, self-service, online tooling, and digital logs means data accumulates. Whether that’s automatically collected from analytics and programs, stored by employees who then leave the company, or part of valuable business assets that are tucked away as they are replaced – dark data exists. Most companies have no real way of knowing what they have, whether it’s valuable, or even whether they’re legally allowed to delete it. Retaining dark data is primarily about compliance. Yet, storing data for compliance-only purposes means incurring expenses and risks without deriving any real value. And simply shifting dark data to cloud storage means incurring huge costs for the future organization – when dark data will have grown to even more unmanageable proportions.
Driving Value with Dark Data
Dark data is expensive, difficult to store, and difficult to migrate as you move from on-premises to cloud-hosted infrastructure. But it doesn’t have to be that way. If you know what data you have, you can set it into scope, delete data you no longer need, and properly manage what you do need. While you’ll never use dark data on a daily, weekly, or even monthly basis – it can drive considerable value, while preventing regulatory issues that might arise if you fail to unlock that data.
- Large-scale asset replacement can result in requiring decades-old data stored on legacy systems.
- GDPR and other regulations may require showing total data assets, which means unlocking dark data to pass compliance requirements
- Performing proper trend analysis means utilizing the full extent of past data alongside present data and future predictions.
Dark Data is a Business Problem
As your organization shifts to the cloud, it can be tempting to leave the “problem” of dark data to IT staff. Here, the choice will often be to discard or shift it to storage without analysis. But dark data is not an IT problem (although IT should have a stake in determining storage and risk management). Instead, dark data represents real business opportunities, risks, and regulatory compliance. It influences trend and performance analysis, it influences business operations, and it can represent significant value.
For example, when Stedin, a Dutch utility company serving more than 2 million homes, was obligated to install 2 million smart meters within 36 months, they turned to dark data. Their existing system, which utilized current asset records in an ERP was only enabling 85% accuracy on “first time right” quotes for engineer visits. The result was millions in avoidable resource costs and significant customer dissatisfaction. With Synerscope’s help, Stedin was able to analyze historical data from 11 different sources – creating a complete picture of resources and creating a situational dashboard covering more than 70% of target clients. The result was an increase to a 99.8% first time right quote – saving millions and helping Stedin to complete the project within deadline.
Synerscope delivers the tools and expertise to assess, archive, and tag archived data – transforming dark data from across siloed back-ends and applications into manageable and useable assets in the Azure cloud. This, in turn, gives business managers the tools to decide which data is relevant and valuable, which can be discarded, and which must be retained for compliance purposes.
If you’d like to know more, feel free to contact us to start a discussion around your dark data.
