Ixivault, a managed app on Microsoft Azure
Your organization’s dark data presents challenges when you move to the cloud. Yet, leaving it in a current location is also not the solution.
Dark data includes digital data which is stored but never mobilized for analysis or to deliver information. If you have dark data, your organization is already missing opportunities to derive value from it. However, if you don’t take dark data with you to the cloud, it drifts even further from your other data assets. Meanwhile, the flexible computation and memory infrastructure of the cloud offers a very cost-effective solution to mobilizing that data. Most importantly, it does so at any scale your organization needs.
However, there are still challenges here. For example, overcoming the risks of governance and compliance, increased storage costs, and storage tiering choices. Do you choose to store data in close proximity to synchronize with other data – but at a higher storage cost?
Migrating Dark Data to the Azure Cloud
For most organizations, failure to create and execute a dark data plan as part of the cloud transition is undesirable at best and breaching data compliance at worst. Synerscope delivers the tools to analyze and “unlock” that data during the transition, making efficient use of cloud computing, while keeping data in your full control. This means no additional risks arise for compliance, security, etc.
Synerscope also helps you mobilize dark data, using a combination of machine learning, AI, and human expertise. Unlocking dark data is essential for most organizations. That remains true whether you’re shifting from legacy systems to Azure, are reducing your governance footprint, or are pressed into unlocking data for compliance or a regulatory audit. Synerscope’s Ixivault comes into play at any point where you need detailed and broad overviews of complex data. This is achieved through sorting, categorizing, and revealing patterns and giving domain experts the tools to label categories at speed, with high accuracy.
Your Data, Your Azure Tenant
Ixivault is a managed app on Microsoft Azure. When you deploy the tool, it installs on top of your Azure Blob or ADLS where the data stays in your control. We power Ixivault on Azure computing, meaning that it dynamically scales up computing power to meet the size and complexity of the data you direct to it for scanning and computation. At no point does the data leave your Azure tenant or any assigned secured storage used before separating sensitive data out. SynerScope’s design suits the most stringent demands for compliance and governance. Our Ixivault feels and operates like a SaaS but does so in your tenant, without any proprietary back-end for storing your data assets. Therefore, Synerscope allows you to categorize, sort, and label your dark data without introducing additional regulatory complexities. Your data stays in your cloud, the process is fully transparent, and you control and monitor your tenant for all matters related to data sovereignty.
That applies whether you’re importing data to Azure for the first time to inspect before deciding where to store it or already have data in a Blob or ADLS and must inspect it or want to open data on legacy infrastructure.
Sorting and Categorizing Dark Data
Ixivault leverages AI and machine learning for sorting and text extraction. Here, visual displays offer domain experts rich and discerning context from which to choose the most suitable labels of descriptive metadata. Our technology is a weak supervised system, first unsupervised computing handles the data in bulk, followed by a human operator to validate labels and bulk sorted data categories. The system works on raw data inputs directly, without training. Using raw data sets with human validation to add labels means we can make the system smarter over time. Future raw data sets are automatically checked for similarities with previously processed data sets. So, high value can be achieved from day one, but the system learns over time. .
Ixivault abstracts data to hypervectors – comparing the similarity between data algorithmically. Using algorithms, the AI can accurately sort data into “Stacks” of similar files. Format, lay-out and content of documents are all used by the algorithms to separate common business documents e.g., contracts, letters, offers, invoices, emails, brochures, claims, and different tables. And our algorithms separate sub-groups according to actual content within each of these. Our language extraction presents distinctive groups of words from each “Stack”, allowing humans to select the most appropriate labels. The same extracted words can also be matched to business glossaries and data catalogs already available to your organization. Hypervectors allow our algorithms to detect similarities across documents ‘holistically’, at a scale beyond unaided human capacity. The resulting merge of rich ontologies and semantic knowledge are re-usable throughout the organization and the many applications it runs.
Machine Learning with Human Context
Ixivault creates outputs that allow your data experts to step in at maximum velocity and scale. The application displays a dashboard showing the stack of data, visual imaging of what’s in this stack, and keywords or tags pulled from that data and metadata. Where descriptive metadata is lacking or absent, our system presents new candidates for labels. The system supports users in running fast and powerful data discovery cycles, which link search, sorting, natural language programming, and labeling. The output is knowledge about your organization’s dark data which can be used and reused by other users and software systems.
This approach allows data experts to look at files and keywords and very quickly add tags. More importantly, it creates room for human expertise, to recognize when data is outside of the norm – e.g., files are related to a special circumstance, which machines simply cannot reliably do. The result is a powerful, fast and flexible system, usable with a variety of data.
Once you select the machine proposed labels, you only have to individually inspect a small number of the actual files to confirm the labeling for an entire group of sorted files.
Unlocking Dark Data as You Move to the Cloud
Moving to Azure forces most organizations to do something with, or certainly think about, their dark data. You can’t move untold amounts of data to the cloud without knowing what’s in it. You would not be able to extract enough additional value from such a blind move. Directing data to the right storage solutions for easy governance, compliance, and management demands knowledge of its content. E.g., so you can prioritize data for further processing and computation, or save on storage for less value-added content. Data intelligence can mostly be paid for by decreasing ‘dark storage’. Meanwhile, your organization can improve its governance footprint and ensure compliance.
Synerscope can deliver the potential value in dark data by increasing knowledge, helping with retention, access management, discovery, data cleansing efforts, data privacy protection measures, and compliance. Most importantly, dark data mining gives organizations the information needed to make business as well as IT and compliance decisions with that data – because Data intersects between the three.
To learn more about Synerscope’s software and our approach, contact us to schedule a demo and see the software in action.