Tag Archive for: Ixivault

Moving to the Azure cloud: unpacking dark data

Moving to the Azure cloud?

Today, more and more businesses are moving to the cloud – to automate and take advantage of AI and scalable storage, and to reduce costs over existing legacy infrastructure. In fact, in 2021, an estimated 19.2% of large organizations made the move to the cloud. And Microsoft Azure is close to leading that shift – with a 60% market adoption.

Often organizations focus on selected applications during a cloud transition. However, existing data might actually present the bigger complexity.  A majority of organizations use less than 50% of the data they own. At the same time, there is no oversight of data that is owned. This unused, unclassified, and unlabeled data is otherwise known as “dark data”, because it remains in the shade until abundant time is allocated to sort, label, and classify it.

Moving to the Azure Cloud is Like Moving House

We believe there is merit to comparing moving to the Azure cloud and moving house. You decide where to move, you choose your new infrastructure, and you get everything ready to move in. Then, you pack up your old belongings and move it with you. The problem is you likely already have plenty of boxes lying around. Think about your attic, your basement, and storage. Things from earlier relocations. You might have lost all knowledge of what’s in there. The same holds true when your organization’s applications and data must move house. But this time you also have to deal with ‘boxes’ of data left unlabeled by people leaving the organization, data left unused for a longer time, and data left behind from already obsolete applications. Moving this and other less well-known data may create bigger issues in the future.

  • Data is accumulating faster than it ever did before. You’ll have more of it tomorrow. Therefore now is the best time to go through data and categorize it
  • Proper governance of data is impossible without knowing its contents first. Older data collected from before GDPR regulations is still there. Compliance and Risk officers and CISOs dread this unknown data and fear it may fall out of compliance regulations.
  • It can be difficult to pass regulatory compliance audits with dark data ar If you can’t open a ‘box’ of data to show auditors what’s inside, you can’t prove you’re compliant.
  • You’re also not allowed to simply delete data. Industries and governments must comply with laws and regulations on archiving and maintaining open data.
  • When you know what data you have you can strategize and move towards controlled decisions on cold/warm/hot storage to optimize both costs and access. Moving data that is still dark may bring about irreversible data loss or at least expensive repairs in the future
  • Locating and accessing data requires the kind of information best-captured in classifications and labels, historical data analysis needs this metadata.
  • The parts of data that make up dark data leaves organizations vulnerable as it makes designing and taking security precautions extra hard.
  • Sometimes you can or must delete information. However, you can only do so if you know its contents beforehand and can determine regulatory compliance and have the foresight for future valuable analytics.

How can you optimize accessing this data? When one of our clients, the Drents Overijsselse Delta Waterschappen, looked at archiving and storing its past project documentation in the cloud, it found the necessary manual labeling a daunting task. The massive time-investment needed is very similar for other organizations making a cloud transition. Manually reviewing data is simply too labor-intensive for most organizations to undertake within a feasible timeframe.

Unpacking Data with Synerscope’s Ixivault

With Synerscope, you can achieve the data clarity you need. As a weakly supervised AI system, our solutions are built to perform where standard AI approaches would fail. Synerscope’s Ixivault implements onto your Azure Tenant – with no backend of its own. This means that all data stays inside your tenant, which is a big plus for all matters and concerns regarding security, governance, and compliance. Our friction-less implementation then allows you to open up, categorize, and label dark data using a combination of machine learning with manual review to speed up the full process by an average of 70%.

Ixivault analyzes your full data pool of structured and unstructured data, creating categories based on data similarities, pulling keywords and distinctive terms, and generating images of those data stacks – which your domain expert can then sit down to quickly label. Most importantly, Ixivault has built-in learning capabilities, meaning that it gets better at categorizing and labeling your specific data as you use it.

All this makes Ixivault the perfect tool to help you move – by unpacking boxes of data as you move them to the cloud. You can then choose appropriate storage, governance and access controls, even if you need or don’t need to keep the data. For the first time you can have a near edge-to-edge overview of all your data with zoom in options to very granular levels so you can make the best choice what to do next with this newly discovered data. Having new information about your data can make you money and save you money all at the same time.

If you need help with unboxing your dark data as you move, contact us for more information about how Synerscope can help. You may also purchase the Ixivault app directly at Microsoft’s Azure Marketplace.

Ixivault Helps Labeling and Categorizing Dark Data in the Azure Cloud

Ixivault, a managed app on Microsoft Azure

Your organization’s dark data presents challenges when you move to the cloud. Yet, leaving it in a current location is also not the solution.

Dark data includes digital data which is stored but never mobilized for analysis or to deliver information. If you have dark data, your organization is already missing opportunities to derive value from it. However, if you don’t take dark data with you to the cloud, it drifts even further from your other data assets. Meanwhile, the flexible computation and memory infrastructure of the cloud offers a very cost-effective solution to mobilizing that data. Most importantly, it does so at any scale your organization needs.

However, there are still challenges here. For example, overcoming the risks of governance and compliance, increased storage costs, and storage tiering choices. Do you choose to store data in close proximity to synchronize with other data – but at a higher storage cost?

Migrating Dark Data to the Azure Cloud

For most organizations, failure to create and execute a dark data plan as part of the cloud transition is undesirable at best and breaching data compliance at worst. Synerscope delivers the tools to analyze and “unlock” that data during the transition, making efficient use of cloud computing, while keeping data in your full control. This means no additional risks arise for compliance, security, etc.

Synerscope also helps you mobilize dark data, using a combination of machine learning, AI, and human expertise. Unlocking dark data is essential for most organizations. That remains true whether you’re shifting from legacy systems to Azure, are reducing your governance footprint, or are pressed into unlocking data for compliance or a regulatory audit. Synerscope’s Ixivault comes into play at any point where you need detailed and broad overviews of complex data. This is achieved through sorting, categorizing, and revealing patterns and giving domain experts the tools to label categories at speed, with high accuracy.

Your Data, Your Azure Tenant

Ixivault is a managed app on Microsoft Azure. When you deploy the tool, it installs on top of your Azure Blob or ADLS where the data stays in your control. We power Ixivault on Azure computing, meaning that it dynamically scales up computing power to meet the size and complexity of the data you direct to it for scanning and computation. At no point does the data leave your Azure tenant or any assigned secured storage used before separating sensitive data out. SynerScope’s design suits the most stringent demands for compliance and governance. Our Ixivault feels and operates like a SaaS but does so in your tenant, without any proprietary back-end for storing your data assets. Therefore, Synerscope allows you to categorize, sort, and label your dark data without introducing additional regulatory complexities. Your data stays in your cloud, the process is fully transparent, and you control and monitor your tenant for all matters related to data sovereignty.

That applies whether you’re importing data to Azure for the first time to inspect before deciding where to store it or already have data in a Blob or ADLS and must inspect it or want to open data on legacy infrastructure.

Sorting and Categorizing Dark Data

Ixivault leverages AI and machine learning for sorting and text extraction. Here, visual displays offer domain experts rich and discerning context from which to choose the most suitable labels of descriptive metadata. Our technology is a weak supervised system, first unsupervised computing handles the data in bulk, followed by a human operator to validate labels and bulk sorted data categories. The system works on raw data inputs directly, without training. Using raw data sets with human validation to add labels means we can make the system smarter over time. Future raw data sets are automatically checked for similarities with previously processed data sets. So, high value can be achieved from day one, but the system learns over time. .

Ixivault abstracts data to hypervectors – comparing the similarity between data algorithmically. Using algorithms, the AI can accurately sort data into “Stacks” of similar files. Format, lay-out and content of documents are all used by the algorithms to separate common business documents e.g., contracts, letters, offers, invoices, emails, brochures, claims, and different tables. And our algorithms separate sub-groups according to actual content within each of these. Our language extraction presents distinctive groups of words from each “Stack”, allowing humans to select the most appropriate labels. The same extracted words can also be matched to business glossaries and data catalogs already available to your organization. Hypervectors allow our algorithms to detect similarities across documents ‘holistically’, at a scale beyond unaided human capacity. The resulting merge of rich ontologies and semantic knowledge are re-usable throughout the organization and the many applications it runs.

Machine Learning with Human Context

Ixivault creates outputs that allow your data experts to step in at maximum velocity and scale. The application displays a dashboard showing the stack of data, visual imaging of what’s in this stack, and keywords or tags pulled from that data and metadata. Where descriptive metadata is lacking or absent, our system presents new candidates for labels. The system supports users in running fast and powerful data discovery cycles, which link search, sorting, natural language programming, and labeling. The output is knowledge about your organization’s dark data which can be used and reused by other users and software systems.

This approach allows data experts to look at files and keywords and very quickly add tags. More importantly, it creates room for human expertise, to recognize when data is outside of the norm – e.g., files are related to a special circumstance, which machines simply cannot reliably do. The result is a powerful, fast and flexible system, usable with a variety of data.

Once you select the machine proposed labels, you only have to individually inspect a small number of the actual files to confirm the labeling for an entire group of sorted files.

Unlocking Dark Data as You Move to the Cloud

Moving to Azure forces most organizations to do something with, or certainly think about, their dark data. You can’t move untold amounts of data to the cloud without knowing what’s in it. You would not be able to extract enough additional value from such a blind move. Directing data to the right storage solutions for easy governance, compliance, and management demands knowledge of its content. E.g., so you can prioritize data for further processing and computation, or save on storage for less value-added content. Data intelligence can mostly be paid for by decreasing ‘dark storage’. Meanwhile, your organization can improve its governance footprint and ensure compliance.

Synerscope can deliver the potential value in dark data by increasing knowledge, helping with retention, access management, discovery, data cleansing efforts, data privacy protection measures, and compliance. Most importantly, dark data mining gives organizations the information needed to make business as well as IT and compliance decisions with that data – because Data intersects between the three.

To learn more about Synerscope’s software and our approach, contact us to schedule a demo and see the software in action.

Ixivault™ – Complete View & Control of Your Data in the Cloud

Ixivault™ – software to get on top of all your enterprise data

The cloud offers compute, memory and storage horsepower and efficient cost to get every bit of information extracted from data. But it needs new software instead of a lift&shift of traditional analytical software to really obtain the benefits. And then there is the question what data to transfer to the cloud? Data silos, structured and unstructured data, Dark Data all stand in the way of an easy transfer. GDPR exacerbates this as it introduces three main challenges that Compliance, Risk and Data Protection functions must deal with, and on which they base their advice to the LOBs and Executive leadership:

  • Does your cloud computing connect with the sensitivity of the data you entrusted or want to entrust to the cloud? If a cloud computing solution is chosen where data processing and/or storing are shared between enterprise customers, the risk of data leakage is present.
  • The question which law applies: again the choice of software or software platform determines if sovereignty principles about the physical location at which certain sensitive data is held can be met.
  • The externalization of privacy requires that no cracks exist between the contracts an enterprise makes with software or platform vendors, the services companies, and the cloud vendor.

These are real hurdles in taking maximum advantage of the speed, scalability, flexibility and cost efficiency of the cloud. Much of this potential is lost when an enterprise at best feels safe transferring only parts of their data. The holes in this ‘Emmental’ cheese of data gets even bigger when we realize that most enterprises have near 70% of Dark Data. (Satya Nadella at Microsoft Inspire 2019)

SynerScope Ixivault™

We propose to scan all content of the enterprise data and using the cloud to perform that in a safe way. For this purpose, SynerScope introduces its Ixivault™ on Microsoft Azure. The setup is entirely within the enterprise’s own Azure tenant through Azure marketplace. The loading to the cloud and bulk scanning happens there (called a vault). The unknown dark data is transformed into known data and silo-ed data is linked so that a grounded decision on its release for further use can be made. Data that cannot be released for wider use in the enterprise cloud tenant is deleted from the vault. Data that’s safe is published for BI, data science and domain expert’s use by different functional departments in the company. All the original data sets stay safely in the company’s data center.

SynerScope turns Dark Data into Bright Data, ready to be used by human combined with machine intelligence for extracting information and value. Our three main solutions Ixivault™, Ixiwa™ and Iximeer™ are designed to handle any type of data in any combination, fast and flexible. Unstructured text, image and IoT data can easily be linked with structured data from ERP, CRM and other operational systems. To facilitate widespread secure and safe working with the data we support the (self) publishing of data to analytic processes (human-machine) under full linkage to the Data Protection Impact Assessment (DPIA) requirements. Granular content-based access control, integration of masking and hashing functions ensure that no eyes will see, and no process will use any un-eligible data (NoXi principle).

To further service the compliance, risk and data protection functions of the company, our systems will log every touch point of the data. Humans and machines can be audited for having worked inside the boundaries of Standard Operating Procedures (SOPs) set with each Project-DPIA. Providing evidence for an always-appropriate use of data is efficiently supported in SynerScope.

Azure

We have built our solution specifically to operate in your own Azure cloud-tenant environment. As an enterprise you directly agree with Microsoft Azure on the SLA’s with regards to data security. Adding SynerScope doesn’t affect these SLA’s and publishing of data inside your enterprise is linked to your own Azure AD set up. SynerScope prefers an Azure set up where all data lives in the ADLS/Blob store (and in its original application source system). We believe data should remain open for use in different applications.

SynerScope is proud to be a partner of Microsoft. We continuously look to exploit the expanding functions of Azure modules. Our serverless architecture provides the flexibility to deploy in any enterprise’s Azure tenant; we gladly welcome you to discuss opportunities to take advantage of your own tenant architecture.

 

IxivaultTM Azure

SynerScope Solution

Flexibility and speed of the SynerScope solution is secured by our patented data scanning, matching and visualization technology.
All this we have developed in aid of your move towards a data driven architecture in the cloud, including full compliance and with a view to provide you with significant cost reduction.
The solution is tried and tested to full satisfaction by our clients in the financial & insurance industry & the critical infrastructure industry.

“SynerScope’s platform might best be described as a data lake management product that covers everything from automated ingestion, through discovery and cataloguing to data preparation.” (BloorInDetail – 2018)

SynerScope will bring you:

Cost Reduction

  • Data warehouse optimization / application rationalization
  • Optimized use of data, cloud data storage and your cloud data warehouse architecture
  • SynerScope as migration and data publishing tool
  • Broad use of data in the LOB by domain experts, citizen data scientists or data scientists proper.

Efficiency improvement IT-operations

  • Incident/ problem root cause analysis reducing time to repair
  • Reduction of incident and problems
  • Usage of historical data and archives

The SynerScope Solution:

  • Extracting and accelerating insight from data with patented products
  • Open technology on Microsoft Azure
  • Deployment via Microsoft marketplace
  • Stand alone deployment also possible
  • Is complementary to third party data cataloguing tools, and adds considerably to the ease of use of unstructured data
  • Security and compliance proof (traceable & transparent)
  • Fast deployment

SynerScope References:

  • Financial & Insurance industry
  • Critical Infrastructure
  • Government (safety regions and smart city)

Tag Archive for: Ixivault

You Can Leave No Data Behind

Accelerating discovery and data identification is a big problem for businesses! Without descriptive metadata most IT systems are helpless. AI, ML and DL cannot hope to deliver without the right data.

Find out more