Dark Analytics: Unearthing Insights from Dark Data
Dark Analytics: Unearthing Insights from Dark Data
Until now, analytics were limited to structured data. The situation is changing with the recognition of dark data’s potential and technology advancements.
Join the DZone community and get the full member experience.Join For Free
Data is money in today’s technology-driven world. Dark data refers to the data collected and stored within an enterprise, but not used beyond its intended purpose. Though coined ‘Dark’, the data contains hidden insights that can bring potential business opportunities to light.
According to IDC, almost 90% of unstructured data is never analyzed. Since most of the dark data is scattered across the organization, in a highly unstructured form, it is extremely difficult to explore and analyze it. However, advancements in technology in recent years can help harness dark data and unleash its potential.
What Are Dark Data Analytics?
Analysis of dark data using tools is termed as data analytics. Dark data analysis also targets deep web (basically everything on the internet that is not indexed by search engines). Dark analytics focuses on raw text-based unstructured data like emails, text messages, audio, video, customer log files, account information, financial statement, etc. present within an organization.
Why Is It Important?
As of today, companies haven’t explored much of the digital universe or harnessed big data for analytics value. According to IDC, by 2020, nearly 37% of the digital universe will contain data that can be converted into valuable information if analyzed. IDC also posits that ‘By 2020, organizations that mine and analyze all relevant data to glean actionable insights are likely to achieve an additional $430 billion in productivity gains compared to their less analytically oriented peers’.
Mining and analyzing the already available untapped data could help companies extract potential information on pricing, customers, employees, competitors, markets, and operations. For example, consider an amusement park - Analyzing the video files from their security camera footage will help them glean valuable consumer insights. For instance, they can determine their customer demographics like how many customers visit during weekdays and weekends, at what times during the day, how many come by foot, car, public transport, etc.
Another example could be the retail industry. Companies in the retail sector can take market research to the next-level with dark data. Instead of analyzing customer feedback to identify their pain points, expectations & churn factors, using advanced pattern recognition, video, and sound analytics, they can gain a more nuanced understanding of their customers’ mood, intent, and expectation in near real-time.
In simple terms, dark analytics is essential to illuminate hidden opportunities within an organization’s unstructured data.
Dimensions That Dark Analytics Focus On
Following are the three dimensions that dark data comprises of and dark analytics focuses on:
Traditional unstructured data: Includes untapped data that is already present within an organization’s repository. The idle, unexplored, and unstructured data is mostly in the text-based form. Emails, documents, messages, etc, are a few examples of this untouched data.
Non-traditional unstructured data: Unstructured data such as image, audio, and video files fall under this dimension of dark analytics. These unstructured data cannot be processed or analyzed with traditional analytics techniques.
Data in the deep web: The last dimension is the deep web - the largest body of untapped information. The deep web comprises of an incredible amount of data curated by academics, government agencies, communities, third-party domains, etc. The sheer size and the distinct lack of structure are the key factors that make it difficult to search in the deep web.
Risks Associated With Dark Data Analytics
Data source and authenticity: Trusting the information from the dark analysis could be difficult if you can’t confirm your data’s accuracy, authenticity, completeness, and transparency. As you explore the dark data trend, you could also expose your company to financial, brand, regulatory, etc, risks.
Legal & regulatory risk: Dark data might contain sensitive information like credit card data or any other confidential data related to business relationships, activities, etc. More than a privacy issue, this may lead to legal and regulatory risks. This not only will affect a company’s reputation, but also will reduce consumers’ confidence and trust in a company.
Cybersecurity: Dark data is just a small portion of an incredibly larger deep web. Although deploying analytics technologies can help glean actionable insights, augmenting data and analysis efforts on non-traditional data derived from external sources can lead to critical questions that organizations can’t afford to ignore today; questions about data integrity, veracity, legality, etc. Hence, from a cyber-risk perspective, the risks are likely to be magnified. On the bright side, this could boost a company’s cyber and risk management efforts.
Dark data has been the talk of the town in recent times. Until now, analytics was limited mainly to structured data. However, the situation is changing now with the recognition of dark data’s potential along with the advancements in technology. In fact, businesses are gearing up to take advantage of dark data to drive innovation and enhance competitiveness in the upcoming data-driven decade.
Opinions expressed by DZone contributors are their own.