Is Poor Data Science the Cause of Digital Transformation Failure?
We look into why data science projects fail at such an alarming rate, and what data science teams can do to increase their success.
Join the DZone community and get the full member experience.Join For Free
Companies are forging ahead with digital transformation at an unprecedented rate. 49 percent of CIOs are reporting that their enterprises have already changed their business models or are in the process of doing so. These transformations, powered by data science and machine learning, include significant technology investments from enterprises. According to MarketsandMarkets™, the overall data science platform market is expected to grow from $19.58 billion in 2016 to $101.37 billion by 2021.
The outcomes from these investments, unfortunately, are stark: 85 percent of big data projects fail. I spoke to Dr. Ryohei Fujimaki, Ph.D., founder, and CEO of dotData, an NEC spinoff focused on data science automation for enterprise. dotData has developed a fully-automated data science platform which speeds time to value by democratizing, operationalizing, and accelerating the entire data science process, from source data to machine learning. It utilizes AI-powered Feature Engineering, a method of transforming the given data into a form which is easier to interpret, and automatically applies data transformation, cleansing, normalization, aggregation, and combination and processes hundreds of tables with complex relationships and billions of rows into a single feature table, automating the most manual data science projects. The platform thus eliminates the most time-consuming and labor- and skill-intensive aspects of the full data science process.
I wanted to find out why data science projects fail at such an alarming rate, and what data science teams can do to increase their success. Ryohei, provided a range of insights into the challenges and how his company is combatting them through data science:
Long Turnaround Time and Upfront Effort Without Visibility Into the Potential Value
One of the fundamental tenets of digital transformation is the use of data to identify and drive business use cases. However, if data is inadequate or not good enough, the results are poor. Ryohei believes that this, coupled with slow processes is a crucial point of failure as it is "too much upfront effort before knowing the outcome. Companies put in the effort but because the data is poor and/or insufficient, so the results are not good enough. This is a huge issue because of the long turn around."
With dotData's machine learning platform, Ryohei details: "What would normally take months of analysis, we can shorten the turn around to two or three days. We can tell if they will fail and see where they need to put more effort."
Misalignment of Technical and Business Processes and Aims
Ryohei shared, "Data science teams can deliver data learning models and machine learning in experimental environments; however, machine learning or data science projects have to be deployed and integrated into the business process."
He recounted a scenario of speaking with a Chief Data Officer at a US Telco where an experimental model failed to live up to the hype once applied in practice. Part of the problem is that data science teams don’t know the production environment, which may lead to a limited outcome or failure:
"Experimental and production models are different, and business requirements and IT requirements are all very different. What works in an experimental environment does not necessarily work in a production environment."
Part of this process is also about managing expectations. Ryohei explains, "Sometimes business people literally believe data science is a magic box, so our short turn around quickly establishes what can be done by data science and what cannot. We don’t want to say after 3 months of effort if their expectations were misaligned."
Lack of Data Practitioners
Data scientists are a scarce resource. Traditionally, digital transformation has required a skilled data engineer who understands data science. Ryohei asserts, "To succeed we traditionally need a very strong software team who understands machine learning — hence many initiatives fail due to lack of staff." dotData's acceleration of data science frees up skilled data scientists from manual work and allows them to focus on what to solve rather than how to solve it. This means teams can deliver ten times more projects and augment business insights driven by its AI-powered feature engineering. dotData’s solution also democratizes data science by enabling existing resources to perform data science tasks. With the dotData GUI, the data science task becomes a five-minute operation, requiring neither significant data science experience nor SQL/Python/R coding.
dotData, was recently awarded the winner of the Best Machine Learning Platform award from AI Breakthrough, a leading market intelligence organization that recognizes the top companies, technologies, and products in the global AI market. Their clients originate from a range of sectors including banking, finance, manufacturing, aviation, telcos, and analytics service consultants and include Fortune Global 250 clients.
Opinions expressed by DZone contributors are their own.