What Star Wars Can Teach Us About Analytics Infrastructure

Knowledge drives the Empire of Big Data, but achieving true understanding is a challenge of galactic proportions. To become one with business insights, data warriors must use the mysterious power of Business Intelligence (BI) to reach deep inside big data. Yet Data can be a dark and seductive master. To learn to trust its power, and take full advantage of the depths of its wisdom, the aspiring Insight Lord needs to be aware of the lessons of the (Data) Force.

OK, we’re not really taking ourselves that seriously.

Here is some Star Wars wisdom that is incredibly applicable to the world of Big Data and analytics infrastructure.

Lesson 1: “I Find Your Lack of Faith Disturbing.”

Executives still mistrust their own data. Organizations say they are increasingly focusing on data-driven decision-making to guide their businesses, but a majority of business leaders lack confidence in the insights generated from data and analytics. This lack of faith hinders progress and can have a real impact on the bottom line.

Lesson 2: “That… Is Why You Fail.”

The lack of faith in data results from the ongoing struggle with analytics in many organizations. There are three root causes of this struggle: initial data, application integration, and data maintenance. These range across system consolidation, batch feeds, manual integration, data processing, and cleansing.

Companies need change as they grow. Technically, it is unnecessary to build a dedicated data infrastructure — especially when the company is small and a limited amount of data is flowing in. Yet shortcuts like merging production and data servers often result in a data Armageddon.

Not separating data and analytics servers is a common mistake many companies make. To avoid the pitfalls of this and to enable data-driven growth, when you begin collecting data, set up a slave database with read-only accounts so analytic queries don’t interfere with production systems.

Lesson 3: “Sorry About the Mess.”

Everything in business has a price tag — and the lack of faith in data is no different. IBM estimates that the 2016 cost of poor quality data, in the U.S. alone, was $3.1 trillion. You could probably fix the hyperdrive in the Millennium Falcon for a lot less…

Lesson 4: “Everything Is Proceeding as I Have Foreseen.”

Data-driven decisions can offer companies competitive advantage, but don’t expect too much, too quickly. The Chief Data Scientist at Lockheed Martin, for example, believes we are still three to five years away from “advanced analytics being critical to the viability of a company...”

Lesson 5: “I Saw…a City in the Clouds.”

Cloud data warehousing is disrupting the data analytics space, yet a recent survey demonstrated that evil forces of chaos may still lurk in Cloud City.

60% of respondents claimed that their data warehouse is difficult to manage, and the same 60% reported that complexity was their primary challenge. Of these, 20% use a cloud-based solution and 41% were Redshift users.

Most respondents were technical professionals. Thus, it is clear that ease-of-use is still a weak spot for data warehouse providers and streamlining operations is still a challenge in the cloud.

Lesson 6: “I Have a Bad Feeling About This.”

Amazon Web Services (AWS) went down for nearly four hours earlier this year — knocking out service to some of the world’s most trafficked web services. By the time it was fully operational again, experts estimated that over $310 million was lost. The lesson learned? Outages are inevitable, even in the cloud. Prepare for them as best you can.

Lesson 7: “It’s a Trap.”

“Cloud services go down. Therefore, I must move everything on-prem.”

This type of thinking, while admirably self-sufficient, is fundamentally flawed. If Microsoft, Google, and Amazon’s clone armies can’t keep their own services running 100% of the time, do you have the IT resources to do so?

Lesson 8: “Would It Help If I Got Out and Pushed?!!”

It actually might!

To deal with nearly limitless volumes of data, organizations are more frequently turning to AI, Machine Learning, and NLP for analytics. NLP may offer the most disruptive potential since most analytics infrastructures nearly entirely geared towards managing structured information. NLP enables accurate analysis of more unstructured data and may carry significant implications for design, construction, and management of analytics infrastructure.

The Bottom Line

In the Star Wars universe, humans are served by technology. A move in this direction in the analytics world requires us to consider technology that lowers the burden of IT and data engineers who are overwhelmed by time-intensive tasks like schema building, data mining, complex modeling, and performance tuning.

To unleash the power of the (Data) Force, small and medium businesses should seek out and adopt an easy-to-use infrastructure analytics platform that can harness Big Data and get analytics quickly — enabling them to make faster and better business decisions.

Oh, and also rule the galaxy.