This article is featured in the DZone Guide to Big Data, Business Intelligence, and Analytics – 2015 Edition. Get your free copy for more insightful articles, industry statistics, and more.
To more thoroughly understand the state of Big Data, and where it’s going, we interviewed 14 executives with diverse backgrounds and experience with Big Data technologies, projects, and clients.
Specifically, we spoke to:
Margaret Roth, Co-Founder and CMO, Yet Analytics
Dr. Greg Curtin, CEO and Founder, Civic Resource Group
Guy Kol, Founder and V.P. R&D, NRGene, Ness Ziona, Israel
Gena Rotstein, CEO and Founder, Dexterity Ventures, Inc.
Scott Sundvor, CTO, 6SensorLabs
Ray Kingman, CEO, Semcasting
Puneet Pandit, Founder and CEO, Glassbeam
Mikko Jarva, CTO Intelligent Data, Comptel Corporation
Vikram Gaitonde, Vice President Products, Solix Technologies
Dan Potter, CMO, Datawatch
Paul Kent, SVP Big Data, SAS
Matt Pfeil, CCO Co-Founder, DataStax
Philip Rathle, VP Products, Neo Technology, Inc.
Hari Sankar, VP of Product Management, Oracle
There is alignment with regards to what Big Data is, how it can be used, and its future. Discrepancy lies in the perception of the state of Big Data today. Some companies have been working with Big Data for years, others feel unable to perform “real” analytics work due to the data hygiene required, as well as the necessary integration of disparate databases.
Here’s what we learned from the conversations:
- The definition of Big Data is consistent across executives and industries—volume, velocity, and variety of data that is always changing and growing exponentially beyond what companies can traditionally handle. Scalability is critical, as are data management and retention policies. Data collection requires a more strategic approach. Companies will evolve from collecting/storing every piece of data to collecting and storing data based on need.
- Executives stay abreast of industry trends by meeting with clients and prospects, learning pain points, and determining which data is available to solve the problem. Just as there’s a tsunami of data, there’s also a tsunami of information and hype about Big Data. Stay above the noise by having a “big picture” perspective of the problem you are solving.
- Real world problems solved by Big Data are myriad. I spoke with companies sequencing the wheat genome; enabling smart cities; and evolving healthcare, automotive, retail, education, media, and beyond. Every initiative is using data to help clients move from being reactive to proactive. Accessing data, integrating multiple sources, and providing the analysis to solve problems requires patience, vision, and knowledge. You will gain all three by working in a real-world environment solving real problems. None will come from contemplating Big Data in the abstract. Once you appreciate the amount of time spent on data hygiene—an absolute requirement before any analysis can take place—you’ll structure data collection and integration so hygiene is less tedious and time-consuming.
- The composition of a data analytics team requires a number of skills: development of algorithms and software implementations, data science, design, engineering, and input from analysts with domain expertise. The most important qualities for team members are creativity, collaboration, and curiosity. No one person or skill-set is the solution to every Big Data project. Big Data provides an opportunity for developers to contribute beyond their typical scope of influence. It’s best for developers to have a broad range of interests and expertise. The more perspectives they can bring to bear on the problem, the better.
- According to Ginni Rometty, CEO of IBM, Big Data is the “next oil.” Several executives pointed out that this oil will be “unrefined” for the next 10 to 20 years. The future of Big Data is in providing real-time data to connect people, machines, experiences, and environments to improve life in a more personal way—from fewer traffic jams to more sustainable agriculture. Some executives I spoke with believe no one is really dealing with Big Data yet. There is so much data in repositories the challenge is to figure out how to aggregate data so it can be analyzed. We also need to determine the right questions to ask, and the right data to store, to transform business and the customer experience. Other executives are already doing these things for their clients; however, even these executives see unrealized possibilities. Demand for Big Data services is growing quickly as the business world sees the possibilities. Once you empower business people, they ask for more information. They ask smarter questions. Speed and agility gain importance. Real-time operational and business data allows people to make well-informed decisions quickly, thereby saving time and money. Effective use of Big Data is becoming an expectation.
- Hadoop was the most frequently mentioned software, with Cloudera and Hortonworks being the most frequently mentioned management applications. However many other solutions—including Cassandra, Clojure, Datomic, Hype, NoSQL, PostgreSQL, SQL Server, and Tableau for visualization—were discussed. There’s enormous demand for Big Data developers, so you don’t need to know all of the software and applications. Pick what you want to become an expert in and write your ticket with that software. Taking the time to learn Hadoop is a good place to start.
- Executives identified a broad range of obstacles for success with clients. The only obstacle for success mentioned by multiple executives was the lack of sufficiently knowledgeable and experienced people. Other concerns included: legacy software systems, fear of what’s in legacy data, lack of understanding of the value Big Data can provide, knowing the right questions to ask, knowing who owns the data in the cloud, vendors making unsubstantiated claims, and too much hype around Big Data. These obstacles are not unexpected given how early we are in the development and execution of Big Data projects. However, be aware of them as you get involved with specific projects. Ask the right questions up front, set the right expectations, and save a lot of time and rework.
- Concerns around Big Data are similar to IoT except privacy is more important than security. Industrial data is one thing, personal data is a whole other animal. As long as personal data is used for good, people will get comfortable as they benefit from Big Data. While Big Data will result in greater knowledge, it should also result in greater transparency—by governments, companies, and advertisers—and help prevent fraud and identity theft. “Data lakes” should be weighed against the danger to privacy posed by centralized data stores. Before we build huge data repositories, we need to know how we’re going to use and safeguard that data. As the data infrastructure matures, these problems will become increasingly easy to solve.
- The future of Big Data is the ability to make well-informed decisions quicker and easier than ever before. People will not be doing what a machine can do, so they’ll be free to use their minds to do creative things. The blue-sky vision is: Big Data will be the central technology to human existence, since it will affect all aspects of life (e.g., weather, transportation, healthcare, nutrition, energy, etc.).
Based on the vision above, following are three takeaways for developers:
- Big Data is evolutionary, not revolutionary. Big Data problems are similar to problems you’ve faced before. Leverage and improve upon the knowledge you already have by learning new architectures and languages.
- Be prepared to be part of the bigger picture. Become more well-rounded and more prepared to collaborate with, and contribute to, your team.
- Understand the real-world problems you are solving. (It may help to think in terms of Domain-Driven Design.) Think about creating the next destructive idea that’s lurking in the open source community. Share more, borrow more, be more open-minded about the possibilities of what you are working on.
The executives we spoke with are working on their own products or serving clients. We’re interested in hearing from developers, and other IT professionals, to see if these insights offer real value. Is it helpful to see what other companies are working on from a more industry-level perspective? We welcome your feedback at firstname.lastname@example.org.
For more insights on workload and resource management, real-time reporting, and data analytics, get your free copy of the Guide to Big Data, Business Intelligence, and Analytics – 2015 Edition!