Big Data Trends and Predictions for 2017

As we bring 2016 to a close, we’ve reflected on some of the trends we have seen gathering momentum throughout our customer engagements. Based on this insight, we’ve selected our top 5 big data trend predictions for 2017 to share with you.

1. Big Data in the Cloud Goes Mainstream

We saw a rise in cloud adoption in 2016 that took us by surprise. Initially, there was resistance to moving data to the cloud due to privacy and security concerns.  But better, cheaper, faster has pushed folks to test cloud-based big data environments - mostly through greenfield projects, such as building new applications. In 2017 we expect to see this continue, with Cloud-first and hybrid data architectures being built and matured. We even expect the eventual offloading of segments of data warehouses to the cloud in time.

2. Streaming and the Internet of Things Becomes a Reality

Similar to cloud, we saw companies begin to test and build infrastructures for streaming data and the Internet of Things (IoT) in 2016. We’ve seen technology driving a lot of the use cases for streaming and IoT.  In 2016 we saw more and more smart devices in the market from Fitbits to smart homes and smart devices. These devices drove the need for companies to stream data and technologies like Spark, Kafka and Flume made streaming into the data lake a reality.  As we move into 2017, we will continue to see these types of devices increase in the marketplace and need for companies to be able to stream data with low latency and at high volumes into their data lakes.

3. Maturing of the Big Data Market

In 2016 we saw the market begin to mature for big data. We started to see companies move into full big data production instead of big data being a side or test project. We’ve seen companies move from their traditional data architecture to big data environments. Companies are seeing the value in big data and it’s becoming an integral part of an organization's decision-making process. In 2017, as companies become more reliant on big data, we will see the need for data governance, lineage, as well as data lifecycle management growth.

4. The Fate of the CDO

As we saw the big data market begin to mature in 2016, we saw the rise of the Chief Data Officer with organizations. In a recent Gartner survey they discovered that 54% of their respondents already had a CDO office and 20% planned to implement one within the next year (Popkin, Logan, Logan, & Faria, "Second Gartner CDO Survey — The State of the Office of the CDO", 2016).  The CDO understands how important data is to their organization’s digital transformation and is focused on the governance and management issues surrounding data usage. Two years ago it was the year of the data scientist, in 2016 it was the data engineer and will 2017 be the year of the CDO?. It will be interesting to see if the trend towards digital transformation with an organization continues to increase the gravitas of the CDO or if the activities of the CDO will end up a part of the basic operations of an enterprise.

5. Geo-location and the Adoption of Smart Cities

Geo-location will provide rich insights into individuals which will help bring advances in smart cities. We expect the adoption of smart cities across the world to pick up steam driven by geo-location technology assisting in everything from police deployments to optimizing the use of Express Lanes for traffic control to helping find a parking space.  Geo-location information will also lead to more personalized marketing.

As I look beyond 2017, I wonder if we’ll begin to see the contemplation of industry-oriented big data clouds. Every industry has a wealth of ecosystem-specific data and a pay for play or open source industry-specific cloud-based environment would go far in reducing costs, driving consistency and perhaps progress human interests more effectively.