Devs and Data, Part 1: Big Data on the Rise

This article is part of the Key Research Findings from the new DZone Guide to Big Data: Volume, Variety, and Velocity. 

Introduction

For this year’s big data survey, we received 459 responses with a 78% completion rating. Based on this response rate, we have calculated the margin of error for this survey to be 5%. Using the data from these responses, we've put together an article on how various sub-fields of big data are on the rise and how devs are becoming more data-driven. 

Big Data and Developers

Over the course of the past year, our respondents have reported becoming much more data-driven. When asked what types of big data they tend to work with, 78% reported working with large volumes of data, 51% with a large variety of data, and 42% with data at high velocity. While the year-over-year change in the percentage of respondents working with large volumes of data and high-velocity data fell within the margin of error for this report (a 4% increase and 2% decrease, respectively), these numbers, nonetheless, remain rather high. And those working with a large variety of data increased 7% year-over-year. Additionally, respondents’ experience in all areas of big data increased considerably from our 2018 big data survey. Here’s a quick breakdown of the percentages of respondents who had experience with a certain topic, comparing our 2018 survey data to the 2019 results:

• Data architecture

  • 2018: 26%

  • 2019: 37%

• Data engineering

  • 2018: 21%

  • 2019: 34%

• Data visualization

  • 2018: 21%

  • 2019: 31%

• Data science

  • 2018: 19%

  • 2019: 28%

• Data mining

  • 2018: 17%

  • 2019: 22%

In addition to these impressive increases, the percentage of respondents reporting to have experience in none of those big data sub-fields fell from 45% in 2018 to 32% in 2019.

Given this increased interest in and experience with big data practices, it comes as no surprise that the adoption rates for big data-focused languages and frameworks also saw an increase over last year’s survey. In 2018, 68% of respondents reported using Python for data science and machine learning; in this year’s survey, 79% of respondents reported using Python. Spark and TensorFlow adoption also increased by 6%, bringing them to a 47% and 46% use rate, respectively, among our survey-takers. And, while we didn’t see a dramatic increase over last year’s survey, 51% of respondents told us they use R for data science and machine learning projects.

Looking Forward

For the rest of this series, we’ll look at the various processes associated with each step of the big data pipeline (ingestion, management, and analysis), and see how they’ve changed over the past year.

This article is part of the Key Research Findings from the new DZone Guide to Big Data: Volume, Variety, and Velocity.