Data Engineers are in Greater Demand than Data Scientists

It seems like these days, everybody wants to be a data scientist.  Harvard declared it to be one of the hottest jobs of the decade back in 2012.  And since then, people have been clamoring to claim that role, whether the shoe fit or not.  So there is no question that the job is hot, hot, hot.

But if a data scientist creates a breakthrough algorithm, and there is no data engineer to put it into production for use by the business, does it have any value?  I will repeat my favorite statistic from Gartner that only 15% of big data projects ever make it into production. And while they never dig into the reasons why 85% of big data projects never make it there, I will propose that there are several key reasons why they fail:

1. They never find an insight worth putting into production
2. They find an insight and build a model but fail to build a production pipeline that can run within the service level agreement on a repeatable basis
3. They don’t need an insight, because the data analysis they want to run isn’t dependent on some complicated model, but still fail to build a production pipeline that can run within the service level agreement on a repeatable basis

That is why for every one data scientist companies need at least two data engineers and according to Jesse Anderson’s blog this week on oreilly.com, you may need as many as 5 data engineers per every 1 data scientist.  In a recent blog, also this week, posted by Dave Wells of The Eckerson Group, he references that  “Michelle Goetz of Forrester Research reported finding twelve times as many unfilled data engineering jobs as data science jobs.”  And while Michelle also has an article from this past November, titled, Data Engineers Will Be More Important Than Data Scientists , I would argue that ship has already sailed.

If I may quote Buno Pati, the Executive Chairman of Infoworks.io, “AI” without the data is just “A”.  You have to get the data to the data scientist first, and then once they have the data they need to do their data science magic, and have identified an insight useful to the business, that insight must be operationalized.  And as it turns out, operationalizing a machine learning algorithm at scale, and just managing data pipelines at scale is its own magic.

As I noted in my last blog, the reason for Cloudera’s stock drop was the difficulty in implementing Hadoop to production.  That issue is caused by the complexity of Hadoop and the lack of enough talented big data engineers who can properly build production workflows.  Coincidentally, or maybe not so coincidentally, this past week, there have been two blogs, referenced above, that all talk about the importance of the data engineer.  

And one thing is clear.  As of this writing, data engineers are in increasing demand.  Fortunately, the more tedious aspects of the data engineering role can be automated to let the data engineer focus more on the logic of the pipelines.   So while data engineers may be more important than data scientists, there is hope in the form of automation which can make today’s data engineers 10x more productive.

In the same way that Integrated Development Environments, IDEs, made software developers significantly more productive,  data engineering automation will do the same in the big data space. So while data engineering is hard, data engineers are rare and demand is high, it isn’t coincidental that this blog post happens to be located on a website that is all about automating big data engineering.  So if you are a big data engineer, and you want to get more efficient at doing your job, or you know a big data engineer, or someone who wants to become one, read the rest of this website to learn more about how you can help respond to the current demand for data engineering and get the data science algorithms into production.