The Rabbit and the Mole: the Web Developer and the Data Scientist
Six years ago I led Zarathustra as the company and product manager. We started this new company to help SMB companies to manage their sales, operations and accounting in one system. I was young and handsome but with not a lot of experience in regards to recruiting and employing web software developers.
It was obvious to us that we didn’t have enough capital to hire experienced developers and while searching for guys to help us with the R&D we realized that we could find great talented young geeks. These guys accumulated their experience by establishment of websites for their own purposes and habits such as movies, sports and dating sites. This always reminds me of the Facebook story which started with a super fast rabbit, Mr. Zuckerberg. At the start he actually created ”thefacebook” service using his own hands and keyboard spontaneously understanding the need to create a simple UI/UX, support operation scalability and demonstrate great value to the end user.
Anyway, our geeks had all started their own sites from scratch and they had all established of one-man-show operation, some of them even made a cool small business out of that. When I say “all”, I mean just that. The team consisted of about 7-8 guys and all of them owned (probably still do own) services or websites that they maintained while helping us building the company’s SaaS offering. They were self-taught and were experienced with the cheap open source programming, server deployment and maintenance, website publishing and promotion, and more. Some of them actually held an academic degree and some were fresh young students. As the product manager, I took the responsibility for support and from time to time I forwarded customers’ calls to our “simple” developers. As you probably understand, there is no doubt that these “simple” developers were and actually are small entrepreneurs and great developers.
Open source today receives wide support from the developers’ community and I find traditional ISVs plan adoption of these technologies. Web-services are not second in the list but do have a major impact on the development strategy. In fact, some will relate to that as a revolution - I had read this nice Forbes article on how to explain the API Revolution to your CEO. Developers are an integral part of non-development meetings and are expected to understand the big business picture and be able to produce without needing a comprehensive definition. Read more about that on this post blog - Developers are from Mars
At first, and due to lack of experience the stories I had read about the great giant’s software development methodologies made me think that our SaaS development team was still “green” (amateurish). As time passed, I slowly but surely understood that things change and that eventually every ISV would want to adopt the spontaneous way that we used to develop our service. Agile is a leading method for R&D with the product manager serving sales, support and evangelist roles. Yet I think that the key change is the one undergone by the developer. The increasing demand for software manufacturing efficiency pushes the individual developer to become versatile, agile and fast. Returning to my geek development team, I can tell you that today these guys possess amazing capabilities and hence great job positions. Each of them can establish a new service on-demand, or a POC for that matter on their own, in just a matter of weeks.
But I have yet to mention the Cloud in this article (at least for the sake of Google indexing). To become a super fast rabbit, the developer must be able to leverage the cloud facilities. The super web developer knows to leverage the IaaS API. He/she must be familiar with some of the PaaS options in the market and understand how they can be used to maintain quick development and scalable service operation. The web developer needs to be agile and flexible, and must understand and support SaaS advantages, specifically the concept of “you don’t need to develop what already exist”. Whether it is a BI engine, CMS, CRM or any other already existing services. I find that the lack of super web developers in the industry becomes more intense due to the need of the ISV to demonstrate new development skills for its new cloud service deployments.
Another player in the this new “cloud development team” is the data scientist - the mole. In our geek team I served as the mole. We integrated simple sql and BI tools to drill down and analyze the data, but this is not enough in today’s Big Data environment. The primary job of a data scientist is to order the unstructured data and utilize it to support business decisions. The Data Scientist must have experience in data mining, analysis, visualization and forecast modeling.
“Netflix does something similar in their online movie business. When you sign up, they strongly encourage you to add to the queue of movies you intend to watch. Their data team has discovered that once you add more than than a certain number of movies, the probability you will be a long-term customer is significantly higher. With this data, Netflix can construct, test, and monitor product flows to maximize the number of new users who exceed the magic number and become long-term customers. They’ve built a highly optimized registration/trial service that leverages this information to engage the user quickly and efficiently.” I invite you to read this great (and long) article about “Building data science teams”
The data scientist does work in the back-end. He/she knows to relate the data to the business, plans the structure and implement the algorithms that will generate business decisions. Forester blogger Mr. James Kobielus refers to the change from traditional finite data as the Big Data Revolution:
“The big data revolution is making that constraint a thing of the past. Think of this new paradigm as “whole-population analytics,” rather than simply the ability to pivot, drill, and crunch into larger data sets. Over time, as the world evolves toward massively parallel approaches such as Hadoop, we will be able to do true 360-degree analysis. For example, as more of the world’s population takes to social networking and conducts more of its lives in public online forums, we will all have comprehensive, current, and detailed market intelligence on every demographic available as if it were a public resource.” Read more
A recent article on GigaOM written by the cloud blogger Mr. Derick Harris, talks about the big data scientist as a mathematician. Harris takes that one step further and brings as an example the Google engineering team blog post describing automated bug prediction tools:
“Companies such as Google and Facebook are doing alright in solving some of their problems, but what they’re doing now is just the tip of the iceberg. As Lewis indicated, there’s real value in evolving their current efforts further, to the point where machine learning and other techniques will let computers do everything from review code to, perhaps, predict problems with overall system health. And as the next generation of web companies start scaling up, they’ll start running into their own unique systems issues that they’ll have to solve. “
Analyzing the data and bringing great insights offer a significant competitive advantage for the enterprise. The SaaS vendor that presents valuable out-of-the-box analytics to its consumers will play an important leadership role in its market (just imagine the next comprehensive “Salesforce” that will demonstrate the next business intelligent Siri).
The new cloud company will continue to struggle with bringing on super fast rabbits and smart moles. The trends for recruitment are aligned with the growth of the cloud market. The newly needed skills generate new opportunities for both the individuals and new start-up companies. On one hand the Cloud - the new IT world brings simplicity and a rapid pace of development but on the other hand it also brings the complexity of Big Data of new service deployments.