Developers and Big Data: Thoughts to Consider
Developers and Big Data: Thoughts to Consider
Be agile, know the business problem you're trying to solve, and who you are solving it for.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
To get a better understanding of the state of Big Data today, we interviewed 14 executives with diverse backgrounds and experience with Big Data technologies, projects, and clients.
Specifically, we spoke to:
Dr. Greg Curtin, CEO and Founder, Civic Resource Group • Mikko Jarva, CTO Intelligent Data, Comptel Corporation • Matt Pfeil, CCO Co-Founder, DataStax • Dan Potter, CMO, Datawatch • Gena Rotstein, CEO and Founder, Dexterity Ventures, Inc. • Puneet Pandit, Founder and CEO, Glassbeam • Philip Rathle, VP Products, Neo Technology, Inc. • Guy Kol, Founder and V.P. R&D, NRGene • Hari Sankar, VP of Product Management, Oracle • Paul Kent, SVP Big Data, SAS • Ray Kingman, CEO, Semcasting • Scott Sundvor, CTO, 6SensorLabs • Vikram Gaitonde, Vice President Products, Solix Technologies • Margaret Roth, Co-Founder and CMO, Yet Analytics
We asked our respondents what recommendations they had for developers working on Big Data projects. Here's what they told us:
Be prepared and be a part of the bigger picture. Get outside the box. Learn Clojure. People need to learn something new to build new architectures and languages. Whatever creative thing you work on outside of work (blogging, writing, making beer) do more of that creative thing to teach your brain to approach problems in a new way. Rewire your brain to think in a new way.
Big Data can make such an impact on the public sector to improve lives and improve the sustainability of communities. What should we be telling and educating public sector entities about that will build a good foundation to improve people’s lives? Keep a professional mindset. Do no harm. We are shepherds of a public asset, put it to good use.
It’s best for everyone if developers see themselves as a part of a global effort to build layers of analysis and IT that can serve people that are not developers—APIs, IT, vocabulary. Empower people to get more from their data. Enable people to do data analysis from their phone. Genomics has the potential to affect humanity as much as the internet did. It's best for humanity if people think about the value genomics can provide.
Look beyond the immediate industry you’re working in for information and applications. Be industry agnostic. Think beyond the immediate business, to all the other applications, for what you are building. Democratize the system like GitHub. Allow people to own their online presence. Empower people to leverage the power of their actions.
Know who you’re developing for. The head of marketing needs data in a different format than the engineers. Know who will be using the data. Understand how it will be used. Work collaboratively. Don’t use a cookie cutter approach. All data is different. Figure out what will impact your business specifically.
Developers are used to having a single, stable data stream. They need to become more agile. Keep their eyes open. The world is not one dimensional. Embrace the bigger picture. Build with the anticipation that everything is evolving tomorrow.
How can you create the next destructive idea that’s lurking in the open source community? Where’s the next Hadoop? Become more friendly to open source. Share more, borrow more, be more open-minded about the possibilities of what you are working on.
Developers need to understand the real-world problems for which they are developing solutions. Develop disconnected applications to show how they work and then show how it integrates to solve problems. Keep the big picture in mind. Concentrate on real-world problems and actionability.
Learn the subtle new way of programming. Apache Hadoop is built on top of Java but you must program differently. Learn to build on top of Spark or Spark SQL.
Know the business requirements. Do not build in a sandbox for technology’s sake. Focus on what the business needs and identify the best way to solve the problem. Find the fastest path to deliver useful data to the business user.
Problems are similar to what we’ve gone through before—data management, good hygiene. Don’t feel like your existing knowledge is worthless. Learn a new programming approach - think team versus individual. A lot of people skills will carry forward. There’s a fascination with the new thing but this is an evolution, not a revolution. There is no discontinuity. Take new ideas and apply them to new projects.
Developers need to see the world is embracing Big Data and there are many problems to solve. Learn the new technologies. They’re open source and there’s a lot of free training available. It’s less restrictive than previous technologies. Unlearn the restrictions you were taught. Think about how to use data to make people’s lives easier.
Developers need to be asking themselves, “what tools do I need to know?” “What is my stack for the next five years?” “How do I engage more with the business?” Understand what the business is trying to do and help the company understand what the technology is capable of doing.
Are you a developer involved with Big Data? What advice do you have based on your experience?
Opinions expressed by DZone contributors are their own.