Big Data Concerns
Big Data Concerns
The proliferation of data and tools is on par with privacy and security as the biggest concern around the state of big data today.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
To gather insights on the state of big data today, we spoke with 22 executives from 20 companies who are working in big data themselves or providing big data solutions to clients.
Here's what they told us when we asked, "What are your biggest concerns regarding the state of big data today and who is addressing these concerns?" Here's what they told us:
- Confusion around technologies. Balance new innovations and identify veteran technology vendors who represent the future of big data. Stable and work in a production environment.
- Society – security and privacy. Google, Apple collecting all this data on us and we don’t know what they’re doing with it. A lot of citizens have given up their rights and need to take back control by being able to know how your data is being used. The same thing with privacy – companies are collecting more data than they can protect. The black side is further ahead than the white side.
- It’s early in the IoT cycle. Tremendous opportunity for innovation and efficiencies. What data to collect, keep, where to store, how long to keep? See more use cases of analyzing disparate data sets for innovation.
- Too much information to ingest. We can extract anything but it’s easy to miss things as evidenced by the recent election.
- Only one: security. As the last few years have shown, a lot of big data technology was not built in a secure manner. The default is non-secured, in fact. Everyone needs to take this very seriously, and everyone needs to address this within their own big data projects. Make sure you are controlling access to your big data platform and repository, and always have a plan for what to do if data gets leaked. Only the paranoid survive.
- No concerns. Human psychology can deal with rapid changes.
- Vendors are creating a separate tool for every point of the problem. How do we enable innovation without adding complexity? What are ways to address the challenges that don’t require a patchwork of solutions?
- Lack of knowledge and understanding of what’s involved with big data projects. Old data never dies; it just degrades over time. You need the skill set to deal with legacy environments.
- Companies are not successful with complex tooling. They need to pick the right tools, evolve and educate. There are so many tools out there, it can be difficult picking the right ones without the right people on the team with the knowledge to make the right decisions. Skillsets are evolving. We will be getting actionable insights faster.
- Big data is evolving. How to take advantage of the data they have. Movement from open source to enterprise grade. Opportunities for companies to make a difference.
- Not enough emphasis on security and reliability. More emphasis on widgets. Not enough emphasis on where data is coming from and how to keep it safe and clean. Provide a safety check to know who the data came from, and where it came from without impacting the process.
- Speed of change. Walking on quicksand. MapReduce is a great example. Conceptually it does the right job but Spark has addressed concerns. Loose coupling. Keep things in modules. There are battles among cloud providers with regards to the machine learning tools they offer. Hosted Hadoop will be in great demand. More enterprise businesses are moving to Azure given their machine learning tools.
- Not everyone is as ethical and careful as we are dealing with clients in highly regulated industries. We operate with guard rails, processes, and infrastructure giving our clients confidence that their data will be secure.
- Prepare for adoption of cloud when constructing the architecture where data will live. What does big data look like on the cloud?
- So many companies are doing things that overlap. This causes confusion in the marketplace. High noise = low signal. I’d like to see V.C.’s and entrepreneurs become more sophisticated. Investing in additive rather than duplicative, one-off solutions.
- The hype and the number of solutions. Too many vendors and no process, or qualified individuals to evaluate the vendors. Need certification framework. We use AWS certification for ourselves.
- Ability to scale, security and governance, skill set gap, the rapid evolution of the technology stack (i.e., it’s outdated once it’s been implemented), lack of out-of-the-box production-ready systems. SAP is addressing all the above.
- As we can track and correlate more and more data effectively, privacy and ethical considerations are amongst the biggest concerns. The European Union just updated their data protection directives and it is very strict. Although the United States also has some strict directives around the use of data, it is not enforced similarly, which makes it difficult to navigate in a global economy.
- I don’t have any concerns. I think it is a massive opportunity, and those who can understand the possibilities and then deliver will have a big competitive advantage. There will be large and successful companies who fail to take advantage of the technology advances and will wind up left behind. Like in the early internet days when traditional brick and mortar companies succumbed to young dynamic companies who grasped the new internet technology, the same will happen again. If you can successfully model the world in data terms, then you can control it. There is so much data and Big Data techniques give you the capability.
Do you have any concerns regarding the state of big data today?
By the way, here's who we talked to!
- Nitin Tyagi, Vice President Enterprise Solutions, Cambridge Technology Enterprises.
- Ryan Lippert, Senior Marketing Manager and Sean Anderson, Senior Product Marketing Manager, Cloudera.
- Sanjay Jagad, Senior Manager, Product Marketing, Coho Data.
- Amy Williams, COO, Data Conversion Laboratory (DCL).
- Andrew Brust, Senior Director Market Strategy and Intelligence, Datameer.
- Eric Haller, Executive Vice President, Experian DataLabs.
- Julie Lockner, Global Product Marketing, Data Platforms, Intersystems.
- Jim Frey, V.P. Strategic Alliances, Kentik.
- Eric Mizell, Vice President Global Engineering, Kinetica.
- Rob Consoli, Chief Revenue Officer, Liaison.
- Dale Kim, Senior Director of Industrial Solutions, MapR.
- Chris Cheney, CTO, MPP Global.
- Amit Satoor, Senior Director, Product and Solution Marketing, SAP.
- Guy Levy-Yurista, Head of Product, Sisense.
- Jon Bock, Vice President of Product and Marketing, Snowflake Computing.
- Bob Brodie, CTO, SUMOHeavy.
- Kim Hanmark, Director of Professional Services EMEA, TARGIT.
- Dennis Duckworth, Director of Product Marketing, VoltDB.
- Alex Gorelik, Founder and CEO and Todd Goldman, CMO, Waterline Data.
- Oliver Robinson, Director and Co-Founder, World Programming.
Opinions expressed by DZone contributors are their own.