How Are Databases Evolving?

To gather insights on the state of databases today and their future, we spoke to 27 executives at 23 companies who are involved in the creation and maintenance of databases.

We asked these executives, "Where do you think the biggest opportunities are in the evolution of databases?" Here's what they told us:

Integration/Convergence

• A world where one data catalog link to storage mechanism about the importance of each piece of data via a unified API so you don’t need multiple codes and stores. Data needs to be transactional but not all of it is equally important.
• The database becomes part of the DevOps methodology. What is the biggest bottleneck? Solve the problem. Move to the next biggest bottleneck. We’ve been using DevOps since 2009 because it works. Apply the lessons learn to databases.
• More solutions that just work support all scenarios.
• A more comprehensive data environment. Provide a platform to build more data centric apps. So many choices of libraries to choose from. Developers have to spend time researching versus doing rapid application development. More integrated platforms with more API management with multi-dimensional analytics and text analytics.
• Cloud platform vendors. Relational SQL adapts to work with data both ways. Convergence of the two. One solution rather than a relational versus a document database.
• Integration into DevOps. Adoption of database automation with CI/CD. Automated provisioning. Good governance as you accelerate the process. Further best practices with the use of containers.
• DBaaS to cloud to manage hardware, software, and infrastructure. Hybrid transactional/analytical processing. The feedback loop is getting smaller and is near real-time. Separate workloads with different storage engines are brought under one umbrella. When something is added to the shopping cart a change is pushed into analytical storage.
• We’re living in a new era where everything is connected and technology is making anything possible; where data is our most valuable resource, enabling transformative actions and insights that empower business disruption. But while the world gets more connected, some databases are still trying to build walls around your data and limit how you use it. Businesses are still prevented from harnessing the full power of their data by unnecessary constraints, restrictions, and controls. They’re held back by fixed locations, fixed contracts, and fixed platforms; by inflexible databases that box in their data and put a lid on its potential. We see a great deal of value in powering a cognitive world where all devices (IoT) and processes will be subject to interrogation — that is, to tap into and capitalize on the data that gets generated to better understand the world, to automate processes, and apply intelligence to make applications smarter.

Cloud/Microservices

• Opportunity for the organization to be more competitive. Next generation databases enable to stop paying a ransom for legacy database technology. Deploy quicker. Connect in real-time to provide a superior CX. Tooling is evolving to support a multi-cloud environment of hybrid databases.
• Maturation of things over the last few years. Greater usability of databases. Cloud computing — disposable infrastructure enables you to focus on your core business. How to let people scale the database with the rest of their resources and additional data.
• Distributed relational databases are the future. Cloud and distributed technology. Relational databases have to provide a solution.
• Opportunities around the cloud. Amazon invests in data solutions for the cloud. Microsoft runs Bing on Azure. See releases happen faster. New services are coming online faster. Assistance in moving to the cloud.
• Beyond database but includes the re-architecting of the enterprise stack. The last 30 years, we’ve had relational databases, servers, and apps. Databases store data in enterprise storage. Now we’re driven by the speed of execution. We add servers and scale out to direct attached disks. The database does this automatically creating tables as needed. Microservices are becoming the norm.
• As hardware changes from small memory with one or two cores to 32 and 64 cores, memory is becoming cheaper. No volatile memory may really be less than two years away. This changes how to architect the system and requires different thinking. More cloud adoption, more container friendly. The hype around AI/ML/DL we’re a complementary technology around that.
• Graph with more interesting viewpoints with different technological needs pushing the limits. Solve with polyglot persistence or multi-model to create a unified platform. While appealing today’s technology kind of optimization for graph problems versus columnar versus Amazon’s new DynamoDB. Optimizations are very difficult to integrate and coexist. The solutions lie in architectural patterns and microservices. Must stay abreast of the right tools for the job.
• Converge a high-speed compute platform like Spark in-memory data management with data in a virtualized solution in a unified offering. There will be a complementary layer on top of existing legacy databases — polyglot data management.

Other

• Data in the air. Cyber-intelligence anomaly detection at 10-40 GB per second. Oil and gas analytics and visualization. Opportunities in all verticals — data in the air and data in motion.
• Data continues to get bigger. Modern platform adoption, More functionality.
• Ease of use (for developers) when deploying highly distributed database systems. As these solutions mature, they expose immense power to even the smallest of teams.
• I do think the application of databases will evolve. Artificial intelligence, machine learning, predictive analytics, and quantum computing will all radically shape database needs.
• Every application requires a database. Use data to measure the success of the business. Companies realize data is the new form of capital. Data helps organizations find ways to reduce costs, increase profitability, increase compliance, and reduce fraud. Tapping into more data sources like sensors and social media results in exponentially more data. Non-volatile DRAM for persistent delivery — performance of in-memory stores.
• Real-time analytics due to increased lack of patience. Need to provide data to end customers. Identify opportunities that are not enabling real-time.
• Certainly, the evolution of distributed architectures like Hadoop and how they may evolve to bring Blockchain-like features to the database.
• I believe the biggest opportunities in the evolution of databases are by far with real-time queries on fresh and connected data.
• As data volumes keep growing, the focus will be on the database’s ability to address these issues. We’re empowering users to quickly build and evolve applications while minimizing the dependency on IT and the DBMS configuration. This all leads to a democratization of data and the ability to incorporate all data into analytics to gain even more insights. But it also means bringing the analytics to the data ("push-down") to gain greater efficiencies and better quality when it comes to machine learning models, scoring, etc.

Where do you see the evolution of databases?

Here’s who we talked to:

• Emma McGrattan, S.V.P. of Engineering, Actian
• Zack Kendra, Principal Software Engineer, Blue Medora
• Subra Ramesh, VP of Products and Engineering, Dataguise
• Robert Reeves, Co-founder and CTO and Ben Gellar, VP of Marketing, Datical
• Peter Smails, VP of Marketing and Business Development and Shalabh Goyal, Director of Product, Datos IO
• Anders Wallgren, CTO and Avantika Mathur, Project Manager, Electric Cloud
• Lucas Vogel, Founder, Endpoint Systems
• Yu Xu, CEO, TigerGraph
• Avinash Lakshman, CEO, Hedvig
• Matthias Funke, Director, Offering Manager, Hybrid Data Management, IBM
• Vicky Harp, Senior Product Manager, IDERA
• Ben Bromhead, CTO, Instaclustr
• Julie Lockner, Global Product Marketing, Data Platforms, InterSystems
• Amit Vij, CEO and Co-founder, Kinetica
• Anoop Dawar, V.P. Product Marketing and Management, MapR
• Shane Johnson, Senior Director of Product Marketing, MariaDB
• Derek Smith, CEO and Sean Cavanaugh, Director of Sales, Naveego
• Philip Rathle, V.P. Products, Neo4j
• Ariff Kassam, V.P. Products, NuoDB
• William Hardie, V.P. Oracle Database Product Management, Oracle
• Kate Duggan, Marketing Manager, Redgate Software Ltd.
• Syed Rasheed, Director Solutions Marketing Middleware Technologies, Red Hat
• John Hugg, Founding Engineer, VoltDB
• Milt Reder, V.P. of Engineering, Yet Analytics