What Have Been the Most Significant Changes in Databases?
What Have Been the Most Significant Changes in Databases?
The cloud and containers, the evolution from NoSQL back to SQL, and consolidation are the three most significant changes.
Join the DZone community and get the full member experience.Join For Free
Download the Altoros NoSQL Performance Benchmark 2018. Compare top NoSQL solutions – Couchbase Server v5.5, MongoDB v3.6, and DataStax Enterprise v6 (Cassandra).
To gather insights on the state of databases today, and their future, we spoke to 27 executives at 23 companies who are involved in the creation and maintenance of databases.
We asked these executives, "How have databases changed in the past year?" Here's what they told us:
- Relational sees more cloud platforms due to cost and available technology. Massively parallel on Azure and Redshift. Fewer customers are building their own servers with SQL. Analytics are using AWS RDS to replace MySQL.
- As Microsoft and Oracle become more self-manageable, the need for DBAs and requirements has changed from maintenance and backup since the engines are better at taking care and cleaning up problems. DBAs now need to understand how to plan, strategize, back up, and use data — how to transform data, work with data lakes, and data warehouses. Get more heterogeneous. Ability to have the optional reuse of the database as a caching engine. Containerization of databases.
- Customers tell us that Docker and containers have been the biggest change in the last year. Containers were originally the domain of cloud-native companies (think Netflix, Uber, or Airbnb). Now, mainstream developers and DevOps teams are deploying Docker and using containers in a more traditional enterprise architecture. But containers are ephemeral in nature and don’t handle database workloads well. There are quite a bit of planning and infrastructure considerations needed to make sure containerized databases are portable, persistent, and performant.
- Cloud is part of the continuous development along with multi-tenant architecture. In-memory column storage for real-time analysis of big data SQL. The range of database services compatible with the cloud and on-premises has grown.
- The last year has seen some significant changes. The advent of cloud; the growing need to be more agile and empower business users; the need to manage complex data landscapes — all of these things have impacted the requirements on database management systems. We are also starting to see a trend back to multi-workload. You see NoSQL databases leveraging SQL and you see SQL based being able to access or handle unstructured; they are all coming back together. We also see advancements in how fast we can ingest data, which is being driven in large part by IoT workloads. In these instances, it's not just the size and structure of the data that poses an issue, but the speed at which it comes to you.
NoSQL Back to SQL
- The biggest changes to NoSQL data storage do differently with more capabilities. The most recent ideas want the best of both worlds with distributed and SQL with elastic SQL. Google Cloud Spanner builds apps using SQL but scales out. Distributed SQL databases have made big changes over last year.
- I’m seeing a lot more traction for multi-model or multi-paradigm databases like the OrientDB platform. These can offer a great deal of flexibility and I’m glad to see them mature. The trend towards distributing databases for scale continues, and I’m finding these to be much more developer-friendly than in past. Another interesting development in the past few years has been the union of the database and messaging in distributed persistent queues like Kafka and Kinesis.
- Huge growth in the adoption of DevOps and continuous delivery. Saw this in their State of the Database in DevOps Survey. The database needs to be part of the DevOps process. We provide tools to transfer the database into the DevOps methodology.
- How databases relate to each other. There’s been a fair amount of convergence. Transactions, functions, adding new storage for scalability with security, flexibility, and scalability with a single solution. Containers are disruptive with a new set of requirements and challenges. How to handle persistence, provisioning, and storage.
- The advent of graphics. The graph DB ecosystem sees new products announced every week. Microsoft has announced two, with SAP and Redis coming out with their own. The benefits may or may not be available with each, depending on what’s underneath. You get the surface benefits but not the performance benefits.
- There are a wide variety of data sources. We’ve gone from demographic and transactional data to having a holistic view of the customer. This has resulted in a massive increase in the volume of data from a company have one or two terabytes of data to having hundreds of terabytes.
- The relationship between databases and applications is changing. Databases are essential to delivering better business analytics and more personalized digital customer experiences (CX). Application requirements are driving more robust database requirements. This has led to an explosion of cloud and open-source database platforms. Gartner has also noted this trend. They indicate that by next year, more than 70% of new in-house applications will be developed on an open-source database and 50% of closed-source databases will be on their way to conversion.
- Customers are looking beyond slow/batch computing on stale data. Instead, they are running fast queries on constantly changing dataset. These queries are beyond simple filtering/aggregation and looking at deep links in the data.
- Scaling out versus scaling up to encompass different networks. Speed is more and more important. Battle I/O (input/output). All data in a system-wide DRAM. We shard so we can deploy in a cost-effective manner. Use thousands of cores of GPUs to align vectors for fast computing around machine learning (ML) and artificial intelligence (AI). Better understanding of the enterprise by augmenting data with ML and AI. We provide an “easy button” for enterprises with big data feeds.
- Oracle now has snapshots that allow you to roll back quickly and easily in the event your data or schema changes. Incredibly useful especially for testing. Trying to move out of RDBMS because they are slow and expensive. It is not the right place for all states of data.
- The rise of open source and the diversity of the ecosystem.
Here’s who we talked to:
- Emma McGrattan, S.V.P. of Engineering, Actian
- Zack Kendra, Principal Software Engineer, Blue Medora
- Subra Ramesh, VP of Products and Engineering, Dataguise
- Robert Reeves, Co-founder and CTO and Ben Gellar, VP of Marketing, Datical
- Peter Smails, VP of Marketing and Business Development and Shalabh Goyal, Director of Product, Datos IO
- Anders Wallgren, CTO and Avantika Mathur, Project Manager, Electric Cloud
- Lucas Vogel, Founder, Endpoint Systems
- Yu Xu, CEO, GraphSQL
- Avinash Lakshman, CEO, Hedvig
- Matthias Funke, Director, Offering Manager, Hybrid Data Management, IBM
- Vicky Harp, Senior Product Manager, IDERA
- Ben Bromhead, CTO, Instaclustr
- Julie Lockner, Global Product Marketing, Data Platforms, InterSystems
- Amit Vij, CEO and Co-founder, Kinetica
- Anoop Dawar, V.P. Product Marketing and Management, MapR
- Shane Johnson, Senior Director of Product Marketing, MariaDB
- Derek Smith, CEO and Sean Cavanaugh, Director of Sales, Naveego
- Philip Rathle, V.P. Products, Neo4j
- Ariff Kassam, V.P. Products, NuoDB
- William Hardie, V.P. Oracle Database Product Management, Oracle
- Kate Duggan, Marketing Manager, Redgate Software Ltd.
- Syed Rasheed, Director Solutions Marketing Middleware Technologies, Red Hat
- John Hugg, Founding Engineer, VoltDB
- Milt Reder, V.P. of Engineering, Yet Analytics
Opinions expressed by DZone contributors are their own.