Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

What Have Been the Most Significant Changes in Databases?

DZone's Guide to

What Have Been the Most Significant Changes in Databases?

The cloud and containers, the evolution from NoSQL back to SQL, and consolidation are the three most significant changes.

· Database Zone ·
Free Resource

RavenDB vs MongoDB: Which is Better? This White Paper compares the two leading NoSQL Document Databases on 9 features to find out which is the best solution for your next project.  

To gather insights on the state of databases today, and their future, we spoke to 27 executives at 23 companies who are involved in the creation and maintenance of databases.

We asked these executives, "How have databases changed in the past year?" Here's what they told us:

Cloud/Containers

  • Moved from traditional on-prem databases to the cloud.
  • Exploding data. Migration to the cloud. The number of file formats and types of storage are increasing, as well.
  • How we consume databases. We used to buy servers. Now we go to Amazon RDS. Consume DBaaS. The model has changed; it’s no longer an issue for dev and test.
  • Relational sees more cloud platforms due to cost and available technology. Massively parallel on Azure and Redshift. Fewer customers are building their own servers with SQL. Analytics are using AWS RDS to replace MySQL.
  • As Microsoft and Oracle become more self-manageable, the need for DBAs and requirements has changed from maintenance and backup since the engines are better at taking care and cleaning up problems. DBAs now need to understand how to plan, strategize, back up, and use data — how to transform data, work with data lakes, and data warehouses. Get more heterogeneous. Ability to have the optional reuse of the database as a caching engine. Containerization of databases.
  • Customers tell us that Docker and containers have been the biggest change in the last year. Containers were originally the domain of cloud-native companies (think Netflix, Uber, or Airbnb). Now, mainstream developers and DevOps teams are deploying Docker and using containers in a more traditional enterprise architecture. But containers are ephemeral in nature and don’t handle database workloads well. There are quite a bit of planning and infrastructure considerations needed to make sure containerized databases are portable, persistent, and performant.
  • Cloud is part of the continuous development along with multi-tenant architecture. In-memory column storage for real-time analysis of big data SQL. The range of database services compatible with the cloud and on-premises has grown.
  • The last year has seen some significant changes. The advent of cloud; the growing need to be more agile and empower business users; the need to manage complex data landscapes — all of these things have impacted the requirements on database management systems. We are also starting to see a trend back to multi-workload. You see NoSQL databases leveraging SQL and you see SQL based being able to access or handle unstructured; they are all coming back together. We also see advancements in how fast we can ingest data, which is being driven in large part by IoT workloads. In these instances, it's not just the size and structure of the data that poses an issue, but the speed at which it comes to you.

NoSQL Back to SQL

  • The biggest changes to NoSQL data storage do differently with more capabilities. The most recent ideas want the best of both worlds with distributed and SQL with elastic SQL. Google Cloud Spanner builds apps using SQL but scales out. Distributed SQL databases have made big changes over last year.
  • NoSQL is not as pervasive. RDBMS didn’t work so people started creating niche databases. Now we have a new way of creating multi-model databases as the foundation of the enterprise platform. The introduction of GPU-accelerated databases for extremely fast performance. In-memory data integration is better with big data systems. Cloud solutions are offering classical relational databases (for example, Spanner and Dynamo are sophisticated solutions).
  • Shift away from NoSQL. Spanner, Cockroach, NuoDB are more consistent systems with SQL. Some winners in the NoSQL space have more queries like Cassandra and MongoDB. You can still scale and have some of these features. More features, more queries, and less burden on developers.
  • SQL is coming back built on top of NoSQL. There’s a new pattern of the physical consolidation of data aggregated in real-time giving new life to data virtualization and optimizing queries exposed to SQL. Spark and Flink are in-memory data processing frameworks. The data processing unit of work is immutable. We’re seeing a lot of rolling up of streaming data and in-memory data management.

Consolidation

  • A unified data platform is necessary for mission critical apps. Database vendors are acquiring and bundling products so they can offer a single solution. It's interesting that Gartner is now looking at interoperability and analytics. The market is asking for consolidation into a single database which is driving the acquisition and consolidation.
  • Consolidation in the industry. Move toward cloud-native databases — Amazon Dynamo DB. More mature companies are adopting what was hyped several years ago.
  • I’m seeing a lot more traction for multi-model or multi-paradigm databases like the OrientDB platform. These can offer a great deal of flexibility and I’m glad to see them mature. The trend towards distributing databases for scale continues, and I’m finding these to be much more developer-friendly than in past. Another interesting development in the past few years has been the union of the database and messaging in distributed persistent queues like Kafka and Kinesis.
  • Huge growth in the adoption of DevOps and continuous delivery. Saw this in their State of the Database in DevOps Survey. The database needs to be part of the DevOps process. We provide tools to transfer the database into the DevOps methodology.
  • How databases relate to each other. There’s been a fair amount of convergence. Transactions, functions, adding new storage for scalability with security, flexibility, and scalability with a single solution. Containers are disruptive with a new set of requirements and challenges. How to handle persistence, provisioning, and storage.

Other

  • The advent of graphics. The graph DB ecosystem sees new products announced every week. Microsoft has announced two, with SAP and Redis coming out with their own. The benefits may or may not be available with each, depending on what’s underneath. You get the surface benefits but not the performance benefits.
  • There are a wide variety of data sources. We’ve gone from demographic and transactional data to having a holistic view of the customer. This has resulted in a massive increase in the volume of data from a company have one or two terabytes of data to having hundreds of terabytes.
  • The relationship between databases and applications is changing. Databases are essential to delivering better business analytics and more personalized digital customer experiences (CX). Application requirements are driving more robust database requirements. This has led to an explosion of cloud and open-source database platforms. Gartner has also noted this trend. They indicate that by next year, more than 70% of new in-house applications will be developed on an open-source database and 50% of closed-source databases will be on their way to conversion.
  • Customers are looking beyond slow/batch computing on stale data. Instead, they are running fast queries on constantly changing dataset. These queries are beyond simple filtering/aggregation and looking at deep links in the data.
  • Scaling out versus scaling up to encompass different networks. Speed is more and more important. Battle I/O (input/output). All data in a system-wide DRAM. We shard so we can deploy in a cost-effective manner. Use thousands of cores of GPUs to align vectors for fast computing around machine learning (ML) and artificial intelligence (AI). Better understanding of the enterprise by augmenting data with ML and AI. We provide an “easy button” for enterprises with big data feeds.
  • Oracle now has snapshots that allow you to roll back quickly and easily in the event your data or schema changes. Incredibly useful especially for testing. Trying to move out of RDBMS because they are slow and expensive. It is not the right place for all states of data.
  • The rise of open source and the diversity of the ecosystem.

What are the most significant changes you are observing to the database ecosystem?

Here’s who we talked to:

  • Emma McGrattan, S.V.P. of Engineering, Actian
  • Zack Kendra, Principal Software Engineer, Blue Medora
  • Subra Ramesh, VP of Products and Engineering, Dataguise
  • Robert Reeves, Co-founder and CTO and Ben Gellar, VP of Marketing, Datical
  • Peter Smails, VP of Marketing and Business Development and Shalabh Goyal, Director of Product, Datos IO
  • Anders Wallgren, CTO and Avantika Mathur, Project Manager, Electric Cloud
  • Lucas Vogel, Founder, Endpoint Systems
  • Yu Xu, CEO, GraphSQL
  • Avinash Lakshman, CEO, Hedvig
  • Matthias Funke, Director, Offering Manager, Hybrid Data Management, IBM
  • Vicky Harp, Senior Product Manager, IDERA
  • Ben Bromhead, CTO, Instaclustr
  • Julie Lockner, Global Product Marketing, Data Platforms, InterSystems
  • Amit Vij, CEO and Co-founder, Kinetica
  • Anoop Dawar, V.P. Product Marketing and Management, MapR
  • Shane Johnson, Senior Director of Product Marketing, MariaDB
  • Derek Smith, CEO and Sean Cavanaugh, Director of Sales, Naveego
  • Philip Rathle, V.P. Products, Neo4j
  • Ariff Kassam, V.P. Products, NuoDB
  • William Hardie, V.P. Oracle Database Product Management, Oracle
  • Kate Duggan, Marketing Manager, Redgate Software Ltd.
  • Syed Rasheed, Director Solutions Marketing Middleware Technologies, Red Hat
  • John Hugg, Founding Engineer, VoltDB
  • Milt Reder, V.P. of Engineering, Yet Analytics

Get comfortable using NoSQL in a free, self-directed learning course provided by RavenDB. Learn to create fully-functional real-world programs on NoSQL Databases. Register today.

Topics:
database ,containers ,nosql ,sql ,consolidation

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}