Executive Insights on the Current and Future State of Databases

This article is featured in the new DZone Guide to Databases: Relational and Beyond. Get your free copy for more insightful articles, industry statistics, and more!

To gather insights on the current and future state of the database ecosystem, we talked to IT executives from 22 companies about how their clients are using databases today and how they see use, and solutions, changing in the future. Here’s who we talked to:

• Jim Manias, Vice President, Advanced Systems Concepts, Inc.

• Tony Petrossian, Director, Engineering, Amazon Web Services

• Dan Potter, V.P. Product Management and Marketing, Attunity

• Ravi Mayuram, SVP of Engineering and CTO, Couchbase

• Patrick McFadin, V.P. Developer Relations, DataStax

• Sanjay Challa, Senior Product Marketing Manager, Datical

• Matthew Yeh, Director of Product Marketing, Delphix

• OJ Ngo, CTO, DH2i

• Navdeep Sidhu, Head of Product Marketing, InfluxData

• Ben Bromhead, CTO and Co-founder, Instaclustr

• Jeff Fried, Director of Product Management, InterSystems

• Dipti Borkar, Vice President Product Marketing, Kinetica

• Jack Norris, V.P. Data and Applications, MapR

• Will Shulman, CEO, mLab

• Philip Rathle, V.P. of Products, Neo4j

• Ariff Kasam, V.P. Products, NuoDB

• Josh Verrill, CMO, NuoDB

• Simon Galbraith, CEO and Co-founder, Redgate Software

• David Leichner, CMO, SQream

• Arnon Shimoni, Product Marketing Manager, SQream

• Todd Blashka, COO , TigerGraph

• Victor Lee, Director of Product Management, TigerGraph

• Mike Freedman, CTO and Co-founder, TimescaleDB

• Ajay Kulkarni, CEO and Co-founder, TimescaleDB

• Chai Bhat, Director of Product Marketing, VoltDB

• Neil Barton, CTO, WhereScape

1. The keys to a successful database strategy are understanding the business needs the database needs to fulfill, followed by the quartet of availability, scalability, performance, and security. You need to understand the precise aspects of the workload so you can choose the right database. Determine the database to use on a case-by-case basis – what are you doing, where are you based, what are you trying to achieve, and what are you using in your infrastructure? Understand how the use case ties back to the business and the business outcomes you are trying to achieve. You have to understand the functional and non-functional requirements of the business.

You have to take advantage of the elasticity of cloud platforms. Availability and scalability are key to 80% of database customers. Ideally, all of the components work in concert to provide a safe, available, and high-performance database solution.

2. There were nearly as many opinions of how companies can get a handle on the vast amounts of data they’re collecting as people I spoke with.aloguing, automation, indexing, building for scale, and Hadoop were the suggestions mentioned more than once.

Data cataloging and automation tools using metadata are encouraged in order to know where data is and how it’s being processed. The exponential growth of data, along with the increasing sprawl of data as it spreads around companies, is making the cataloging and identification of data an important issue for every company. Companies need to adopt an automation solution with capabilities that support big data automation to solve two major pain points: integration and scheduling.

Build an index that can be queried quickly. Data are normalized for storage and quick access by indexing and by writing good queries for ingesting and accessing the data. A core competency for any database is how it scales — if you need more capacity you can just add more nodes.

The collection side problem is solved with Hadoop file systems. Move to Hadoop on-premise or cloud and help reassemble a subset of the data as needed for analytics.

3. The biggest changes in the database ecosystem have been the cloud and the proliferation of databases. These changes are being impacted by digital transformation, including the cloud and containers, which can provide built-in high availability, automated patching, dynamic scalability, and backup management with point-in-time recovery. Databases are being deployed within containers for speed, agility, scalability, ease of deployment, use, configuration, and installation. There’s a war between big cloud vendors on the scale and capabilities of their respective database platforms.

We’re seeing a lot of new databases that are specialized for particular use cases. Today it’s a polyglot environment with many companies employing several models with the ability to do different types of processing in the same database. We’re seeing NoSQL going back to SQL, as well as the emergence of document and graph databases based on business needs.

4. While there were more mentions of graph than others when asked about adoption of databases, its clear organizations are pursuing a polyglot, hybrid, multi-model, cloud strategy using the best tool to get the job done. Graph and document databases are growing faster than SQL and relational; however, SQL and relational will always be there. Respondents saw high rates of adoption of cloud platforms because they scale up and down as needed and are much more cost effective.

5. There were more than a dozen applications and another dozen industries in which databases are being used to solve business problems. The most frequently mentioned application was improving the customer experience by offering application personalization and recommendations. The most frequently mentioned verticals were financial services and retail.

In one example a respondent shared, a financial services firm has applications that rely on a core set of databases with different teams working on different applications needing access to the same databases. Any database changes affect applications and thus need to be tested. An ephemeral test production system was implemented without a large scale to keep speed up and cost down. This accelerated the test process and the client was able to test faster with greater fidelity.

A real-time application of a retail company predictively and actively engages customers with highly personalized experiences with tools that can collect, explore, analyze, and act on multiple streams of data instantaneously. These tools allow retail businesses to make data-driven decisions to improve personalization, recommendations, and customer experience.

6. The most common issues companies are having with databases are: 1) data management; 2) database sprawl; and 3) choosing the right database to solve the business problem. indexing are all issues for clients. Design mistakes and data quality issues are more complex, although the wrong database can increase the likelihood of mistakes and data quality harder to maintain.

“Database sprawl” has continued to be one of the biggest issues affecting companies today. As applications continue to evolve, their requirements have led to a growing number of point solutions at the data layer. The ability to coordinate disparate technologies in diverse and intricate conditions with the lack of a single point of control is challenging. You need someone who knows custom scripting.

Another common issue is users who are forcing the database engine to do something it’s not designed to do. Every database has a fit and a purpose or use case. It’s important to understand that use case and then use the appropriate database technology.

7. The biggest opportunities in the evolution of databases are integration and using AI/ML to provide real-time analytics. Given the number of databases, integrating multiple technologies will become more important. We’re seeing the beginning with JSON, Python, and R. Databases are taking ML into production and deploying at large scale. While ML models require considerable time and expense, the value they provide is tremendous.

8. The biggest concerns around databases today are the proliferation of databases and keeping the data in those databases secure. It’s hard to track all of the new databases that come out every week (DB-Engines currently tracks 343). It’s challenging to decide on a technology that will be around for the long haul. Data infrastructure is becoming more fragmented. Coordinating across databases and silos is very difficult. There is a lot of confusion around which database to use to solve a particular problem. DB-Engines is a good resource, but organizations can’t find the talent to take advantage of all of the new technologies.

Unfortunately, security is still an afterthought. With GDPR and the forthcoming California privacy regulations, security capabilities are becoming more important — including encryption, access, auditing AI models, and auditing test and training data. If we don’t improve data security, it will erase any gains made through new technology advances.

9. Developers do not need advanced database knowledge or skills as much as they need to know SQL, understand the different kinds of databases, and which are the most viable in each category. SQL is still the most useful and used language, so it's still important to understand it and how to write scripts. Try the different database technologies, spin up a cloud and begin playing in minutes. Get hands-on with different databases to determine the best fit for your needs. As you get more involved with a particular database, you will learn how to optimize it to solve your business problem.

This article is featured in the new DZone Guide to Databases: Relational and Beyond. Get your free copy for more insightful articles, industry statistics, and more!