Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

What Are the Hurdles Companies Face With Databases Today?

DZone's Guide to

What Are the Hurdles Companies Face With Databases Today?

The three most frequently mentioned industries were financial services, healthcare, and retail; however, it is agreed there are database applications for every industry.

· Database Zone ·
Free Resource

RavenDB vs MongoDB: Which is Better? This White Paper compares the two leading NoSQL Document Databases on 9 features to find out which is the best solution for your next project.  

To gather insights on the state of databases today and their future, we spoke to 27 executives at 23 companies who are involved in the creation and maintenance of databases.

We asked these executives, "What are the most common issues you see companies having with databases?" Here's what they told us:

Lack of Knowledge

  • The lack of education about and understanding of GPUs in the datacenter. Adoption of the cloud. Teaching enterprises about the difference in hardware technology. There are 50 to 100 different types of databases and the customer needs to understand the benefits of each type. We provide the ability to integrate TensorFlow, Café, Torch. No limits on the types of feeds, languages, and interfaces you can use. There’s a lot of Oracle and SAP legacy databases with years of stored procedures that need to be unwrapped to see changes for the better with regards to visibility and greater actionability on the data. Fusing consumer IoT with industrial IoT.
  • Once we take care of the operational challenges, what’s left is understanding the data models and the data access. This requires understanding customers, how they interact with the app, and how the app interacts with the database.
  • Compromise critical business performance with speed, scale, accuracy, and cost. Go relational and you can do as much as you want. How much you’re able to explore is based on how much you’re willing to pay. NoSQL world each niche database. Consistency, availability, partitioning triangle, manual sharding, and scaling are the challenges people are dealing with. Customers with scars are knowledgeable. Customers who have hired a new team know the new techniques but have a hard time — they get to a prototype quickly but don’t see the roadblocks with scalability, consistency, and availability until they’re further down the road and it takes longer and is more expensive to address.
  • Legacy is a big one as the system kind of works with some pain. How bad is the pain relative to the amount of work required to alleviate it? How do I estimate the cost, time, effort, and payoff? The cost of switching technology is often high. Evaluating fit can be challenging. Stakeholders in the company can raise political situations. Work together with IT and app development to determine the proper product fit and address any political issues.
  • When developers work with databases, they can quickly find themselves in over their heads if they try to address database issues. It’s only natural for a developer to turn back to their code as the most familiar path to resolve an issue. In most cases, the database engine will do a better job at finding the most efficient way of completing a task than you could in code — especially when it comes to things like making the results conditional on operations performed on the data.
  • The more data that is stored is inversely proportional to a company’s ability to analyze the data. Companies have no clue how many copies of data they have. No idea of data lineage. Data virtualization helps codify the dependencies of data storage saving significant storage dollars. In-memory technology requires writing to different APIs. We’re trying to introduce a combination of data virtualization and data grids to provide a consolidated view of all of the data.

Complexity

  • We help customers find things more easily. Ingesting into a SQL server schema had to change for every customer. Cloud and database technology can simplify these efforts.
  • Not understanding the kind of workload appropriate for the database. People will start with the relational database they already have but don’t have a checkpoint to determine the need for a graph or key value database. Performance struggles when using technology that’s not the best fit. It works fine on a small scale, but you realize the limitations as you grow.
  • There will always be performance, scale, resiliency, and security challenges. But increasingly, and at a tactical level, we see the shift to containers as a common issue. Moving from bare metal to containers is not your typical “lift and shift.” It’s a true modernization of the architecture. Understanding that data persistence, portability, and performance in a containerized environment are truly different. Building the right microservices architecture and a scalable, distributed server, storage, and network fabric are key to run stateful, database applications in containers.
  • Key functions are performance, scalability, availability, and security. You must consider all of these. Too often, a client will focus on short-term user problems and then have to perform major reworking.
  • Prior to using our technology, all of our customers shared they had query performance issues on joins over hundreds of millions of entities/relationships.
  • Affordability, resiliency, and inflexibility are fairly common concerns in the database market. Concerns like these continue to drive us to produce technologies and offerings that level those speed bumps and get people involved, up and running faster.

Scalability

  • Unanticipated growth. Different teams using different products. Start experimenting with the functionality of the product and end up with thousands of workflows. What happens if you end up with a billion rows in this table?
  • Exploding data, the proliferation of solutions, outsourcing quality or testing but must remain compliant.
  • Invested in a point solution that they’ve outgrown as their needs have changed. A consolidated database eases the patch management process. All PaaS reduce dependencies on versions but you still have 17 different libraries. Systems are more complex. Move from a database to a data platform so you can do more than just store data.
  • Many companies fail to take a critical look at the initial choice of a database solution. A database gets chosen because it is safe (what the developers know), cool (what the developers saw on Hacker News), or paid for (what the company has already licensed). Always put requirements first, and do your best to anticipate realistic scale needs.
  • A source of pain is that the production database is too large to give access to developers. You can’t test and end up with blind spots because production data is different than test data. Making SQL server able to run on Linux. A local copy of SQL server to test against is huge.
  • When developers build apps and test but not at scale — 500 GB versus hundreds of terabytes.

Other

  • As customers deploy to next generation databases, they need automatic backup and recovery. Test and development environments need to meet the two-week DevOps release cycle. Automatically refresh data nightly to provide test and development with the data they needed.

  • The need to align database changes with application changes. Stems from Conway’s Law – the company that designs the process will design the process so it follows the line of communication in the company. That mentality does not work anymore. Move to the cloud with a DevOps methodology. The database model hasn’t changed since 1979.
  • Modernizing the application stack of existing applications moving from Oracle and SQL server to Cloud. How to manage the data tier. Need to modernize data tier at the same time. Architect and infrastructure change quickly how to manage databases to let me change over time.
  • Depends on if you are working with a third-party app versus build your own. Build your own need to be optimized. We help the third party with the infrastructure to help with performance and disaster recovery.
  • Larger disparate teams needing to integrate databases into DevOps. Enable to speak the same language as the application development team. Provide different tooling so they are able to plug the databases into the processes that exist using the same technology. Shift the database integration process left.
  • Adapt to changing infrastructure – cloud and containers. Different use cases serving different requirements. Intelligent payment processing 24/7/365. Different requirements for each use case. Understand how to make the database meet the requirements. Consistency, persistency, partition tolerance. What’s the best way to make the database meet the requirements?

What are some hurdles you see companies facing with databases today?

Here’s who we talked to:

  • Emma McGrattan, S.V.P. of Engineering, Actian
  • Zack Kendra, Principal Software Engineer, Blue Medora
  • Subra Ramesh, VP of Products and Engineering, Dataguise
  • Robert Reeves, Co-founder and CTO and Ben Gellar, VP of Marketing, Datical
  • Peter Smails, VP of Marketing and Business Development and Shalabh Goyal, Director of Product, Datos IO
  • Anders Wallgren, CTO and Avantika Mathur, Project Manager, Electric Cloud
  • Lucas Vogel, Founder, Endpoint Systems
  • Yu Xu, CEO, TigerGraph
  • Avinash Lakshman, CEO, Hedvig
  • Matthias Funke, Director, Offering Manager, Hybrid Data Management, IBM
  • Vicky Harp, Senior Product Manager, IDERA
  • Ben Bromhead, CTO, Instaclustr
  • Julie Lockner, Global Product Marketing, Data Platforms, InterSystems
  • Amit Vij, CEO and Co-founder, Kinetica
  • Anoop Dawar, V.P. Product Marketing and Management, MapR
  • Shane Johnson, Senior Director of Product Marketing, MariaDB
  • Derek Smith, CEO and Sean Cavanaugh, Director of Sales, Naveego
  • Philip Rathle, V.P. Products, Neo4j
  • Ariff Kassam, V.P. Products, NuoDB
  • William Hardie, V.P. Oracle Database Product Management, Oracle
  • Kate Duggan, Marketing Manager, Redgate Software Ltd.
  • Syed Rasheed, Director Solutions Marketing Middleware Technologies, Red Hat
  • John Hugg, Founding Engineer, VoltDB
  • Milt Reder, V.P. of Engineering, Yet Analytics

Do you pay to use your database? What if your database paid you? Learn more with RavenDB.

Topics:
databases

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}