[DZone Research] Data Persistence

The data for this article is from the data collected for the DZone Guide to Databases: Relational and Beyond.

Introduction

For this year's DZone Guide to Databases, we surveyed software professionals from across the IT industry. We received 582 responses with a 79% completion rating. In this article, we discuss how developers work with both persistent and ephemeral data. 

Tenacious D(ata)

Over half of respondents (61%) reported that they persist 75-100% of their data. While this is not surprising, it does offer us an interesting way to explore the ephemeral nature of data in software development. 41% of respondents told us they have less than 10% of ephemeral data. This percentage held fairly steady across both SQL and NoSQL databases (MySQL: 39%; PostgreSQL: 35%; MongoDB: 38%; Oracle: 41%; MS SQL Server: 41%; Redis: 32%). Statistically significant differences between the use of SQL databases and a particular NoSQL database begin to appear, however, as larger amounts of ephemeral data come into play. Across the general survey population, 13% reported that 25-50% of their data is ephemeral. For Redis users, this percentage rises to 19%. Among the general survey population, 14% reported that over 50% of their data is ephemeral. For Redis users, this number again rises to 19%. All measures of ephemeral data among MongoDB users, however, fell within the margin of error for this survey. This finding could, thus, very well be a particularity to Redis rather than NoSQL databases writ large, but it is still interesting to note that Redis users deal with larger amounts of ephemeral data than users of other databases.

As data gets persisted in a database, developers tend to use three main forms of persisted storage models: relational, key-value, and document store. Among all survey-takers, 88% reported using relational storage models, 57% key-value models, and 46% document store models. If we compare these numbers to our four main SQL databases used in production environments, we see that these numbers largely hold. Here’s how this relationship breaks down:

• MySQL users:
  • Functional storage model: 90%
  • Key-value storage model: 58%
  • Document store model: 51%
• PostgreSQL users:
  • Functional storage model: 91%
  • Key-value storage model: 65%
  • Document store model: 50%
• Oracle users:
  • Functional storage model: 92%
  • Key-value storage model: 55%
  • Document store model: 41%
• MS SQL Server Users:
  • Functional storage model: 92%
  • Key-value storage model: 55%
  • Document store model: 43%

As we can see, these numbers all fall within the margin of error for this report (4%), with the only notable exception being the percentage of PostgreSQL users who use the key-value storage model.

When we compare these numbers to our two popular NoSQL databases (MongoDB and Redis) in production environments, interesting differences appear. The use of relational models remained highly popular, with 85% of MongoDB users and 91% of Redis users employing this storage model. Interestingly, the adoption rates of the key-value and document store models proved much higher among NoSQL developers. For MongoDB, 80% reported using the document store model and 72% reported using the key-value model in production environments. Among Redis users, 86% reported using the key-value model and 55% reported using the document store model.

For non-production environments, the SQL and NoSQL databases here under discussion again matched the same patterns displayed in production environments.

The data for this article is from the data collected for the DZone Guide to Databases: Relational and Beyond.