The Day the Database Crashes

In most business applications, the heart of an organization's value in the marketplace is their data or the ability to utilize data based on the niche that is being filled in today's world. That realization is what inspired the article I wrote in 2016 called, What Is the Value of Your RESTful API?

Unfortunately, that same data is often considered the last priority to keep in shape from an Information Technology (IT) perspective. Too many times, the tasks around the database are too complex or too time-consuming or require too much offline time that is approved by senior management. This procrastination can have catastrophic results when the day arrives and the database finally crashes.

So, how do so many corporations get into this position?

Where It All Started

It is really not that hard to figure out how things can get off track so easily. A company needs to store data as part of their daily routine and a relational database is often implemented to meet the current need. The database design typically matches the needs of the business in a manner that ranges from simple to modestly complex.

As the needs of the business evolve, additional attributes are added to the database. During this time, an application (or applications) using this data continues to evolve as well. Aside from the casual conversation on archiving data or creating some indexes, the database remains the core of the application — ready to accept new attributes as the business matures.

With changes in front-end technology, decisions are made to re-write or replace the legacy application with something that is new, perhaps responsive, that will help the business be more efficient, driving productivity and (hopefully) revenue. In almost every case, the design team realizes the need to keep the old application in place while the new application is being built - almost always leaving the original database design in place.

The World of Microservices

Through iterations of clients, the concept of microservices arrived with a headful of steam. The microservice concept allowed application-server business logic to be separated and managed in smaller codebases. As a result, updates to microservice A do not need to worry about microservice B — especially with API versioning in place. A huge sigh of relief could be heard from back-end developers who no longer have to juggle coordinating complex releases loaded with dependencies.

However, what the development team realized is that isolating the data for each microservice isn't as easy as spinning up a new microservice. The program code could be isolated, but the data dependencies at the database level were not something that could be easily solved. Microservice A needed data from tables that were part of microservices B through E, F or maybe even G. Like before, that central database remained in place — but now, there are several microservices connecting to the database instead of the single monolith application.

The Database: Redundant and Often Not Changed

To combat the growing needs from the microservices, the database is often upgraded to better hardware, additional indexes added (where they make sense) and even the concept of clustering is introduced. Under the covers, the same database (which has grown quite large at this point) remains.

The complexity of the database puts IT in a tough position. The desire is to break the legacy database into smaller schema or designs, but the dependencies at the data, application, and business layer serve as blockers. The effort to make the number of changes (which can often be considered to be of massive scale) competes with a list of features that are priorities to the business. In most prioritization meetings, feature work typically wins over technical debt - especially when it is considered the type of work which can impact end-users doing their daily tasks.

When the Database Finally Crashes

Organizations can only look the other way so long before that legacy database takes a turn for the worse. The most recent scenario I can recall is when the amount of storage available for the database became less than what was needed for scheduled database processing. When the process no longer had physical storage to utilize, a flurry of corruption began to pass through the current database instance — as well as the clustered instances put into place for fault tolerance purposes.

Regardless of the cause, when the database finally crashes, the organization finds itself in a doomed state. Until the database is restored, the applications and services which depend on the back-end technology are unusable. The corporation itself must find a way to attempt to do business while IT determines and executes a plan of attack.

The Odds Are Against You

According to the Federal Emergency Management Agency (FEMA), 40% of businesses fail to recover from a disaster. Another 25% fail within one year after a disaster. For smaller businesses, 90% do not survive two years after being faced with a disaster.

While these statistics cover a wide array of "disasters," a technology-based disaster certainly can provide the same (if not worse) type of impact as a natural disaster — especially when the recovery time is based in days instead of minutes or hours. The key in every case is to put safeguards in place to mitigate the risks. Secondly, a business continuation plan should be completed and exercised on a routine basis.

Conclusion

It is the responsibility of IT professionals to maintain their applications, databases, and systems. This includes keeping an eye on existing systems, making sure the necessary steps are taken in order to keep a healthy application server presence. I always recommend a 20% allocation of time be focused on maintenance and technical debt items in tasks. However, this requires buy-in from senior management, product owners and the business itself.

It is not a question of "if" something like a database crash is going to happen, it is always a question of "when" it will happen.