Database Clones and Containers for Enterprise Data Delivery
Database Clones and Containers for Enterprise Data Delivery
This article introduces a new software-based database cloning solution with SQL Server containers, providing enterprises with an open hub for database environment delivery.
Join the DZone community and get the full member experience.Join For Free
New whitepaper: Database DevOps – 6 Tips for Achieving Continuous Delivery. Discover 6 tips for continuous delivery with Database DevOps in this new whitepaper from Redgate. In 9 pages, it covers version control for databases and configurations, branching and testing, automation, using NuGet packages, and advice for how to start a pioneering Database DevOps project. Also includes further research on the industry-wide state of Database DevOps, how application and database development compare, plus practical steps for bringing DevOps to your database. Read it now free.
In the late 1980s, personal computers were poised to transform enterprise computing, and the Ethernet protocol would create a similar phenomenon in local area networking. We soon needed File Sharing, and in the early 1990s, NetApp popularized network attached storage. Interestingly, Cisco and others followed a similar path as NetApp, delivering fixed purpose UNIX appliances. The IT infrastructure that persists to this day was born.
Rationalizing this disparate infrastructure is a continuing issue, particularly as enterprises increasingly explore digital transformation and cloud migration. Fortunately, new methods are available that combine use of database clones with containers for delivery of database environments, for both on-premise and cloud.
My firm, Windocks, delivered an independent port of Docker’s source to Windows, and earlier this year was the first to incorporate database cloning with containers. In this article, we’ll look at the use of database clones that are available on existing Storage Array Networks (SANs), as well as new Windows-based clones.
Database Clones for Dev/Test
Database clones are ideal for short-lived environments in dev, test, and reporting. A writable terabyte-class environment can be delivered in seconds with a small storage footprint. Clones employ a Copy on Write or Redirecton Write method to accommodate changes, relying on a parent snapshot or image as a data source. Clones expand dynamically as changes are made.
Storage Area Network (SAN) Snapshots
Databases require data that are proximate to the host, and Storage Area Networks with iSCSI, Fibrechannel, or fast NFS and SMB networks are the current incumbent storage systems. NetApp, Pure Storage, EqualLogic, and others deliver writable clones in the form of fast snapshots. SANs are widely available but are underutilized for database dev/test support due to the complex scripting required to access data. Use of SAN-based snapshots involve provisioning of LUNs, volumes, and mount points, and ongoing management is needed to clean up mount points when the snapshots are no longer used.
Docker Volume plugins have been developed to allow containers to access SAN data, but the complexity of dealing with storage primitives remain. Time will tell if DBAs are will wade through the storage complexity to use this approach.
SAN V2: "Copy Data Management"
Dev and Test environments don’t share the performance requirements of production systems and new firms are focused on dev/test support. Secondary storage comprises roughly 80 percent of enterprise data, and the Gartner Group has dubbed this space “Copy Data Management.” Cohesity, Rubrik, and others are delivering improvements in what has been a quiet part of the industry.
CDM systems continue the pattern of fixed purpose UNIX appliances, but with new capabilities. CDMs deliver data deduplication, incremental updates, and simplified access to data via restful APIs. These systems aim to provide “incremental forever,” instantly available environments (doing away with full backups). APIs simplify access and use of data, and CDM system are available as software appliances on AWS, Azure, and other public clouds.
Windows Database Cloning
In addition to SAN and CDM appliances, Windows-based cloning is now available from Windocks that combines SQL Server database clones with containers. Windows-based clones employ the same storage virtualization techniques as the SAN and CDMs, and deliver writable Terabyte-class environments in seconds, with minimal storage required for the clones, as Windows-based solution clones are infrastructure agnostic and run on-premise or on any public cloud.
Building Windows-based clones starts with a plain text configuration file, listing the database backups or database files, along with any SQL Server scripts to be run. In this example, the backups are located on a network file share, and a SQL Server script is used to implement data masking.
The image is built by submitting the configuration file and SQL script with the Windocks web UI. The file selector tool is used to select the files (1), which are highlighted and selected (2), and submitted with an image name (3).
Once the new image is available users provision containers by choosing a container name and “create.” The cloned database is delivered in seconds, irrespective of the size of the parent image. Developers can specify fixed connection strings, allowing the back-end to be provisioned with fixed ports and SQL credentials, simplifying integration with .NET, Java, or other front-end or middle-tier applications.
A container-based solution such as Windocks easily integrates with Jenkins and other Continuous Integration tooling and supports delivery of multi-tier environments that can include .NET or Java with SQL Server back-ends. Windocks also supports delivery of snapshots sourced from SANs and CDM systems, and will soon support delivery of these cloned environments to Microsoft containers and other application environments.
Modernize Data Delivery With Existing Infrastructure
Surveys indicate that limited access to database environments for testing is blocking efficient software development and delivery. These processes begin with a step-wise approach to implement and deliver cloned database environments. Clones are critical for supporting use of production database environments in Jenkins and other CI processes.
Database clones provide an opportunity to modernize development and test processes by using existing SAN infrastructure, or new CDM solutions that offer enterprise backup scalability, with deduplication, and improved ease of use by DBAs. For organizations that are interested to explore SQL Server DBA accessible tools, Windocks is a new option.As a software-based approach, Windocks supports free evaluation with a Windocks Community Edition, which is available here.
Opinions expressed by DZone contributors are their own.