The Data Bimodal – Agility vs Security
See the challenges of two contradictory but necessary data delivery approaches.
Join the DZone community and get the full member experience.Join For Free
You’ve probably heard the term bimodal before.
It is a popular term, coined (maybe adopted) by Gartner, to describe the two ways of approaching software delivery. To quote: “Bimodal is the practice of managing two separate but coherent styles of work: one focused on predictability; the other on exploration.”
However, and perhaps more literally, Bimodal literally means "two modes."
In this article, I’d like to stick with the concept of software delivery; however, I’d like to talk about an area in software engineering that is typically most difficult: the management and delivery of your “Data” components.
An area that is not just challenging due to the natural complexity of the data itself, which off course may have heterogeneous data architectures (structured, unstructured, relational, and non-relational etc), complex schemas (thousands of tables, columns, and relationships), and huge data sets (potentially billions of data points).
But also due to opposing organizational priorities, that is
- one: "data delivery and agility" over two: "data compliance and security"
Two modes that inevitably must be married if you are to be successful.
So with that in mind, let’s discuss the needs of these two modes.
Mode 1: Agile DevOps Oriented
Key focus: Rapid delivery of data to support your software engineering effort. Ideally supportive of your Software Delivery Teams (like Development and Testing) and your existing automation efforts, i.e., CICD and DevOps methods.
Key Mode Considerations:
- Data CI — Rapid Data Creation/Fabrication (think Continuous Integration)
Tip: A key objective of this requirement is the rapid build of “Fit for Purpose” data.
- Data CD — Rapid Data Provisioning (think Continuous Delivery)
Tip: A key objective of this requirement is rapid refresh and rollback.
Other Important Delivery Enablers:
- Data Sub-Setting and/or virtualization to reduce storage and/or processing.
- Data Requirements Management ensures needed data is provisioned.
- Data Mining ensures correct data can be found quickly.
- Data Bookings to ensure data can be reserved and safely used by your tribes.
Key Benefits of this Mode
- Improved collaboration, i.e., data teams, is a core part of the solution.
- Standardized through the use of consistent data automation.
- Productivity across development and test i.e., less time doing wasteful activities.
- Better quality, i.e., data based on requirement and provided consistently.
- Innovative, i.e., promotes the use of technology to improve difficult operational activities.
- Release / Deliver Faster
- Because the preparation of new data can be difficult, development teams may ‘negatively’ gravitate to recycling the old data. Note: Stale data is a classic DataOps anti-pattern that will slowly impact your ability to deliver quality outcomes.
- The platform that data resides on may not be an “island”. It is often necessary to understand and align data preparation activities across multiple platform teams to ensure end-to-end integrity and testability.
- ‘Fit for purpose data’ is often difficult to produce in larger more complex systems.
Note: Older monolithic system often has very complex data patterns, derived over time (and changing functionality). This data is not easily replicated by standard fabrication methods nor easily subset (when using production cuts).
Mode 2 — Security DevSec Oriented
Key focus: Securitization of your data, specifically customer data, to ensure sensitive information, for example, Personally Identifiable Information (PII), is not exposed to unauthorized individuals.
Key Mode Considerations:
- Risk Profiling — Discover your data and risks so you can target remediation.
Tip: Focus on actual risks that expose PII. Securing everything may take too long.
- Data Masking — Remediate risks so sensitive information can’t be reversed*.
Tip: In scenarios where you may need to reverse, then use encryption and secret keys.
- Data Validation — Ensure that the data being provisioned is indeed compliant.
Tip: Don’t assume the data masking was successful or complete. Data is complex.
Key Benefits of this Mode
- Avoid data leakage.
- Improved cross-team collaboration i.e., IT, Compliance, and Security.
- Data Literacy i.e., visibility and understanding of data and risks.
- Ensure data is secured inside non-Production before being made available.
- Promotes ongoing improvement of DevSec automation.
- Developers often want to focus on getting the job done i.e., delivery (refer to mode 1). This can often result in security tasks being neglected and exposures sneaking through the net.
- Developers don’t necessarily speak the same language as security or compliance teams. An absence of good collaboration between teams can result in objectives being lost in translation. Objectives need to be clearly discussed, baselined, and prioritized collectively.
- Lack of common-sense, from IT, Security or Compliance, can result in over-engineering a security solution. In over-engineering, I mean it is badly scoped and delivering no reasonable extra value. Poor or non-sensible security definitions can drive up project costs and cause ongoing delivery delays.
- Security tasks are often time-consuming due to a lack of historical investment. That is they have not been automated and/or the process is not easily repeatable. Non-automated and standardized processes will make upkeep difficult and will likely lead to disrepair and technical debt.
Bringing It Together
So we have reviewed the two key Data Delivery modes.
The first mode focussed on Data Delivery and Agility through the embracement of DevOps methods. The second mode focussed on Data Compliance and Security through the embracement of DevSec methods.
However, as I am sure you would realize, the question is not "which mode?"
Instead, the question must be:
- "How does our organization ensure that both modes are done effectively?”
Take a step back and consider how you’re doing it.
Opinions expressed by DZone contributors are their own.