How Data Encryption Can Simplify Infrastructure Architecture

Product and infrastructure engineering teams are not always aligned with the interests of security engineering teams. While product and infrastructure focus on driving business value and delivering practical solutions, security focuses on detection, prevention, and remediation, which can seem less immediately valuable. Like an insurance policy, it's not entirely obvious why it's worth the money or effort when there hasn't been an incident yet. 

Instead of the traditional cycle of identifying vulnerabilities, applying remediation, and following up through case management, I've found it much more effective to advocate for security solutions that also deliver business value. For example, using OAuth and IAM-based access instead of static keys and encryption instead of more granular access control can significantly simplify infrastructure, reduce complexity, and lessen the operational burden, making them very appealing to both product and platform engineering teams.

An Example: Replace Static Keys With IAM-Based OAuth

Traditionally, access between systems is implemented via static key-secret pairs. While common, this method often leads to reliability issues due to the complexity of managing key generation, rotation, and application lifecycle. Platform teams must also invest significant effort in monitoring and detecting anomalies to prevent unexpected key-secret compromises, such as accidental exposure via Slack or GitHub. Even when developers report and remediate leaks, the rotation process can be laborious. Worse, developers may consider it a low-risk leak, and the leak can go unreported.

According to ISO/IEC 27001:2022, A.9.1:

Organizations must implement policies and procedures to control access to information, ensuring it is only accessible to those with a legitimate need.

Platform teams have two choices: 

1. Add more complex access controls and approval processes.
2. Replace static key-secret pairs with IAM-based OAuth.

The first option can be tempting, as it involves simply adding a vendor like ServiceNow without much additional work. However, the second option, while requiring more implementation changes, is more secure and reduces the operational burden on application teams to update secrets, restart pods, and ensure secrets are picked up. In fact, several companies focusing on non-human identity authentication, such as P0 and Clutch, have recently emerged, highlighting the growing trend towards more secure and efficient authentication methods.

This example demonstrates how a different approach to security implementation can improve security standards, simplify infrastructure architecture, and enhance overall developer velocity.

The Case for Data Encryption

Data encryption is another example where, although security teams cannot simply "add a vendor," it significantly reduces complexity and implementation efforts across all platforms from both security and architecture design standpoints.

The typical data flow involves:

• Source application publishes data
• Data is sent to a transport layer (e.g., Kafka, Kinesis)
• Data is stored in a database (MySQL, Postgres), data warehouse (Redshift, Snowflake), or data lake (S3, Databricks)

Different solutions have different interpretations and implementations of "access control," leading platform teams to implement their own versions. This often results in fragmented implementations across the company. For security engineers, the more fragmented the implementations are, the more difficult it is to implement standardized governance, control, and monitoring, ultimately making the system less secure.

Infrastructure/Vendor Auth and Permission Comparison 

Conclusion

With data encryption, access is configured once with a crypto key and can then be assigned to individual workloads at different stages of the data flow. This significantly reduces the complexities involved in implementing and aligning permission policies across different platforms. Encryption ensures that data is consistently protected across all platforms, simplifying governance and control while enhancing overall security.