Designing a Secure Architecture for Distributed Systems

Securing distributed systems is a complex challenge due to the diversity and scale of components involved. With multiple services interacting across potentially unsecured networks, the risk of unauthorized access and data breaches increases significantly. This article explores a practical approach to securing distributed systems using an open-source project. The project demonstrates how to integrate several security mechanisms and technologies to tackle common security challenges such as authentication, authorization, and secure communication.

Understanding Security Challenges in Distributed Systems

Distributed systems involve multiple services or microservices that must communicate securely across a network. Key security challenges in such architectures include:

1. Secure communication: Ensuring that data transmitted between services is encrypted and safe from eavesdropping or tampering
2. Authentication: Verifying the identities of users and services to prevent unauthorized access
3. Authorization: Controlling what authenticated users and services are allowed to do, based on their roles and permissions
4. Policy enforcement: Implementing fine-grained access controls and policies that govern service-to-service and user interactions
5. Certificate management: Managing digital certificates for encrypting data and establishing trust between services

This open-source project addresses these challenges using several integrated technologies and solutions.

Project Setup and Configuration

The project begins with setting up a secure environment using shell scripts and Docker. The setup involves provisioning digital certificates and starting the necessary services to ensure all components are ready for secure communication.

Steps to Set Up the Environment

1. Provisioning Certificates

The project uses a shell script (provisioning.sh) to simulate a Certificate Authority (CA) and generate the necessary certificates for the services.

   ./provisioning.sh

2. Launching Services

Docker Compose is used to start all services defined in the project, ensuring they are configured correctly for secure operation.

   docker-compose up

3. Testing Service-to-Service Communication

To validate service-to-service communication using certificates and JWT tokens, the test_services.sh script is provided. This script demonstrates how different services interact securely using their assigned certificates.

Solving Security Challenges in Distributed Systems

The project integrates several key technologies to address the primary security challenges mentioned earlier. Here's how each challenge is tackled:

1. Secure Communication With Mutual TLS (mTLS)

Challenge

In a distributed system, services must communicate securely to prevent unauthorized access and data breaches.

Solution

The project uses Mutual TLS (mTLS) to secure communication between services. mTLS ensures that both the client and server authenticate each other using their respective certificates. This mutual authentication prevents unauthorized services from communicating with legitimate services.

Implementation

Nginx is configured as a reverse proxy to handle mTLS. It requires both client and server certificates for establishing a secure connection, ensuring that data transmitted between services remains confidential and tamper-proof.

2. Authentication With Keycloak

Challenge

Properly authenticating users and services is critical to prevent unauthorized access.

Solution

The project leverages Keycloak, an open-source identity and access management solution, to manage authentication. Keycloak supports multiple authentication methods, including OpenID Connect and client credentials, making it suitable for both user and service authentication.

• User Authentication:
  Users are authenticated using OpenID Connect. Keycloak is configured with a client (appTest-login-client) that handles user authentication flows, including login, token issuance, and callback handling.
• Service Authentication:
  For service-to-service authentication, the project uses a Keycloak client (client_credentials-test) configured for the client credentials grant type. This method is ideal for authenticating services without user intervention.

Authentication Flow Example

1. Users navigate to the login page.
2. After successful login, Keycloak redirects the user to a callback page with an authorization code.
3. The authorization code is then exchanged for a JWT token, which is used for subsequent requests. The authn.js file in the nginx/njs directory provides a detailed implementation of this flow.

Service Authentication Example Using Client Credentials

curl -X POST "http://localhost:9000/realms/tenantA/protocol/openid-connect/token" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -d "grant_type=client_credentials" \
     -d "client_id=client_credentials-test" \
     -d "client_secret=your-client-secret-here"

3. User Authorization With Open Policy Agent (OPA) and JWT

Challenge

Enforcing fine-grained access controls to ensure that authenticated users and services only have access to authorized resources

Solution

The project utilizes a combination of Open Policy Agent (OPA) and JWT tokens to enforce authorization policies. The project demostrate three different strategies for JWT validation to ensure robust security:

1. Retrieving certificates from Keycloak: Fetches the certificates dynamically from Keycloak to validate the token.
2. Using x5t (Thumbprint): Uses the thumbprint embedded in the token to retrieve the public key from a local trust store.
3. Embedded certificate validation: Validates the token using an embedded certificate, ensuring the certificate is validated against a trusted Certificate Authority (CA).

Refer to the nginx/njs/token.js file for the detailed implementation of these strategies.

4. Policy Enforcement With Open Policy Agent (OPA)

Challenge

Implementing dynamic and flexible access control policies for both services and users

Solution

OPA is used to enforce fine-grained policies for access control. Policies are written in a declarative language (Rego) and stored in the opa/ directory. These policies dictate the conditions under which services can communicate and users can access resources, ensuring that access controls are consistently applied across the system.

5. Certificate Management

Challenge

Managing digital certificates for services to establish trust and secure communications

Solution:
The project includes a robust certificate management system. A shell script (provisioning.sh) is used to simulate a Certificate Authority (CA) and generate certificates for each service. This approach simplifies certificate management and ensures that all services have the necessary credentials for secure communication.

We also added an endpoint to update the service certificate without the need of nginx restart.

curl --insecure  https://localhost/certs  --cert certificates/gen/serviceA/client.crt --key certificates/gen/serviceA/client.key -F cert=@certificates/gen/serviceA/client.crt -F key=@certificates/gen/serviceA/client.key

Conclusion

Building a secure distributed system requires careful consideration of various security aspects, including secure communication, authentication, authorization, policy enforcement, and certificate management. This open-source project provides a comprehensive example of how to integrate multiple security mechanisms to address these challenges effectively.

By following the setup and configurations demonstrated in this project, developers can leverage mutual TLS, Keycloak, Open Policy Agent, and Nginx to build a robust security architecture. These technologies, when combined, provide a strong foundation for securing distributed systems against a wide range of threats, ensuring both data protection and secure access control.