Securing Generative AI Applications

Generative AI is a field of Artificial Intelligence that generates content leveraging a Large Language Model (LLM). In order to build a Gen AI application, the architecture comprises of:

1. Presentation or application layer which are WebApps or Mobile applications
2. Service layer to serve the requests, manage users, sessions, histories, etc.
3. Data layer to manage the data, store the history, user information, etc.
4. An ML Layer to provide the Gen AI functionality by interacting with LLM models
5. An ML Ops layer for continuous fine-tuning, training, or retraining the models

The Gen AI applications by many means are similar to non-Gen AI applications and the paradigm of security applicable to non-Gen AI applications is also applicable to Gen AI applications. However, the capabilities of LLM enable this application to reason, think, and generate output. This opens a whole new area of security at various layers which should be thought through and implemented to ensure the capabilities provided by Gen AI applications should not impact the users negatively by providing false or biased output.

In this article, I will be focusing on the ML Layer of applications and discuss various security measures one can take and should implement to ensure the best security practices.

ML Training

In order to build a Gen AI application, we can leverage existing models available in the marketplaces. These models are general-purpose and would work for various use cases. However, there are scenarios where one needs to train these models on organization/customer-specific data to leverage most out of these models. There are various ways of fine tuning and one can create an automated pipeline to perform the fine tuning at an interval. It should be the developer's responsibility to ensure no critical data like PII (Personally Identifiable Data), PHI (Protected Health Information), Financial data, etc. end up in the pool of fine-tuned data. 

A security control to detect such information and isolate it from ending up in fine-tuned data is a must. These security controls will evaluate the criticality of the data and take necessary actions to isolate such data. Metrics should be evaluated and alarms should be raised to ensure incoming data is not dealing with any kind of critical information.

Secure Testing

Once the model is in place, security testing integrated into the pipeline will test the output model against pre-defined test cases. These test cases should be reviewed and updated frequently to ensure the safety of fine-tuned models. A drift detection mechanism is implemented to ensure the model functionality for given inputs and the same hyperparameters. In case any hyperparameter values are updated, thorough input/output testing should be performed.

Output Validation

Any model output should not be handed over to the users directly. The model performs differently under different circumstances like prompts, hyperparameters, etc. These can provide an output that is not factually correct or could contain sensitive information. In the application, the developer should implement a sanitation layer to validate the model output. This layer could use rule-based, ML-based, or LLM-based methods to check the model output and validate and sanitize it. Once the confidence is high, the output will be served to the end users.

Prompt Validation

Prompts are very crucial for an LLM as these direct the model to act accordingly and provide output. The application developer should architect the application to ensure any communication with LLM is validated and sanitized at the service layer. There are various ways a malicious user can direct an LLM to perform unauthorized actions and produce malicious output. Prompt sanitation will ensure the sanctity of the application. This can be achieved using:

1. Create the prompt at the service layers
2. Perform prompt testing in the continuous integration pipeline
3. Sanitize the user inputs for the prompt creation
4. Validate the generated prompt (rule-based, LLM, or ML Based)
5. LLM output validation

E2E Testing

In the CI/CD world, it is important to ensure end-to-end testing. Developers must outline the testing details, and methodologies and keep updating the document to ensure any new changes do not compromise the security of the applications. These tests are security-focused and should be implemented in the pipeline. This will ensure any model changes or code changes do not introduce any new security bugs.

Finally, I will close this article with a recommendation for having an established process around:

1. Coding guidance: This will include any third-party library usage, coding practices, etc.
2. Review guidance: This will ensure minimum guidance to follow from the security aspect while performing the code reviews
3. Metrics and alarms: Implemented at various layers, this will ensure the health, security, and safety of the system
4. Model performance metrics: To ensure the optimal model performance with any continuous change
5. Mitigation plan: In case of any security incident, the developers/team should have a mitigation plan to quickly perform necessary changes to ensure uptime and reduce the damage.

Thanks for spending time!