Deploying Small Language Models on Sovereign Cloud

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge.

Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. 

The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement.

This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster.

The Target Infrastructure: GDC Air-Gapped

Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises.

However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up.

Base Image Curation With Docker DHI

To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation.

Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries.

# Verify local DHI CLI installation
docker dhi --help

# Mirror a hardened Python or Java base image to your local org
docker dhi mirror start --org enterprise-registry \  dhi/python,enterprise-registry/dhi-python

By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies.

Generating the SLM Artifact and SBOM

As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages.

The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images.

# Retrieve the SLSA provenance and SPDX SBOM for audit
docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json

This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately.

Deploying to the Sovereign Edge

With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device.

Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services.

By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements.

The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.