Air-gapped systems: running critical software with no internet

Some systems must never talk to the internet: a live broadcast control room, a hospital network, a production line, a lab with export-controlled data. For years that meant giving up modern tooling. It doesn't anymore. You can run Kubernetes, GitOps, signed images and proper monitoring inside an enclave that has no route out, as long as you treat the boundary as a process and not just as a missing cable. This is how I build and run them.

Dominik Heinrich

November 7, 202610 min read

1. What an air gap really is

People use the word for three very different things, and the difference matters. A physical air gap means no network path at all: data only moves on media that someone carries across. A one-way gateway, often called a data diode, enforces direction in hardware, so data can flow into the enclave, or out of it for monitoring, but never back. A logical gap is a firewall that is supposed to block everything, which is segmentation, not isolation: one wrong rule and the gap is gone. NIST SP 800-82, the US guide for operational technology security, treats segmentation and unidirectional gateways as separate tools for exactly this reason. Before you design anything, write down which of the three you actually have, because your update process, your monitoring and your threat model all follow from it.

Tip from practice

Draw the real data flows on one page, including USB sticks, laptops of service technicians and remote support tools. That page usually shows the gap you didn't know you had.

2. When you need one, and when you don't

Air gaps make sense where an incident would be catastrophic or where rules demand it: classified and defence networks, industrial control systems in energy and manufacturing, live broadcast production where a compromise takes you off air, medical devices, payment and trading cores, and research data under export control. They cost something real: updates arrive in batches instead of continuously, every new tool needs an offline story, and people need discipline at the boundary. That is why I decide per system, not per company. Usually only the crown jewels go behind the gap, while everything around them stays connected and well segmented. The NIST SP 800-53 controls on boundary protection and information flow enforcement are a good checklist to argue that decision with your auditors.

3. The architecture: three zones and one direction

Every setup I build has the same three zones. The connected zone pulls what you need from the outside world: container images, Helm charts, operating system updates, Git repositories. The transfer zone is a small, hardened staging area where everything is scanned for vulnerabilities, described in a software bill of materials and signed, before it is packed into one bundle with a checksum manifest. The enclave only ever receives that bundle, verifies it again and serves it from its own registry and its own Git. Tools such as Zarf describe the idea well with their own motto, develop connected and deploy disconnected, and they package exactly this flow. Whether you use such a tool or a few scripts, the rule is the same: nothing inside the enclave ever reaches out, everything comes in through one controlled door.

Air-gapped Kubernetes: a connected zone with vendor registries, Git and updates; a transfer zone that scans, creates an SBOM and signs; a one-way transfer into an enclave with a private registry, internal Git with GitOps, a Kubernetes cluster running Linux pods and Windows VMs on KubeVirt, internal monitoring, a local time source and break-glass access, and no route to the internet. — How software reaches an air-gapped Kubernetes enclave: scanned and signed outside, moved one way, verified again inside. Click to open full size.

4. Images, charts and SBOMs, pinned and signed

Mirror only what you run, never a whole public registry. Pin every image by digest and every chart by version, because tags move and there is no way to check later what a tag meant on the day you copied it. kubeadm can list exactly which images a given Kubernetes version needs, and Helm can pull a chart as a file. Generate an SBOM in SPDX or CycloneDX format for the whole bundle; CISA describes SBOMs as the ingredient list that lets you answer the question 'are we affected?' in minutes when the next big vulnerability lands. Then sign. In the connected zone a key pair with cosign works well: the private key stays outside, ideally on a hardware token, and only the public key enters the enclave.

bash

# Connected zone: build one transfer bundle, pinned and signed
set -euo pipefail
mkdir -p bundle/images bundle/charts

kubeadm config images list --kubernetes-version "$K8S_VERSION" > bundle/k8s-images.txt
while read -r img; do
  skopeo copy --all "docker://$img" "oci-archive:bundle/images/$(echo "$img" | tr '/:' '__').tar"
done < bundle/k8s-images.txt

helm pull ingress-nginx/ingress-nginx --version "$CHART_VERSION" -d bundle/charts
syft dir:bundle -o spdx-json > bundle/sbom.spdx.json      # what is inside

(cd bundle && find . -type f ! -name SHA256SUMS -exec sha256sum {} + > SHA256SUMS)
cosign sign-blob --key cosign.key --bundle bundle/SHA256SUMS.bundle bundle/SHA256SUMS

Tip from practice

Scan in the transfer zone and refuse the bundle if a critical finding has no documented decision. Fixing it inside the enclave later costs ten times more.

5. Crossing the gap: one way, verified twice

The bundle crosses either through a data diode or on signed removable media, carried under a two-person rule. On the enclave side it lands in quarantine first. There, a separate host verifies the signature with the public key, checks every file against the checksum manifest and only then imports the images into the private registry and the charts and manifests into internal Git. Because the enclave cannot reach a public transparency log, verification runs against the key alone, which cosign supports explicitly. Every transfer gets a log entry: who, when, what, which checksums, which approval. Agree on a fixed rhythm, for example one bundle a week, plus a documented emergency path for critical security fixes, so nobody is tempted to plug in a cable 'just this once'.

bash

# Enclave side, quarantine host: verify before anything is imported
set -euo pipefail
cd /quarantine/bundle

# Signature against the public key only (no public transparency log reachable)
cosign verify-blob --key /etc/airgap/cosign.pub \
  --bundle SHA256SUMS.bundle --insecure-ignore-tlog=true SHA256SUMS

sha256sum -c SHA256SUMS            # every file, byte for byte

for f in images/*.tar; do
  skopeo copy --all "oci-archive:$f" "docker://registry.enclave.local/mirror/$(basename "$f" .tar)"
done

6. Running Kubernetes inside the enclave

Inside, a private registry such as Harbor is the single source for images, with vulnerability scanning and policies that stop images failing your rules from being pulled. Every node's container runtime points at it as a mirror, so even a manifest that still says docker.io resolves internally and never tries to leave. Deployments come from internal Git through GitOps with Flux or Argo CD, which gives you the same review, history and rollback you would have in the cloud. At admission, a policy engine such as Kyverno, which supports Sigstore cosign signatures, rejects any image that is not signed by your key or does not come from the internal registry. The result feels surprisingly normal for the teams: pull requests, pipelines and dashboards, just with a slower and better-checked supply chain behind them.

toml

# /etc/containerd/certs.d/docker.io/hosts.toml on every node:
# anything that asks for docker.io is served by the internal Harbor proxy project
server = "https://registry-1.docker.io"

[host."https://registry.enclave.local/v2/mirror-docker.io"]
  capabilities = ["pull", "resolve"]
  override_path = true

Tip from practice

Run the admission policy in audit mode for a week first. You will find the one forgotten sidecar image before it blocks a production rollout.

Cloud Native Deployments in Air Gapped Environments, CNCF (YouTube). Loads from YouTube only after you click.

7. Windows and legacy software: VMs on KubeVirt

Real enclaves are rarely all containers. Control software, vendor tools and older systems often only run on Windows. KubeVirt runs those as virtual machines right next to your pods, on the same nodes, with the same GitOps, the same monitoring and the same backup routine, so you don't need a second virtualization platform with its own update problem. I have run exactly this mix: hybrid Kubernetes across bare metal and the cloud for broadcast workloads, Windows applications as VMs on KubeVirt, and the sensitive parts air-gapped. Plan three things early: the virtio drivers Windows needs inside KubeVirt, a way to bring Windows updates in as part of your bundle, and licence activation that works without internet, for example through an activation service inside the enclave.

8. The boring parts that break first

In my experience, air-gapped platforms rarely fail on the clever parts. They fail on time, certificates and visibility. Without a local time source, clocks drift until TLS and tokens break, so bring your own NTP or PTP source, for example a GPS clock. Run an internal certificate authority and automate renewals inside, because nobody can fetch a certificate from the internet. Keep monitoring, logs and alerting inside the enclave; if you have a one-way gateway, you can send metrics outward to a central team without opening anything inward. Test backups by restoring them inside, regularly. And define break-glass access before the first incident: time-limited, two people, fully logged. Even my small offline robot, an ESP32 with local speech recognition and a local language model, taught me the same lesson: design for no network from day one, and the rest becomes routine.

Tip from practice

Write the runbook for 'the transfer path is down for two weeks' before you need it: which fixes can wait, which need the emergency path, who decides.

9. A realistic rollout

Don't start by air-gapping everything. Pick one system that clearly needs it and build the transfer path for it end to end: bundle, scan, sign, carry, verify, import, deploy. Run that loop a few times with harmless updates until it is boring and documented, then move the real workloads in. Measure two numbers from day one: how long a critical security fix takes from release to running inside, and how many transfers were rejected and why. The first tells management what the gap costs in speed, the second proves that the controls actually work. After a few months most teams settle into a rhythm where weekly bundles feel routine and the emergency path is used a handful of times a year, which is exactly where you want to be.

Takeaway
An air gap is only as strong as the process at its edge: make every crossing scanned, signed and verified, and give everything else a home inside.

Real projects behind this

Hybrid Kubernetes for live broadcast

Try it yourself

Air-gap check

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides

Air-gapped systems: running critical software with no internet

1. What an air gap really is

2. When you need one, and when you don't

Is your air gap more than a firewall rule?

3. The architecture: three zones and one direction

4. Images, charts and SBOMs, pinned and signed

5. Crossing the gap: one way, verified twice

6. Running Kubernetes inside the enclave

7. Windows and legacy software: VMs on KubeVirt

8. The boring parts that break first

9. A realistic rollout

Get in touch