AI on Docker and Kubernetes: serving LLM apps
What you’ll learn
Section titled “What you’ll learn”- How to package an LLM-backed API as a small, non-root container image.
- How to run Ollama in Docker and Docker Compose: model storage, GPU access, and why the default port mapping is unsafe.
- How to deploy the model server and the API to Kubernetes with health probes, GPU requests, a network policy and autoscaling.
- How containerized AI services fail in production, and when a managed model API is the better choice.
This page owns running models in containers. Choosing a local model server and sizing hardware are on Running LLMs locally; Docker and Kubernetes basics are in the Docker and Kubernetes sections.
Summary
Section titled “Summary”An AI service in containers is usually two workloads with very different needs. The application (an API, an agent, a RAG service) is small, stateless and scales horizontally like any web service. The model server (Ollama, vLLM, llama.cpp) is large, stateful in practice (gigabytes of weights on disk, a model loaded in GPU memory), slow to start and expensive to replicate. Treat them separately: different images, different scaling rules, and a private network path between them.
If the application calls a hosted model API instead, there is no model server to run at all; only the first workload remains, plus careful secret handling.
Context and forces
Section titled “Context and forces”- Images and weights are big. Model files range from a few gigabytes to hundreds. Baking them into images makes builds and pulls slow; keeping them on a volume makes pods depend on storage.
- GPUs are scarce and must be requested explicitly. A container sees no GPU unless the runtime is configured and the workload asks for one, and many setups fall back to CPU silently.
- Start-up is slow. Loading a model can take tens of seconds to minutes, which interacts badly with naive health checks and autoscaling.
- Model APIs often have no authentication. Ollama’s API has none, and its official image listens on all interfaces.
- Cost scales with GPU hours, not requests, so idle replicas are expensive.
How to run the samples
Section titled “How to run the samples”The files below were tested on November 16, 2026 with Docker Engine 29.9.2, Docker Compose 5.6.1, the ollama/ollama:0.40.16 image and python:3.14-slim. The Compose stack was started and its endpoints called (output below). Model downloads were not possible in the test environment, so the summarize call was exercised only up to the “model not pulled” path. Kubernetes manifests were validated with kubeconform -strict against the Kubernetes 1.37.2 schemas; they were not applied to a cluster.
The design: the application container
Section titled “The design: the application container”The example service, for an invented shop called OrdersPortal, summarizes support tickets with a self-hosted model. It has a liveness endpoint that never touches the model, and a readiness endpoint that checks the model server and that the model is present.
# main.py - OrdersPortal ticket summarizer backed by a self-hosted Ollama serverimport os
import httpxfrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModel
OLLAMA_URL = os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434")MODEL = os.environ.get("SUMMARY_MODEL", "gemma4")TIMEOUT = float(os.environ.get("LLM_TIMEOUT_SECONDS", "60"))
app = FastAPI()
class Ticket(BaseModel): ticket_id: str text: str
@app.get("/healthz")def healthz() -> dict: """Liveness: the process is up. Never call the model here.""" return {"status": "ok"}
@app.get("/readyz")def readyz() -> dict: """Readiness: the model server answers and the model is present.""" try: version = httpx.get(f"{OLLAMA_URL}/api/version", timeout=3).json()["version"] models = [m["name"] for m in httpx.get(f"{OLLAMA_URL}/api/tags", timeout=3).json()["models"]] except httpx.HTTPError as exc: raise HTTPException(503, f"model server unreachable: {exc.__class__.__name__}") if not any(name.split(":")[0] == MODEL.split(":")[0] for name in models): raise HTTPException(503, f"model {MODEL} not pulled (server {version})") return {"status": "ready", "ollama": version, "model": MODEL}
@app.post("/summarize")def summarize(ticket: Ticket) -> dict: body = { "model": MODEL, "stream": False, "messages": [ {"role": "system", "content": "Summarize the support ticket in one sentence."}, {"role": "user", "content": ticket.text}, ], } try: reply = httpx.post(f"{OLLAMA_URL}/api/chat", json=body, timeout=TIMEOUT) reply.raise_for_status() except httpx.HTTPError as exc: raise HTTPException(502, f"model call failed: {exc.__class__.__name__}") return {"ticket_id": ticket.ticket_id, "summary": reply.json()["message"]["content"]}fastapi==0.163.1uvicorn==0.54.0httpx==0.28.1# Dockerfile# syntax=docker/dockerfile:1FROM python:3.14-slim AS buildWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.14-slimWORKDIR /appCOPY --from=build /install /usr/localCOPY main.py .RUN useradd --uid 10001 --no-create-home appUSER 10001EXPOSE 8000CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]Design choices worth copying:
- Pinned dependencies and a two-stage build. The build stage installs packages into
/install; the final image copies only the result, so build caches and tools stay out. - A fixed non-root user. UID 10001 matches the Kubernetes
runAsUserlater, so file permissions behave the same everywhere. - No model and no secrets in the image. The model server is reached over the network, configured by environment variables. If the service called a hosted API instead, its key would come from a secret at run time, never from the image or the build.
- Timeouts on every model call. Generation can take a minute; without a timeout, stuck requests pile up and exhaust workers.
(The test build had to trust a TLS-inspecting proxy’s certificate through a BuildKit secret mount on the pip install line; on a normal network the Dockerfile above builds unchanged.)
Ollama in Docker
Section titled “Ollama in Docker”The official image is ollama/ollama; pin a version tag (0.40.16 was current in November 2026) rather than latest, because defaults change between releases. The Ollama Docker documentation gives the basic commands:
# CPU only. Note the loopback binding: see the warning below.docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama:0.40.16
# NVIDIA GPU (requires the NVIDIA Container Toolkit on the host)docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama:0.40.16
# AMD GPU (ROCm image)docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama:0.40.16-rocm
# Pull a model into the volume, then run itdocker exec -it ollama ollama pull gemma4Three details matter more than they look:
- The image listens on all interfaces. Its Dockerfile sets
OLLAMA_HOST=0.0.0.0:11434, so the official example-p 11434:11434publishes an unauthenticated model API on every network interface of the host. Bind to127.0.0.1as above, or do not publish the port at all and reach it over a private container network. - Models live in
/root/.ollama. Mount a named volume there. Without it, every new container downloads the model again. - GPU fallback is silent. If the toolkit or flags are missing, Ollama runs on the CPU. Check the start-up log line
msg="inference compute": in the test run without a GPU it reportedlibrary=cpu.
Compose: the application and the model server together
Section titled “Compose: the application and the model server together”services: ollama: image: ollama/ollama:0.40.16 volumes: - ollama-models:/root/.ollama environment: OLLAMA_CONTEXT_LENGTH: "16384" OLLAMA_KEEP_ALIVE: "30m" OLLAMA_NO_CLOUD: "1" # No "ports:" - only services on this Compose network can reach the API. healthcheck: test: ["CMD", "ollama", "list"] interval: 10s timeout: 5s retries: 5 # Uncomment on a host with an NVIDIA GPU and the NVIDIA Container Toolkit: # deploy: # resources: # reservations: # devices: # - driver: nvidia # count: all # capabilities: [gpu]
summarizer: build: . environment: OLLAMA_BASE_URL: http://ollama:11434 SUMMARY_MODEL: gemma4 ports: - "127.0.0.1:8000:8000" # loopback only on the host depends_on: ollama: condition: service_healthy
volumes: ollama-models:Start it, then check the endpoints:
docker compose up -d --buildcurl -s localhost:8000/healthzcurl -s localhost:8000/readyzcurl -s --max-time 3 localhost:11434/api/version || echo "ollama not reachable from host"Output from the test run, before any model was pulled:
{"status":"ok"}{"detail":"model gemma4 not pulled (server 0.40.16)"}ollama not reachable from hostReadiness correctly reports that the model is missing, and the model server is not reachable from the host at all. After docker compose exec ollama ollama pull gemma4, readiness turns green. The environment variables set a larger context than the VRAM-based default, keep the model loaded for 30 minutes between requests, and disable Ollama’s cloud-hosted models so prompts stay on this machine; Running LLMs locally explains each setting. The GPU block uses the Compose device-reservation syntax from the Docker documentation; the uncommented variant passed docker compose config.
Kubernetes
Section titled “Kubernetes”The model server
Section titled “The model server”# ollama.yaml - a single Ollama server with persistent model storage and one GPUapiVersion: v1kind: PersistentVolumeClaimmetadata: name: ollama-models namespace: ordersportal-aispec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Gi---apiVersion: apps/v1kind: Deploymentmetadata: name: ollama namespace: ordersportal-aispec: replicas: 1 strategy: type: Recreate # the RWO volume can be mounted by one pod at a time selector: matchLabels: app: ollama template: metadata: labels: app: ollama spec: containers: - name: ollama image: ollama/ollama:0.40.16 ports: - containerPort: 11434 env: - name: OLLAMA_CONTEXT_LENGTH value: "16384" - name: OLLAMA_KEEP_ALIVE value: "-1" # keep the model loaded; this pod exists to serve it - name: OLLAMA_NUM_PARALLEL value: "4" - name: OLLAMA_NO_CLOUD value: "1" resources: requests: cpu: "2" memory: 16Gi limits: memory: 32Gi nvidia.com/gpu: 1 # needs the NVIDIA device plugin on the node readinessProbe: httpGet: path: /api/version port: 11434 periodSeconds: 10 livenessProbe: httpGet: path: /api/version port: 11434 initialDelaySeconds: 30 periodSeconds: 20 failureThreshold: 6 volumeMounts: - name: models mountPath: /root/.ollama volumes: - name: models persistentVolumeClaim: claimName: ollama-models---apiVersion: v1kind: Servicemetadata: name: ollama namespace: ordersportal-aispec: selector: app: ollama ports: - port: 11434 targetPort: 11434---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: name: ollama-from-summarizer-only namespace: ordersportal-aispec: podSelector: matchLabels: app: ollama policyTypes: ["Ingress"] ingress: - from: - podSelector: matchLabels: app: summarizer ports: - port: 11434Why it looks like this:
- GPUs go in
limits. Kubernetes exposes GPUs through vendor device plugins as resources such asnvidia.com/gpuoramd.com/gpu. They are specified inlimits(a request, if given, must equal the limit), and they are not shared between containers by default. Install the vendor’s device plugin and drivers on GPU nodes first; clusters with richer needs can use Dynamic Resource Allocation, stable since Kubernetes 1.34. Recreateand one replica. AReadWriteOncevolume can be attached to one node, so a rolling update would hang with two pods fighting over it. To scale the model server, give each replica its own copy of the weights (a StatefulSet with a volume per pod, or an init step that copies weights from object storage) and put a Service in front.- The NetworkPolicy is the authentication. Ollama has none, so only pods labeled
app: summarizermay connect. NetworkPolicy needs a network plugin that enforces it; on a cluster without one, the object is accepted and does nothing. - Probes check the server, not a generation.
/api/versionanswers in milliseconds. A probe that runs a prompt competes with real traffic and can kill a healthy but busy pod.
The application
Section titled “The application”# summarizer.yaml - the stateless API in front of the model serverapiVersion: apps/v1kind: Deploymentmetadata: name: summarizer namespace: ordersportal-aispec: replicas: 2 selector: matchLabels: app: summarizer template: metadata: labels: app: summarizer spec: securityContext: runAsNonRoot: true runAsUser: 10001 containers: - name: summarizer image: registry.example.test/ordersportal/summarizer:1.4.0 ports: - containerPort: 8000 env: - name: OLLAMA_BASE_URL value: http://ollama.ordersportal-ai.svc.cluster.local:11434 - name: SUMMARY_MODEL value: gemma4 resources: requests: cpu: 100m memory: 128Mi limits: memory: 256Mi readinessProbe: httpGet: path: /readyz port: 8000 periodSeconds: 10 livenessProbe: httpGet: path: /healthz port: 8000 periodSeconds: 20 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"]---apiVersion: v1kind: Servicemetadata: name: summarizer namespace: ordersportal-aispec: selector: app: summarizer ports: - port: 80 targetPort: 8000---apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: summarizer namespace: ordersportal-aispec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: summarizer minReplicas: 2 maxReplicas: 6 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70Validation:
kubeconform -strict -summary -kubernetes-version 1.37.2 ollama.yaml summarizer.yamlSummary: 7 resources found in 2 files - Valid: 7, Invalid: 0, Errors: 0, Skipped: 0Two points that are easy to miss. Readiness depends on the model server: if Ollama is down, the summarizer pods leave the Service instead of returning errors, while liveness stays green so Kubernetes does not restart them pointlessly. And the autoscaler scales only the cheap, stateless tier. CPU on the summarizer is a weak signal of model load, because the work happens on the GPU; for real traffic, scale on request concurrency or queue depth through custom metrics, and remember that adding API pods cannot make one GPU faster.
Using a hosted model instead
Section titled “Using a hosted model instead”If the service calls a hosted API (Anthropic, OpenAI, Google, or a cloud platform such as Amazon Bedrock), drop the model server and the GPU, and load the key from a Secret:
# Fragment: container env for a hosted-model client; not applied to a clusterenv: - name: ANTHROPIC_API_KEY valueFrom: secretKeyRef: name: llm-api-keys key: anthropicCreate the Secret from your secret manager or CI, not by typing the key into a shell history, and prefer workload identity (IAM roles for service accounts, Workload Identity Federation) where the provider supports it, so no long-lived key exists at all.
Trade-offs and alternatives
Section titled “Trade-offs and alternatives”| Option | Gain | Cost | Choose it when |
|---|---|---|---|
| Hosted model API from a container | No GPUs, newest models, elastic capacity | Per-token cost; data leaves your network under the provider’s terms | Most applications |
| Ollama in Compose on one host | Simple, private, cheap to try | One machine; limited concurrency | Internal tools, development, small teams |
| Ollama on Kubernetes | Private, uses existing cluster tooling | GPU node management; one model server per volume | Moderate internal load on an existing cluster |
vLLM on Kubernetes (vllm/vllm-openai image, v0.33.0 in November 2026) |
Much higher throughput under concurrency | More VRAM per model, more tuning | Shared GPU service for many users |
| Managed platform endpoints (Bedrock, Vertex AI, Azure AI Foundry) | Provider-run scaling inside your cloud account | Platform lock-in, regional model availability | Organizations standardized on one cloud |
Serverless functions (for example AWS Lambda, with up to 15 minutes per invocation and 10 GB of memory) suit short calls to hosted APIs, not self-hosted models: there is no GPU, and cold starts would reload weights.
Failure modes and operations
Section titled “Failure modes and operations”- Silent CPU fallback. Missing drivers, toolkit or GPU request leaves the model on the CPU, many times slower. Alert on the start-up log line and on tokens per second.
- Exposed model API.
-p 11434:11434or a LoadBalancer Service publishes an unauthenticated API. Use loopback bindings, ClusterIP Services and NetworkPolicy. - Re-downloading models. No volume, or a new volume per pod, means every restart pulls gigabytes. Persist the model directory, or pre-load weights from object storage.
- Probes that generate text. They time out under load and restart healthy pods. Probe a cheap endpoint.
- Cold starts after scale-up. A new model-server pod can take minutes to become useful. Keep a minimum replica count and scale ahead of known peaks.
- Unpinned images and models.
latesttags for the server and moving model tags change behavior without a deploy. Pin image versions and model digests, and re-run evaluations before changing either (regression testing). - Context set too small. A low context length truncates long tickets or agent histories without an error; set it explicitly and size memory for it.
- Logs full of customer data. Prompts and outputs in container logs inherit the log system’s retention and access. Log metadata by default and follow your data rules (AI governance).
When not to use it
Section titled “When not to use it”- You have no GPU operations experience and modest privacy needs. A hosted API in a plain container is cheaper and better than a self-run model.
- Traffic is spiky and low on average. GPUs that sit idle most of the day cost more than per-token pricing.
- You need frontier-model quality. The largest proprietary models are available only as hosted APIs.
- A single developer needs a local model. Run Ollama directly on the laptop (Running LLMs locally); containers add little.
Interview questions
Section titled “Interview questions”1. Why separate the application and the model server into different deployments? They scale and fail differently. The application is stateless and cheap, so it scales on traffic; the model server is GPU-bound, slow to start and tied to large weight files. Separate deployments let each have the right probes, resources and update strategy, and let you swap a self-hosted model for a hosted API without touching the application’s scaling.
2. How do you give a container a GPU in Docker and in Kubernetes?
In Docker, install the vendor’s container toolkit on the host and run with --gpus=all (or a Compose device reservation with capabilities: [gpu]). In Kubernetes, install drivers and the vendor’s device plugin on GPU nodes, then request the resource (nvidia.com/gpu: 1) under limits. In both cases, verify from inside the workload that the GPU is used, because many servers fall back to CPU silently.
3. What should liveness and readiness probes check for an LLM-backed API? Liveness checks only that the process responds, so a slow model never causes restarts. Readiness checks dependencies: that the model server responds and the expected model is available. Neither should run a generation.
4. Senior scenario: after moving OrdersPortal’s summarizer to Kubernetes, latency is fine at night and terrible at peak, and the HPA keeps adding API pods. What is happening and what do you change? The bottleneck is the single model server, not the API tier. More API pods just queue more requests on one GPU, and CPU-based autoscaling reacts to waiting threads. Measure GPU utilization, queue depth and tokens per second. Then cap concurrency at the API (return 429 or queue), raise the model server’s parallelism within its memory, add model-server replicas with their own copies of the weights, or move to a higher-throughput server such as vLLM. Scale the API on request concurrency, not CPU, and consider routing overflow to a hosted model.
5. Senior scenario: a security review finds an Ollama endpoint reachable from the office network. How did it happen and how do you prevent it?
Most likely a port mapping like -p 11434:11434 (the image listens on all interfaces) or a Service of type LoadBalancer or NodePort. Remove the exposure, check logs for unknown clients, and rotate anything the endpoint could reveal. Prevent it with loopback-only bindings in Compose, ClusterIP plus NetworkPolicy in Kubernetes, an admission policy that blocks external Services for model-server labels, and an authenticating gateway when access from outside the cluster is genuinely needed.
Key takeaways
Section titled “Key takeaways”- Run the application and the model server as separate workloads with separate scaling and probes.
- Pin image and model versions; persist model files on volumes.
- Ollama’s image listens on all interfaces and has no authentication: bind to loopback, use private networks and NetworkPolicy.
- Request GPUs explicitly and verify they are used.
- Autoscale the stateless tier on concurrency; scale model servers deliberately.
- A hosted model API in a plain container is the right default unless privacy or cost at volume says otherwise.
Related pages
Section titled “Related pages”- Running LLMs locally
- Docker overview and multi-stage builds
- Docker Compose
- Kubernetes introduction, deployments and ConfigMaps and Secrets
- AI governance
- LLM model selection
Official references
Section titled “Official references”- Ollama, Docker documentation (source: ollama/ollama
docs/docker.mdx) and the officialDockerfile;ollama/ollamaon Docker Hub (0.40.16) - Docker, Enable GPU access with Docker Compose
- NVIDIA, Container Toolkit installation guide
- Kubernetes, Schedule GPUs, Dynamic Resource Allocation, Network policies, Configure liveness and readiness probes and Horizontal Pod Autoscaling (Kubernetes 1.37)
vllm/vllm-openaion Docker Hub (v0.33.0)- kubeconform (v0.8.0, used for validation)