Hosting AI Agents on Docker & Kubernetes

Why Containerise AI Applications?

AI applications — LLM apps, agent pipelines, RAG services — have the same operational challenges as any software: dependency management, scaling, environment consistency, and deployment reproducibility. Containers solve all of these.

Additional AI-specific reasons:

• GPU isolation — containers can request specific GPU resources
• Model versioning — bundle a specific model checkpoint with its serving code
• Reproducibility — the same container behaves identically in dev, staging, and production
• Scaling — run 10 instances of your agent service during peak load, scale down at night

---

Containerising a Python LLM App

Basic Dockerfile

# Dockerfile
FROM python:3.12-slim

WORKDIR /app

# Install dependencies first (cached layer)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

# main.py — FastAPI AI agent service
from fastapi import FastAPI
from pydantic import BaseModel
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

app = FastAPI()
llm = ChatOpenAI(model="gpt-4o")
chain = ChatPromptTemplate.from_template("Answer concisely: {question}") | llm | StrOutputParser()

class Question(BaseModel):
    question: str

@app.post("/ask")
async def ask(q: Question):
    return {"answer": await chain.ainvoke({"question": q.question})}

docker build -t my-ai-service .
docker run -p 8000:8000 -e OPENAI_API_KEY=sk-... my-ai-service

Multi-stage Build (smaller image)

FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/app/deps -r requirements.txt

FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /app/deps /app/deps
COPY . .
ENV PYTHONPATH=/app/deps
EXPOSE 8000
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

---

Running Ollama (Local LLM) in Docker

docker-compose.yml

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    # For NVIDIA GPU support:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  ai-service:
    build: .
    ports:
      - "8000:8000"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
    depends_on:
      - ollama

volumes:
  ollama_models:

# Using Ollama from the ai-service container
from langchain_ollama import ChatOllama
import os

llm = ChatOllama(
    model="llama3.2",
    base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
)

docker compose up -d
# Pull the model into the running Ollama container
docker compose exec ollama ollama pull llama3.2

---

Kubernetes Deployment

Deploying an AI Service

k8s/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-agent-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-agent-service
  template:
    metadata:
      labels:
        app: ai-agent-service
    spec:
      containers:
        - name: ai-agent-service
          image: myregistry/ai-agent-service:v1.2.0
          ports:
            - containerPort: 8000
          env:
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ai-secrets
                  key: openai-api-key
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "1000m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: ai-agent-service
spec:
  selector:
    app: ai-agent-service
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

Storing Secrets

# Create secret for API keys
kubectl create secret generic ai-secrets \
  --from-literal=openai-api-key=sk-... \
  --from-literal=langchain-api-key=ls__...

GPU Node for Local Models

# GPU deployment for Ollama on Kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 1
  template:
    spec:
      nodeSelector:
        accelerator: nvidia-gpu    # Schedule on GPU nodes
      containers:
        - name: ollama
          image: ollama/ollama:latest
          resources:
            limits:
              nvidia.com/gpu: 1    # Request 1 GPU
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models-pvc

Horizontal Pod Autoscaler

Scale agent instances based on CPU/memory or custom metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-agent-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-agent-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

---

Deploying to AWS

AWS ECS (Fargate) — Simplest Managed Container

# Push image to ECR
aws ecr create-repository --repository-name ai-agent-service
docker tag my-ai-service:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/ai-agent-service:latest
aws ecr get-login-password | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/ai-agent-service:latest

// ECS Task Definition (excerpt)
{
  "family": "ai-agent-service",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "containerDefinitions": [{
    "name": "ai-agent-service",
    "image": "123456789.dkr.ecr.us-east-1.amazonaws.com/ai-agent-service:latest",
    "portMappings": [{"containerPort": 8000}],
    "secrets": [{
      "name": "OPENAI_API_KEY",
      "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789:secret:openai-api-key"
    }]
  }]
}

AWS Lambda — Serverless AI

# lambda_function.py — lightweight agent on Lambda
import json
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4o")   # initialised outside handler (warm start caching)

def handler(event, context):
    question = json.loads(event["body"])["question"]
    chain = ChatPromptTemplate.from_template("{q}") | llm
    response = chain.invoke({"q": question})
    return {
        "statusCode": 200,
        "body": json.dumps({"answer": response.content})
    }

Lambda considerations for AI:

• Cold starts — first invocation is slow (loading model weights). Use Provisioned Concurrency for latency-sensitive workloads
• Timeout — max 15 minutes; LLM calls can be slow for complex tasks
• Memory — up to 10GB; sufficient for small models and all cloud LLM API calls
• Container images — use up to 10GB container images for heavier dependencies

AWS Bedrock — Managed Foundation Models

Skip running your own LLM and use AWS Bedrock to call Claude, Llama, or Titan models directly:

from langchain_aws import ChatBedrock

llm = ChatBedrock(
    model_id="anthropic.claude-sonnet-4-6-20251001-v1:0",
    region_name="us-east-1"
)

response = llm.invoke("Explain retrieval augmented generation")

Benefits: no model hosting, automatic scaling, AWS IAM-based auth, data stays in your AWS account.

---

CI/CD Pipeline for AI Services

.github/workflows/deploy-ai-service.yml

name: Build and Deploy AI Service

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t ai-agent-service .

      - name: Run evaluation tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGCHAIN_API_KEY: ${{ secrets.LANGCHAIN_API_KEY }}
        run: |
          pip install pytest langsmith
          pytest tests/evals/ --tb=short

      - name: Push to ECR
        run: |
          aws ecr get-login-password | docker login ...
          docker push ...

      - name: Deploy to ECS
        run: aws ecs update-service --cluster prod --service ai-agent-service --force-new-deployment

---

Edge Deployment

For AI at the edge (IoT devices, on-premise, air-gapped):

• NVIDIA Jetson — GPU-enabled edge devices running containerised AI workloads
• K3s — lightweight Kubernetes for edge clusters
• ONNX Runtime — run optimised model inference on CPU/GPU with small footprint
• Ollama on-premise — full local LLM stack on your own server, no cloud dependency

# Run Ollama as a system service (on-premise server)
sudo systemctl enable ollama
sudo systemctl start ollama
# Serves the same API as cloud Ollama — your apps need zero changes