Running Local LLMs

Why Run a Local LLM?

Cloud LLMs (ChatGPT, Claude, Gemini) are powerful but have constraints:

Reason	Details
Data privacy	Confidential code, legal documents, patient data — can’t send to external APIs
Cost	High-volume inference at scale gets expensive quickly
Offline use	Air-gapped environments, travel, unreliable internet
Latency	Local inference can be faster for small models
Customisation	Fine-tune on your own data without sharing it with a provider
Compliance	Data residency requirements — data must not leave a specific region

The trade-off: local models are smaller and generally less capable than frontier models (GPT-4o, Claude Sonnet). But for many tasks — code assistance, summarisation, classification, chat — they’re more than sufficient.

---

Ollama — The Simplest Way

Ollama is the easiest way to run LLMs locally. One command to install, one command to run.

Installation

# macOS / Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows — download installer from ollama.ai
# Or via winget:
winget install Ollama.Ollama

Running Models

# Pull and run a model (downloads automatically)
ollama run llama3.2           # Meta Llama 3.2 3B — fast, good for most tasks
ollama run llama3.1:70b       # Llama 3.1 70B — more capable, needs 48GB+ RAM
ollama run mistral            # Mistral 7B — fast and capable
ollama run codellama          # Meta's code-focused model
ollama run phi4               # Microsoft Phi-4 — small but surprisingly capable
ollama run gemma3             # Google Gemma 3
ollama run deepseek-r1        # DeepSeek reasoning model
ollama run qwen2.5-coder      # Alibaba's code model — excellent for coding tasks

# List downloaded models
ollama list

# Remove a model
ollama rm llama3.2

Ollama REST API

Ollama runs a local HTTP server at http://localhost:11434:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain dependency injection in one sentence.",
  "stream": false
}'

Ollama + LangChain

from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.2")
response = llm.invoke("Summarise the key ideas in Clean Architecture")
print(response.content)

# Works as a drop-in replacement for ChatOpenAI
# — same chain/agent code, different model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

chain = ChatPromptTemplate.from_template("Explain {topic} simply.") | llm | StrOutputParser()
result = chain.invoke({"topic": "vector databases"})

Ollama + OpenAI SDK

Ollama exposes an OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"   # any non-empty string
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a Python function to sort a list of dicts by key"}]
)
print(response.choices[0].message.content)

---

LM Studio — GUI for Local LLMs

LM Studio provides a desktop GUI for discovering, downloading, and running models — no command line required.

• Download from lmstudio.ai
• Browse and download models from Hugging Face with one click
• Built-in chat interface for testing models
• Local API server compatible with OpenAI SDK
• Supports GPU acceleration on Apple Silicon, NVIDIA, and AMD

When to use LM Studio vs Ollama:

• LM Studio: non-technical users, browsing models, GUI chat
• Ollama: developers, scripting, CI/CD integration, Docker deployment

---

Model Selection Guide

Choose a model based on your hardware and task:

RAM	Recommended models	Notes
8 GB	Llama 3.2 3B, Phi-4 mini, Gemma 3 2B	Fast, basic tasks
16 GB	Mistral 7B, Llama 3.1 8B, Qwen 2.5 7B	Good quality, reasonable speed
32 GB	Llama 3.1 14B, Phi-4 14B, DeepSeek-R1 14B	Near cloud quality for many tasks
64 GB+	Llama 3.1 70B, Qwen 2.5 72B	Approaching frontier model quality

GPU acceleration dramatically speeds up inference:

• Apple Silicon (M1/M2/M3/M4) — Metal GPU, excellent efficiency
• NVIDIA — CUDA, best performance for large models
• AMD — ROCm support improving

---

vLLM — Production-Grade Serving

vLLM is designed for high-throughput, production inference — serving many users concurrently.

pip install vllm

# Serve a model (OpenAI-compatible API)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

vLLM key features:

• PagedAttention — handles long contexts efficiently
• Continuous batching — processes many requests in parallel
• Tensor parallelism — splits large models across multiple GPUs
• OpenAI-compatible API — swap in/out without changing your application code

---

Running Local LLMs in Docker

# Dockerfile — Ollama in a container
FROM ollama/ollama:latest

# Pre-pull a model during build (optional)
RUN ollama serve & sleep 5 && ollama pull llama3.2

EXPOSE 11434
CMD ["ollama", "serve"]

docker-compose.yml

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama   # persist downloaded models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama_data:

docker compose up -d
# Pull a model into the running container
docker compose exec ollama ollama pull llama3.2

---

Privacy Best Practices

When using local LLMs for sensitive data:

# Verify network isolation — model must not call home
import socket

# Run inference in an environment with no outbound internet (firewall rule)
# Verify with:
try:
    socket.create_connection(("8.8.8.8", 80), timeout=1)
    print("WARNING: network access available — verify model is truly local")
except OSError:
    print("Network isolated — safe for confidential data")

• Air-gap the inference machine for the most sensitive workloads
• Audit the model weights source (Hugging Face, ollama.com) before use
• Consider quantised models (GGUF format) — smaller but still accurate for many tasks