Running LLMs locally: Ollama, llama.cpp, vLLM
What you’ll learn
Section titled “What you’ll learn”- The real reasons to run a model on your own hardware, and the costs that come with it.
- How to estimate whether a model fits in memory, and what quantization trades away.
- How Ollama, llama.cpp, LM Studio and vLLM differ, and which suits a laptop, a workstation or a shared server.
- How to call any of them through the OpenAI-compatible API so application code stays portable.
- The failure modes that catch teams out: silent context truncation, CPU fallback, exposed ports and unexpected cloud features.
Summary
Section titled “Summary”Running a model locally means the weights, the prompts and the outputs stay on hardware you control. You choose it for data control, offline use, predictable cost at steady volume, or freedom to customize. You pay with weaker models than the hosted frontier, hardware and operations you now own, and performance engineering that a provider used to do for you.
Four tools cover most needs. Ollama is the simplest way to pull and serve a model on one machine. llama.cpp is the engine underneath much of the ecosystem, with fine control and broad hardware support. LM Studio is a desktop application with a graphical model browser and a local server. vLLM is a high-throughput server for GPUs, built for many concurrent users. All four expose an OpenAI-compatible HTTP API.
When local makes sense, and when it does not
Section titled “When local makes sense, and when it does not”| Reason to run locally | What to check first |
|---|---|
| Data may not leave your network (contracts, regulation, air-gapped sites) | Whether a hosted provider’s zero-retention or regional offering already satisfies the requirement; that is often cheaper |
| Offline or unreliable connectivity | Whether a small model is good enough for the task |
| High, steady volume where per-token pricing dominates cost | Utilization: an idle GPU costs the same as a busy one |
| Fine-tuning or custom models | Whether prompt changes or retrieval solve it without training |
| Latency for small models close to the user | Measure it; a large hosted model is often faster than a large local one on modest hardware |
Local is usually the wrong choice when the task needs frontier-level reasoning, when volume is low or bursty, or when nobody will own the server. For the hosted alternatives, see LLM model selection.
Sizing: will the model fit?
Section titled “Sizing: will the model fit?”A model’s memory need is driven by its parameter count and the precision of each weight:
memory for weights ≈ parameters × bits per weight ÷ 8
- A 7-billion-parameter model at 16-bit precision needs about 14 GB for weights.
- The same model quantized to roughly 4 bits per weight needs about 4 GB.
- Add memory for the context window (the key-value cache grows with context length and with the number of parallel requests) and some overhead.
Quantization stores weights at lower precision. It is what makes local inference practical, and the common format for it is GGUF, used by llama.cpp and tools built on it. Around 4 to 5 bits per weight is the usual balance; quality drops faster below that. Always evaluate the quantized model on your own task, not the full-precision one from a model card.
Where the memory must be: inference is fast when the whole model sits in GPU memory (VRAM), or in unified memory on Apple silicon. If it does not fit, layers spill to the CPU and speed can fall by an order of magnitude. A smaller model fully on the GPU usually beats a larger one split across GPU and CPU.
This page deliberately gives no table of “best models per RAM size”: open model releases change monthly. Use the tool’s model library to see current options and their sizes, then apply the arithmetic above.
The tools
Section titled “The tools”| Tool | What it is | Best for | Notes |
|---|---|---|---|
| Ollama | CLI and background server with a model library | One developer machine, quick setup, scripting | Listens on 127.0.0.1:11434 by default; also offers optional cloud-hosted models |
| llama.cpp | C/C++ inference engine with a CLI and an HTTP server | Control over quantization, context, GPU layers; unusual hardware | Runs GGUF models; server supports schema-constrained JSON output |
| LM Studio | Desktop app with model browser, chat and local server | People who prefer a GUI; also has a headless daemon (llmster) and the lms CLI |
Server defaults to http://localhost:1234 |
| vLLM | GPU inference server with continuous batching | A shared service with many concurrent requests | Linux and NVIDIA first; serves on port 8000 by default |
Ollama
Section titled “Ollama”The commands below come from the Ollama README and documentation. They were not executed for this page, because the build environment cannot download models. The model tag gemma4 is the one used in Ollama’s own examples as of November 2026; browse ollama.com/library for current models.
# Install on macOS or Linux (Windows: irm https://ollama.com/install.ps1 | iex)curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4 # download if needed, then chat in the terminalollama pull gemma4 # download onlyollama list # models on diskollama ps # models loaded in memory, with the CPU/GPU split and context sizeollama stop gemma4 # unload from memoryollama rm gemma4 # delete from diskThe native REST API:
curl http://localhost:11434/api/chat -d '{ "model": "gemma4", "messages": [{ "role": "user", "content": "Summarize: order A1001 arrived damaged." }], "stream": false}'Settings that matter, from the Ollama documentation as of Ollama 0.40 (latest release 0.40.16, November 2026). Defaults have changed between releases, so check the documentation for your version:
- Context length. Ollama picks a default from available VRAM: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, 256k at 48 GiB or more. The docs recommend at least 64,000 tokens for agents and coding tools. Set it with
OLLAMA_CONTEXT_LENGTH=64000 ollama serve, and expect memory use to rise. - Keep-alive. Models unload after 5 minutes idle by default (
OLLAMA_KEEP_ALIVEchanges it), so the first request after a pause pays the load time. - Parallelism.
OLLAMA_NUM_PARALLELdefaults to 1; required memory scales with parallel requests times context length. - Network exposure. The server binds to
127.0.0.1by default.OLLAMA_HOST=0.0.0.0:11434exposes it to the network, and the API has no authentication of its own. - Cloud features. Current Ollama can also run cloud-hosted models, where prompts are processed on Ollama’s servers. For a strictly local setup, set
OLLAMA_NO_CLOUD=1(or"disable_ollama_cloud": truein~/.ollama/server.json) and restart.
llama.cpp
Section titled “llama.cpp”# Not executed for this page. Install script from the llama.cpp README:curl -LsSf https://llama.app/install.sh | sh
# Chat with a GGUF model pulled from Hugging Facellama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Start the OpenAI-compatible HTTP serverllama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
# The same server under its program name, as in the server documentation and release binariesllama-server -hf ggml-org/Qwen3.5-0.8B-GGUF --port 8080Two spellings appear in the official material. The README’s quick start uses the llama command installed by its install script (llama cli, llama serve). The server’s own documentation, the release binaries and source builds name the programs llama-cli and llama-server. Use whichever your installation provides. The server takes flags such as -c or --ctx-size for context length, -ngl or --n-gpu-layers for how many layers go to the GPU, --port, --api-key, and -j or --json-schema to constrain output to a JSON Schema. Pass --port explicitly instead of relying on the default, which has changed between releases.
LM Studio
Section titled “LM Studio”LM Studio is a desktop application for macOS on Apple silicon, Windows and Linux. You search for a model, download it, chat with it, and optionally start a local server. For servers without a display, the llmster daemon provides the same engine:
# Not executed for this page. From the LM Studio headless documentation:curl -fsSL https://lmstudio.ai/install.sh | bashlms daemon uplms get <model>lms server start # OpenAI-compatible API on http://localhost:1234/v1The LM Studio desktop app is not open source; check its license terms for your organization’s use.
# Not executed for this page (needs Linux and a supported GPU). From the vLLM quickstart:uv venv --python 3.12 --seedsource .venv/bin/activateuv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct --api-key "$VLLM_API_KEY"curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"vLLM (0.33.0 on PyPI as of November 2026, Python 3.11 or later) is designed for throughput: PagedAttention manages attention memory efficiently, and continuous batching merges requests that arrive at different times into shared GPU work. The server hosts one model at a time and loads full-precision or GPU-oriented quantized weights from Hugging Face, so it typically needs more VRAM per model than a 4-bit GGUF setup.
One client for all of them
Section titled “One client for all of them”Because every tool above speaks the OpenAI chat completions format, application code can target a local server by changing the base URL. The example domain is support tickets for an invented shop, OrdersPortal, that must stay on the company network.
To run it: pip install openai, start a local server with a model loaded, then python summarize_ticket.py. The script was verified with Python 3.11 and openai 3.51.0 against a stub server that returns a fixed OpenAI-format response on port 11434; it was not run against a real model, so the summary text below is the stub’s canned reply.
# summarize_ticket.py - call a local model through the OpenAI-compatible APIimport os
from openai import OpenAI
client = OpenAI( base_url=os.environ.get("LOCAL_LLM_URL", "http://localhost:11434/v1"), # The SDK requires a key. Ollama ignores it; vLLM started with --api-key checks it. api_key=os.environ.get("LOCAL_LLM_API_KEY", "ollama"),)
ticket = ( "Order A1001 arrived today with the box crushed and the lamp broken. " "I would like a replacement, not a refund.")
response = client.chat.completions.create( model=os.environ.get("LOCAL_LLM_MODEL", "gemma4"), messages=[ {"role": "system", "content": "Summarize the support ticket in one sentence."}, {"role": "user", "content": ticket}, ], temperature=0,)
print(response.choices[0].message.content)print("tokens:", response.usage.prompt_tokens, "in,", response.usage.completion_tokens, "out")Output (from the stub server):
Customer reports order A1001 arrived damaged and asks for a replacement.tokens: 40 in, 14 outChange LOCAL_LLM_URL to http://localhost:1234/v1 for LM Studio or http://localhost:8000/v1 for vLLM, and LOCAL_LLM_MODEL to the model name that server reports at /v1/models. For a vLLM or llama-server instance started with --api-key, set LOCAL_LLM_API_KEY to the same value from your secret store. Frameworks have the same seam: LangChain’s langchain-ollama package provides ChatOllama, and anything that accepts an OpenAI-compatible base URL works with all four servers. See LangChain.
“Compatible” covers the request and response shape, not behavior. Tool calling, structured output and streaming details vary by server and by model, so test the features you depend on.
Running in containers
Section titled “Running in containers”Ollama, llama.cpp and vLLM all publish official container images; the commands, GPU setup and Kubernetes patterns are on AI on Docker and Kubernetes. The production rules are the same whichever server you run:
- Pin the image tag instead of the implicit
latest, and upgrade deliberately, because defaults such as context length change between releases. - Keep models on a volume so they survive container restarts, and pre-load them in an init step so the first user request does not trigger a multi-gigabyte download.
- Publish the port only to the hosts that need it. Ollama’s API has no authentication; put it behind a gateway or use a server’s
--api-keyoption. - Set context length, parallelism and keep-alive explicitly in the container’s environment, sized to the GPU memory you actually have.
Trade-offs: local versus hosted
Section titled “Trade-offs: local versus hosted”| Local model | Hosted API | |
|---|---|---|
| Data location | Your hardware | The provider’s infrastructure, under its terms |
| Capability | Open-weight models; below the hosted frontier on hard reasoning | Frontier models |
| Cost shape | Fixed: hardware, power, staff. Cheap per token only at high utilization | Variable: pay per token, nothing when idle |
| Operations | You patch, monitor, scale and upgrade models | Provider does |
| Latency | No network hop; limited by your hardware | Network hop; very fast hardware |
| Customization | Full: fine-tune, quantize, constrain decoding | What the API offers |
| Licensing | Each open model has its own license; some restrict commercial use | Covered by the service terms |
A common production design is hybrid: a local or small model for classification, redaction and routine drafting, and a hosted model for the hard cases that policy allows to leave the network.
Failure modes and operations
Section titled “Failure modes and operations”- Silent context truncation. With a 4k default context, a long document or an agent’s tool history is cut without an error, and the model answers from what remains. Set the context length explicitly and check it with
ollama ps. - CPU fallback. A model slightly too large for VRAM runs partly on the CPU and becomes very slow. Check the processor split, then choose a smaller model or quantization.
- Cold starts. After the keep-alive expires, the next request reloads the model. Pre-warm or extend the keep-alive for interactive use.
- Exposed, unauthenticated ports. Binding to
0.0.0.0puts an open inference API on the network. Keep the default loopback binding, or place the server behind a reverse proxy with authentication and TLS. Use--api-keyon llama.cpp and vLLM. - “Local” that is not local. Cloud model options, web-search features and telemetry in desktop tools can send data out. Read the tool’s privacy documentation, disable cloud features, and enforce the rule at the network layer with egress filtering instead of trusting application settings.
- Untrusted model files. Download weights from the publisher’s official repository, prefer data-only formats such as GGUF and safetensors over pickle-based checkpoints, and record the model’s hash and license.
- Concurrency collapse. A desktop-style server handles one or a few requests at a time. For shared use, measure throughput under realistic concurrency and move to a batching server such as vLLM.
- Quality assumptions. A quantized small model may fail tasks the full model passes. Keep a graded test set and run it on every change of model, quantization or prompt.
- Wrong chat template. Each model family expects a specific prompt format. Servers apply the template stored with the model; a missing or wrong one produces rambling or repeated output.
- Model and tool drift. Model tags can be updated in place. Pin exact tags or digests and upgrade deliberately.
Interview questions
Section titled “Interview questions”1. What does quantization do, and what does it cost? It stores weights at lower numeric precision, for example about 4 bits instead of 16, cutting memory and often increasing speed. The cost is some loss of accuracy, small at moderate levels and steep at aggressive ones, and it varies by task. You measure it on your own test set.
2. How do you estimate whether a model fits on a given GPU? Multiply parameters by bits per weight and divide by 8 to get the weight size, then add the context cache, which grows with context length and parallel requests, plus overhead. If the total exceeds VRAM, layers move to the CPU and throughput drops sharply.
3. Ollama or vLLM? Ollama for a single machine and a few users: simple install, model library, sensible defaults. vLLM for a shared GPU service: continuous batching gives much higher throughput under concurrent load, at the price of more setup and more VRAM. Many teams develop against Ollama and serve with vLLM, relying on the common OpenAI-compatible API.
4. A local model answers questions about long documents incorrectly, but short prompts work. What do you check? The effective context length first. If the server’s context is smaller than the prompt, the input was truncated silently. Raise the context setting within memory limits, confirm with the server’s status output, and consider retrieval so that only relevant passages are sent.
5. Senior scenario: legal asks for an assistant that summarizes confidential contracts and nothing may leave the company network. Design it. Clarify the requirement: whether a hosted provider with contractual zero retention and regional processing is acceptable, since that changes the cost by a large factor. If it truly must be on-premises: choose two or three candidate open-weight models with licenses that permit the use, build a graded set of real contract summaries with lawyers, and pick the smallest model and quantization that pass. Serve it with vLLM on GPUs sized for peak concurrency, behind an internal gateway with authentication, per-user logging and rate limits. Block egress from the inference hosts at the network layer, mirror model files internally and record their hashes. Add retrieval so long contracts are chunked instead of truncated, show source passages with each summary, and keep a human review step. Plan for operations: monitoring, capacity, a tested upgrade path when a better model appears, and a fallback when the service is down.
Key takeaways
Section titled “Key takeaways”- Run locally for data control, offline use or steady high volume, not by default.
- Memory for weights is roughly parameters times bits per weight divided by 8; keep the whole model on the GPU.
- Ollama for simplicity, llama.cpp for control, LM Studio for a GUI, vLLM for concurrent serving.
- Target the OpenAI-compatible API so code moves between local servers and hosted providers.
- Set context length explicitly, keep ports private, disable cloud features and verify with network controls.
- Evaluate the exact model and quantization you will ship.
Related pages
Section titled “Related pages”- LLM model selection
- AI on Docker and Kubernetes
- Microsoft small language models
- LangChain
- AI governance
- AI and machine learning basics
Official references
Section titled “Official references”- Ollama documentation: context length, FAQ (binding, keep-alive, cloud features), Docker, OpenAI compatibility
- ollama/ollama on GitHub (README and docs source used for verification, November 2026)
- llama.cpp README and server documentation
- LM Studio documentation: headless mode and llmster, llmster as a startup task, system requirements
- vLLM quickstart and
vllmon PyPI (0.33.0) langchain-ollamaon PyPI (1.1.0)