Ollama vs llama.cpp vs vLLM: choosing a local LLM server

For one developer on a laptop, run Ollama: it installs in minutes, downloads models by name, and loads them on demand. For a small team, a Mac, a CPU-only server, or a consumer GPU, run llama.cpp's llama-server, which serves GGUF models with explicit control over parallel slots, API keys, and Prometheus metrics. For production traffic on data-center GPUs, run vLLM, whose continuous batching and PagedAttention keep throughput high as concurrency grows. All three expose an OpenAI-compatible API, so you can start with one and move to another by changing a base URL and a model name.

What each server is optimized for

Ollama optimizes for time to first answer. It runs as a desktop app or background service, pulls models by name, loads a model on the first request, and unloads it after an idle period, five minutes by default according to the Ollama FAQ. A few OLLAMA_* environment variables replace fine-grained scheduling control, and that simplicity is the point.

llama.cpp optimizes for portability. It is a C/C++ inference engine built on the ggml library, with backends for Metal on Apple Silicon, CUDA, HIP for AMD, Vulkan, SYCL, and tuned CPU paths for AVX, AVX2, AVX-512, AMX, and ARM NEON. It can place some layers on the GPU and keep the rest in system RAM when a model does not fit in VRAM. Its HTTP server, llama-server, is one binary plus a GGUF file plus flags. The project README now shows a unified llama serve command in its quick start, while the llama-server documentation and the Docker images still ship the llama-server executable used in this article.

vLLM optimizes for throughput on accelerators. Its README lists PagedAttention for KV cache management, continuous batching, chunked prefill, prefix caching, multi-LoRA serving, tensor and pipeline parallelism, and a long list of quantization formats. It supports NVIDIA, AMD, and Intel GPUs and x86, ARM, and PowerPC CPUs, with further accelerators through hardware plugins.

The short version: Ollama and llama.cpp fit modest hardware and a handful of simultaneous users, while vLLM earns its extra setup once many requests arrive at the same time. The hardware side of the decision, including how much memory weights and KV cache need, is covered in the local LLM hardware guide.

One OpenAI client, three base URLs

All three servers implement /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, and /v1/responses. The Ollama OpenAI compatibility page marks its Responses support as non-stateful. llama-server also offers an Anthropic-compatible /v1/messages, and vLLM lists Anthropic Messages, audio transcription, and pooling APIs next to the OpenAI routes.

Keep the server choice in configuration so the client code never changes:

import os
from openai import OpenAI

# Ollama:       http://127.0.0.1:11434/v1
# llama-server: http://127.0.0.1:8080/v1
# vLLM:         http://127.0.0.1:8000/v1
client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],
    api_key=os.environ.get("LLM_API_KEY", "not-used"),
)

reply = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "user", "content": "Explain HTTP 429 in two sentences."}],
    temperature=0.2,
)
print(reply.choices[0].message.content)

The OpenAI SDK requires a key string, so pass a placeholder when the server has none. Ollama ignores the key locally, while llama-server and vLLM check it only when you start them with one.

The model field is where the servers differ. Ollama expects a library tag such as qwen3:8b. vLLM expects the Hugging Face repository name or the value of --served-model-name. A single-model llama-server serves whatever it loaded, --alias sets the name that /v1/models reports, and in router mode the name selects the model.

Compatibility stops at the edges of the OpenAI specification. vLLM takes extra parameters such as top_k through extra_body, and by default it applies the model's generation_config.json, which can change sampling defaults unless you pass --generation-config vllm. Chat templates and tokenizers can also differ between a GGUF conversion and the original checkpoint, so run the same evaluation set against both servers before switching, as described in the AI agent evals guide.

Model formats and where the weights come from

The three servers do not read the same files, and that often settles the question before performance does.

Ollama pulls models from its library by name, runs GGUF repositories from Hugging Face with ollama run hf.co/{user}/{repo}:{quant}, and imports local GGUF files or Safetensors directories through a Modelfile whose FROM line points at the weights. It does not quantize GGUF files during import, so quantize them with llama.cpp tools first.

llama.cpp reads GGUF. You convert a Hugging Face checkpoint with convert_hf_to_gguf.py and quantize it, or download a ready GGUF with -hf user/repo:quant. The README lists integer quantization from 1.5-bit to 8-bit. Vision models also need a projector file, which -hf fetches automatically.

vLLM reads Hugging Face model repositories with Safetensors weights, plus quantized checkpoints such as GPTQ, AWQ, FP8, INT8, INT4, and compressed-tensors. Its documentation describes GGUF support as highly experimental, and that support has moved to an out-of-tree vllm-gguf-plugin.

# Ollama: a library model, or a GGUF repository on Hugging Face
ollama pull qwen3:8b
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

# llama.cpp: download a GGUF and serve it
llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF --port 8080

# vLLM: serve a Hugging Face repository
vllm serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 8000

Concurrency, batching, and KV cache memory

Concurrency costs memory. Each active sequence keeps attention keys and values for every token in its context, so memory grows with the number of simultaneous sequences multiplied by their length, on top of the weights. A useful estimate for a standard transformer is:

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value
Qwen3-8B, 16-bit cache: 2 × 36 × 8 × 128 × 2 = 147,456 bytes = 144 KiB
One 32,768-token sequence: 144 KiB × 32,768 = 4.5 GiB

The layer count, KV head count, and head dimension come from the model's config.json. Four such sequences need 18 GiB of cache before the weights are counted. Models with sliding-window or hybrid attention layers need less, so treat the formula as an upper bound for them.

Ollama

OLLAMA_NUM_PARALLEL sets how many requests each loaded model processes at once, and the FAQ lists a default of 1. OLLAMA_MAX_LOADED_MODELS defaults to three times the number of GPUs, or three for CPU inference. OLLAMA_MAX_QUEUE defaults to 512 queued requests, after which the server answers with a 503 error. The FAQ also warns that required memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. The documented default context length varies between pages and with available VRAM, so set it explicitly and confirm it in the CONTEXT column of ollama ps.

OLLAMA_HOST=127.0.0.1:11434 \
OLLAMA_NUM_PARALLEL=4 \
OLLAMA_CONTEXT_LENGTH=16384 \
OLLAMA_KEEP_ALIVE=30m \
ollama serve

llama-server

llama-server divides work into slots. Each slot holds one conversation, and -np sets the number of slots, where the default -1 means automatic. Continuous batching is enabled by default, so new requests join the running batch between decode steps. When the slot count is automatic, all slots share one unified KV buffer. When you set -np yourself, the unified buffer is off by default and -c is split across the slots: -c 32768 -np 4 gives each conversation 8,192 tokens. Check the result with GET /slots, which reports n_ctx for every slot.

GPU offload is set with -ngl, which accepts a number, auto (the default), or all. When the model does not fit, the layers left on the CPU are bound by system memory bandwidth, so a partly offloaded model works but usually generates tokens more slowly.

llama-server -m /models/qwen3-8b-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -np 4 -ngl all \
  --api-key-file /etc/llama/api-keys.txt \
  --metrics

vLLM

vLLM's continuous batching admits new sequences at every decode iteration whenever KV cache blocks are free, and PagedAttention allocates that cache in fixed-size blocks as sequences grow instead of reserving the maximum length up front. At startup vLLM claims the fraction of GPU memory set by --gpu-memory-utilization and turns what remains after the weights and activation workspace into KV cache blocks. When blocks run out under load, it preempts some requests and recomputes them once space is free again, which shows up in the vllm:num_preemptions metric and in tail latency.

The main levers are --max-model-len for the longest context you accept, --max-num-seqs for the most sequences per iteration, --max-num-batched-tokens, --gpu-memory-utilization, and --tensor-parallel-size for splitting one model across GPUs. Lowering --max-model-len to what the application actually needs is usually the cheapest way to fit more concurrent requests.

export VLLM_API_KEY="$(openssl rand -hex 32)"
vllm serve Qwen/Qwen3-8B \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 32768 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.90

Structured output and tool calling

All three servers can constrain output to a JSON schema through the OpenAI response_format field, which makes this request portable:

schema = {
    "type": "object",
    "properties": {
        "severity": {"type": "string", "enum": ["low", "medium", "high"]},
        "summary": {"type": "string"},
    },
    "required": ["severity", "summary"],
    "additionalProperties": False,
}

reply = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "user", "content": "Triage this log line: disk /var is 97% full"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "triage", "schema": schema},
    },
)

Ollama's native API also takes "json" or a schema in its format field, and its docs recommend repeating the schema in the prompt. llama-server converts JSON schemas into its GBNF grammar format and also accepts a raw grammar for other output shapes. vLLM supports response_format and a structured_outputs object in extra_body with json, regex, choice, grammar, and structural_tag keys. The older guided_* parameters were removed in v0.12.0, as the vLLM structured outputs page notes. Constrained decoding guarantees syntax, not truth, and a response cut off by max_tokens is still invalid JSON, so validate every result against the schema in your own code.

Tool calling works on all three, with different setup:

• Ollama accepts tools on /api/chat and on /v1/chat/completions, including parallel tool calls, for models whose templates support tools.
• llama-server supports OpenAI-style tools through Jinja chat templates, which are enabled by default. The llama.cpp function calling notes list native handlers for several model families and a generic fallback that uses more tokens. Parallel calls stay off unless the request sets "parallel_tool_calls": true.
• vLLM needs --enable-auto-tool-choice and a --tool-call-parser matched to the model family, such as hermes for Qwen2.5 or llama3_json for Llama 3.1 and 3.2. The vLLM tool calling page explains that named functions and tool_choice="required" use structured outputs, while auto does not guarantee that arguments parse.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --enable-auto-tool-choice --tool-call-parser hermes

Treat tool arguments as untrusted input. The model chooses them, and text inside a retrieved document can steer that choice, as the prompt injection and MCP security guide explains.

Running in Docker with GPU access

On Linux with NVIDIA GPUs, install the NVIDIA Container Toolkit first so that --gpus works. The commands below follow each project's documented image and flags, with one change: every port is published on 127.0.0.1 only. The Docker port publishing documentation states that ports mapped without a host address are published on all host addresses, and Docker's firewall notes add that published-port traffic is diverted before ufw rules see it.

# Ollama with NVIDIA GPUs
docker run -d --name ollama --gpus=all \
  -v ollama:/root/.ollama \
  -p 127.0.0.1:11434:11434 \
  ollama/ollama

# llama-server, CUDA build
docker run -d --name llama --gpus all \
  -v /srv/models:/models:ro \
  -p 127.0.0.1:8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/qwen3-8b-q4_k_m.gguf --host 0.0.0.0 --port 8080 -ngl all

# vLLM with NVIDIA GPUs
docker run -d --name vllm --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B

Inside a container the server must listen on 0.0.0.0, otherwise the published port cannot reach it. The 127.0.0.1 on the host side is what keeps it private. vLLM's documented command adds --ipc=host because PyTorch shares data between processes through shared memory, particularly for tensor-parallel inference. For AMD GPUs, use ollama/ollama:rocm with --device /dev/kfd --device /dev/dri, the server-rocm or server-vulkan llama.cpp images, or vllm/vllm-openai-rocm. In production, pin image tags or digests instead of latest.

Security: bind locally, authenticate at a proxy

Treat an inference port like a database port. Anyone who reaches it can spend your GPU time, read whatever the model returns, and in some configurations call administrative routes.

Start with the bind address. Ollama binds 127.0.0.1:11434 by default, and llama-server binds 127.0.0.1:8080. vLLM is different: when --host is not set, its launcher binds every interface, so always pass --host 127.0.0.1 outside a container.

Then check what each server's key actually protects:

• Ollama's local API has no API key setting. Anything that can reach the port can use it, so share it only through a proxy that authenticates.
• llama-server checks --api-key or --api-key-file on its API, while /health stays public by design. Its README notes that CORS reflects any Origin by default and recommends --cors-origins on local networks.
• vLLM's --api-key covers only a few path prefixes, including /v1. The vLLM security guide lists unprotected routes such as /invocations, which reaches the same inference functions, and /pause or /abort_requests, and recommends a reverse proxy that allowlists endpoints.

A small Nginx front end covers both needs. It terminates TLS, checks a bearer token, forwards only /v1/, and returns 404 for everything else.

map $http_authorization $llm_client {
    default "";
    "Bearer REPLACE_WITH_A_LONG_RANDOM_TOKEN" "team";
}

server {
    listen 443 ssl;
    server_name llm.example.internal;
    ssl_certificate     /etc/nginx/tls/llm.crt;
    ssl_certificate_key /etc/nginx/tls/llm.key;
    client_max_body_size 10m;

    location /v1/ {
        if ($llm_client = "") { return 401; }
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }

    location / {
        return 404;
    }
}

proxy_buffering off lets streamed tokens reach the client as they are generated. If the upstream server has its own key, give it the same token so that a request that bypasses the proxy still fails. For Ollama, point proxy_pass at port 11434 and add proxy_set_header Host localhost:11434, as the FAQ example does. Add limit_req when several users share one GPU. On a developer laptop, remember that OLLAMA_ORIGINS=* lets any web page you visit call the local server through your browser.

Monitoring and health checks

Start with readiness. llama-server's /health returns 503 while the model loads and 200 once it is ready, and vLLM exposes /health as well. Use these for container health checks and load balancer probes, not as proof that generation quality is fine.

vLLM exposes Prometheus metrics at /metrics by default. The vLLM metrics page lists vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, histograms for time to first token, inter-token latency, and end-to-end latency, and a preemption counter. llama-server exposes /metrics only when started with --metrics, including llamacpp:requests_processing, llamacpp:requests_deferred, llamacpp:prompt_tokens_seconds, and llamacpp:predicted_tokens_seconds, while GET /slots shows per-slot state.

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: ["127.0.0.1:8000"]
  - job_name: llama-server
    static_configs:
      - targets: ["127.0.0.1:8080"]

Ollama's documented API has no Prometheus endpoint. Use GET /api/ps for loaded models, their VRAM use, context length, and unload time, and read the timing fields in each response. Durations are in nanoseconds, so generation speed is eval_count / eval_duration × 10^9 tokens per second:

curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model": "qwen3:8b", "prompt": "Say ready.", "stream": false}' |
  jq '{tokens: .eval_count, tokens_per_second: (.eval_count / .eval_duration * 1e9)}'

Alert on signals that mean users are waiting: a waiting or deferred queue that stays above zero, KV cache usage close to 1, a rising preemption count, and time to first token drifting up at p95. Server metrics describe capacity. They do not tell you which prompt, tool call, or retrieval step made a request slow, so connect them to request traces as described in the AI agent observability guide.

Side-by-side comparison

	Ollama	llama.cpp llama-server	vLLM
Optimized for	Fast setup and model management	Portability on CPU, Apple Silicon, and consumer GPUs	Throughput on data-center GPUs
Model formats	Library models, GGUF, Safetensors import	GGUF	Hugging Face Safetensors, GPTQ, AWQ, FP8 and more; GGUF experimental
Default listen address	127.0.0.1:11434	127.0.0.1:8080	All interfaces, port 8000
Built-in API key	None	--api-key, --api-key-file	--api-key or VLLM_API_KEY, limited path prefixes
Concurrency	OLLAMA_NUM_PARALLEL per model, default 1	Slots (-np) with continuous batching	Continuous batching with PagedAttention
Structured output	format, response_format	response_format, JSON schema, GBNF	response_format, structured_outputs
Tool calling	tools on native and OpenAI routes	Jinja templates, native or generic handlers	--enable-auto-tool-choice with a parser
Metrics	/api/ps and response timings	/metrics with --metrics, /slots	/metrics by default
Best fit	One developer, prototypes	Small teams, modest or mixed hardware	Many concurrent users, production SLOs

A decision path for development, teams, and production

Work through these questions in order and stop at the first clear answer.

1. Is this one person exploring models on a laptop or workstation? Use Ollama. Move on when you need metrics, authentication, or per-request control.
2. Is the hardware a Mac, a CPU-only server, or a consumer GPU where the model barely fits in VRAM or does not fit at all? Use llama-server. GGUF quantization and partial offload let it run where vLLM cannot load the model.
3. Is it a small team with a few concurrent users on one machine? Use llama-server with -np set to peak concurrency, an API key, --metrics, and a proxy. Ollama also works if everyone shares one or two models and the proxy authenticates.
4. Do you expect sustained concurrent traffic, latency targets, several GPUs, or GPU quantization formats such as FP8 or AWQ? Use vLLM, bound to localhost behind a gateway, with pinned images and Prometheus scraping from day one.

Many teams end up with two of them: Ollama or llama-server on developer machines, vLLM in staging and production, and the same OpenAI client code everywhere. That only works if the evaluation suite runs against the production artifact, because a GGUF quantization and an FP8 checkpoint of the same model behave as different models in practice. The production AI agent architecture guide covers the gateway, timeouts, retries, and fallbacks that should sit in front of any of these servers.

Deployment checklist

• Choose the server with the decision path, and record the exact model file, quantization, and chat template you will serve.
• Calculate KV cache memory for peak concurrency and maximum context, then set OLLAMA_CONTEXT_LENGTH, -c and -np, or --max-model-len and --max-num-seqs to match.
• Bind to 127.0.0.1, pass --host 127.0.0.1 to vLLM, and publish Docker ports as 127.0.0.1:port:port.
• Put TLS, authentication, an endpoint allowlist, and rate limits in a reverse proxy. Never expose an unauthenticated inference port.
• Set a key on llama-server or vLLM as a second layer, and keep it out of shell history and image layers.
• Scrape /metrics from llama-server and vLLM, and poll Ollama's /api/ps, over a private network.
• Load-test with your own prompts at the expected concurrency, recording time to first token, tokens per second, queue length, and memory.
• Validate structured output against its schema in code, and treat tool arguments as untrusted input.
• Pin versions and image digests, and rerun your evaluation suite whenever the server, model file, quantization, or image changes.