vLLM tutorial: run an OpenAI-compatible LLM server on one GPU

Key takeaways

• vLLM is an open-source (Apache-2.0) inference engine that turns a Hugging Face model into an OpenAI-compatible HTTP server with one command: vllm serve <model>.
• On NVIDIA hardware it needs Linux, Python 3.11 to 3.14 and a GPU with compute capability 7.5 or higher (T4, RTX 20-series, A100, L4, H100 and newer), according to the vLLM installation guide.
• The server listens on port 8000 and speaks the same Chat Completions, Completions, Embeddings and Responses routes as OpenAI, so the official openai Python client works by changing only base_url.
• Four flags solve most first-run problems: --max-model-len, --gpu-memory-utilization, --tensor-parallel-size and --quantization.
• Choose vLLM for many concurrent users on a GPU server; Ollama or llama.cpp remain simpler for a single user on a laptop.

This vLLM tutorial turns an open Hugging Face model into an OpenAI-compatible server on one Nvidia GPU on a Linux machine in about ten minutes: install the vllm package, run vllm serve, then call http://localhost:8000/v1/chat/completions from curl or Python.

vLLM started at UC Berkeley's Sky Computing Lab and describes itself on GitHub as "a high-throughput and memory-efficient inference and serving engine for LLMs." Its core trick is PagedAttention, which manages the attention key-value (KV) cache in small blocks, combined with continuous batching, so a single GPU can answer many requests at once. If you only want to chat with a model on your own laptop, our guide to running a local LLM with Ollama is the easier path. If you need an endpoint that a team, an app or an agent can hit all day, read on.

vLLM requirements: GPU, CUDA and Python

The vLLM GPU installation page lists the following for NVIDIA CUDA. AMD ROCm, Google TPU, Intel XPU and Apple Silicon (through the separate vLLM-Metal project) have their own guides.

Source: vLLM documentation, GPU installation and quickstart pages, checked 27 October 2026.

VRAM is the real limit. The model weights must fit on the GPU, and whatever memory is left becomes KV cache, which decides how many tokens of context and how many parallel requests you can hold. A small instruct model such as Qwen2.5-1.5B-Instruct, which the vLLM quickstart uses, is a safe first test on almost any supported card.

Step 1: Install vLLM with uv or pip

The vLLM team recommends uv, a fast Python package manager, because its --torch-backend=auto option picks the right PyTorch build for your installed CUDA driver:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

If you prefer plain pip, point it at the PyTorch CUDA 12.9 index:

python3 -m venv .venv
source .venv/bin/activate
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
vllm download-kernels

Do not skip vllm download-kernels. According to the installation guide it installs precompiled FlashInfer kernels; without them vLLM compiles kernels at startup, which can take several minutes on Hopper and newer GPUs. Run it again after every upgrade, in the same environment. To try vLLM without creating a permanent environment, uv run --with vllm vllm --help works too.

Gated models and Hugging Face tokens

vLLM downloads weights from the Hugging Face Hub on first use and caches them in ~/.cache/huggingface. Gated models such as Meta's Llama family require you to accept the licence on the model page and export a token first:

export HF_TOKEN=hf_your_token_here

If Hugging Face is slow or blocked on your network, you can switch the download source to ModelScope with export VLLM_USE_MODELSCOPE=True before starting the engine.

Step 2: Start the OpenAI-compatible server with vllm serve

One command downloads the model, loads it onto the GPU and starts the API:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

By default the server listens at http://localhost:8000 and serves one model per process. Use --host 0.0.0.0 to accept connections from other machines and --port to change the port. Before exposing it beyond localhost, require a key with --api-key or the VLLM_API_KEY environment variable; the quickstart notes that several keys can be supplied and any of them is accepted.

Check that the model is live:

curl http://localhost:8000/v1/models

Key vllm serve flags explained

These are the engine arguments you will touch most often. Descriptions and defaults come from the vLLM engine-arguments reference.

Source: vLLM documentation, engine arguments and parallelism pages, checked 27 October 2026.

A realistic single-GPU command for a mid-sized model looks like this:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --served-model-name qwen7b \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --api-key change-me

Multi-GPU: tensor and pipeline parallelism

The vLLM parallelism guide gives a simple rule. If the model is too large for one GPU but fits on one machine, use tensor parallelism. If it is too large for one machine, combine tensor and pipeline parallelism. On GPUs without NVLink, the docs say pipeline parallelism can deliver higher throughput with less communication overhead. Its own examples:

# Four GPUs in one server
vllm serve facebook/opt-13b --tensor-parallel-size 4

# Eight GPUs total
vllm serve gpt2 --tensor-parallel-size 4 --pipeline-parallel-size 2

Quantized models

Quantization shrinks weights so a bigger model fits in the same VRAM. vLLM's quantization docs cover AWQ, GPTQ, FP8, INT8 and INT4 schemes, BitsAndBytes, and tools such as LLM Compressor and NVIDIA Model Optimizer. In most cases you simply serve an already-quantized checkpoint and vLLM reads the method from the model config. GGUF files, the format popular with llama.cpp and Ollama, also load, but the vLLM docs warn that GGUF support "is highly experimental and under-optimized" and recommend taking the tokenizer from the base model:

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

For the trade-offs between 4-bit, 8-bit and full-precision weights, see our explainer on LLM quantization and GGUF.

Step 3: Run vLLM with Docker

The project publishes an official image, vllm/vllm-openai. The command in the vLLM Docker guide is:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-0.6B

The volume mount keeps downloaded weights between container restarts. --ipc=host (or a large --shm-size) matters because PyTorch passes data between processes through shared memory, especially under tensor parallelism. Any engine argument from the table above can be appended after the image name. The host needs the NVIDIA Container Toolkit installed for --gpus all to work.

Step 4: Query the vLLM API with curl

According to the vLLM online-serving docs, the server exposes the Completions, Chat Completions, Responses, Embeddings and audio Transcription and Translation APIs under /v1. A chat request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer change-me" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain PagedAttention in two sentences."}
    ],
    "max_tokens": 200
  }'

Leave out the Authorization header if you did not start the server with an API key, and use your --served-model-name alias in the model field if you set one. Chat requests only work when the model ships a chat template in its tokenizer configuration; otherwise the docs say every chat request will error, and you must pass one with --chat-template ./template.jinja.

Step 5: Call vLLM from Python with the OpenAI client

Because the routes match OpenAI's, the official openai package works unchanged. Install it with pip install openai, then:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # or the value you passed to --api-key
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Give me three uses for a self-hosted LLM."},
    ],
    max_tokens=300,
)
print(response.choices[0].message.content)

Streaming uses the standard OpenAI pattern:

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

This is the main reason teams pick vLLM: an app written for a hosted API can move to a self-hosted model by swapping the base URL and model name. Frameworks that accept an OpenAI-compatible endpoint, from LangChain to a home-grown local RAG app in Python, can point at vLLM the same way. If you also work with Anthropic's hosted models, compare the request shape in our Claude API Python tutorial.

Watch out for the model's default sampling settings

vLLM applies the generation_config.json from the model's Hugging Face repository when one exists, so the model creator's recommended temperature and top-p can override what you expect. Start the server with --generation-config vllm to use vLLM's own defaults instead, or set sampling values explicitly in every request.

Step 6: Offline batch inference without a server

For one-off jobs, such as labelling a dataset or scoring evaluation prompts, you can skip HTTP and use the LLM class directly. This example follows the vLLM quickstart:

from vllm import LLM, SamplingParams

prompts = [
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.prompt, "->", output.outputs[0].text)

Note that llm.generate does not apply a chat template. For instruct or chat models, the docs recommend llm.chat, which accepts messages in the OpenAI format:

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
messages = [{"role": "user", "content": "Summarise vLLM in one line."}]
result = llm.chat(messages, SamplingParams(max_tokens=64))
print(result[0].outputs[0].text)

If you fine-tuned your own adapter with our LoRA fine-tuning guide, offline batch runs like this are a cheap way to compare it with the base model on hundreds of prompts.

vLLM vs Ollama vs llama.cpp: which to use

All three tools can expose an OpenAI-style API, but they target different jobs.

Sources: vLLM GitHub repository and docs; Ollama OpenAI-compatibility docs; llama.cpp server README; checked 27 October 2026.

A practical rule: prototype with Ollama, and move to vLLM when you have more than a handful of simultaneous users, need multi-GPU tensor parallelism, or want to serve full-precision or FP8 checkpoints straight from Hugging Face.

Common vLLM errors and fixes

• CUDA out of memory at startup: the model plus KV cache does not fit. Lower --max-model-len, use a quantized checkpoint, or add GPUs with --tensor-parallel-size.
• Another program needs the GPU: reduce --gpu-memory-utilization, for example to 0.7.
• Very slow first start: run vllm download-kernels so FlashInfer kernels are not compiled at launch.
• Chat requests fail but completions work: the model has no chat template; pass --chat-template.
• 401 errors from clients: the server was started with --api-key; send it as a Bearer token or in the client's api_key.
• Odd or repetitive output: check whether the model's generation_config.json is changing sampling, and test with --generation-config vllm.

What it means

vLLM has become one of the standard ways to self-host open models: its GitHub repository showed about 95,600 stars when checked on 27 October 2026, and its licence is Apache-2.0, so it can be used commercially. For developers, the practical payoff is portability. Code written against the OpenAI Chat Completions format can run on your own GPUs by changing two lines, which keeps sensitive data in-house and turns per-token API bills into a fixed hardware cost.

The trade-off is operational work. You own GPU drivers, CUDA versions, memory tuning and uptime. Start with a small model on one GPU, confirm the curl and Python calls work, then scale context length, concurrency and GPU count one flag at a time.