> ## Content Index > Fetch the complete content index at: https://bullquill.com/llms.txt > Use this file to discover other available public pages before exploring further. # vLLM tutorial: run an OpenAI-compatible LLM server on one GPU - URL: https://bullquill.com/how-to-serve-llm-with-vllm/ - Published: 2026-10-27T03:15:46.000Z - Updated: 2026-10-27T03:26:06.000Z - Description: Install vLLM, start an OpenAI-compatible server with one command, tune max-model-len, GPU memory and tensor parallelism, then query it from curl and the openai Python client. Includes Docker, offline batch inference and a vLLM vs Ollama vs llama.cpp comparison. - Author: Rowan Ashby - Tags: AI, Tutorial, How-To, vLLM, Hugging Face, OpenAI API, Python, Docker, #lang-en, #evergreen, #tr-how-to-serve-llm-with-vllm ## Key takeaways - vLLM is an open-source (Apache-2.0) inference engine that turns a Hugging Face model into an OpenAI-compatible HTTP server with one command: `vllm serve `. - On NVIDIA hardware it needs Linux, Python 3.11 to 3.14 and a GPU with compute capability 7.5 or higher (T4, RTX 20-series, A100, L4, H100 and newer), according to the vLLM installation guide. - The server listens on port 8000 and speaks the same Chat Completions, Completions, Embeddings and Responses routes as OpenAI, so the official `openai` Python client works by changing only `base_url`. - Four flags solve most first-run problems: `--max-model-len`, `--gpu-memory-utilization`, `--tensor-parallel-size` and `--quantization`. - Choose vLLM for many concurrent users on a GPU server; Ollama or llama.cpp remain simpler for a single user on a laptop. This vLLM tutorial turns an open Hugging Face model into an OpenAI-compatible server on one Nvidia GPU on a Linux machine in about ten minutes: install the vllm package, run vllm serve, then call http://localhost:8000/v1/chat/completions from curl or Python. vLLM started at UC Berkeley's Sky Computing Lab and describes itself on GitHub as "a high-throughput and memory-efficient inference and serving engine for LLMs." Its core trick is PagedAttention, which manages the attention key-value (KV) cache in small blocks, combined with continuous batching, so a single GPU can answer many requests at once. If you only want to chat with a model on your own laptop, our guide to [running a local LLM with Ollama](https://bullquill.com/how-to-run-local-llm-with-ollama/) is the easier path. If you need an endpoint that a team, an app or an agent can hit all day, read on. ## vLLM requirements: GPU, CUDA and Python The vLLM GPU installation page lists the following for NVIDIA CUDA. AMD ROCm, Google TPU, Intel XPU and Apple Silicon (through the separate vLLM-Metal project) have their own guides. | Requirement | What vLLM needs (NVIDIA) | Notes | | ---------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------- | | Operating system | Linux | Windows is not supported natively; the docs suggest WSL | | Python | 3.11 to 3.14 | The quickstart uses 3.12 in its examples | | GPU | Compute capability 7.5 or higher | Examples given: T4, RTX 20xx, A100, L4, H100, B200 | | CUDA build | Default wheels built with CUDA 12.9 | Builds for CUDA 12.8 and 13.0 are offered on GitHub releases; Blackwell (B200, GB200) needs at least 12.8 | | Environment | A fresh virtual environment | Compiled kernels can clash with other CUDA or PyTorch builds | *Source: vLLM documentation, GPU installation and quickstart pages, checked 27 October 2026.* VRAM is the real limit. The model weights must fit on the GPU, and whatever memory is left becomes KV cache, which decides how many tokens of context and how many parallel requests you can hold. A small instruct model such as Qwen2.5-1.5B-Instruct, which the vLLM quickstart uses, is a safe first test on almost any supported card. ## Step 1: Install vLLM with uv or pip The vLLM team recommends `uv`, a fast Python package manager, because its `--torch-backend=auto` option picks the right PyTorch build for your installed CUDA driver: ``` uv venv --python 3.12 --seed source .venv/bin/activate uv pip install vllm --torch-backend=auto vllm download-kernels ``` If you prefer plain pip, point it at the PyTorch CUDA 12.9 index: ``` python3 -m venv .venv source .venv/bin/activate pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129 vllm download-kernels ``` Do not skip `vllm download-kernels`. According to the installation guide it installs precompiled FlashInfer kernels; without them vLLM compiles kernels at startup, which can take several minutes on Hopper and newer GPUs. Run it again after every upgrade, in the same environment. To try vLLM without creating a permanent environment, `uv run --with vllm vllm --help` works too. ### Gated models and Hugging Face tokens vLLM downloads weights from the Hugging Face Hub on first use and caches them in `~/.cache/huggingface`. Gated models such as Meta's Llama family require you to accept the licence on the model page and export a token first: ``` export HF_TOKEN=hf_your_token_here ``` If Hugging Face is slow or blocked on your network, you can switch the download source to ModelScope with `export VLLM_USE_MODELSCOPE=True` before starting the engine. ## Step 2: Start the OpenAI-compatible server with vllm serve One command downloads the model, loads it onto the GPU and starts the API: ``` vllm serve Qwen/Qwen2.5-1.5B-Instruct ``` By default the server listens at `http://localhost:8000` and serves one model per process. Use `--host 0.0.0.0` to accept connections from other machines and `--port` to change the port. Before exposing it beyond localhost, require a key with `--api-key` or the `VLLM_API_KEY` environment variable; the quickstart notes that several keys can be supplied and any of them is accepted. Check that the model is live: ``` curl http://localhost:8000/v1/models ``` ### Key vllm serve flags explained These are the engine arguments you will touch most often. Descriptions and defaults come from the vLLM engine-arguments reference. | Flag | What it does | Default | When to change it | | ------------------------- | ------------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------- | | \--max-model-len | Context length (prompt plus output) | Read from the model config | Lower it (e.g. 8192) when you hit KV-cache out-of-memory errors on a long-context model | | \--gpu-memory-utilization | Fraction of GPU memory vLLM may use, 0 to 1 | 0.92 | Lower it when another process shares the GPU | | \--tensor-parallel-size | Splits each layer across GPUs | 1 | Set to your GPU count when the model is too big for one card | | \--pipeline-parallel-size | Splits the model by layers | 1 | Multi-node setups or GPUs without NVLink | | \--quantization | Weight quantization method | Read from the model's quantization config | Rarely needed for pre-quantized AWQ, GPTQ or FP8 checkpoints | | \--dtype | Weight and activation precision | auto | Force half on older cards without BF16 support | | \--served-model-name | Name clients send in the model field | Same as the model argument | Give clients a short, stable alias | | \--max-num-seqs | Maximum sequences processed per iteration | Set by the engine | Cap concurrency to protect latency | *Source: vLLM documentation, engine arguments and parallelism pages, checked 27 October 2026.* A realistic single-GPU command for a mid-sized model looks like this: ``` vllm serve Qwen/Qwen2.5-7B-Instruct \ --served-model-name qwen7b \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --api-key change-me ``` ### Multi-GPU: tensor and pipeline parallelism The vLLM parallelism guide gives a simple rule. If the model is too large for one GPU but fits on one machine, use tensor parallelism. If it is too large for one machine, combine tensor and pipeline parallelism. On GPUs without NVLink, the docs say pipeline parallelism can deliver higher throughput with less communication overhead. Its own examples: ``` # Four GPUs in one server vllm serve facebook/opt-13b --tensor-parallel-size 4 # Eight GPUs total vllm serve gpt2 --tensor-parallel-size 4 --pipeline-parallel-size 2 ``` ### Quantized models Quantization shrinks weights so a bigger model fits in the same VRAM. vLLM's quantization docs cover AWQ, GPTQ, FP8, INT8 and INT4 schemes, BitsAndBytes, and tools such as LLM Compressor and NVIDIA Model Optimizer. In most cases you simply serve an already-quantized checkpoint and vLLM reads the method from the model config. GGUF files, the format popular with llama.cpp and Ollama, also load, but the vLLM docs warn that GGUF support "is highly experimental and under-optimized" and recommend taking the tokenizer from the base model: ``` vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B ``` For the trade-offs between 4-bit, 8-bit and full-precision weights, see our explainer on [LLM quantization and GGUF](https://bullquill.com/what-is-llm-quantization-gguf/). ## Step 3: Run vLLM with Docker The project publishes an official image, `vllm/vllm-openai`. The command in the vLLM Docker guide is: ``` docker run --runtime nvidia --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=$HF_TOKEN" \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ --model Qwen/Qwen3-0.6B ``` The volume mount keeps downloaded weights between container restarts. `--ipc=host` (or a large `--shm-size`) matters because PyTorch passes data between processes through shared memory, especially under tensor parallelism. Any engine argument from the table above can be appended after the image name. The host needs the NVIDIA Container Toolkit installed for `--gpus all` to work. ## Step 4: Query the vLLM API with curl According to the vLLM online-serving docs, the server exposes the Completions, Chat Completions, Responses, Embeddings and audio Transcription and Translation APIs under `/v1`. A chat request: ``` curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer change-me" \ -d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain PagedAttention in two sentences."} ], "max_tokens": 200 }' ``` Leave out the `Authorization` header if you did not start the server with an API key, and use your `--served-model-name` alias in the `model` field if you set one. Chat requests only work when the model ships a chat template in its tokenizer configuration; otherwise the docs say every chat request will error, and you must pass one with `--chat-template ./template.jinja`. ## Step 5: Call vLLM from Python with the OpenAI client Because the routes match OpenAI's, the official `openai` package works unchanged. Install it with `pip install openai`, then: ``` from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY", # or the value you passed to --api-key ) response = client.chat.completions.create( model="Qwen/Qwen2.5-1.5B-Instruct", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Give me three uses for a self-hosted LLM."}, ], max_tokens=300, ) print(response.choices[0].message.content) ``` Streaming uses the standard OpenAI pattern: ``` stream = client.chat.completions.create( model="Qwen/Qwen2.5-1.5B-Instruct", messages=[{"role": "user", "content": "Write a haiku about GPUs."}], stream=True, ) for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True) ``` This is the main reason teams pick vLLM: an app written for a hosted API can move to a self-hosted model by swapping the base URL and model name. Frameworks that accept an OpenAI-compatible endpoint, from LangChain to a home-grown [local RAG app in Python](https://bullquill.com/how-to-build-rag-app-python-local/), can point at vLLM the same way. If you also work with Anthropic's hosted models, compare the request shape in our [Claude API Python tutorial](https://bullquill.com/how-to-use-claude-api-python/). ### Watch out for the model's default sampling settings vLLM applies the `generation_config.json` from the model's Hugging Face repository when one exists, so the model creator's recommended temperature and top-p can override what you expect. Start the server with `--generation-config vllm` to use vLLM's own defaults instead, or set sampling values explicitly in every request. ## Step 6: Offline batch inference without a server For one-off jobs, such as labelling a dataset or scoring evaluation prompts, you can skip HTTP and use the `LLM` class directly. This example follows the vLLM quickstart: ``` from vllm import LLM, SamplingParams prompts = [ "The capital of France is", "The future of AI is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) llm = LLM(model="facebook/opt-125m") outputs = llm.generate(prompts, sampling_params) for output in outputs: print(output.prompt, "->", output.outputs[0].text) ``` Note that `llm.generate` does not apply a chat template. For instruct or chat models, the docs recommend `llm.chat`, which accepts messages in the OpenAI format: ``` llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct") messages = [{"role": "user", "content": "Summarise vLLM in one line."}] result = llm.chat(messages, SamplingParams(max_tokens=64)) print(result[0].outputs[0].text) ``` If you fine-tuned your own adapter with our [LoRA fine-tuning guide](https://bullquill.com/how-to-fine-tune-llm-with-lora/), offline batch runs like this are a cheap way to compare it with the base model on hundreds of prompts. ## vLLM vs Ollama vs llama.cpp: which to use All three tools can expose an OpenAI-style API, but they target different jobs. | | vLLM | Ollama | llama.cpp (llama-server) | | --------------------- | ------------------------------------------------------ | ------------------------------- | ------------------------------------------------------ | | Best for | Many concurrent users on GPU servers | One user on a desktop or laptop | Lightweight, portable inference, including CPU | | Main model format | Hugging Face weights (safetensors), AWQ, GPTQ, FP8 | GGUF via the Ollama library | GGUF | | GGUF support | Experimental | Native | Native | | OpenAI-compatible API | Yes, default port 8000 | Yes, at localhost:11434/v1 | Yes, chat completions, responses and embeddings routes | | Concurrency | PagedAttention plus continuous batching | One-command local runtime | Parallel decoding and continuous batching | | Distributed serving | Tensor, pipeline, data, expert and context parallelism | Not its focus | Not its focus | | Platforms | Linux first; plugins for TPU, Gaudi, Apple Silicon | macOS, Windows, Linux | Almost any OS, CPU or GPU | *Sources: vLLM GitHub repository and docs; Ollama OpenAI-compatibility docs; llama.cpp server README; checked 27 October 2026.* A practical rule: prototype with Ollama, and move to vLLM when you have more than a handful of simultaneous users, need multi-GPU tensor parallelism, or want to serve full-precision or FP8 checkpoints straight from Hugging Face. ## Common vLLM errors and fixes - **CUDA out of memory at startup:** the model plus KV cache does not fit. Lower `--max-model-len`, use a quantized checkpoint, or add GPUs with `--tensor-parallel-size`. - **Another program needs the GPU:** reduce `--gpu-memory-utilization`, for example to 0.7. - **Very slow first start:** run `vllm download-kernels` so FlashInfer kernels are not compiled at launch. - **Chat requests fail but completions work:** the model has no chat template; pass `--chat-template`. - **401 errors from clients:** the server was started with `--api-key`; send it as a Bearer token or in the client's `api_key`. - **Odd or repetitive output:** check whether the model's `generation_config.json` is changing sampling, and test with `--generation-config vllm`. ## What it means vLLM has become one of the standard ways to self-host open models: its GitHub repository showed about 95,600 stars when checked on 27 October 2026, and its licence is Apache-2.0, so it can be used commercially. For developers, the practical payoff is portability. Code written against the OpenAI Chat Completions format can run on your own GPUs by changing two lines, which keeps sensitive data in-house and turns per-token API bills into a fixed hardware cost. The trade-off is operational work. You own GPU drivers, CUDA versions, memory tuning and uptime. Start with a small model on one GPU, confirm the curl and Python calls work, then scale context length, concurrency and GPU count one flag at a time. ## Frequently asked questions ### What GPU do I need to run vLLM? On NVIDIA hardware vLLM needs a GPU with compute capability 7.5 or higher, such as a T4, RTX 20-series, A100, L4, H100 or B200, running Linux with Python 3.11 to 3.14\. The model weights plus KV cache must fit in the card's VRAM. ### What port does vllm serve use? By default the vLLM server listens at http://localhost:8000, with OpenAI-compatible routes under /v1 such as /v1/chat/completions. Change it with the --host and --port flags. ### Can I use the OpenAI Python library with vLLM? Yes. Create an OpenAI client with base\_url set to http://localhost:8000/v1 and any api\_key (or the key passed to --api-key), then call client.chat.completions.create with the model name the server is serving. ### Should I use vLLM or Ollama? Ollama is simpler for one person running models on a laptop. vLLM is built for serving many concurrent requests on GPU servers, with continuous batching and multi-GPU tensor parallelism.