How to Use vLLM: A Practical First-Day Guide

# How to Use vLLM: A Practical First-Day Guide

If you only need the one-line answer: vLLM is the open-source LLM inference and serving engine out of the Sky Computing Lab at UC Berkeley, and on Linux you can install it with uv pip install vllm --torch-backend=auto, then run a model offline with from vllm import LLM, SamplingParams followed by llm = LLM(model="facebook/opt-125m") and outputs = llm.generate(prompts, sampling_params), or serve it as an OpenAI-compatible API with vllm serve Qwen/Qwen2.5-1.5B-Instruct.

How to use vLLM, concretely: pick the install path that matches your hardware, run offline inference through LLM.generate, then expose the model behind an OpenAI-compatible API with vllm serve. The rest of this guide unpacks the whole arc and flags the two production footguns the vLLM team calls out in their own docs.

What vLLM Is, and Why It Wins on Throughput

vLLM started as a research project in UC Berkeley’s Sky Computing Lab and is now maintained by a community of more than 2,000 contributors across many dozens of academic institutions and companies. The project README describes it as “a fast and easy-to-use library for LLM inference and serving” and lists its design pillars as PagedAttention for KV cache management, continuous batching of incoming requests, chunked prefill, and prefix caching.

If you came from my guide to running local LLMs, that piece stays at the wrapper level; learning how to use vLLM starts at the engine layer underneath, where batched throughput is the whole point.

The reason it shows up in every production LLM stack is PagedAttention. The SOSP 2023 paper from the original Berkeley team describes it as an attention algorithm inspired by classic OS virtual memory: each sequence’s KV cache is partitioned into fixed-size blocks, the blocks can live anywhere in physical GPU memory, and a block table tracks the mapping between logical and physical blocks. The result is near-zero KV-cache waste and large batch sizes at the same latency.

How much faster, exactly? Treat these as configuration-specific claims, not universal guarantees. The launch blog reports up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher throughput than Text Generation Inference on LLaMA-7B / A10G and LLaMA-13B / A100 with ShareGPT-sampled inputs and outputs. The SOSP paper reports vLLM sustaining 1.7x-2.7x the request rate of Orca (Oracle) and 2.7x-8x of Orca (Max), and up to 22x the request rate of FasterTransformer, on ShareGPT at comparable latencies. The blog also notes PagedAttention limits memory waste to the last block of each sequence, producing “under 4%” waste in practice, and that PagedAttention’s memory sharing cuts memory use for parallel sampling and beam search by “up to 55%”, translating to “up to 2.2x” throughput improvement for those decoding algorithms.

Every number above is tied to a specific model, GPU, and dataset, so treat any how-to-use-vLLM multiplier as a hypothesis until your own workload confirms it.

Picking the Right Install Path

The official Quickstart on Read the Docs says the prerequisites are Linux plus Python 3.10 through 3.13, and that vLLM also runs on macOS through a separate vLLM-Metal community plugin for Apple Silicon GPU acceleration. vLLM doesn’t support Windows natively; the docs point at community-maintained WSL forks for that. Picking the right door is the first real skill in how to use vLLM, because the wrong door usually fails during install, before you ever reach a prompt.

NVIDIA CUDA

The recommended path for how to use vLLM on NVIDIA silicon runs through uv, Astral’s fast Python environment manager. Create a fresh environment and let uv detect your driver:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

--torch-backend=auto tells uv to inspect the installed CUDA driver and pick the matching PyTorch index at runtime. To force a specific CUDA version, pass --torch-backend=cu129 (or set UV_TORCH_BACKEND=cu129), which the docs say targets CUDA 12.9. If you prefer pip, the equivalent install is pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129.

The GPU install page spells out two floors that bite people. First, the minimum NVIDIA compute capability is 7.5, which covers T4, RTX 20xx, A100, L4, H100, and B200. Second, NVIDIA Blackwell GPUs (B200, GB200) require CUDA 12.8 or higher; vLLM’s prebuilt binaries currently target CUDA 12.9 by default, with additional 12.8 and 13.0 wheels available.

AMD ROCm

For AMD, the docs currently support ROCm 6.3 and above, with prebuilt wheels for ROCm 7.0 (rocm700, Python 3.12, glibc 2.35+) and ROCm 7.2.1 (rocm721). The install command is uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/. Supported GPUs include MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950, needs ROCm 7.0+), Radeon RX 7900 series (gfx1100/1101), Radeon RX 9000 series (gfx1200/1201), and Ryzen AI MAX / AI 300 (ROCm 7.0.2+).

Intel, Google TPU, Huawei Ascend, and Apple Silicon

Intel XPU is in initial support: you need the vllm-xpu-kernels package, Python 3.12, and either an Intel Data Center GPU or an Intel ARC GPU. Official Docker images for Intel start from v0.26.0, and a nightly image is at vllm/vllm-openai-xpu:nightly. On Google TPU you install the separate vllm-tpu package. On Huawei Ascend NPUs you install the community-maintained vLLM Ascend plugin. On Apple Silicon Macs you install the community-maintained vLLM-Metal plugin, which uses MLX as the compute backend and works only with MLX-optimized models from the mlx-community org on Hugging Face, not with arbitrary PyTorch checkpoints.

The docs also call out a constraint worth honoring: to be performant, vLLM compiles many CUDA or ROCm kernels, and the resulting binaries may not be compatible with other CUDA or ROCm versions or PyTorch builds. The official guidance is to install vLLM in a fresh environment and use the bundled PyTorch. Plenty of first attempts at how to use vLLM break right here, from dropping the package into an environment that already owns PyTorch.

Running Offline Inference

Once vLLM is installed, the canonical first run is the offline path. The Quickstart’s example uses the small facebook/opt-125m checkpoint:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

A few non-obvious behaviors are worth knowing up front. By default vLLM applies the model’s generation_config.json from Hugging Face if one is present, so the sampling defaults come from the model creator; if you prefer vLLM’s defaults, pass generation_config="vllm" when constructing LLM. llm.generate doesn’t apply a chat template; for instruct or chat models you must either call llm.chat(messages_list, sampling_params) with OpenAI-style messages, or apply the template yourself with tokenizer.apply_chat_template(messages_list, tokenize=False, add_generation_prompt=True). Models are downloaded from Hugging Face by default; to use ModelScope instead, set VLLM_USE_MODELSCOPE=True before initializing the engine.

Once the basics click, the offline side of how to use vLLM runs deeper than generate and chat. The Offline Inference docs list async queue APIs (LLM.enqueue, LLM.enqueue_chat, LLM.wait_for_completion), profiling APIs (LLM.start_profile, LLM.stop_profile), sleep mode (LLM.sleep, LLM.wake_up), cache management (LLM.reset_mm_cache, LLM.reset_prefix_cache), and a Prometheus metrics snapshot (LLM.get_metrics). For RL training there is a weight-transfer group: LLM.init_weight_transfer_engine, LLM.start_weight_update, LLM.update_weights, LLM.finish_weight_update, LLM.update_weight_version, and LLM.get_weight_version. For pooling models (embedding, classification, retrieval) you use LLM.classify, LLM.embed, LLM.score, or LLM.encode instead of LLM.generate and LLM.chat. Cross-worker fan-out is available through LLM.collective_rpc and LLM.apply_model.

Serving a Model: How to Use vLLM as an OpenAI-Compatible API

The online half of how to use vLLM is vllm serve <model>. The Quickstart example is vllm serve Qwen/Qwen2.5-1.5B-Instruct. By default the server listens on http://localhost:8000; pass --host and --port to override. The server hosts one model at a time and supports list-models, create-chat-completion, and create-completion endpoints in the OpenAI format.

The OpenAI-Compatible Server docs enumerate the supported endpoints. /v1/completions covers text generation (the suffix parameter is not supported). /v1/chat/completions covers text-generation models with a chat template (the user parameter is ignored). /v1/chat/completions/batch, /v1/responses, /v1/embeddings (embedding models only), and /v1/audio/transcriptions and /v1/audio/translations (ASR models) round out the surface. The Quickstart’s curl http://localhost:8000/v1/models and client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") patterns show how to point the official openai Python client at vLLM with no real auth, and vLLM accepts the same client.chat.completions.create(...) calls you would send to OpenAI.

That base URL is also the seam between a local model and your agent tooling: anything that speaks the OpenAI protocol can treat vLLM as the backend, which is the pattern behind my Hermes Agent review.

Tuning is where how to use vLLM drifts from the OpenAI spec: you can pass extra sampling parameters that vLLM accepts but OpenAI doesn’t, including top_k, use_beam_search, min_p, repetition_penalty, length_penalty, stop_token_ids, include_stop_str_in_output, ignore_eos, min_tokens, response_format, structured_outputs, priority, request_id, session_id, prompt_logprobs, logprob_token_ids, bad_words, prefix_cache_salt, enable_response_messages, vllm_xargs, kv_transfer_params, ec_transfer_params, and chat_template_kwargs. Two extra HTTP headers are also supported: X-Request-Id, which is only emitted when you launch the server with --enable-request-id-headers, and X-Vllm-Priority, an integer that overrides the priority field in the request body and requires priority scheduling to be enabled on the served model.

If you want to swap the attention backend, vllm serve accepts --attention-backend. The Quickstart lists the choices: on NVIDIA CUDA, FLASH_ATTN or FLASHINFER; on AMD ROCm, TRITON_ATTN, ROCM_ATTN, ROCM_AITER_FA, ROCM_AITER_UNIFIED_ATTN, TRITON_MLA, ROCM_AITER_MLA, or ROCM_AITER_TRITON_MLA; on Intel XPU, FLASH_ATTN, TRITON_ATTN, TRITON_MLA, XPU_MLA_SPARSE, TORCH_SDPA, or TURBOQUANT. Note that there are no prebuilt vLLM wheels that bundle FlashInfer, so you must install it in your environment first.

Production Footguns

Two callouts in the vLLM docs deserve special attention before you put a vLLM server on the open internet, because this is where how to use vLLM stops being a demo and starts being infrastructure.

The first is API-key scope. --api-key (or VLLM_API_KEY) only authenticates requests to endpoints under /v1, /v2, and /inference. Other endpoints on the same HTTP server are not authenticated, and the most notable example is /invocations, which exposes the same inference capabilities as the /v1 endpoints. Don’t rely on --api-key alone to secure vLLM; put it behind a reverse proxy.

The second is binary compatibility. The GPU install docs warn that vLLM compiles many kernels for performance, which means the resulting wheels may not be compatible with other CUDA or ROCm versions or PyTorch builds. The official guidance is to install vLLM in a fresh environment and use the bundled PyTorch; if you need a different CUDA version or want to use an existing PyTorch installation, you have to build vLLM from source. Installing vllm into an environment that already has a mismatched PyTorch is the most common “vLLM is broken” failure mode.

Models, Modeling Backends, and Custom Checkpoints

Wondering how to use vLLM with a checkpoint the native backend doesn’t cover? The Supported Models docs describe two backends. The native backend implements a long list of architectures under vllm/model_executor/models, covering decoder-only LLMs (Llama, Qwen, Gemma), mixture-of-experts LLMs (Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS), hybrid attention and state-space models (Mamba, Qwen3.5), multimodal models (LLaVA, Qwen-VL, Pixtral), and embedding and retrieval models (E5-Mistral, GTE, ColBERT), plus reward and classification models such as Qwen-Math. The Transformers modeling backend falls back to the upstream Transformers implementation, covering encoder-only, decoder-only, and mixture-of-experts models with full or sliding attention, at identical performance. You can check which one is in use with llm.apply_model(lambda model: print(type(model))): a Transformers backend prints a class name starting with Transformers.

Custom Transformers models not yet in vLLM or Transformers upstream can still be loaded. From the Hub, pass trust_remote_code=True for offline inference or --trust-remote-code for serving. From a local directory, pass the directory to model=<MODEL_DIR> or vllm serve <MODEL_DIR>. The Supported Models docs define four testing tiers for each model: Strict Consistency (greedy-decoding parity vs. Transformers), Output Sensibility (perplexity and no obvious errors), Runtime Functionality (loads and runs), and Community Feedback (everything else).

Frequently Asked Questions

Can vLLM run on Windows?

Not natively. The official docs say vLLM doesn’t support Windows; you need Linux directly, or WSL, or a community-maintained fork.

Do I need an H100 to use vLLM?

No. The minimum NVIDIA compute capability is 7.5, which covers everything from a T4 up to a B200. Performance scales with VRAM and FLOPs, but the engine runs on a wide range of hardware.

Why does my install of vLLM not see my GPU?

The most common cause is a CUDA version mismatch. vLLM’s prebuilt binaries target CUDA 12.9 by default, with additional 12.8 and 13.0 wheels, and Blackwell GPUs require CUDA 12.8 or higher. The docs recommend installing vLLM into a fresh environment and using the bundled PyTorch.

How do I expose vLLM as a drop-in replacement for OpenAI?

Run vllm serve <model>, then point the official openai Python client at http://<host>:<port>/v1 with api_key="EMPTY". The Quickstart’s OpenAI client snippet and the OpenAI-Compatible Server docs cover the full set of supported endpoints. That base-URL swap is the entire trick to how to use vLLM as a drop-in OpenAI replacement.

Is the API key enough to lock vLLM down?

No. --api-key only authenticates the /v1, /v2, and /inference endpoint prefixes; /invocations and other endpoints are not protected. For production, put vLLM behind a reverse proxy that enforces authentication and TLS.

What is PagedAttention, in one sentence?

PagedAttention stores continuous keys and values in non-contiguous physical memory by partitioning each sequence’s KV cache into fixed-size blocks and tracking the mapping between logical and physical blocks, achieving near-zero KV-cache waste and large batch sizes at the same latency.

Can I run vLLM on a Mac?

Yes, through the community-maintained vLLM-Metal plugin. vLLM-Metal uses MLX as the compute backend, not PyTorch, and only works with MLX-optimized models from the mlx-community organization on Hugging Face.

Reviewed & Written By

Tony Simons

Independent tech reviewer and creator of Tony Reviews Things. 14 years of hands-on testing, software auditing, and workflow automation. I test the gear so you don't waste your money on junk.

About Me How I Test