How to run local LLMs is one of the few questions in AI where the honest answer has gotten dramatically simpler in the last two years. You no longer need a data center or a research budget. A laptop with a decent GPU, or a Mac with unified memory, can hold a genuinely useful open model, and the tools that do the heavy lifting are free and open source.
I’ve been watching this space since the llama.cpp days, when learning how to run local LLMs meant compiling C++ and praying the quant worked. The gap between possible and practical keeps closing, and this guide is the setup I’d hand someone who wants private AI on hardware they actually own.
The plan: pick a path, install the tool, pull a model that fits your memory, and verify it works. You can be done in five minutes with Ollama, or spend an afternoon with llama.cpp if you want exact control. I’ll show both, plus when a vLLM server makes sense. Once you know how to run local LLMs, the rest is just choosing the right path for your workload.
How to Run Local LLMs: Pick the Path That Fits
Local inference is governed by two numbers, and they decide everything. The first is VRAM capacity, which decides whether a model fits on your hardware. The second is memory bandwidth, which decides how fast the model generates tokens. Capacity is a yes or no. Bandwidth is a speed limit.
A 2026 tutorial on running LLMs locally explains the math this way: for a dense model, tokens per second is roughly memory bandwidth divided by the bytes read per token, because the GPU streams essentially all the weights for every token. That’s why an RTX 5090 with 1.79 TB/s of bandwidth can blow past a Mac for raw generation speed, while a Mac with 128 GB of unified memory can hold a 70B-class model no consumer GPU can fit.
That tradeoff is the whole game. Fast hardware generates quickly but caps your model size. Big memory runs bigger models but slower. You pick the tool based on which side of that tradeoff you care about, and most people who want to learn how to run local LLMs end up caring about both at different moments.
What You Need Before You Run Local LLMs
Before you start, know your memory number. On a machine with an NVIDIA GPU, run nvidia-smi --query-gpu=name,memory.total --format=csv and read the total. On a Mac, unified memory is your VRAM, so check About This Mac or run sysctl hw.memsize and divide by a billion.
Then pick a model with a quantized size that fits with room to spare. At 4-bit, the common local sweet spot, a 12B model needs roughly 7-8 GB, a 27B model needs 15-17 GB, and a 70B-class model needs around 40 GB. The file size is a floor, not the full requirement. Add KV cache for your context length and runtime overhead, and leave a few gigabytes of headroom. If your card sits at the bottom edge of a row, drop to the row above.
You also want a project or a task that’s safe to point an open model at. Local is private, but private doesn’t mean consequence-free. Keep production credentials out of anything you paste into a prompt, and commit or stash unrelated work before you let a model touch your files.
Path One: Ollama for a Five-Minute Setup
Ollama wraps llama.cpp behind a one-command interface, and its official repository describes it as a tool for running large language models locally. It lists llama.cpp among its supported inference backends and points to reference docs for the CLI, REST API, model imports, and Modelfile format. Start here if you want a model running in the next five minutes, which is the fastest way to learn how to run local LLMs on a single machine.
Install on macOS or Linux with the official one-command script:
curl -fsSL https://ollama.com/install.sh | shWindows users download the installer from ollama.com and run it. There’s also an official Docker image on Docker Hub if containers are more your style. Verify the install before moving on:
ollama --versionIf that fails, the install script didn’t put ollama on your PATH. Restart the terminal or check the install log before continuing.
Pull and run a model:
ollama run gemma4:12bThe first run downloads several gigabytes of weights, which takes a few minutes on a typical connection, then drops you into an interactive chat. Type a prompt and confirm you get a response. If you see output, the model is loaded and running entirely on your hardware. Exit with /bye.
Verify the Ollama API Is Reachable
Here’s the part that surprises most people. Ollama’s official OpenAI compatibility documentation shows the local server exposing an OpenAI-compatible endpoint at http://localhost:11434/v1, where an API key is required but ignored. Existing OpenAI client code can point at Ollama with no other setup.
Confirm it from a second terminal while a model is loaded:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "gemma4:12b", "messages": [{"role": "user", "content": "Reply with one word: confirmed."}]}'A JSON response with your model’s name means the server is up and any OpenAI-compatible application can use it. For most single-machine use, this is the whole guide. Stop here unless you need finer control over quantization, GPU offload, or concurrent request throughput.
Path Two: llama.cpp for Exact Control
llama.cpp is a plain C/C++ inference engine for running large language models on consumer CPUs and GPUs, and its repository documents integer quantization from 1.5-bit to 8-bit, CUDA kernels for NVIDIA GPUs, Metal support on Apple silicon, and CPU plus GPU hybrid inference. Ollama hides all of that behind its own interface. When you want to tune exactly how a model fits, you build llama.cpp yourself.
The quickest path is the prebuilt binaries from the releases page or the llama.app installer. If you build from source, the official build guide covers CUDA, Metal, Vulkan, and the rest:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -jThen download a GGUF model directly from Hugging Face and run it:
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUFFor serving, llama.cpp ships an OpenAI-compatible API server:
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUFThe quant matters here. Q4_K_M is the common default for good reason: roughly a quarter the size of the original weights with minimal quality loss. Spend spare memory on more parameters before more precision. This path is where you end up when you need to know how to run local LLMs with a specific quant profile or exact GPU offload.
Path Three: vLLM for Serving Under Load
A vLLM server is for when the model has to back an application under concurrent requests. Ollama and llama.cpp will answer one request happily. vLLM’s continuous batching shines when ten people, or ten automated jobs, hit the same model at once.
The tradeoff is operational weight. vLLM wants a GPU that can hold the model and a deployment that justifies the setup time. For a single user on a laptop, it’s overkill. For a team self-hosting an internal assistant, it’s the right tool, and it’s the path you want once you understand how to run local LLMs for more than one person.
The test that tells you batching is working: send ten parallel requests and watch the throughput. If the server is doing its job, the total time for ten requests barely exceeds the time for one, because the GPU fills idle slots instead of waiting.
Pick a Model That Fits
Model choice is where most first attempts go wrong, and it’s almost always a memory problem, not a capability problem. At 4-bit, a 12B model runs comfortably on a 16 GB laptop. A 27B dense model wants a 16-24 GB card. A 70B-class model needs dual 24 GB cards, a 48 GB card, or a Mac with enough unified memory.
Mixture-of-experts models break the dense assumption in your favor. A model like Qwen3.6-35B-A3B has 35B total parameters but activates only about 3B per token. You still hold all the experts in memory, which is the capacity cost, but each token reads only the active subset, which is the bandwidth saving. That’s why MoE models generate three to five times faster than dense models of similar size on identical hardware. When you’re learning how to run local LLMs, that’s the shortcut worth remembering.
Connect Your Apps to a Local Model
Once a model is serving, pointing applications at it is the same trick for every path. Ollama exposes http://localhost:11434/v1. llama.cpp’s server does the same. Both speak the OpenAI shape, so tools that accept a custom base URL work without patches.
That’s also how agent frameworks get local backends. My Hermes Agent review covers a setup where the whole stack can point at an Ollama box instead of a cloud API. The pattern is identical to the one OpenAI’s own docs describe: set the base URL, set a dummy key, and the client talks to your hardware. If you want to know how to run local LLMs for an agent workflow, this is the connection point.
Verify It Worked
A model that installed but never answered is a model you don’t have. Verification is three steps. First, confirm the tool reports a version. Second, run a real prompt and confirm output. Third, if you plan to serve, hit the API endpoint with curl and confirm a JSON response.
Two quick troubleshooting checks. If a pull hangs at manifest, check your network. If chat loads but crawls, run ollama ps in a second terminal and check the processor column. You want to see 100 percent GPU. Split CPU inference still works, but it’s the difference between a useful tool and a frustrating one.
When Local LLMs Aren’t the Answer
Local is a system decision, not a default. Run local for privacy, steady-volume cost, and control. Use an API for frontier quality, low ops, and bursty traffic. A 70B quant on your desk is impressive and private, but it is not GPT-class frontier behavior, and if your workload spikes unpredictably, a server you don’t have to babysit wins.
The good news is that the two worlds connect. Google's DiffusionGemma announcement is a useful look at where local hardware is heading, and the tooling around it keeps getting better. If you can run a model on your own machine, you’ve already crossed the hardest line: your data stays yours.
