Trail guide

How to run an LLM locally

Running an open source model on your own hardware keeps every prompt, document, and answer inside your fence. No API keys, no per-token bill, no third party reading your data. Here is the whole ride, from picking a model size to serving it behind a local endpoint.

01
Size the model to your hardware
8 GB of VRAM or unified memory comfortably runs a 7–9B model at 4-bit. 16 GB opens up 13–14B. 24 GB handles most 30B quantized builds. CPU-only works for 3B-class models if you can accept a few tokens per second.
02
Download the weights
Open the model page on ModelCorral, check the license and file list, then pull the repo. Weights are free; you only pay for the hardware that runs them.
```
huggingface-cli download Qwen/Qwen2.5-7B-Instruct --local-dir ./qwen-7b
```
03
Quantize if memory is tight
4-bit quantization cuts memory roughly to a third with modest quality loss. GGUF builds run under llama.cpp and Ollama; AWQ and GPTQ builds run under vLLM and Transformers on NVIDIA GPUs.
04
Serve it and point your app at it
Most local runtimes expose an OpenAI-compatible endpoint, so existing client code works with a changed base URL. Nothing leaves your machine.
```
python -m vllm.entrypoints.openai.api_server --model ./qwen-7b
```

Local runtimes worth saddling

llama.cpp / GGUF

Widest hardware support, including CPU and Apple Silicon.

Ollama

Simplest install and model management for desktops.

vLLM

High-throughput GPU serving with an OpenAI-compatible API.

Transformers

Maximum flexibility for research and custom pipelines.

Compare open source LLMs Browse the herd

Questions from the trail

What hardware do I need to run an LLM locally?

For a 7B model at 4-bit quantization, plan on roughly 6–8 GB of VRAM or unified memory. Apple Silicon Macs with 16 GB and NVIDIA cards with 8 GB or more both work well. CPU-only inference is possible for smaller models but noticeably slower.

Is running an LLM locally private?

Yes. Once the weights are on disk, inference happens entirely on your machine — no prompt, document, or output is sent to a third party. That is the core reason regulated teams run models on their own infrastructure.

Can I run a local LLM offline?

Yes. After the initial download, local runtimes need no network connection. Cache the model files and the tokenizer, and the stack runs air-gapped.

How much does it cost to run an LLM locally?

The models are free to download. Your cost is hardware and electricity, or hourly GPU rental if you run in the cloud — with no per-token API charges.