Local AI Field Guide

Start

Why bother — 02

The cloud is fine, until it isn't.

Local AI flips the model. Your hardware, your weights, your data — no API key, no rate limits, no third party reading your prompts.

01

Privacy

Sensitive prompts never leave the device. Crucial for legal, medical, internal, and personal data.

02

Cost

No per-token billing. Once you own the GPU, marginal cost approaches zero.

03

Latency

No network round-trip. Tools and agents respond as fast as your hardware allows.

04

Offline

Airplanes, secure facilities, remote sites. Inference doesn't require connectivity.

05

Control

Pin a model version forever. No silent upgrades changing behavior overnight.

06

Customization

Fine-tune, quantize, swap samplers, edit chat templates. Frontier APIs expose only a subset of that control.

Fundamentals — 03

Anatomy of a language model.

Before runtimes and quantization, three numbers define what a model is and what it can do.

Parameters

The learned weights. Sized in billions: 1B, 7B, 13B, 70B, 405B. More parameters generally mean stronger reasoning — but also more VRAM. Mainstream local sweet spots: 7B–14B for everyday, 30B–70B for serious work.

Context Window

How many tokens the model can see at once. Modern locals range from 8K to 128K to 1M tokens. Longer context costs more memory because of the KV cache (more on that later).

Tokens

The units the model reads and writes. Roughly ¾ of a word in English. The string "hello world" is two tokens; "antidisestablishmentarianism" may be six. Pricing, speed, and limits are all in tokens.

Architecture

Decoder-only transformers, with variants that matter. Dense vs MoE (only a slice of params activate per token). Attention flavors change the memory/speed tradeoff: GQA shares key/value heads across groups of query heads, shrinking the KV cache; MLA compresses attention state through latent vectors, pushing long-context efficiency further. Positional encoding tells the model where tokens sit: RoPE rotates attention features by position, while YaRN stretches RoPE so models can handle longer context windows. These names show up on every model card because they affect VRAM, context length, and throughput.

MTP

Multi-Token Prediction. Instead of training to predict only the next token, the model predicts several future tokens via auxiliary heads. DeepSeek-V3 popularized it at scale, and newer model cards increasingly call out MTP or built-in speculative heads. Two payoffs: denser training signal and faster decoding when the runtime supports it. Availability is model- and backend-specific, so always check the model card and serving docs.

The trick that makes it all possible — 04

Quantization, or: how to fit a 70B model on a gaming GPU.

Models train in 16-bit floats. At inference, you can compress weights to 8, 5, 4, or even 2 bits with minimal quality loss. This is the single biggest unlock for local AI.

Precision	Bits	Quality	Size (7B)
FP16	16	Reference	~14 GB
Q8_0	8	~Identical	~7.5 GB
Q5_K_M	5.5	Excellent	~5.0 GB
Q4_K_M	4.5	Sweet spot	~4.1 GB
Q2_K	2.5	Degraded	~2.7 GB

Formats to know

GGUF — The local-chat default. llama.cpp-native, CPU/Mac-friendly, used by Ollama and LM Studio.
Safetensors — The common Hugging Face checkpoint format. Often the source before conversion or quantization.
EXL2 — ExLlama's mixed-precision format. Fast quantized inference on NVIDIA GPUs.
GPTQ / AWQ — GPU quant formats. Still common in text-generation-webui, vLLM, and older model drops.
MLX — Apple Silicon-native weights for the MLX ecosystem.
ONNX — Portable deployment format for apps, edge devices, Windows GPUs, and NPUs.

Rule of thumb GGUF for local chat Safetensors as source EXL2/GPTQ/AWQ for NVIDIA

The landscape — 05

The open-weight families.

A handful of labs ship the models worth knowing. Each family has a personality, a license, and a release cadence.

Alibaba · Qwen3 / Qwen3.6

Qwen

Qwen3 spans dense 0.6B-32B models plus MoE 30B-A3B and 235B-A22B open weights. Qwen3.6 adds newer coding and agentic releases such as 35B-A3B and 27B Dense. Strong tool use, hybrid thinking, Apache-2.0 open-weight options, and a very fast release cadence.

Moonshot · Kimi K2.7 Code

Kimi

K2-style MoE models are tuned for long-context coding and agent workflows. Kimi K2.7 Code is the current coding-focused entry, while K2.6 remains a strong general long-context option. Check the model list before choosing a suffix.

DeepSeek · V4 / R1-0528

DeepSeek

V4 Pro & V4 Flash, V3.2, and refreshed R1 distills. MTP pioneer at scale. The current line emphasizes long context, thinking mode, tool use, and agentic coding.

Z.ai · GLM-5.2

GLM

GLM-5.2 is the current flagship family, with GLM-5.1 still listed for compatibility. Strong long-context coding, tool use, and agentic workflows. Zhipu's line moves fast, so verify the exact endpoint.

Meta · Llama 4

Llama

Scout and Maverick are native multimodal MoEs, with Behemoth as the larger teacher/preview model. Still the West's flagship open-weight family; the community license has caveats.

Google · Gemma 4

Gemma

Current Google open-weight family for edge, workstation, and multimodal use. Gemma 4 adds small edge variants and larger dense/MoE options; check the Gemma terms and exact model card before commercial use.

Mistral

Mistral · Devstral

Medium 3.5, Small 4, Large 3, Devstral 2 for coding, Pixtral for vision, and Ministral edge models. Many open-weight releases use permissive licenses, but always check the individual model card.

Microsoft · Phi-4

Phi

Phi-4 (14B), Phi-4-reasoning, Phi-4-mini-instruct (3.8B, 128K, CPU-runnable), Phi-4-multimodal. Synthetic-data philosophy: textbook-quality corpora.

MiniMax · LG · IBM

The rest

MiniMax M3, LG EXAONE 4.5, and IBM Granite Vision are worth tracking for long-context, enterprise, and multimodal workloads, but they are rarely the first model you pull.

Also in the mix: Nemotron (NVIDIA), OLMo (AI2 — fully open data + code), SmolLM (HF), Hermes (Nous), Command models (Cohere), Granite (IBM), and Yi (01.AI). Treat leaderboards as a weekly snapshot; verify the current Hugging Face and vendor pages before pulling.

Choose your fighter — 06

Model types and their merits.

"7B model" tells you almost nothing. What kind of 7B determines whether it fits your job. The taxonomy that matters.

Base

Pretrained, raw

Pure next-token predictor. Completes patterns, doesn't follow instructions. The starting point for fine-tuning. No "Instruct" or "Chat" suffix on the name.

Instruct · Chat

The default

Post-trained with SFT then RLHF / DPO to follow instructions and converse. What 99% of users actually want. "Instruct," "Chat," or "IT" in the name.

Reasoning

Think first, answer second

Emits long chains of thought before answering. Strong at math, code, logic. Slower & token-hungry. DeepSeek V4 thinking mode, Qwen3 thinking mode, Phi-4-reasoning, GLM-5.2 Think.

Coder · Agentic

Built for code

Continued pretraining on code, fill-in-the-middle, long-horizon agentic loops. Kimi K2.7 Code, Qwen3-Coder-Next, Qwen3.6, DeepSeek V4, Devstral 2.

Vision · VLM

Multimodal input

Accepts images alongside text. Llama 4, Qwen-VL, Gemma, Pixtral, InternVL, EXAONE, Granite Vision. OCR, screenshots, charts, document Q&A.

MoE

Mixture of Experts

Total parameters massive, only a fraction active per token. Kimi K2, GLM-5.x, DeepSeek-V4, Llama 4, Qwen3-Next, and Gemma MoE variants. High quality per active-param — but the full model still has to fit in memory.

Adjacent specialists: Embedding (retrieval), Rerankers (cross-encoders), Draft models (tiny pairs for speculative decoding), Distilled (small students of bigger teachers — DeepSeek-R1-0528-Qwen3-8B, etc.).

What actually runs the model — 07

Runtimes: llama.cpp and its friends.

A runtime loads weights and turns prompts into tokens. Pick one based on your interface preference and use case.

Runtime	Best for	Interface	Notable
llama.cpp	Maximum compatibility, CPU + GPU	CLI / C++ library	The engine under nearly everything else. GGUF native. Speculative decoding support keeps improving; check current build flags for MTP-style features.
Ollama	Beginners, fast setup	CLI + REST API	One-liner installs. `ollama run qwen3` · `ollama run gemma3` or newer model tags.
LM Studio	GUI users, model discovery	Desktop app	Built-in chat, model browser, OpenAI-compatible server.
vLLM	Production serving, throughput	Python server	PagedAttention, continuous batching. Datacenter-grade.
ExLlamaV2	Single-GPU NVIDIA speed	Python library	Best raw inference speed for EXL2 quants.
MLX / MLX-LM	Apple Silicon native	Python framework	Uses unified memory directly. Built by Apple.
ONNX Runtime	Edge, mobile, Windows GPUs	Cross-platform library	Microsoft's portable inference engine. Phi ships ONNX-native. DirectML on Windows, CoreML on iOS, NPU acceleration on Snapdragon X.
text-generation-webui	Power-user tinkering	Web UI	Every backend, every sampler, every knob.

The bottleneck — 08

It's VRAM, almost always.

More than CPU, more than RAM, more than disk: video memory determines what you can run. Compute speed determines how fast. Prices move fast, but the fit tiers are stable enough to plan around.

GPU tiers — rough fit map

GPU	VRAM	Buy	Rent / hr	Runs (Q4)
RTX 3060	12 GB	~$250 used	spot only	7B comfortably
RTX 3090 / 4090	24 GB	~$700–2.3k	~$0.20–0.80	13B–30B
RTX 5090	32 GB	$1,999+ street	~$0.70–1.20	30B comfortable
A100 / H100	80 GB	~$8k–25k used	~$1–3	70B; Q8 tight
H200	141 GB	server/OEM	~$3–6	120B–235B Q4 class
B200 (Blackwell)	192 GB	server/OEM	~$5–10	235B Q4 class
M3 Ultra Mac Studio	96 GB current store	$3,999+	n/a	70B Q8 / 120B Q4 class

Prices & rentals approximate, USD, June 2026 snapshot. Spot rates on RunPod / Vast.ai / Lambda vary 2–3× by demand. Current Mac Studio M3 Ultra store configs top out below the old 256/512 GB planning examples, so check memory options before budgeting a 405B-class local box.

Backends

CUDA — NVIDIA only. The mature path. Best ecosystem support.
Metal — Apple Silicon. Native, fast, low setup.
ROCm — AMD. Improving rapidly. Linux primarily.
Vulkan — Cross-platform fallback. Intel Arc and older GPUs.
DirectML — Windows abstraction over any DX12 GPU.
CPU — Always works. Usable for 7B Q4 with AVX-512.

The Apple advantage

Unified memory means CPU and GPU share one pool. High-memory Apple Silicon boxes are excellent for quiet local inference, but current availability changes by configuration. Do not plan a 405B-class run unless the machine actually has 256–512 GB unified memory.

Measuring speed — 09

Performance: the three numbers that matter.

t/s

Tokens per second

TTFT

Time to first token

KV

Key-value cache size

Tokens/sec — generation throughput once decoding starts. A 7B Q4 on an M2 Pro hits ~40 t/s. A 70B on an RTX 4090 hits ~15 t/s. Reading speed is ~10 t/s, so anything above that feels real-time.
Time-to-first-token (TTFT) — how long the model takes to process your prompt before generating. Scales with prompt length. A 50K-token document might take seconds to ingest.
KV cache — every previous token's attention state, kept in VRAM. Grows linearly with context. Doubling context can mean doubling memory pressure.

Levers you can pull

GPU layer offload — push as many layers to VRAM as fit; CPU handles the rest.
Batch size — bigger batches = better throughput for multiple parallel requests.
Flash Attention — fused attention kernel. Free speed when supported.
Speculative decoding — a tiny draft model proposes tokens, the big one verifies. 2–3× speedups.
Context length — only allocate what you need. Halving max context can free gigabytes.

The invisible interface — 10

Chat templates: every model has its own format.

Behind every "chat" is a precise string format the model was trained on. Get it wrong and quality collapses silently.

Llama 3

// Llama 3 chat template
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

What is quantization?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

ChatML

// ChatML — used by Qwen, OpenAI-style
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is quantization?<|im_end|>
<|im_start|>assistant

Modern runtimes apply templates automatically via the tokenizer's chat_template field — but you should know it's happening.
System prompts go in the system turn. Tool calls and reasoning sometimes get their own special tokens.
Mismatched templates are the #1 cause of "this model is dumb" reports. Always verify the format.

Where the template fits

01

HTTP request

POST /v1/chat/completions with JSON messages: [...].

02

Parse roles

Server reads each turn: system, user, assistant.

03

Apply template

Tokenizer's chat_template wraps each turn with the model's special tokens.

04

Tokenize

Resulting string becomes token IDs — integers the model actually reads.

05

Model runs

Forward pass, sampling, tokens stream back as text to the client.

Two ways to teach a model — 11

Fine-tuning vs. RAG.

When the base model doesn't know your stuff, you have two paths. They solve different problems and are often used together.

Dimension	Fine-tuning	RAG
Changes	Model weights	Model input (prompt)
Best for	Style, format, behavior	Facts, documents, current data
Update cost	Hours of GPU time per update	Re-embed changed documents
Hallucinations	Can still invent	Grounded in retrieved text
Citations	Difficult	Built-in (you have the source)
Tooling	Unsloth, Axolotl, TRL, LLaMA-Factory	LangChain, LlamaIndex, Haystack
Use when	You need a specific tone or skill	You need the model to know things

Rule: try RAG first. It's cheaper, more flexible, and easier to debug. Fine-tune only when prompting and retrieval have hit their limits.

The democratized fine-tune — 12

LoRA & QLoRA: train a 70B model on one GPU.

Full fine-tuning of a large model requires hundreds of gigabytes of VRAM. LoRA — Low-Rank Adaptation — sidesteps it entirely.

Freeze the base model. Original weights stay untouched.
Inject small trainable matrices. Two low-rank matrices (rank 8, 16, 32) per layer that add to the original weights.
Train only those. Maybe 0.1% of the parameters. Fits on a single consumer GPU.
Ship the adapter. A LoRA file is megabytes, not gigabytes — load on top of any base.

QLoRA

The one-GPU trick.

Load the base model in 4-bit, attach a LoRA in 16-bit, train. A 70B fine-tune drops from "8× A100" to "single 24GB GPU." Released 2023, still the default approach.

Tooling

Unsloth Axolotl TRL LLaMA-Factory PEFT

Meaning, as numbers — 13

Embeddings: turning language into geometry.

An embedding model converts a chunk of text into a high-dimensional vector. Similar meanings land near each other in space. This is what powers semantic search.

Embedding models

pplx-embed — Perplexity's 2026 release. 0.6B & 4B, MIT, Qwen3-base with bidirectional attention. Strong MTEB Multilingual v2 results; up to 32× storage reduction via quantization. No task prefixes needed.
Qwen3-Embedding — 0.6B / 4B / 8B from Alibaba. The peer model pplx-embed is built on. Strong multilingual.
EmbeddingGemma — Google's compact 300M-class on-device embedding model. Strong choice when memory, battery, or mobile deployment matter.
BGE-M3 — BAAI's multilingual, multi-function retrieval baseline. Dense, sparse, and multi-vector retrieval in one model.
nomic-embed-text-v2-moe — Open-weight multilingual MoE embeddings; the newer Nomic default to try before the older v1/v1.5 line.
E5 / mE5 — Microsoft's family. Older now, but still a solid baseline.
Jina · GTE — Long-context (Jina) and Alibaba's GTE general text embeddings.

Vector databases

FAISS — Facebook's library. In-memory, blazing fast, library not a server.
Chroma — Easy local dev. SQLite-style for vectors.
Qdrant — Production-grade Rust server. Excellent filtering.
LanceDB — Embedded, file-based, modern format.
Weaviate · Milvus · pgvector — Heavier, distributed options.

Similarity = cosine distance between vectors. Closer vectors → more semantically related text.

Putting it together — 14

The RAG pipeline, end to end.

Retrieval-Augmented Generation. Fetch relevant context, paste it into the prompt, let the model answer with sources.

Indexing — done once

01

Load

PDFs, HTML, code, transcripts. Extract clean text.

02

Chunk

Split into 256–1024 token windows with overlap.

03

Embed

Run each chunk through the embedding model.

04

Store

Vectors + chunks + metadata in the vector DB.

Querying — every request

05

Embed query

Same embedding model as indexing.

06

Retrieve

Top-k nearest chunks by cosine similarity.

07

Rerank

Optional: cross-encoder reorders for relevance.

08

Generate

Stuff into prompt, model answers with citations.

Advanced patterns: hybrid search (BM25 + vectors), HyDE (hypothetical answer embedding), agentic retrieval (model decides what to search for).

Beyond chat — 15

Structured output & function calling.

Free-form text is the demo. Production systems need parseable output and the ability to call code. Local runtimes support both — better than most people realize.

Structured output

JSON mode — runtime forces valid JSON. Available in llama.cpp, Ollama, vLLM.
GBNF grammars — llama.cpp's grammar-constrained sampling. Define exactly what tokens are legal.
Outlines / LMQL — Python libraries that compile regex and Pydantic schemas into grammars.
XGrammar — modern, fast grammar engine used by vLLM.

Function / tool calling

Model emits a structured request to call a named function with arguments.
Your code runs the function, returns the result, model continues.
Most modern open models are tool-call trained: Llama 4, Qwen3+, Kimi K2.7, DeepSeek V4, GLM-5.2, Mistral, Hermes.
This loop, repeated, is what people mean by an agent.

// Typical tool call emitted by a local model
{
  "name": "search_web",
  "arguments": { "query": "stanford red hex code" }
}

The new connective tissue — 16

MCP: the USB-C for AI tools.

The Model Context Protocol is an open standard for connecting models to tools, data sources, and prompts. One protocol, many servers, any client.

Servers expose tools, resources, and prompts. Anyone can write one.
Clients (Claude Desktop, IDEs, agents) connect to servers and route the model's tool calls.
Transport is stdio or HTTP. Servers can run locally or remotely.
Replaces a tangle of bespoke integrations with a single shared contract.

Why it matters

Tools, decoupled.

Before MCP, every assistant invented its own plugin format. Now your filesystem server, your GitHub server, your Postgres server work the same way in any MCP-aware client — including local-LLM frontends.

Open standard Stdio / HTTP JSON-RPC

A pragmatic starting point — 17

A recommended first stack.

If you remember nothing else, this is the path of least resistance from zero to running.

Runtime

Ollama for ease, LM Studio if you want a GUI, llama.cpp directly if you're a developer. Each can expose an OpenAI-compatible local API.

First model

Qwen3 8B or Gemma 4 E4B at Q4 — broadly capable, runs on most hardware. Phi-4-mini-instruct if you're CPU-only on a laptop. Enable Qwen3's thinking mode or pull DeepSeek-R1-0528-Qwen3-8B when you want a small reasoning model.

Coding model

Qwen3-Coder-Next or Qwen3.6 35B-A3B for local coding; DeepSeek V4 Flash/Pro if you're using API or large-model serving. Kimi K2.7 Code if you want a long-horizon coding agent. Use through opencode, Claude Code, Zed, or Cursor's local-model mode.

Embedding model

qwen3-embedding or embeddinggemma via Ollama for one-line install. Try nomic-embed-text-v2-moe or bge-m3 for multilingual retrieval, and upgrade to pplx-embed-v1 when you need top-tier retrieval.

Vector store

Chroma for prototypes, Qdrant when it gets real. LanceDB if you want embedded.

RAG framework

LlamaIndex for retrieval focus, LangChain for orchestration, or roll your own — it's a few hundred lines.

Where to browse

Hugging Face for weights, r/LocalLLaMA for news, bartowski, unsloth, and mradermacher for fresh quantized GGUFs. TheBloke's older repos are still useful history, not the current feed.

Things that will trip you up — 18

Mistakes everyone makes first.

01

Running too large a quant

You can fit Q8 in VRAM, but generation will be slow. Q4_K_M is usually faster AND nearly as good. Quality vs speed isn't linear.

02

Ignoring the chat template

Manually constructing prompts without the model's expected special tokens. Output quality silently tanks.

03

Default sampler settings

Temperature 0.7 is fine for chat, terrible for code. Greedy (temp=0) for deterministic tasks. Learn min_p, top_p, repetition_penalty.

04

Forgetting the KV cache

Loading a 7B Q4 model uses 4 GB. Then a 32K context conversation adds another 4 GB. Plan for it.

05

Comparing on vibes

Try the same prompts across models. Use lm-eval-harness or your own test set. Benchmarks rarely reflect your workload.

06

Chasing parameter count

A well-tuned 14B often beats a sloppy 70B for real tasks. Stop fetishizing size. Test before committing.

End of guide — 19

¶

The fastest way to learn this is to start running things.

Install Ollama. Pull a 7B model. Ask it something. Everything in this deck makes more sense once tokens are streaming out of your own machine.

Weights

huggingface.co

The open-model hub. Every weight, every quant.

Community

r/LocalLLaMA

News, benchmarks, drama, working examples.

Reference

llama.cpp · ollama

The engines worth reading the source of.

Standards

modelcontextprotocol.io

Where tools, agents, and models converge.