LLM Quantization Explained: Fit Bigger Models in Less VRAM

A 70B model in full precision needs more memory than most desktop GPUs hold. Quantization changes that math. It shrinks model weights from 16-bit floats to 8-bit, 4-bit, or fewer bits per weight, and the model still answers questions, writes code, and holds a conversation. The tradeoff sits in quality, speed, and setup effort. This guide explains the mechanics, the memory math, the main methods, and the bit width that fits most local setups.

Verdict: Which Quantization Route Fits You

Start with the short version. If you run models on a laptop or a single consumer GPU, download a pre-quantized GGUF file at 4 bits and run it in Ollama or llama.cpp. That path needs the least setup and works on the widest range of hardware. If you serve models on NVIDIA hardware and want the best 4-bit accuracy, use AWQ or GPTQ weights, which calibrate against sample data before they ship. If you want to quantize a model yourself at load time with no calibration step, bitsandbytes in 4-bit or 8-bit mode loads straight from a standard checkpoint. Hugging Face sums up the tiers: 8-bit lands very close to bf16, 4-bit holds high accuracy with the right method, and anything below 4 bits shows a clear drop, sharpest at 2 bits.

The default rule holds for most readers: 8-bit costs almost nothing in quality, 4-bit is the everyday sweet spot for local use, and sub-4-bit formats belong to experiments and research. Pick your route from that rule, then benchmark the model on your own prompts before you commit to it. A choice that looks good on a benchmark table can still stumble on your task, so a quick test on your own workload beats any generic recommendation.

Quantization in Plain Terms

A language model stores its knowledge as weights, and each weight is a number. In bf16, each weight takes 2 bytes. Quantization maps those 16-bit values onto a smaller set of integers, such as the 256 values an 8-bit integer can hold, or the 16 values a 4-bit integer can hold. The model then stores the small integers plus a scale factor that maps them back to an approximate version of the original range. That scale factor is the key detail. Without it, rounding would crush the values into a useless range. With it, the model keeps most of its behavior while using a fraction of the memory.

Also Read:

LoRA Fine-Tuning Explained: Adapt LLMs for Less

Two design choices shape how much quality survives. The first is symmetric versus asymmetric quantization. Symmetric quantization centers the range on zero with a single scale. Asymmetric quantization adds a zero-point offset, which helps when weights cluster away from zero. The second choice is granularity. Per-channel quantization applies one scale to a whole row of weights. Per-group quantization splits weights into small groups, and GPTQ defaults to groups of 128 weights, with a separate scale for each group. Smaller groups follow the weight distribution better, so they lose less accuracy, at the cost of a little extra bookkeeping at runtime.

Weight-only quantization shrinks the stored weights and leaves activations in higher precision. Weight-and-activation quantization, written as W8A8 for 8-bit weights and 8-bit activations, shrinks both. Weight-only formats dominate local use because weights are the bulk of the memory, and dequantizing them on the fly is cheap enough. W8A8 matters more for servers that batch many requests, where activation memory and compute both grow.

The rounding itself is a design choice. An int4 value holds 16 levels, so each weight must land on one of 16 rungs. An int8 value holds 256 levels, so the same weight lands much closer to its original value. Methods differ in how they pick those rungs. GPTQ adjusts neighboring weights to compensate for each rounding error as it works through the model. AWQ protects a small set of salient weights, about 1 percent of the total, and the AWQ paper reports that protecting that small fraction cuts quantization error by a wide margin. The Hugging Face quantization concept guide explains these mechanics in detail, including the group sizes and the integer ranges behind each format.

Also Read:

AI Agents vs Workflows: When to Use Which

The Memory Math: What Fewer Bits Buy You

The arithmetic is simple, and it drives every decision in this guide. Hugging Face states the rule in its quantization overview: models load in bf16 at 2 bytes per parameter, so an 8B model needs about 16GB for its weights in bf16, about 8GB at 8 bits, and 4 to 5GB at 4 bits. Halve the bits, halve the memory. A 70B model follows the same rule: about 140GB in bf16, about 70GB at 8 bits, and about 35 to 40GB at 4 bits by the same arithmetic. That 4-bit figure is what moves a 70B model from a multi-GPU server onto a single high-end desktop card, and it moves a 7B or 8B model onto a laptop with 8GB of shared memory.

Weights are half the story. The KV cache, the memory a model uses to remember tokens it has already processed, grows with context length and batch size. A long conversation or a large batch can push the KV cache past the weights in size. Quantizing the cache helps, and Hugging Face documents a QuantizedCache option that stores KV values at lower precision. The same docs warn that quantizing the cache can harm latency if the context length is short and enough GPU memory is already free for generation without cache quantization. Short chats gain little from a quantized cache. Long documents and large batches gain a lot.

One more number belongs in the budget: the runtime itself. The model file is the headline figure, but the inference engine, the CUDA context, and the operating system take their own share. Leave 1 to 2GB of headroom above the model size on a consumer card. A 4-bit 8B model at about 5GB fits a 8GB card with room to spare. A 4-bit 70B model at about 40GB needs a 48GB card or a split across two cards. Our local LLM setup guide walks through the hardware side of that decision, and our Ollama vs llama.cpp comparison covers the two runtimes most readers will use to load these files.

Also Read:

Prompt Injection Explained: How It Works and How to Stop It

Quantization Methods Compared

Six methods cover almost all local and server use. They differ in bit width, in whether they need calibration data, and in whether you quantize ahead of time or at load time. The Hugging Face method picker lists the same tradeoffs, and the table below condenses it into one view.

GGUF deserves the top row for local users. It is the format llama.cpp reads, and model hubs host thousands of pre-quantized GGUF files, so you download and run with no quantization step of your own. AWQ and GPTQ target NVIDIA serving, where their calibration step buys back accuracy that plain rounding would lose. bitsandbytes trades a little accuracy for zero setup: point it at a standard checkpoint and it quantizes in memory as it loads. BitNet sits in a research lane. Its weights are ternary, minus 1, 0, or 1, and the BitNet paper reports that it matches a full-precision model of the same size on its benchmarks. That result is promising, and it is not yet a practical choice for most readers.

Calibration deserves a plain explanation because it sounds more complex than it is. A calibration step runs a small sample of text through the model and watches which weights matter most for the output. AWQ uses that information to protect salient weights. GPTQ uses it to adjust weights as it rounds them. The cost is time: calibration runs from minutes to hours depending on model size and hardware. Pre-quantized files skip that cost because someone else already paid it. If you quantize a model yourself, budget that time, and keep the calibration sample close to your target domain: chat text for a chat model, code for a code model.

Also Read:

How AI Assistants Remember You: Memory Explained

Picking a Bit Width: Where Quality Starts to Drop

Bit width is the single biggest lever on quality. Hugging Face publishes benchmark tiers that make the tradeoff concrete, measured on Llama 3.1 70B in bf16 running on two NVIDIA A100 80GB GPUs, at batch size 1 while generating 64 tokens. Those conditions matter. A different model, a different batch size, or a different task can shift the numbers, so treat the tiers as a guide, not a guarantee.

Three patterns sit behind those tiers. First, 8-bit quantization is close to free. The LLM.int8() paper reports that it cuts inference memory in half while retaining full precision performance, with more than 99.9 percent of values multiplied in 8-bit, and it reports inference for models up to 175B parameters without performance degradation. That paper dates from 2022, and the claim covers the models and tasks its authors tested, so treat it as a published result for their tests. Second, 4-bit quantization holds up well with a good method. AWQ and GPTQ lead the field here, and both need calibration to reach their best numbers. Third, quality falls off a cliff below 4 bits. A 3-bit model can still work for casual chat, and a 2-bit model can still produce coherent text, but both lose ground on reasoning, math, and long outputs. Our guide to why small AI models win in production covers when a smaller model at higher precision beats a larger model at lower precision.

The practical test is simple. Load the same model at 8-bit and 4-bit, run ten prompts from your real workload, and compare answers side by side. Include a math question, a long summary, and a coding task. Those three expose quality loss faster than small talk does. If the 4-bit answers match the 8-bit answers on your prompts, keep the 4-bit file and enjoy the memory savings. If they drift, step up to 8-bit or pick a smaller model at higher precision.

Speed, Decode, and the KV Cache

Quantization saves memory first and speed second, and the reason sits in how a model generates text. Decoding one token at a time is memory-bound: the GPU spends most of its time reading weights from memory. A 4-bit weight file moves a quarter of the data that a bf16 file moves, and that saving can outweigh the cost of dequantizing each weight before use. The result is faster token generation on memory-limited hardware, which covers most laptops and single-GPU desktops. Prefill, the step that processes your prompt in one pass, is compute-bound, so weight-only quantization helps it less. W8A8 helps here because it shrinks activations as well as weights, and it helps servers that batch many requests together.

Published speed claims need their conditions attached. The GPTQ paper reports that GPTQ can quantize a 175B model in about four GPU hours, reducing bit width to 3 or 4 bits per weight with negligible accuracy loss against the uncompressed baseline. That figure describes the quantization step, and it comes from a 2022 paper on hardware of that era. The AWQ paper reports that its TinyChat runtime offers more than 3x speedup over the Hugging Face FP16 implementation on desktop and mobile GPUs. That claim covers the TinyChat runtime on its test hardware, and your runtime and GPU will produce a different number. Treat both as evidence that the technique works, then measure your own setup.

The KV cache shapes speed in a second way. A quantized cache frees memory for longer contexts, which lets a server hold more concurrent sessions. The Hugging Face KV cache docs note that cache quantization can harm latency when the context is short and memory is already plentiful, so the win appears at long contexts and tight memory, not in every case. Short chats on a card with spare memory can skip cache quantization. Long documents and parallel users benefit from turning it on and measuring the change. Readers who pay per token on hosted APIs can also cut costs on the serving side: our guide to cutting LLM API bills with prompt caching covers the API equivalent of the same idea.

One last speed note concerns model loading. A smaller file loads faster, and a 4-bit GGUF file starts in seconds where a bf16 checkpoint takes much longer to read from disk. That difference matters for tools that restart often, such as coding assistants, and less for a server that loads once and runs for weeks.

Conclusion: Start at 4 Bits, Then Benchmark

Quantization turns a simple idea into practical freedom: fewer bits per weight means a bigger model fits on the hardware you already own. Three points carry the whole guide. First, 8-bit is close to free in quality, 4-bit is the sweet spot for local runs, and sub-4-bit formats trade clear quality for size. Second, the method matters as much as the bit width: GGUF for the widest hardware support, AWQ or GPTQ for the best 4-bit accuracy on NVIDIA, bitsandbytes for zero-setup loading. Third, published benchmarks describe specific rigs and tasks, so your own prompts are the final judge.

Put it to work in one sitting: pick a model you want to run, download its 4-bit GGUF file, and load it in your runtime of choice. Our guide to running private AI on your laptop covers the Ollama setup step by step. Run ten prompts from your real work, compare the answers against the full-precision version, and keep the quantized file if the answers hold up. Share the bit width you run in the comments, with the model and GPU it runs on.