LLM quantization explained: 70B model from 141 GB to 43 GB

Key takeaways

• LLM quantization stores a model's weights at lower numeric precision, such as 8 or 4 bits instead of 16, so the same model needs far less memory and disk space.
• GGUF is the single-file model format used by llama.cpp and Ollama. Names like Q4_K_M, Q5_K_M and Q8_0 describe how many bits each weight gets.
• Llama 3.1 70B shrinks from 141 GB at fp16 to 43 GB at q4_K_M on Ollama's tag page. That is the difference between a server rack and one high-end workstation.
• Measured loss is small down to about 4 bits: on LLaMA 7B, Q4_K_M raised perplexity by under 1% versus fp16, while 2-bit Q2_K raised it by almost 15%.
• A sensible default for most people is Q4_K_M. Move to Q5_K_M, Q6_K or Q8_0 if you have spare memory, and drop below 4 bits only when nothing else fits.

What is LLM quantization?

LLM quantization stores a model's weights at lower precision, such as 8 or 4 bits instead of 16, so it needs far less memory. On Ollama, Llama 3.1 70B shrinks from 141 GB at fp16 to 43 GB at q4_K_M, with little measured loss.

Most open models ship in 16-bit formats (fp16 or bf16), which take 2 bytes per weight. A 7-billion-parameter model therefore needs about 14 GB just to hold its weights. Store each weight in 8 bits and the file roughly halves; store it in 4 bits and it falls to about a quarter. Because local inference is usually limited by how much video memory (VRAM) or RAM you have, quantization is what makes running a model on a laptop or a single gaming GPU possible.

If you followed our guide on how to run a local LLM with Ollama, you have already used a quantized model: Ollama's default tags point to 4-bit builds.

How quantization works, in plain terms

A weight in a 16-bit model can take any of about 65,000 values. A 4-bit weight can take only 16. Quantization maps the original values onto that small grid while keeping as much of the original pattern as possible.

The trick that makes this work is block-wise scaling. Instead of using one scale for a whole layer, modern methods split weights into small blocks, for example 32 or 256 weights, and store a scale factor (and sometimes a minimum) for each block. When the model runs, each weight is rebuilt as roughly w = q × block_scale + block_min, where q is the small integer that was stored. Those extra scales are why a "4-bit" file really costs about 4.5 to 4.9 bits per weight.

The main families differ in how carefully they choose the rounded values:

• Round-to-nearest simply rounds each weight. It is fast and needs no data, and it works well at 8 bits.
• Calibration-based methods such as GPTQ and AWQ run a small sample of text through the model and use it to decide which weights matter most. That keeps quality higher at 4 bits and below.
• Importance-matrix (imatrix) quants in llama.cpp apply the same idea to GGUF files: a calibration pass records which weights are most sensitive, and the quantizer protects them.

What is GGUF?

GGUF is a binary file format for storing models for inference with GGML, the tensor library behind llama.cpp. It was created by Georgi Gerganov, the developer of llama.cpp, and it succeeds the older GGML, GGMF and GGJT formats. According to the GGUF specification, its design goals are single-file deployment, memory-mapping for fast loading, and extensible metadata stored as typed key-value pairs. In practice, one .gguf file holds the weights, the tokenizer and the settings the model needs to run.

Hugging Face's Hub filters GGUF models under the GGUF library tag, and the format is read by llama.cpp, Ollama, LM Studio and GPT4All. GGUF is a container format, not a single quantization method: the same file type can hold fp16 weights or 2-bit weights.

How to read GGUF quant names

The suffix in a file name such as Llama-3.1-8B-Instruct-Q4_K_M.gguf tells you the quantization type:

• The number (Q2 to Q8) is the base bit width of most weights.
• K marks the "k-quants" introduced to llama.cpp by Iwan Kawrakow in June 2023. They use super-blocks of 256 weights with their own quantized scales.
• S, M, L (small, medium, large) mean more or fewer of the sensitive tensors are kept at higher precision. Q4_K_M, for example, keeps some layers at higher precision than Q4_K_S.
• _0 and _1 (Q4_0, Q8_0) are the older round-to-nearest types. Hugging Face's documentation lists all of them, Q8_0 included, as legacy, although Q8_0 is still widely offered as a near-lossless option, including on Ollama's tag pages.
• IQ types (IQ4_XS, IQ3_XXS, IQ2_XS) are importance-matrix quants aimed at very low bit widths.

How much memory do 16-bit, 8-bit and 4-bit models need?

The weight file size is close to parameters × bits per weight ÷ 8. The table below uses the effective bits per weight that llama.cpp reports for each GGUF type on Llama 3.1 8B, applied to common model sizes. Figures are decimal gigabytes for the weights alone.

Calculation: Bullquill, using bits-per-weight values from the llama.cpp quantize README (Q8_0 8.50, Q5_K_M 5.70, Q4_K_M 4.89, F16 16.00). The estimates match Ollama's published download sizes closely: Llama 3.1 70B is listed at 75 GB (q8_0), 50 GB (q5_K_M) and 43 GB (q4_K_M).

Real memory use is higher than the file size. The model also needs a KV cache, which grows with the length of the conversation, plus some runtime overhead. A long context can add several gigabytes; our explainer on what a context window is covers why. As a rule, leave 1 to 2 GB of headroom for short chats and more for long documents.

Does quantization hurt quality? What the measurements show

The standard test is perplexity, a measure of how well a model predicts held-out text; lower is better. When Kawrakow introduced k-quants, the pull request published perplexity for each type on Meta's original LLaMA 7B and 13B models:

Source: llama.cpp pull request #1684 (k-quants), merged June 5, 2023; percentage change calculated by Bullquill.

The pattern is consistent: quality is almost unchanged at 5 and 6 bits, the loss is still small at 4 bits, and it climbs steeply below that. The 13B model lost proportionally less at 2 bits (about 11%) than the 7B model, which matches the common observation that larger models tolerate compression better.

The research papers behind the other formats point the same way:

• LLM.int8() (Dettmers and colleagues, August 2022) showed a 175B-parameter model could run in 8-bit "without performance degradation", halving inference memory by keeping rare outlier values in 16-bit.
• GPTQ (Frantar and colleagues, October 2022) quantized 175B-parameter GPT models to 3 or 4 bits in about four GPU hours with what the authors call negligible accuracy loss, and reported speedups of about 3.25x on Nvidia A100 GPUs over fp16.
• AWQ (Lin and colleagues, MIT, June 2023) found that protecting only 1% of the most important weights greatly reduces quantization error.
• QLoRA (May 2023) introduced the 4-bit NormalFloat (NF4) type and fine-tuned a 65B model on a single 48 GB GPU while matching full 16-bit fine-tuning performance.

Perplexity is only one measure. Tasks that need precise recall, maths or long reasoning chains can suffer more than the averages suggest. If a quantized model makes errors the full model does not, try one step higher before blaming the model.

GGUF vs GPTQ vs AWQ vs bitsandbytes

Source: Hugging Face Transformers quantization overview, compatibility table.

The key practical difference: GGUF, GPTQ and AWQ are usually downloaded already quantized, while bitsandbytes quantizes a normal 16-bit checkpoint as it loads. In Transformers that is one line of configuration:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=nf4_config)

That is the same setup our LoRA and QLoRA fine-tuning tutorial uses to train on a single consumer GPU.

How to pick a quant in Ollama and llama.cpp

In Ollama, quantization is part of the tag. ollama pull llama3.1 fetches the 8B q4_K_M build (4.9 GB). To choose another level, name it explicitly:

ollama pull llama3.1:8b-instruct-q8_0     # 8.5 GB, near-lossless
ollama pull llama3.1:8b-instruct-q5_K_M   # 5.7 GB
ollama pull llama3.1:8b-instruct-q4_K_M   # 4.9 GB, the default
ollama ps                                  # check the model fits on the GPU

To run a GGUF file you downloaded from Hugging Face, write a Modelfile containing FROM /path/to/file.gguf and run ollama create my-model. Ollama's import documentation notes that it does not quantize GGUF models during import, so pick the quant level before you download.

In llama.cpp, you can make your own quant from a Hugging Face checkpoint in two steps:

python convert_hf_to_gguf.py ./my-model --outfile my-model-bf16.gguf --outtype bf16
./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M

A simple decision path:

1. Find your usable memory: GPU VRAM, or unified memory on Apple Silicon, minus 1 to 2 GB.
2. Choose the largest model whose Q4_K_M file fits in that space. A bigger model at 4 bits usually beats a smaller one at 8 bits for the same memory.
3. If there is room left, step up to Q5_K_M or Q6_K. Use Q8_0 when you want results almost identical to the original.
4. Use Q3 or IQ2 types only to squeeze a model that otherwise would not fit, and test it on your own tasks first.

What it means

Quantization is the reason open-weight models moved from data centres to desktops. A 4-bit GGUF file puts an 8B model in under 5 GB and a 70B model within reach of a single workstation with 48 GB or more of memory, with a quality loss most users will struggle to notice in everyday chat. For builders, it lowers the cost of private setups such as a local RAG app in Python, where documents never leave the machine. The trade-off is real but measurable. Start at Q4_K_M, check your own outputs, and spend any spare memory on more bits or a bigger model, not on 16-bit weights.

Sources

• Hugging Face Hub documentation: GGUF and quantization types
• GGML project: GGUF file format specification
• llama.cpp: quantize tool README (bits per weight, sizes, commands)
• llama.cpp pull request #1684: k-quants (perplexity results)
• Hugging Face Transformers: quantization overview
• Hugging Face Transformers: bitsandbytes
• Ollama library: llama3.1 tags and sizes
• Ollama documentation: importing a model
• Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339)
• Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323)
• Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978)
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)