I quantized a 72B LLM and then proved it didn't get dumber: benchmarking perplexity with llama.cpp

I recently quantized Qwen2.5-72B-Instruct from its original 145GB down to 60GB on a Mac Studio — and instead of just trusting that “Q6_K is near-lossless” like every guide claims, I decided to measure it myself. This post covers what perplexity benchmarking is, why it matters, how I ran it, what the numbers said, and where I’m going next.

The Setup

All of this ran on a single machine:

• Mac Studio, M3 Ultra — 28-core CPU (20 performance + 8 efficiency), 256GB unified memory
• llama.cpp (build b10280) with Metal GPU acceleration
• Model: Qwen2.5-72B-Instruct, downloaded as the original bf16 safetensors from Hugging Face (~145GB across 37 shards)

The pipeline before benchmarking: convert the Hugging Face checkpoint to a f16 GGUF “master” file with convert_hf_to_gguf.py, then quantize with llama-quantize to Q6_K. Fun fact about this machine: the 145GB f16 conversion wrote to disk in about 4 minutes at ~600MB/s, and the quantization of all 963 tensors took just over 2 minutes with 20 threads. The Mac Studio is a genuinely absurd machine for this kind of work.

What Is Perplexity?

Quantization compresses a model’s weights from 16 bits down to ~6 bits per weight (in Q6_K’s case) by storing them as small blocks of compact codes plus scale factors. It’s lossy compression. The obvious question: how much did the model actually lose?

Perplexity (PPL) is the standard intrinsic metric for this. It measures how “surprised” a model is by real text: you feed it a corpus it hasn’t been trained to memorize, and at every position, ask how much probability the model assigned to the token that actually came next. Mathematically it’s the exponential of the cross-entropy loss.

Intuition: a perplexity of ~4.5 means that at each step, the model was effectively choosing between ~4.5 equally likely candidates. A perfect oracle would score 1. A model outputting random noise would score in the thousands. Lower is better.

For quantization work, the absolute number matters less than the delta: quantize the model, run the exact same test on the original and the quantized copy, and the difference tells you how much capability the compression destroyed.

Why Measure Instead of Trust?

Three reasons:

1. The published numbers are for a different model. The reference figures in llama-quantize --help (e.g. Q6_K = +0.0217 ppl) were measured on Llama-3-8B. My model is a 72B with a different architecture. Rules of thumb say bigger models tolerate quantization better — I wanted to see it.
2. Quantization involved my choices. My conversion, my quantize run, my hardware. An end-to-end sanity check catches mistakes anywhere in the pipeline (a corrupted conversion or a broken chat template would show up as a wildly wrong number).
3. This is what separates engineering from vibes. “The model feels fine” is not a measurement. Getting comfortable with evals — even a simple one like this — is a transferable skill for all LLM work: prompt changes, fine-tunes, RAG pipelines, everything.

How: Running the Benchmark

llama.cpp ships a perplexity tool. The standard corpus is WikiText-2 (raw Wikipedia test text):

# get the corpus
wget https://huggingface.co/datasets/ggml-org/ci/resolve/main/wikitext-2-raw-v1.zip
unzip wikitext-2-raw-v1.zip

# run against the f16 baseline
llama-perplexity -m ./qwen2.5-72b-f16.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99

# identical run against the quantized model
llama-perplexity -m ./qwen2.5-72b-Q6_K.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99

(-ngl 99 offloads all layers to the Metal GPU.)

What the tool actually does, decoded from its log line — 584 chunks, n_ctx=512, batch_size=2048, n_seq=4:

• Slices the corpus into 584 windows of 512 tokens (~299K tokens evaluated in total)
• Processes 4 windows in parallel, so each GPU pass crunches 2048 tokens
• Within every window, scores the model’s prediction for each token given its predecessors
• Prints a running estimate as it goes, converging to a final number with an error bar

Each full run took ~40–45 minutes on the M3 Ultra, GPU pinned at 100% the entire time.

A surprise worth understanding: the quantized model was slower to benchmark

The Q6_K run took slightly longer per pass than f16 (18.3s vs 16.5s). Counterintuitive — isn’t the quantized model supposed to be faster?

It is — but only in the right phase. LLM inference has two regimes:

• Decode (generating tokens one at a time, i.e. normal chat): bound by memory bandwidth, because every token requires re-reading all the weights. Smaller quantized weights = fewer bytes read = significantly faster. This is where the Q6_K wins big.
• Prefill (processing many tokens in parallel — long prompts, and 100% of what a perplexity run does): bound by compute, because one weight-read is reused across the whole 2048-token batch. Here quantization gives no bandwidth win, and adds a small tax: the GPU can’t multiply 6-bit block codes directly, so every weight is dequantized to floats on the fly before the math. f16 skips that step entirely — it’s the GPU’s native format.

So: quantized models are faster where it matters for daily use (generation), and marginally slower in batch-processing workloads. Knowing which bottleneck you’re in explains almost everything about LLM performance.

To check the decode side of that claim, I ran the head-to-head in chat — the same essay prompt through both models via llama-cli:

Model	Prompt	Generation
f16	65.7 t/s	5.1 t/s
Q6_K	67.0 t/s	10.1 t/s

Exactly the pattern the two regimes predict. Generation doubled on the quant — in line with the bandwidth math (2.4× fewer bytes read per token, minus overhead that doesn’t shrink with the weights: KV-cache reads and the on-the-fly dequant). The f16 number itself confirms the bottleneck: 819GB/s of memory bandwidth divided by 145GB of weights caps decode at ~5.6 t/s, and it measured 5.1 — the model was reading weights about as fast as the machine physically allows. Prompt processing came out identical at this prompt length; the dequant tax from the perplexity runs only becomes visible at large batches. (Single-run spot checks, not a sweep — and generation slows further as the context fills.)

Findings So Far

Model	Size	Bits/weight	PPL (WikiText-2, 584 chunks)	Δ vs f16
f16 (baseline)	145GB	16.00	4.4751 ± 0.0275	—
Q6_K	60GB	7.08	4.4808 ± 0.0275	+0.0057

The headline: the quality difference is +0.0057, roughly five times smaller than the ±0.0275 measurement uncertainty. The two models are statistically indistinguishable on this benchmark. I cannot demonstrate, with this test, that the 60GB file is any worse than the 145GB one.

A few observations:

• “Near-lossless” is not marketing. Q6_K on this model measurably cost nothing detectable while cutting size by 59%.
• Big models really do quantize more gracefully. The reference Q6_K delta on Llama-3-8B is +0.0217; my 72B came in at a quarter of that. The folk wisdom held up.
• The practical consequence: the f16 master file is now provably redundant for using the model — its only remaining job is as source material for producing other quant levels. (Golden rule of quantization: always quantize from the full-precision master, never re-quantize an already-quantized file — the rounding errors stack.)
• The efficiency math is wild when you sit with it: same knowledge, same measured capability, 2.4× smaller, and 2× faster token generation (10.1 vs 5.1 t/s measured in chat) because decode speed scales with bytes read per token.

What’s Next

• Q4_K_M (~44GB, ~4.9 bits/weight): the community’s default “sweet spot” quant. The 8B reference says +0.175 ppl; based on the pattern above I’m predicting the 72B lands much lower. Same benchmark, third row for the table.
• Extreme quantization with importance matrices: 2-bit quants (IQ2_M, ~22GB for a 72B!) are usable only with an imatrix — a calibration pass that identifies which weights matter most so the quantizer protects them. I want to run the ablation: IQ2 with imatrix vs Q2_K without, and quantify exactly how much the calibration rescues.
• KL-divergence measurement: perplexity measures against ground-truth text; KL-divergence (llama-perplexity --kl-divergence) measures how much the quant’s output distribution shifted from the f16’s — arguably a more direct probe of “how much did quantization change the model.”
• KV-cache quantization: weights aren’t the only memory consumer — the attention cache grows with context length (~10GB at 32K context for this model in f16). llama.cpp can quantize the cache too (--cache-type-k q8_0 --cache-type-v q8_0 with FlashAttention). Composing weight + cache quantization is how you fit big-model-long-context on one machine.
• Task-level evals: wiring up a benchmark harness against llama-server’s OpenAI-compatible API to score things like MMLU/GSM8K, which measure downstream ability rather than intrinsic prediction quality.

Limitations

Honest caveats about what this experiment does and doesn’t show:

• One corpus, one metric. WikiText-2 is generic English Wikipedia prose. Perplexity on it can understate quantization damage in specialized domains — math, code, multilingual text — where precision matters more. A complete evaluation would add task benchmarks.
• Short context. The standard test uses 512-token windows. Quantization effects can be larger at long contexts; this benchmark says nothing about 32K-token behavior.
• Intrinsic ≠ downstream. Predicting Wikipedia well correlates with, but doesn’t guarantee, quality on instruction-following, reasoning chains, or tool use. The cheap complement: run your own real prompts through both models side by side.
• One model. These numbers are for Qwen2.5-72B-Instruct specifically. The pattern (big models quantize well) is consistent with community experience, but every architecture differs — measure your own.
• Absolute PPL values aren’t portable. They depend on the tokenizer, corpus slicing, and context settings. Only deltas measured under identical conditions are comparable — don’t line my 4.4751 up against numbers from other posts.

Closing Thought

The whole arc — download, convert, quantize, benchmark — took one afternoon on a single desk-side machine, most of it waiting on progress bars. The result is a 72-billion-parameter model, running locally at interactive speeds, with receipts proving the compression cost nothing measurable. If you have the hardware to run a big quant, the tooling has gotten remarkably approachable — and if you’re going to quantize, spend the extra 90 minutes to measure it. The confidence is worth more than the disk space.

This experiment is part of the ongoing local LLM benchmarking project, which tracks this work as it develops.

---

Toolchain: llama.cpp (b10280) · Model: Qwen/Qwen2.5-72B-Instruct · Hardware: Mac Studio M3 Ultra, 256GB unified memory · Corpus: WikiText-2 raw test set