Local LLM orchestration and benchmarking

What

A personal lab for running open-weight models locally — Qwen and Llama variants served through llama.cpp, fronted by Open WebUI. The point isn’t to use them; it’s to measure them under real constraints and build intuition for which size/quant combinations are actually usable on the hardware I have.

Why

Two reasons:

1. I want to stop paying per-token tax for work that a 4B–8B model can do fine, and I want to know empirically where that line is.
2. The more interesting question — if you’re doing RAG or agent work — is latency per token, not benchmark score on MMLU. Those numbers you only get by running things yourself.

How

• Runtime: llama.cpp with GGUF quantized weights.
• Front-end: Open WebUI for chat-style sanity checks; direct llama-cli for benchmarking runs.
• Quantization pipeline: convert Hugging Face safetensors to a f16 GGUF master with convert_hf_to_gguf.py, quantize with llama-quantize, then measure quality as the perplexity delta against the f16 baseline (llama-perplexity on WikiText-2).
• What I measure: tokens per second (prompt eval + generation), context window behavior as it fills up, memory footprint, perplexity deltas for quantization quality, and — separately — correctness on a small private instruction-following set.

Findings so far

Numbers are per-machine: the 4B/8B findings below came from a MacBook Pro (M2 Pro, 12-core, 16GB); the 72B work runs on a Mac Studio (M3 Ultra, 28-core, 256GB unified memory).

Qwen 4B: fast, but brittle

~50 TPS on my setup. Fine for summarization, classification, and short-form generation. Falls over on multi-step instructions and anything that requires holding structure across a longer response — it drifts or drops constraints. Useful as a component, not as a general model.

8B models: ran into a hardware wall

On 16GB RAM, the default quant for 8B-class models pushes the machine into heavy swap — I saw ~5GB of memory swapping during longer-context generation, and context exhaustion when I tried to stretch the window. The machine doesn’t crash; it just slows to a crawl and the generation quality gets non-deterministic as cache pressure rises.

Takeaway from the 16GB phase

On 16GB, 4B is the speed king for simple tasks. 8B is usable but only if you drop to a more aggressive quant and keep the context reasonable. The “just run an 8B locally” recommendation you see online assumes more headroom than 16GB provides once you account for the OS, the embedding model (if RAG), and the browser you forgot to close.

Qwen2.5-72B on the M3 Ultra: quantization was measurably free

Quantized Qwen2.5-72B-Instruct from bf16 (145GB) to Q6_K (60GB), then benchmarked both with llama-perplexity on WikiText-2: 4.4751 ±0.0275 for f16 vs 4.4808 ±0.0275 for Q6_K. The delta — +0.0057 — is five times smaller than the measurement uncertainty, so the two models are statistically indistinguishable on this test at 59% of the size saved. A side finding worth knowing: the quantized model was slower in prefill (dequantization is a compute tax in batch processing) while 2× faster in decode, where memory bandwidth dominates — 10.1 vs 5.1 t/s generation in a same-prompt chat spot check. Full methodology, numbers, and caveats: I quantized a 72B LLM and then proved it didn’t get dumber.

What’s next

• Q4_K_M for the 72B (~44GB): third row of the perplexity table. The 8B reference delta is +0.175; the pattern so far predicts the 72B lands much lower.
• Imatrix ablation: 2-bit quants (IQ2_M, ~22GB for a 72B) need an importance-matrix calibration pass to be usable — run IQ2 with imatrix vs Q2_K without and quantify how much the calibration rescues.
• KL-divergence (llama-perplexity --kl-divergence): measure how much the quant’s output distribution shifted from f16’s, a more direct probe than perplexity against ground-truth text.
• KV-cache quantization: compose weight + cache quantization (--cache-type-k q8_0 --cache-type-v q8_0) to fit big-model-long-context on one machine.
• Task-level evals: a benchmark harness against llama-server’s OpenAI-compatible API for MMLU/GSM8K-style downstream scores.
• Full TPS sweep: Qwen 4B and 8B, Llama 8B, at Q4_K_M / Q5_K_M / Q6_K, measuring TPS + a small correctness set — including the runs where the “better” quant wasn’t worth it.
• Wire the best-performing local model into aegis-rag as the default generation backend.

Limitations

• Numbers are per-machine and labeled as such — 4B/8B findings from a 16GB MacBook Pro (M2 Pro), 72B results from a 256GB Mac Studio (M3 Ultra). Neither generalizes to CUDA hardware.
• The 72B quality result is one corpus, one metric (perplexity on WikiText-2, 512-token windows) — the note spells out what that does and doesn’t show.
• Correctness set is small and hand-written — not a standard eval. Good for catching obvious regressions, not for publishing leaderboard claims.