turboquant_plus/scripts/README.md

TurboQuant Diagnostic Scripts

Hardware profiling, benchmarking, and quality validation for TurboQuant cache types.

---

Quality & Speed At A Glance

Quality (Qwen3.5-35B-A3B MoE Q8_0, M5 Max 128GB):

Config	PPL vs q8_0	NIAH Single (4K-32K)	NIAH Multi-Key (4K-32K)
Uniform turbo3	+1.1%	18/20 (q8_0: 17/20)	4/4 perfect
Mode 2 (layer-adaptive)	+0.14%	—	—

turbo3 matches q8_0 on needle retrieval. The few single-needle misses at 32K overlap with positions where q8_0 also fails.

Speed (turbo3/q8_0 ratio):

Hardware	Prefill	Decode (short)	Decode (32K)	Notes
M5 Max 128GB	0.95-1.00x	0.92x	0.75x	Healthy gradient
M2 Pro 32GB	1.32-1.73x	~0.73x	pending	Prefill wins big on lower bandwidth
M1 Max 64GB	0.93-0.98x	0.83x	0.39x	Constant memory LUT regression
Dual 4090 (CUDA)	0.98x	0.37x (MoE, 42K)	—	buun investigating, fix in progress

Pre-M5 decode regression is a known hardware limitation, not a quality issue. Fully documented in decode-speed-hardware-analysis.md. 12 kernel approaches tested, root cause is constant memory LUT divergence on Apple7/Apple8 GPU families.

Compression:

Config	Effective ratio	Bits per value
Uniform turbo3	4.6x	3.5
Mode 2 (q8_0 last 8 layers)	3.5x	~4.6
+ Temporal decay	+30-34% savings at 32K+	—

---

Quick Start

# 1. Clone the repo
git clone https://github.com/TheTom/turboquant_plus.git
cd turboquant_plus

# 2. Run the diagnostic
bash scripts/turbo-diag /path/to/llama.cpp /path/to/model.gguf

That's it. The launcher automatically:

1. Finds or creates an isolated Python venv (.turbo-diag-venv/)
2. Installs rich for a color-coded terminal UI (falls back to ASCII if it fails)
3. Runs the full diagnostic suite

Estimated runtime: 20–40 minutes depending on hardware. Use --skip-ppl and --skip-stress to cut that roughly in half.

Requirements: Python 3.10+, a built llama.cpp with turbo3 support, and a .gguf model file.

Which fork do I build?

Hardware	Fork	Branch
Apple Silicon (M1-M5)	TheTom/llama-cpp-turboquant	feature/turboquant-kv-cache
NVIDIA (CUDA)	spiritbuun/llama-cpp-turboquant-cuda	feature/turboquant-kv-cache
AMD (ROCm)	spiritbuun's fork (has RDNA v_dot2 path)	feature/turboquant-kv-cache

AMD users: ROCm support is untested and experimental. Flash attention is reportedly broken on RDNA 2 (6800 XT) with ROCm > 7.0.1, and turbo3 requires flash attention. RDNA 3/4 may work but nobody has confirmed yet. Expect rough edges and please report what you find.

# Apple Silicon
git clone https://github.com/TheTom/llama-cpp-turboquant.git
cd llama-cpp-turboquant && git checkout feature/turboquant-kv-cache
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# NVIDIA CUDA
git clone https://github.com/spiritbuun/llama-cpp-turboquant-cuda.git
cd llama-cpp-turboquant-cuda && git checkout feature/turboquant-kv-cache
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# AMD ROCm
git clone https://github.com/spiritbuun/llama-cpp-turboquant-cuda.git
cd llama-cpp-turboquant-cuda && git checkout feature/turboquant-kv-cache
cmake -B build -DGGML_HIP=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Important: Flash attention must be enabled. Add -fa on (server) or -fa 1 (bench/CLI) to all commands. Without it, turbo3 will silently fail on pre-M5 hardware.

How It Runs

                        turbo-diag
                            |
              +-------------+-------------+
              |                           |
         Creates venv              Installs rich
      (.turbo-diag-venv/)          (optional UI)
              |                           |
              +-------------+-------------+
                            |
                 turbo_hardware_diag.py
                            |
         +------------------+------------------+
         |                  |                  |
   Detect Hardware    Start Monitor      Start Display
   (CPU/GPU/RAM)     (10s CSV poll)    (rich or ASCII)
         |                  |                  |
         +------------------+------------------+
                            |
              Run 13 Diagnostic Sections
              +---------------------------------+
              |  1. Hardware Inventory           |
              |  2. System Load (pre)            |
              |  3. Model Info                   |
              |  4. GPU Capabilities             |
              |  5. Build Validation             |
              |  6. Prefill Speed (5 depths)     |  <-- llama-bench
              |  7. Decode Speed (5 depths)      |  <-- llama-bench
              |  8. Stress Test (11 depths)      |  <-- llama-bench
              |  9. Combined Workload            |  <-- llama-bench
              | 10. Perplexity (quality gate)    |  <-- llama-perplexity
              | 11. Memory Breakdown             |  <-- llama-cli
              | 12. System Load (post)           |
              | 13. Summary + Anomaly Report     |
              +---------------------------------+
                            |
              +-------------+-------------+
              |             |             |
         .txt log     .json profile  .csv monitor
              |             |             |
              +-------------+-------------+
                            |
                     turbo-diag-*.zip
                    (send this to us!)

---

What It Generates

The diagnostic produces a single zip file you send back to the team:

turbo-diag-20260326-143022.zip
├── turbo-diag-20260326-143022.txt       # Human-readable log (all sections)
├── turbo-hwprofile-20260326-143022.json  # Machine-parseable hardware profile
└── turbo-monitor-20260326-143022.csv     # Background system metrics (polled every 10s)

Output Files

File	Format	Purpose
turbo-diag-*.txt	Plain text	Full diagnostic log — hardware inventory, benchmark tables, anomaly flags, and how-to-read guide. Compatible with hw_replay.py for programmatic parsing.
turbo-hwprofile-*.json	JSON	Structured hardware profile: CPU, RAM, GPU family, model metadata, and build info. Used for cross-machine comparison.
turbo-monitor-*.csv	CSV	Background metrics captured every 10 seconds during the entire run.

Monitor CSV Columns

Column	Description
timestamp	UTC ISO 8601 timestamp
load_1m	1-minute load average
mem_pressure_pct	Memory pressure percentage (active+wired pages)
swap_used_mb	Swap usage in MB
gpu_temp_c	GPU temperature in Celsius (NVIDIA only; N/A on macOS)
cpu_speed_limit	macOS CPU speed limit percentage (100 = no throttling)
gpu_mem_used_mb	GPU memory used in MB (unified on Apple Silicon)
gpu_util_pct	GPU utilization percentage (NVIDIA only)

---

How to Read Results

The .txt log is organized into 13 numbered sections. Here's what to look at, in order of importance:

Check quality first (Section 10)

Perplexity (PPL) must be within 2% of q8_0. If PPL is broken, all speed numbers are meaningless.

Decode speed — the critical metric (Section 7)

This is the test that matters most. Look at the turbo3/q8_0 ratio at each context depth:

• Healthy (M5 Max): 0.92x (short) → 0.72x (48K) — gradual, predictable degradation
• Problem (M1): 0.90x (short) → 0.09x (42K) — constant cache thrashing, falls off a cliff

Prefill speed (Section 6)

turbo3/q8_0 ratio should be flat at 0.95–1.00x across all depths. If it degrades with context length, that's a context scaling regression.

Stress test inflection point (Section 8)

Fine-grained decode scaling gradient. Shows the exact context depth where turbo3 decode starts falling apart. This is the most useful data for diagnosing constant cache issues on specific hardware.

Thermal comparison (Sections 2 vs 12)

Compare pre-benchmark and post-benchmark system load. If CPU_Speed_Limit dropped below 100 during the test, results may be artificially low due to thermal throttling.

Memory breakdown (Section 11)

KV cache sizing at multiple context lengths. If the system is near its recommendedMaxWorkingSetSize, swap pressure will kill decode performance.

Section-by-Section Reference

Section	Title	What It Tells You
1	Hardware Inventory	CPU, RAM, GPU, cache hierarchy, and power state (no PII)
2	System Load (pre-benchmark)	Baseline load, memory pressure, top CPU consumers, and disk I/O
3	Model Info	Model architecture, params, layer count, expert count, and file type
4	GPU Device Capabilities	GPU family, Metal version, Tensor API support, and unified memory
5	Build Validation	Confirms turbo3 type is available, Metal library loads, and build commit
6	Prefill Speed	Prefill tok/s at 2K/4K/8K/16K/32K for q8_0, turbo3, and mode 2
7	Decode Speed	Decode tok/s at short/4K/8K/16K/32K — the critical test
8	Constant Cache Stress Test	Fine-grained decode gradient at 2K–32K in ~2K increments
9	Combined Prefill+Decode	Realistic workload simulation (prefill then decode)
10	Perplexity	Quality validation — PPL for q8_0, turbo3, and turbo3 mode 2
11	Memory Breakdown	KV cache sizing at multiple context lengths
12	System Load (post-benchmark)	Post-test load, thermal throttling check
13	Diagnostic Summary	All anomalies, how-to-read guide, and next steps

---

Known Baselines

M5 Max Reference Numbers (Qwen 35B MoE Q8_0)

Metric	Expected	Notes
Prefill ratio (turbo3/q8_0)	0.95–1.00x	Flat across all depths
Decode ratio (short context)	~0.92x	Small overhead from WHT transform
Decode ratio (32K context)	~0.75x	Gradual degradation, still usable
PPL delta vs q8_0	< 2%	Typically < 1%

M1 Max 64GB Community Results (Qwen 35B MoE Q8_0)

Tester 1: @Ambisphaeric

2026-03-26. GPU: Apple M1 Max, 32 cores, 64GB unified.

Prefill (tok/s):

Context	q8_0	turbo3	Ratio
2K	855.7	836.1	0.977x
4K	805.2	784.0	0.974x
8K	724.6	696.3	0.961x
16K	600.5	569.6	0.949x
32K	444.9	414.0	0.931x

Decode (tok/s, tg128):

Context	q8_0	turbo3	Ratio	Flag
short	41.2	34.5	0.837x
4K	31.9	20.4	0.640x
8K	27.5	14.8	0.538x
16K	20.7	9.4	0.454x	ANOMALY
32K	13.7	5.3	0.387x	ANOMALY

Tester 2: @mariotomich

2026-03-26. GPU: Apple M1 Max, 32 cores, 64GB unified. Clean system (94% free memory, low load). Full diagnostic run with mode 2.

Prefill (tok/s):

Context	q8_0	turbo3	mode2	turbo3 ratio	mode2 ratio
2K	859.2	844.5	848.2	0.983x	0.987x
4K	810.5	789.6	795.6	0.974x	0.982x
8K	731.2	698.9	707.1	0.956x	0.967x
16K	599.9	570.9	575.9	0.952x	0.960x
32K	450.0	416.4	423.5	0.925x	0.941x

Decode (tok/s, tg128):

Context	q8_0	turbo3	mode2	turbo3 ratio	mode2 ratio	Flag
short	42.9	35.6	36.9	0.830x	0.860x
4K	34.3	20.8	22.1	0.606x	0.645x
8K	28.1	14.8	16.3	0.526x	0.580x	ANOMALY
16K	21.0	8.7	9.7	0.414x	0.462x	ANOMALY
32K	13.8	5.4	6.1	0.391x	0.442x	ANOMALY

Cross-validation: Both M1 Max testers show nearly identical results, confirming the M1 decode regression is hardware-dependent, not user-specific. Mode 2 (layer-adaptive) consistently adds ~5% decode improvement over uniform turbo3.

Decode degrades significantly at long context. Known M1 issue: constant memory LUT contention in the flash attention dequant path. M5 Max (with tensor API) does not have this problem.

M2 Pro 32GB (Mac Mini) Results (Qwen2.5-7B Q4_K_M) — Partial

Internal test (2026-03-26). GPU: Apple M2 Pro, 19-core GPU, Apple8 family (1008), has_tensor=false, 32GB unified, Metal 4. Build: 02268fc (norm correction). Clean run after killing competing workloads.

Prefill (tok/s):

Context	q8_0	turbo3	Ratio
2K	155.7	263.6	1.69x
4K	143.2	245.3	1.71x
8K	119.5	206.3	1.73x
16K	93.5	144.3	1.54x
32K	63.3	83.7	1.32x

turbo3 prefill is 32-73% faster than q8_0 on M2 Pro. Much bigger gap than M5 Max (~parity). Hypothesis: M2 Pro's lower memory bandwidth (200 GB/s vs M5 Max 546 GB/s) makes the compressed cache win harder since turbo3 moves less data per token.

Decode: Test hung at 32K context, no decode data collected. Rerun pending with timeout fix. Expect M1-like decode regression (has_tensor=false, same constant memory LUT issue).

Mode 2 prefill: Hung at pp16384. High variance at 8K+ suggests memory pressure beginning (32GB with 4GB model + KV cache + system overhead).

---

What "Good" vs "Bad" Looks Like

Indicator	Good	Bad
Decode ratio curve	Gradual slope, stays above 0.70x at 32K	Cliff drop below 0.50x at any depth
Prefill ratio	Flat 0.95–1.00x	Drops with context depth
PPL delta	< 2%	> 5% (quality regression)
CPU Speed Limit	100 before and after	Drops below 90 during test
Swap growth	< 50 MB	> 100 MB (memory pressure)

Anomaly Flags

The diagnostic auto-detects these issues and tags them [ANOMALY] in the log:

• Steep decode degradation — ratio drops >15% between consecutive depths
• Thermal throttling — CPU_Speed_Limit dropped below 100%
• Memory pressure/swapping — swap grew >100 MB during benchmarks
• Low q8_0 baseline — system was under load (q8_0 decode < 5 tok/s)
• Quality regression — turbo3 PPL >10% worse than q8_0
• Unreliable 1K measurement — Metal async dispatch artifact (>10,000 tok/s at 1K context)

---

Hardware Replay System

The hw_replay.py module (turboquant/hw_replay.py) parses diagnostic output into structured Python objects for programmatic analysis.

Load a diagnostic profile

from turboquant.hw_replay import HardwareProfile

profile = HardwareProfile.from_diag_file("turbo-diag-20260326.txt")

# Or from the JSON profile
profile = HardwareProfile.from_json("turbo-hwprofile-20260326.json")

Extract performance curves

# Decode speed vs context depth
decode_curve = profile.get_decode_curve("turbo3")
# {0: 42.1, 4096: 38.7, 8192: 35.2, ...}

# turbo3/q8_0 ratio at each depth
ratio_curve = profile.get_ratio_curve("turbo3", "q8_0", "decode")
# {0: 0.92, 4096: 0.89, 8192: 0.85, ...}

# Find the inflection point (where decode falls off a cliff)
inflection = profile.find_decode_inflection("turbo3")
# 16384  (or None if the curve is smooth)

Compare two profiles

from turboquant.hw_replay import compare_profiles

m5 = HardwareProfile.from_diag_file("diag-m5-max.txt")
m1 = HardwareProfile.from_diag_file("diag-m1-ultra.txt")

report = compare_profiles(m5, m1)
print(report.to_markdown())

Predict performance on untested hardware

from turboquant.hw_replay import predict_decode_from_baseline

# Predict M1 (family 1007, no tensor API) from M5 baseline
predicted = predict_decode_from_baseline(
    baseline=m5,
    target_gpu_family_id=1007,
    target_has_tensor=False,
)
# {0: 0.92, 4096: 0.85, 8192: 0.71, 16384: 0.34, ...}

---

Command Reference

usage: turbo_hardware_diag.py [-h] [--model MODEL] [--llama-dir LLAMA_DIR]
                              [--skip-ppl] [--skip-stress] [--verbose]
                              [--output-dir OUTPUT_DIR]
                              [llama_dir] [model_path]

Positional Arguments

Argument	Description
llama_dir	Path to llama.cpp directory (default: current directory)
model_path	Path to .gguf model file (auto-detected if not specified)

Optional Flags

Flag	Description
--model MODEL	Path to .gguf model file (alternative to positional arg)
--llama-dir DIR	Path to llama.cpp directory (alternative to positional arg)
--skip-ppl	Skip perplexity tests — saves ~10 minutes
--skip-stress	Skip the fine-grained stress test (Section 8)
--verbose, -v	Enable verbose/debug-level logging
--output-dir DIR, -o DIR	Output directory for diagnostic files (default: current directory)

Examples

# Positional args (most common)
bash scripts/turbo-diag ~/llama.cpp ~/models/Qwen3.5-35B-A3B-Q8_0.gguf

# Named flags
bash scripts/turbo-diag --llama-dir ~/llama.cpp --model ~/models/model.gguf

# Quick run — skip slow tests
bash scripts/turbo-diag ~/llama.cpp ~/models/model.gguf --skip-ppl --skip-stress

# Verbose output to a specific directory
bash scripts/turbo-diag ~/llama.cpp ~/models/model.gguf -v -o ~/diag-results/

---

Before You Run

Plug in your charger and close heavy apps. On Apple Silicon Macs, battery power triggers CPU/GPU throttling that produces artificially low speed numbers — we observed 2-3x slower benchmarks on an M5 Max running on battery vs plugged in. The diagnostic will detect throttling (CPU_Speed_Limit in Section 12), but the damage is already done to earlier results. NVIDIA laptops have similar power management behavior.

Close Chrome, video calls, and anything GPU-heavy before running. The diagnostic captures top CPU/GPU consumers (Section 2), so interference will be visible in the log, but clean results are always better.

---

Troubleshooting

"turbo3 not recognized" or cache type errors

Your llama.cpp build doesn't have turbo types compiled in. Rebuild with the TurboQuant patches applied:

cd /path/to/llama.cpp
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

"No .gguf model found"

The auto-detection searches llama.cpp/models/, ../models/, and ~/local_llms/models/. If your model lives somewhere else, pass the path explicitly:

bash scripts/turbo-diag ~/llama.cpp /path/to/your/model.gguf

Benchmarks are painfully slow

Use the skip flags to cut runtime roughly in half:

bash scripts/turbo-diag ~/llama.cpp model.gguf --skip-ppl --skip-stress

PPL tests take ~10 minutes, and the stress test adds another ~10 minutes of fine-grained decode sweeps.

Script crashes on startup

• Python version: Requires 3.10+. The launcher checks this automatically, but if you're running turbo_hardware_diag.py directly, verify with python3 --version.
• Missing binaries: The script needs llama-bench, llama-perplexity, and llama-cli in <llama-dir>/build/bin/. If any are missing, rebuild llama.cpp.
• Permissions: On macOS, system_profiler and pmset may require user consent the first time. If hardware detection fails, the diagnostic still continues — it just logs warnings.

Rich terminal UI not showing

If you see plain ASCII instead of color tables, the rich package didn't install. This is cosmetic — all results are identical. To fix:

pip install rich
# Or delete the venv and let turbo-diag recreate it:
rm -rf scripts/.turbo-diag-venv

---

Other Scripts

turbo-quality-gate.sh

Pre-push quality and speed gate. Run before pushing any TurboQuant changes.

What it checks:

1. Perplexity — turbo3 PPL must be within 5% of q8_0 baseline (8 chunks of wikitext-2)
2. Context scaling — turbo3/q8_0 speed ratio at 4K context must be > 0.95x

Usage:

bash scripts/turbo-quality-gate.sh
# Exit 0 = PASS, Exit 1 = FAIL

Configure paths via environment variables: LLAMA, MODEL, WIKI.

turbo-realworld-bench.sh

Real-world decode benchmark using a long PDF document (70+ pages). Compares turbo3 vs q8_0 at realistic context depths by spinning up llama-server instances, sending the full PDF as a prompt, and measuring prefill + decode speed.

Usage:

bash scripts/turbo-realworld-bench.sh [path-to-pdf]

Requirements: A long PDF file (not included in repo), pdftotext (from poppler), and a built llama.cpp with turbo3 support. Configure via LLAMA, MODEL, PORT_BASE, THREADS, and MAX_TOKENS env vars.

---

Sharing Results

After the diagnostic completes, you'll see:

============================================
  DIAGNOSTIC PACKAGE READY
============================================

  Zip file: /path/to/turbo-diag-20260326-143022.zip
  Contents:
    turbo-diag-20260326-143022.txt
    turbo-hwprofile-20260326-143022.json
    turbo-monitor-20260326-143022.csv

  Send this zip file to the TurboQuant team.

How to share:

1. GitHub issue (preferred): Open an issue at turboquant_plus with title "Diagnostic: [your hardware]" and attach the zip. GitHub supports attachments up to 25MB — the zip is typically under 100KB.
2. DM on X/Twitter: Send the zip file directly to @no_stp_on_snek

The zip is self-contained — everything we need is inside. No need to copy-paste logs or screenshot terminal output. The .txt log is human-readable if you want to look at it yourself first.

---

Privacy

No PII is collected. The diagnostic is designed to be safe to share publicly.

What IS collected

• Hardware specs: CPU model, core count, RAM size, GPU family, cache hierarchy
• GPU capabilities: Metal/CUDA version, Tensor API support, VRAM/unified memory size
• Benchmark numbers: tok/s for prefill, decode, combined, and perplexity
• System load: load average, memory pressure, swap usage, CPU speed limit
• Build info: llama.cpp commit hash
• Model metadata: filename, file size, architecture, layer/head/expert counts

What is NOT collected

• Usernames, home directory paths, or hostnames
• Network configuration, IP addresses, or MAC addresses
• File system contents or directory listings
• Environment variables (except TURBO_LAYER_ADAPTIVE when explicitly set)
• Model weights or prompt content
• Any data that could identify you or your machine beyond hardware specs