From 80ab3ded47669da2c2134a2245b2b22db2f6e2cf Mon Sep 17 00:00:00 2001 From: TheTom Date: Fri, 27 Mar 2026 07:48:10 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20sparse=20V=20validated=20on=20q8=5F0=20?= =?UTF-8?q?=E2=80=94=20+5%=20decode,=20confirms=20generality?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit q8_0 with sparse V: 80.7 → 84.7 tok/s (+5.0%) on M5 Max. Not TurboQuant-specific — any quantized KV cache benefits. Added TURBO_SPARSE_V=0 disable override for A/B testing. Raw logs included. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/papers/sparse-v-dequant.md | 13 ++++++++++++ .../q8_0_sparse_v_ablation_m5.txt | 20 +++++++++++++++++++ 2 files changed, 33 insertions(+) create mode 100644 docs/threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt diff --git a/docs/papers/sparse-v-dequant.md b/docs/papers/sparse-v-dequant.md index ae78249..61896a0 100644 --- a/docs/papers/sparse-v-dequant.md +++ b/docs/papers/sparse-v-dequant.md @@ -221,6 +221,19 @@ Sparse V dequantization is not specific to TurboQuant. It applies to any quantiz This includes NVFP4, KIVI, CacheQuant, and other KV cache quantization methods. The 3-line implementation is kernel-level and requires no model changes, no retraining, and no calibration data. +### 7.1 Empirical Validation on q8\_0 + +To validate generality beyond TurboQuant, we tested sparse V on llama.cpp's standard q8\_0 KV cache (8-bit quantization, 2× compression) using the same model and hardware. A `TURBO_SPARSE_V=0` override was added to force-disable the optimization for A/B comparison: + +| Test | q8\_0 + sparse V | q8\_0 (no sparse V) | Improvement | +|------|-----------------|---------------------|-------------| +| Decode (short, tg128) | 84.7 tok/s | 80.7 tok/s | **+5.0%** | +| Blended (pp32768+tg128) | 1145.2 tok/s | 1096.7 tok/s | **+4.4%** | + +Sparse V provides a **5% decode speedup on q8\_0** — a cache format with far cheaper per-position dequantization than turbo3. The benefit is smaller than turbo3's +22.8% at 32K because q8\_0's dequant is lightweight (simple scale-and-add vs centroid LUT + WHT rotation), but the attention sparsity still allows meaningful work to be skipped. + +This confirms that sparse V is a general flash attention optimization, not a TurboQuant-specific trick. Any quantized cache format — including NVFP4, q4\_0, and future formats — should benefit. Raw benchmark logs: [`threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt`](../threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt). + --- ## 8. Limitations and Future Work diff --git a/docs/threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt b/docs/threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt new file mode 100644 index 0000000..ca958c0 --- /dev/null +++ b/docs/threshold-ablation-logs/q8_0_sparse_v_ablation_m5.txt @@ -0,0 +1,20 @@ +Sparse V impact on q8_0 (non-TurboQuant) KV cache +Hardware: Apple M5 Max 128GB +Model: Qwen3.5-35B-A3B Q8_0 +Date: 2026-03-27 + +=== Q8_0 WITH SPARSE V (auto-enabled on M5) === +ggml_metal_library_init: turbo3 sparse V dequant enabled +| qwen35moe 35B.A3B Q8_0 | 34.36 GiB | 34.66 B | MTL,BLAS | 6 | q8_0 | q8_0 | 1 | tg128 | 84.69 ± 0.38 | +| qwen35moe 35B.A3B Q8_0 | 34.36 GiB | 34.66 B | MTL,BLAS | 6 | q8_0 | q8_0 | 1 | pp32768+tg128 | 1145.19 ± 2.19 | + +=== Q8_0 WITHOUT SPARSE V (TURBO_SPARSE_V=0 force-disabled) === +| qwen35moe 35B.A3B Q8_0 | 34.36 GiB | 34.66 B | MTL,BLAS | 6 | q8_0 | q8_0 | 1 | tg128 | 80.69 ± 0.67 | +| qwen35moe 35B.A3B Q8_0 | 34.36 GiB | 34.66 B | MTL,BLAS | 6 | q8_0 | q8_0 | 1 | pp32768+tg128 | 1096.73 ± 3.39 | + +=== SUMMARY === +Short decode (tg128): 80.69 → 84.69 tok/s (+5.0%) +Blended (pp32768+tg128): 1096.73 → 1145.19 tok/s (+4.4%) + +Sparse V benefits q8_0 even though q8_0's per-position dequant is much cheaper than turbo3's. +The skip still saves work at positions with negligible attention weights.