From b4fa8b68101b9483fcedbd2bf78b3a70bf0f5389 Mon Sep 17 00:00:00 2001 From: TheTom Date: Wed, 25 Mar 2026 20:49:49 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20TurboQuant+=20experiments=20master=20do?= =?UTF-8?q?cument=20=E2=80=94=20status,=20findings,=20dead=20ends?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers all 9 experiments: 2 merged, 2 active, 1 blocked, 1 dead, 3 not started. Key learnings: layer sensitivity, dequant bottleneck, byte-level GPU optimization. Co-Authored-By: Claude Opus 4.6 (1M context) Co-Authored-By: tturney@psyguard.ai --- docs/turboquant-plus-experiments.md | 105 ++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 docs/turboquant-plus-experiments.md diff --git a/docs/turboquant-plus-experiments.md b/docs/turboquant-plus-experiments.md new file mode 100644 index 0000000..01ed250 --- /dev/null +++ b/docs/turboquant-plus-experiments.md @@ -0,0 +1,105 @@ +# TurboQuant+ Experiments — Status and Findings + +## Overview + +TurboQuant v1 is complete: 4.6x KV cache compression at 99% of q8_0 speed across all context depths, with 1.1% PPL loss. These experiments explore improvements beyond the base paper. + +## Experiment Status + +### MERGED (in production) + +| Experiment | Branch | Result | +|-----------|--------|--------| +| **Speed optimization** | `experiment/speed-optimization` | 739 → 2747 tok/s (3.72x speedup). fp16 WHT, half4 butterfly, graph-side rotation, block-32. | +| **Context scaling fix** | `experiment/context-scaling-fix` | Custom GGML_OP_TURBO_WHT + optimized dequant. Flat 99% of q8_0 through 32K context. | + +### ACTIVE (promising, not yet merged) + +#### Layer-Adaptive KV Cache +**Branch:** `experiment/layer-adaptive` +**Finding:** The last 8 of 40 layers account for essentially ALL of turbo3's quality loss. + +| Config | PPL | vs q8_0 (6.111) | +|--------|-----|-----------------| +| Uniform turbo3 (all 40 layers) | 6.211 | +1.6% | +| Mode 2: q8_0 last 8, turbo3 first 32 | 6.120 | +0.1% | + +**Implication:** By spending q8_0 precision on just 20% of layers, we get q8_0 quality at ~3.5x effective compression. This finding also informs temporal decay — old tokens in early layers can be compressed more aggressively. + +**Next:** Test at extended context depths (2K-32K) to verify mode 2 holds. Currently only tested at 512 context. + +#### Temporal Decay +**Branch:** `experiment/temporal-decay` +**Status:** Design complete, implementation pending. +**Idea:** Progressive requantization based on token age: recent tokens at 3.5-bit, old tokens at 2.5-bit or lower. + +**Key design decisions (from research):** +- Best trigger: `llama_kv_cache::update()` — runs between batches, exclusive cache access +- Token age tracking: already exists in `llama_kv_cells` (pos per cell) +- MVP: 2 tiers (turbo3 → turbo2), check every 256 tokens +- Blocker: turbo2 block type doesn't exist yet + +**Updated with layer-adaptive findings:** Temporal decay should be layer-aware. Early layers (insensitive) decay faster. Late layers (sensitive) decay slower or not at all. Combined with layer-adaptive mode 2, this means: +- Recent tokens: turbo3 everywhere (current behavior) +- Old tokens in layers 0-31: turbo2 (aggressive compression) +- Old tokens in layers 32-39: stay at turbo3 or q8_0 (preserve quality) + +**Estimated impact:** 30-50% memory reduction for long-context workloads beyond what turbo3 already provides. + +### BLOCKED (needs engineering work) + +#### Asymmetric K/V Compression +**Branch:** `experiment/asymmetric-kv` +**Idea:** K at lower precision, V at higher — V carries content, K carries direction. + +**Blocker:** Flash attention kernel template only has same-type instantiations. Mixed K/V types (e.g., turbo3 K + q8_0 V) need new template instantiations. Significant work for unclear payoff since uniform turbo3 already achieves 99% speed parity. + +**Shipped fix:** V un-rotation now correctly checks `v->type` instead of `k->type`, which is needed for any future asymmetric support. + +### DEAD (invalidated) + +#### MoE-Aware Expert Gating +**Branch:** `experiment/moe-aware-gating` +**Idea:** Track expert firing frequency, give frequently-activated experts' KV higher precision. +**Why it's dead:** Expert routing is an FFN signal. KV cache stores attention projections computed BEFORE expert routing. K and V values are identical regardless of which experts fire. The premise doesn't apply to KV cache compression. + +Codex independently reached the same conclusion: "Expert routing is an FFN signal; KV cache importance is an attention signal. The premise may not map cleanly." + +### NOT YET STARTED + +#### Rotation-Free via Outlier Channeling (Idea E from notes) +**Potential:** Highest single improvement if viable. Eliminates WHT rotation entirely by isolating outlier channels at fp16 and quantizing the rest without rotation. +**Test:** One-line kurtosis measurement on real KV tensors with top-k outlier channels removed. If kurtosis drops to ~3.0 without rotation, the approach works. +**Status:** Not started. The dequant is now fast enough that eliminating rotation may not be the highest priority. + +#### Fused Compressed Attention (Priority 6 from notes) +**Potential:** Compute Q·K dot products directly on quantized indices without full dequant. Precompute Q·centroid table (8 values), then each K element is a table lookup instead of centroid lookup + multiply. +**Status:** Deferred. The optimized dequant closed most of the gap. This would help decode speed at very long context but is complex (custom flash attention kernel). + +#### Speculative Cache Warming (Idea B from notes) +**Status:** Skip. Codex and the Obsidian notes analysis both recommend against — complexity doesn't justify marginal gains. + +#### Codebook Interpolation (Idea C from notes) +**Status:** Skip. Soft assignment (storing interpolation weights) costs more bits than the quality improvement justifies. Compression theory agrees. + +## Key Learnings + +1. **Layer sensitivity is extremely non-uniform.** The last 20% of layers account for ~100% of turbo3's quality loss. This is the single most important finding for future work. + +2. **Dequant compute, not rotation, is the real bottleneck.** The graph-side WHT rotation (whether dense matmul or custom O(d log d) op) adds <1% overhead. The per-position centroid lookup in flash attention is what scales with context. + +3. **Byte-level optimization matters on GPU.** Reading the same qs/signs bytes once per 4 elements instead of per-element eliminated the context scaling regression. GPU constant memory access patterns dominate. + +4. **WHT and RoPE don't commute — but it doesn't matter.** Graph-side WHT applied after RoPE (same pipeline point as KV quantize) works correctly. The earlier failure was a matrix orientation bug, not a commutativity issue. + +5. **MoE expert routing is irrelevant for KV cache.** KV vectors are computed in shared attention before expert routing. Don't waste time on MoE-aware KV compression. + +6. **Always run perplexity.** "Coherent text" evaluation caught nothing when PPL was 165. Speed numbers are meaningless without quality validation. + +## Recommended Next Steps (Priority Order) + +1. **Test layer-adaptive at extended contexts** — verify mode 2 holds 2K-32K +2. **Measure decode speed** — testers report 4 tok/s vs 11 tok/s at 42k. Optimized dequant should help but unmeasured. +3. **Implement turbo2 for temporal decay** — 2-bit block type, the blocker for temporal decay MVP +4. **Rotation-free kurtosis test** — one measurement to determine if Idea E is viable +5. **Upstream PR preparation** — llama.cpp CONTRIBUTING.md requires perplexity, KL divergence, and CPU perf baselines