mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 02:53:13 -05:00
TheTomandClaude Opus 4.6
7fc9fafe94
A minimal hybrid memory policy for long-context KV cache eviction on top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921). V3 adds two structural constraints to the paper's scoring: - Hard prefix window: first N tokens never evicted - Per-segment quota: eviction spreads evenly across position buckets Validated envelope (full data in the paper): - Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS - Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS - TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction - Wall time overhead: within baseline noise after four optimization passes Where it does not work: - Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers cleanly but NIAH silently fails at middle and end positions. The failure mode suggests the scoring is evicting semantically important tokens in these architectures, not that the implementation is broken. Open question documented in Section 6. Paper includes: full implementation journey (V1 paper-faithful, V2 per-segment quota only, V3 prefix-protect + quota), how-to-run section with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix flags, a dedicated section asking the community for input on the missing pieces, and appendices with complete numeric data and the commit-level rollback ladder on the llama.cpp fork. This paper accompanies the draft pull request on the experiment/triattention-integration branch at TheTom/llama-cpp-turboquant (not for merge yet; community review and comparison notes welcome). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>