Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
- turbo4v2 is faster than no-turbo (not "can be"), explained by Apple
Silicon matmul favoring smaller types during dequant
- removed "Gap to Python" row — Swift is at parity or ahead now
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Blackwell decode penalty drops from 25-40% to 7-9% with signalnine's
fused shmem mmvq kernel. 3.7-5.4x improvement. Prefill flips from
penalty to bonus. RTX 5090 now matches RTX 4090 penalty levels.
signalnine authored the kernel; TheTom collaborated on integration.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primoco diagnosed AmesianX's 30.8% F16 baseline as RoPE frequency
scaling from ctx_size < native context. At native 32768: all configs
(f16, tbq4_1, q8_0/tbq4_1) hit 100% accuracy. Zero cost from TBQ.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-user confirmation on Blackwell consumer hardware. All 6 K/V
combinations tested with philosophical, mathematical, and 23K-token
inputs. Originally hit corrupt output on Madreag's fork.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric tbqp3/tbq3 matches f16/f16 exactly (20/65, 30.8%) at temp=0.
Confirms no TBQ degradation on head_dim=128 models. Notable that
symmetric turbo3 works on Qwen3-14B Q4_K_M unlike Qwen2.5-7B Q4_K_M.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hannu Varjoranta's PR #1 on vibhavagarwal5/vllm changed default
value_quant_bits from 4 to 8 (FP8), fixing 0% gsm8k. Cites @TheTom
as source for "value precision is the quality bottleneck." Independent
vLLM validation of asymmetric thesis and turbo4-resurrection findings.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Independent validation of asymmetric KV compression on Ada (SM 89)
and Blackwell (SM 120). Asymmetric q8_0-K/turbo4-V confirmed best
config on Ada (+8.4% prefill, -6.2% decode). Blackwell shows larger
decode regression (-25.2%), likely dp4a vs fp8/fp4 tensor core mismatch.
LA=1 boundary layers improve decode speed in addition to quality.
Source: https://github.com/ggml-org/llama.cpp/discussions/20969
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I on Gemma 4 26B MoE: -2.3% size, neutral/better PPL, 100% speed.
Minimal weight compression benefit on MoE (few dense layers). KV compression
is the real win for this architecture.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new community reports:
- theaaronhughes: RTX 5090 Gemma 4 26B KV at 64k/128k
- tryingET: RTX PRO 6000 Config I+turbo3 beats Q8_0+turbo3 by 7% decode
- pjsgsy: RTX 3050 APEX-I-Quality pre-quantized models with CPU offload
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First Volta (V100, SM 7.0) validation: Config I +8.7% faster decode.
First Pascal (1080 Ti, SM 6.1) validation: Config I +6-11% faster.
Config I is now faster than Q8_0 baseline on every CUDA architecture
tested (Pascal, Volta, Ada, Blackwell).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vilenkin basis decomposition on Phi-2 suggests K and V occupy disjoint
spectral bands, providing potential mathematical basis for asymmetric
compression. Caveat: single model, needs validation across architectures.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I 30% faster than Q8_0 decode on RDNA 4. First successful AMD
HIP weight compression result. PPL +1.8%.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
14-config Metal benchmark independently confirms asymmetric K/V as
default recommendation. BitNet compound case is new finding. NIAH
survives catastrophic PPL (retrieval degrades independently).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@tryingET: Qwen3.5-27B, 96GB VRAM. Config I is faster than Q8_0 on
Blackwell Workstation (+11.9% prefill, +8.5% decode). PPL +0.048%.
Config I + turbo3 beats Q8_0 + turbo3/turbo4 at 32K.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vLLM TurboQuant integration explicitly cites turboquant_plus for
boundary layers and asymmetric KV. Community ablations on A100/H100
confirm our findings.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Weight compression is now on feature/turboquant-kv-cache (main).
No separate branch needed. PR #45 remains open for community logs.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 4.4: TurboQuantDC (662 tests, MIT) independently confirmed
stacking, boundary layer protection, and QJL removal on RTX 4090.
Links to community results page for full hardware coverage.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12+ GPUs, 11+ models, 6 families, 10+ independent testers. All data
from PR #45 community testing. Includes GPU compatibility matrix,
model results by family, stacking validation, decode speed comparison,
known issues, confidence ratings, and contribution template.
Linked from getting-started.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
19-version kernel optimization log proves CUDA TQ4_1S decode gap is
fundamental (float32 bandwidth + FMA density), not an optimization gap.
Q4_K_M is the practical CUDA runtime answer. TQ4_1S optimal for Metal,
storage/distribution format on CUDA.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
clang on Windows does not inherit CMake Release flags. Without -O3 -DNDEBUG
the build runs ~50x slower. Tested on RX 9070 XT (gfx1201, RDNA 4).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add TQ3_4S mention and reframe contribution to include follow-on work.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- asymmetric paper: correct Sjoerd's q8_0/turbo3 results (corrupt output
on head_dim=256, speed data accurate but content garbage)
- recommendations: add MTMD_BACKEND_DEVICE=CUDA1 for multi-GPU vision
- community count: 30+ to 50+ testers
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CUDA port now available via signalnine (PR #45)
- Add CUDA GPUs to tested hardware (1080 Ti, RTX 5090)
- Replace "Metal only" limitation with CUDA decode gap analysis
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update backend support: Metal + CUDA (was Metal only)
- Add copy-paste Quick Test section: clone, build, compress, bench
- Recommend Qwen3.5-27B as test model (known good, fits 32GB)
- Explicit instructions to paste results back in PR #45
- Note Windows MSVC support
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Asymmetric paper: Sjoerd 122B on 2x L40S at 82K. q8_0/turbo3 100%
decode recovery. q8_0/turbo2 also 100% recovery (V free to 2-bit).
- MoE paper: turbo2 no-crossover at 82K on hybrid MoE (12 attn layers),
architecture-dependent finding.
- KLD results: new context-length scaling data (2K/4K/8K on M2 Pro).
Mean KLD stable/improving with context, does not drift.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Models with head_dim=64 (e.g., GPT-OSS-120B) may crash or produce
degraded output with turbo compression. WHT requires sufficient
dimensionality for CLT convergence. Reported by NigelTufnel12345.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
The turbo4-K PPL explosions were caused by kernel bugs (our fork) and
head_dim misdetection (AmesianX fork), not an IQ4_XS-specific issue.
AmesianX independently confirmed turbo4-K works correctly after fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Add compression reference table and config warning to getting-started.md
- Add independent validation from sztlink, Madreag, and mudler (APEX/LocalAI) across asymmetric, layer-aware, MoE, and sparse V papers
- Update sparse-v-dequant.md with Madreag 4-GPU cross-validation
- Minor README wording fix (branch -> repository)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
New paper: q8_0/turbo2 with Boundary V on Qwen3.5-35B MoE achieves
7.53x V compression with PPL within 1% of q8_0 at 512c/8K/32K.
Quality matches or beats q8_0/turbo3 at every tested length. 32K
decode ~2-3% faster than turbo3-V (n=2, M5 Max). Scoped to one
tested MoE model. Includes decode investigation negative results
(20+ approaches exhausted) and dense-vs-MoE bottleneck analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Replace parquet link with direct raw text download. Add HF CLI as
alternative. Credit: @henkiespenkie44
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TheTom