REFRACT is meant to be a general KV-quant quality oracle, applicable
across model families (gemma, qwen, mistral, phi, llama, etc.) and not
TurboQuant-specific. Top-level placement reflects that scope and gives
a cleaner CLI: `python3 -m refract.cli` instead of nested under turboquant.
Updates:
- git mv turboquant/refract/ -> refract/
- All `from turboquant.refract` -> `from refract` imports rewritten
- pyproject.toml: include both turboquant* and refract* in setuptools
- All "turboquant/refract/" path references in README/tests updated
Verified: 14/14 unit tests pass, CLI module path works.
Two-axis composite (GTM + KLD@D corpus proxy) with bit-exact-zero floor
verification, replacing corpus PPL as the primary correctness oracle for
KV-cache quantization changes. Motivated by the §4.3/§4.5/§4.8 findings
in docs/papers/attn-rotation-and-ppl-artifact.md, where corpus PPL inverts
sign on gemma-4-it while KLD vs the fp16-KV reference ranks configs
correctly.
- Axis A (GTM): greedy trajectory match across 30 CC0 prompts, 128 tokens
each, via llama-cli --single-turn (avoids the interactive-mode hang).
- Axis B (KLD@D): mean KL divergence via llama-perplexity --kl-divergence
against an fp16-KV --kl-divergence-base reference (v0.1 corpus proxy;
v0.2 will compute true trajectory KLD).
- Composite: harmonic_mean(GTM, KLD), banded as
EXCELLENT/PASS/DEGRADED/FAIL.
- Floor: REFRACT(ref, ref) must be >= 99.5 or the run aborts.
- CLI: python3 -m turboquant.refract.cli score ...
- Reports: ANSI text card + JSON twin for ML pipelines.
- Tests: 14 fast unit tests + 1 integration test reproducing the
paper's gemma-4 26B-A4B q8/turbo4 OFF cell (must land DEGRADED/FAIL).
Strictly additive under turboquant/refract/; no upstream files touched.
No new pip dependencies (stdlib only).
Co-authored-by: tturney@psyguard.ai
Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
Native Swift/Metal vLLM backend for OpenAI-compatible serving
with TurboQuant+ support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- turbo4v2 is faster than no-turbo (not "can be"), explained by Apple
Silicon matmul favoring smaller types during dequant
- removed "Gap to Python" row — Swift is at parity or ahead now
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Active collaboration with Eric on the Swift MLX fork — turbo4v2 working,
Gemma4 fixes in progress. M-series users should use that for fastest decode.
llama.cpp repo remains the cross-platform (CUDA, ROCm, CPU, Metal) path.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated context scaling table with fresh Qwen2.5-7B benchmarks (GPU idle,
2 runs per context, best-of-2). Previous numbers (61-83%) were measured
before the delegated KVCache optimization (7ad7500).
Updated Quick Start to recommend TurboKVCache (delegated architecture)
over the older TurboKVCacheLite/make_turbo_cache pattern.
New numbers: 97-100% baseline decode at all context lengths 512-16K.
PPL: +0.51% asym, +1.70% sym.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Float32 scales fix resolved the quality issue. C++ kernel now works
at all contexts: 98-100% baseline at 128-4K, 87.5% at 32K.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compact_turbo_cache() uses custom sdpa_vector_qv C++ Metal kernel.
97-100% baseline speed with real V memory savings.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
87-98% baseline speed with ~60% KV memory savings via mx.quantized_matmul.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TurboKVCacheLite: 100% baseline decode, 74% V savings.
Both 7B dense and 35B MoE tested from 128 to 128K context.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
make_turbo_cache() moved into TheTom/mlx with auto monkey-patch.
No mlx-lm fork required.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
35B MoE: 94.2 tok/s (96% of 98.3 baseline). Only 6/10 KV layers
compressed after boundary 2+2.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 93% was measured on prompt that barely triggered compression.
Clean benchmark with full compression active: 81% baseline decode.
PPL improved to -0.05% (turbo slightly better than baseline).
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First/last 2 KV layers stay at FP16 to prevent NaN from extreme
V norms on boundary layers. Matches the production recommendation.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7B dense: 60.0 tok/s (93% of 64.5 baseline), +0.04% PPL, zero extra
memory. Previous best was 82% (unpatched asymmetric).
Root cause of prior NaN: extreme V norms (184x) on boundary layers.
Boundary protection resolves completely.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric turbo4 is catastrophic on dense models (KLD=6.86, NIAH 0/15).
Asymmetric K=FP16 fixes everything (KLD=0.003, NIAH 15/15, top-1 98.1%).
Tested on Qwen2.5-7B-Instruct-8bit (28 layers, all KV cache).
Quality test suite added: scripts/mlx_quality_suite.py
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Detect per-layer cache types so hybrid models (Qwen3.5 linear+attention)
get TurboKVCache only on KVCache layers, not ArraysCache layers
- Use min_compress_tokens=32 for KLD test so compression actually fires
- Extend KLD text to ~500 tokens for meaningful divergence measurement
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New standalone script that tests TurboKVCache quality using mlx-lm
without touching existing llama.cpp test scripts.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Blackwell decode penalty drops from 25-40% to 7-9% with signalnine's
fused shmem mmvq kernel. 3.7-5.4x improvement. Prefill flips from
penalty to bonus. RTX 5090 now matches RTX 4090 penalty levels.
signalnine authored the kernel; TheTom collaborated on integration.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primoco diagnosed AmesianX's 30.8% F16 baseline as RoPE frequency
scaling from ctx_size < native context. At native 32768: all configs
(f16, tbq4_1, q8_0/tbq4_1) hit 100% accuracy. Zero cost from TBQ.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-user confirmation on Blackwell consumer hardware. All 6 K/V
combinations tested with philosophical, mathematical, and 23K-token
inputs. Originally hit corrupt output on Madreag's fork.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TheTom