Atlas now ships the full TurboQuant+ KV cache upstream on both
backends: CUDA/GB10 (PR #133 — canonical Hadamard, 9 asymmetric
variants, Turbo2) and Apple Metal (PR #147 — turbo8/4/3/2 +
safer-asym + sparse-V dequant).
Expand the §QJL note to state explicitly that production drops the QJL
stage on both K and V. Name TheTom/llama-cpp-turboquant as the production
path and document the recommended config
(--cache-type-k q8_0 --cache-type-v turbo3). Keep the 5-group consensus
citation and add guidance for downstream users: use TurboQuantMSE for V
or straight PolarQuant for K. Only enable QJL classes for paper
reproducibility or K-side research below 8-bit, and validate at target
context length since QJL noise historically accumulates past ~16K.
Follow-up to #93 (QJL math fix). The reference impl is now correct, but
the production guidance hadn't been said this plainly anywhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sprint journal documenting the in-progress block-selector sparse
attention work on Apple Silicon: companion to sparse-v-dequant.md
(post-softmax) covering the pre-attention K-skip lever.
Includes the preliminary local M5 Max bench grid (1.16-1.73× decode
across B=4-8 × ctx=16K-48K on Qwen2.5-14B-1M-4bit), the Apple-Silicon-
specific findings (MLX_SDPA_BLOCKS heuristic mis-tune for the float-mask
kernel variant, pass-2 reducer modulo bug, MetalAllocator non-determinism
cliff), the failed-paths catalogue (custom Metal kernel 14× slower at B=8,
compose-gather 13.7× slower, bool mask 25% regression), and the
per-family port pattern across 10 model families.
Framed as a sprint journal — the work is not yet merged to upstream
alpha (PRs ekryski/mlx#34, ekryski/mlx-swift#36, ekryski/mlx-swift-lm#224,
ekryski/mlx-swift-lm#225 all open). Numbers are local M5 Max only;
recommendations are current best guess rather than settled defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Two new pieces of paper-form documentation covering longctx's two
operating modes and the V3 rescue fix.
NEW: docs/papers/longctx-1m-and-triattention.md
- Long-form paper on longctx — open long-context retrieval companion
for inference engines. Two modes: (1) standalone retrieval scored
on MRCR v2 8-needle on AMD MI300X with Qwen2.5-32B-Instruct, (2)
rescue layer for our TriAttention V3 eviction policy on Apple
Silicon, validated end-to-end via 256K planted-fact NIAH on
Qwen3.5-2B-4bit.
- Headline numbers: MRCR v2 1M MultiQ Selector + bge-rerank + det
copy n=30 directional 0.688 (above SubQ 0.659; mass-val pending).
V3+longctx ✓HIT every rung 32K → 256K via ChatSession;
V3-only ✗miss every rung.
- Architecture diagrams: full-stack (clients → vllm-swift →
ChatSession → mlx-swift-lm → V3 engine + rescue → longctx-svc),
V3 turn-by-turn flow (eviction-write on turn N, rehydrate-retrieve
before turn N+1).
- Appendices: module map (mlx-swift-lm side), longctx-svc endpoint
surface, V3 environment variable reference table.
- Cites Mao et al., "TriAttention: Efficient Long Reasoning with
Trigonometric KV Compression" (arXiv:2604.04921, 2026) as the
original paper. V3 in this stack is our independent Swift port +
hybrid extension; cross-references docs/papers/triattention-v3.md.
UPDATED: docs/papers/triattention-v3.md
- Adds §10 "Addendum: Swift Port + longctx Rescue — Fixing the
Hybrid NIAH Failure" after the §9 vLLM addendum.
- Reports the Apple Silicon Swift port test results that resolve the
Section 7.1 question "why does V3 break on hybrid Mamba+Attention
models?". Same failure mode reproduces on Swift port; longctx
rescue layer flips it.
- Includes the 12-cell receipt table, the V3 turn-by-turn diagram,
the driver-gotcha section explaining why bare container.generate()
doesn't rescue (Tier-3 rehydrate is ChatSession-only per commit
fe1a3b0 in mlx-swift-lm), and what's still not measured (>256K,
larger hybrids, V3+TQ+ stack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed scripts/eden-investigation/ as a public artifact
pointer, but the directory contents are scratch and don't belong in the public
repo. Methodology in §2 is sufficient for reproduction; the file pointer was
a leftover from cleaning up the earlier /tmp/ paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed personal scratch paths under /tmp/eden-investigation/
that aren't useful to public readers and look confusing. Replaces those plus the
two private Obsidian-note pointers with a single line pointing to the
scripts/eden-investigation/ directory in this repo, where the actual experiment
scripts live (h*.py, F1–F3 KL test, universal-K candidate centroids).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two independent third-party confirmations to the appendices:
asymmetric-kv-compression.md: @jagmarques tested asymmetric K/V on
Mistral-7B with E8 lattice VQ (a different quantization family from
TurboQuant) on Cerebrium A10/A100. Confirms asymmetric advantage scales
with context length: at 1742-tok prefix K3V2 +0.82% vs K2V2 +0.91%; at
3544-tok prefix K3V2 +0.35% vs K2V2 +2.26% (6.4× improvement). Clean
GQA-amplification explanation: 8 query heads per K head means K
quantization error is effectively 8× worse than V error in output
impact. Cross-method validation that the asymmetric conclusion is not
TurboQuant-specific.
turbo4-resurrection.md: @unamedkr / quantumaikr (quant.cpp) ported the
TurboQuant paper end-to-end in single-header C and found the QJL stage
contributed byte-identical zero to attention scores regardless of
input. Disabling QJL produced bit-for-bit identical perplexity.
Theoretical explanation: the √(π/2)/m constant in the unbiased
estimator is derived for Gaussian projection rows; on Rademacher (±1)
rows the per-key contribution gets dwarfed by residual-norm scaling.
Independently took the freed bits and reinvested in 4-bit codebook
(8 → 16 centroids), same path as turbo4 resurrection. Llama 3.2 3B:
turbo_kv_4b PPL 14.28 (+5.3% vs FP32), beating llama.cpp q4_0 KV
(~14.99, +10.6%). Targets embedded/edge (iOS, Android, WASM,
microcontrollers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the empirical study covering the broader 35-hypothesis investigation
spawned by the EDEN/Mitzenmacher/Portnoy critique (#87, #89). Companion
to eden-optimal-s-revisit.md.
Headline result: a K-cache centroid table that improves per-vector
reconstruction MSE by 1–13% across five model families (Qwen3-0.6B,
Llama-3.2-1B, Mistral-7B, Phi-3-mini, Gemma-2-2b) causes 70–90% mean
KL@D regressions and 50–60% catastrophic-rate failures at the model
output. This is a sign inversion, not a small mismatch: methods that
are strictly better under MSE are systematically worse under KL@D in
deployment.
The paper is structured in three acts:
- Act 1: K cache is sub-Gaussian post-WHT (H8); V is Gaussian
- Act 2: per-prompt fitting fails (PDS), ensemble (ens_4way) reaches an
irreducible ~8% catastrophic floor at scale (N=100)
- Act 3: the universal-K cross-model counterexample (F3) demonstrates
the MSE→KL inversion at the cleanest possible level
Includes a 2-D toy attention illustration of the softmax bucket-flip
mechanism, an explicit decision boundary for when MSE proxies quality
(linear consumers) vs when it doesn't (attention softmax), an
ablation summary distinguishing first-order from second-order factors,
operational guidance for practitioners, and a "why this wasn't caught
earlier" diagnosis of the methodology blind spots.
Companion paper handles the algorithmic claim head-on. This paper is
the broader evaluation-methodology critique.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a short paragraph in Appendix A documenting the algebraic identity
sqrt(S_unb * S_bias) = ||x||/||c||, which coincides with the matched-norm
scale. Frames it as an algebraic interpolation between the two
EDEN-derived forms, notes it is not derived in DRIVE, EDEN, or the
TurboQuant/EDEN note paper (arXiv:2604.18555), and does not correspond
to an optimization objective in those works.
The empirical head-to-head numbers in the table directly above the new
paragraph already show that each form attains the optimum of its own
objective and only its own; matched-norm behaves as a distinct operating
point, not as a tuning of either EDEN scale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three findings from the Mac mini PyPI install walkthrough (2026-05-03):
1. `--prompts` was required even though wheel ships them — defaulted to
bundled refract/prompts/v0.1.jsonl via importlib.resources (v0.3.2.2)
2. macOS system python3 is 3.9, mlx-lm needs >=3.10 — documented in
QUICKSTART Step 0 with brew install hint
3. MLX run_kld returned mean_kld=NaN on Qwen2.5-1.5B-Instruct-8bit +
q8_0/q8_0 KV. Root cause: log(softmax + 1e-12) underflows on
sharply-peaked logits from already-8bit weight models. Rewritten
in terms of log-softmax (logits - logsumexp) which is stable.
Non-finite per-position KL values now filtered before aggregation;
mean computed across valid positions only. If every position is
non-finite the backend raises BackendCapabilityError with an
actionable hint instead of silently returning NaN.
Test mock updated for the new mx symbols (logsumexp, exp, isfinite,
where, zeros_like). 416 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Mac mini install walkthrough (2026-05-03) found that `refract score`
required explicit --prompts even though the wheel ships the v0.1
prompts file. After `pip install refract-llm` a user shouldn't need
to know where the bundled file lives.
Now: when --prompts is omitted, _run_score resolves the bundled
refract/prompts/v0.1.jsonl via importlib.resources. Minimum viable
score invocation drops to:
refract score --backend mlx --model /path/to/model --candidate ctk=q8_0,ctv=q8_0
README install section flipped to lead with `pip install refract-llm`
(PyPI path) and demote the source-clone path to "for hacking".
Walkthrough also surfaced two non-blocking findings (kept as separate
follow-ups):
- macOS system python3 is 3.9, mlx-lm needs 3.10+; users hit this
- KLD axis returned NaN on Qwen2.5-1.5B-Instruct-8bit MLX — separate
numerical-stability bug, not a packaging issue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
The eden.py default formula S = ||x||²/⟨c,x⟩ was loosely labeled as
"biased optimal-S" throughout. On closer inspection that's imprecise:
the strict per-vector MSE-minimum is S* = ⟨c,x⟩/||c||² (textbook
least-squares; asymptotically → 1 for Lloyd-Max optimal centroids).
The eden.py default is a different formula, satisfying the inner-
product-preserving identity ⟨Sc, x⟩ = ||x||² per vector.
Empirical comparisons unaffected (we tested what eden.py ships).
Just the labeling. Added Terminology note in §4 with worked example
and credit to @amitport for surfacing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Light-touch addition to the existing 'Additional experiments and
writeups' aggregator line. Discoverable for anyone following the
academic discussion without dominating the deployment-focused README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical response to issues #87 (Mitzenmacher) and #89 (Portnoy) on
turboquant_plus. 8 hypothesis tests across synthetic Gaussian and
Qwen3-0.6B + Llama-3.2-1B real KV cache.
Headline: EDEN's optimal scale is real but second-order. Rotation choice
(WHT vs dense Haar) is first-order and accounts for ~all of the published
gap. Production already uses the first-order fix. On real KV with WHT,
matched-norm beats EDEN-S by 0.5-9% across Qwen and Llama; tightens to
0.67-0.72% on Llama. EDEN-S genuinely wins at b=8 on synthetic Gaussian
(70%+ MSE reduction) — credibility moat showing the lever exists in its
own regime.
Concedes attribution argument (#89): DRIVE/EDEN line is real prior art.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Distribution name: turboquant -> refract-llm.
Top-level package shipped: refract* only (turboquant/ stays in source
tree for dev work but is no longer in the wheel).
Bundled via package-data:
- refract/prompts/v0.1.jsonl (the 30-prompt CC0 set)
- refract/examples/*.json + *.html (4 reference reports)
- refract/llama-cpp-patches/refract-trajectory.patch (for users with
their own llama.cpp build to apply themselves)
readme = refract/README.md (PyPI long_description focused on REFRACT).
Adds a [full] extra that pulls all backend deps in one shot. Adds dev
extras for build + twine.
Refract __version__ bumped 0.3.2 -> 0.3.2.1 to match the wheel.
Local smoke: pip install dist/refract_llm-0.3.2.1-py3-none-any.whl in
a fresh venv, `refract --help` and `refract selftest` work, bundled
data accessible via importlib.resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Audit follow-up to the earlier skip-axis fix (728cd84). The stub axis
result dataclasses still carry score=100, so without these changes:
- json_report still wrote 'axes.gtm.score: 100.0' for a skipped axis,
visible to anyone parsing the JSON or running 'refract compare'
- 'refract compare' showed 100.0 in the Traj/KLD column for skipped
reports, which made cross-report comparison falsely flatter the
skipped run
Now: when CompositeScore.gtm_score is None we write axes.gtm.score = null
in the JSON. 'refract compare' reads 'skipped' or score=None and prints
'skip' in that column instead of the stub 100.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Two reader-facing fixes from AJ's screenshot (2026-05-02 X thread):
1. --skip-gtm / --skip-kld previously fed a stub score=100 to
composite_score, so a single real KLD axis of 99.47 produced a
misleading composite of 99.74 EXCELLENT. The skipped axis now
contributes None to composite_score() and is excluded from the
harmonic mean. CompositeScore.gtm_score / .kld_score are now
Optional[float]. Text + HTML reports render "n/a" with a
"skipped" badge for skipped axes instead of fake 100/EXCELLENT.
The "harmonic mean of N axes" gloss reflects the real count.
Diagnostics blocks (GTM diagnostics, KLD diagnostics) are
replaced with a single "axis was skipped" line when the axis
wasn't measured.
2. The HTML "Run details → Environment" card now surfaces the
backend name (llama.cpp / MLX / vLLM / SGLang) plus engine
version fields (llama_cpp_commit, mlx_lm_version, vllm_version,
sglang_url, served_model_id) when the active backend populates
them. Previously a report didn't say which engine ran it, which
made cross-engine comparison reports ambiguous in the inbox.
Tests added (test_score.py):
- composite_score(gtm=None, kld=99.47) returns ~99.47 (not 99.74)
- composite_score(gtm=62.40, kld=None) returns ~62.40
- composite_score(None, None) returns 0.0 FAIL (no crash)
- 3-axis skip case computes correctly
- interpret_pattern handles None
- text_report renders "n/a" + "harmonic mean of 1 axes" on skip-gtm
Refract test count: 410 -> 416.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
6 new tests in test_command_construction.py:
- _llama_extra_flags() returns [] when env unset/empty/whitespace
- _llama_extra_flags() shlex-splits multi-word values + handles quoted args
- run_completion appends user flags AFTER REFRACT defaults so last-wins
on -ngl works (regression bound: tests both -ngl indices)
- run_perplexity_kld_base appends extra flags
- run_perplexity_kld appends extra flags
- run_completion_trajectory (the trajectory axis hot path AJ would hit)
appends extra flags
Refract test count: 404 -> 410.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
User report: AJ hit CUDA OOM running Qwen3.6-35B-A3B on a 12 GB 3060
because REFRACT hardcodes -ngl 99 with no escape hatch for the MoE
offload flag (-ncmoe 32) the user needs to fit the model.
Add REFRACT_LLAMA_EXTRA_FLAGS env var. Appended (shlex-split) to every
llama-cli, llama-completion, and llama-perplexity subprocess invocation
in refract.runner. llama.cpp uses last-wins on repeated flags, so:
REFRACT_LLAMA_EXTRA_FLAGS="-ngl 28 -ncmoe 32"
overrides REFRACT's default -ngl 99 and adds the MoE expert offload
needed for the 12 GB GPU + 35B-A3B configuration.
Docs: QUICKSTART has a "Constrained VRAM?" section before Step 2;
PITFALLS has a new entry crediting AJ's report; CHANGELOG v0.3.2.1
mentions the env var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
The 4 vLLM "skeleton" tests in test_backends.py asserted that
VLLMBackend methods raise NotImplementedError; they break in v0.3.2.1
where the backend is a working implementation. Replace with:
vLLM (5 tests):
- _kv_str_to_vllm_dtype known mappings (f16 -> auto, q8_0 -> fp8_e4m3)
- TurboQuant preset round-trips (turbo3, turbo4, q8_0/turbo4)
- BackendCapabilityError on unmappable ctk/ctv
- run_completion signature has the kw-only contract args
- all 5 backend methods are present and callable
SGLang (5 tests, new):
- get_backend("sglang") returns SGLangBackend
- _validate_kv_str accepts f16/bf16/q8_0
- TurboQuant rejected loudly (no SGLang TurboQuant path)
- run_kld without REFRACT_SGLANG_REF/CAND_URL raises with actionable msg
- all 5 backend methods are present and callable
Also: MLX backend uses mlx_lm.generate.maybe_quantize_kv_cache when
mlx_lm.models.cache no longer exposes it (0.31+); fall back via
getattr(cache_mod, ...) so tests that mock the old surface still work.
Refract test count: 399 -> 404 (+10 new, -4 stale, 1 mlx fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Add a Step 0 that walks through:
- git clone TheTom/turboquant_plus + cd + pip install -e .[backend-extras]
- optional sibling clones for the patched llama.cpp fork (with cmake
build steps and LLAMA_CPP_BIN_DIR export) and the vLLM fork
- pointer to the SGLang Docker image used in the cross-engine bench
Renumber the existing Steps 0..4 -> 2..6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Documents the 3-engine apples-to-apples cross-engine benchmark:
- BF16 baseline (load, PPL, prefill, decode, KV size) across vLLM, SGLang, llama.cpp
- REFRACT --full 4-axis fidelity scoring (gtm, kld, rniah, plad) on each engine's native 8-bit KV
- Headline finding: same fp8_e4m3 dtype string produces 1.8x different KL drift between vLLM and SGLang on the same model
- 8 engine-side bugs encountered and worked around (vLLM max_num_seqs, prompt_logprobs cap, GPU mem carryover; SGLang aiter.dtypes, AITER fp8 prefill broken, KV dtype fixed at server launch)
- Full reproducibility scripts and REFRACT framework changes documented
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
PLAD axis previously interleaved ref/cand per (prompt, perturbation) cell.
On vLLM with hybrid Qwen3.6 (~120GB per LLM, two won't fit 192GB MI300X),
each ref/cand swap triggers eviction-and-reload of ~30-90s. With 30 prompts
× 4 cells (anchor + 3 perturbations) = 240 swaps, the axis runs ~3 hours
just for eviction overhead.
Now plans all cells upfront, runs all ref-side completions, then all
cand-side completions, then computes drift. Two LLM loads total.
No effect on backends that can hold both configs in cache cheaply
(llama.cpp, MLX).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
R-NIAH axis calls runner.tokenize_to_ids to estimate chars-per-token. The
runner unconditionally shelled out to llama-tokenize binary, which fails
when the host's llama.cpp checkout has drifted (Symbol not found:
_llama_memory_breakdown_print) or when running on droplets without llama.cpp
on PATH at all.
Now dispatches to _ACTIVE_BACKEND.tokenize_to_ids when the backend is not
llamacpp. MLX/vLLM/SGLang backends already implement this method.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Memory-pressured backends (vLLM on hybrid Qwen3.6-35B-A3B can't fit two
simultaneous LLM instances on 192GB) need each KV config loaded exactly
once per axis. The trajectory axis previously interleaved ref/cand calls
per prompt, which forces 2N model loads (catastrophic when each load
takes 40 min on hybrid models).
Now collect all ref trajectories, then all cand trajectories, then
diff. Two model loads total instead of 2N. No effect on llama.cpp/MLX
which keep configs cached cheaply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
vLLM backend: hybrid Qwen3.6-35B-A3B doesn't fit two simultaneous LLM
instances (~120GB each on 192GB MI300X). Cache evicts prior LLM
before loading new one. Adds max_num_seqs env knob (default 32) so
Mamba cache block count doesn't underflow vs default max_num_seqs=1024.
Default gpu_memory_utilization bumped to 0.85 since serial loading
needs the headroom.
MLX backend: mlx_lm 0.31.x moved maybe_quantize_kv_cache from
mlx_lm.models.cache to mlx_lm.generate. Import from new location.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Replaces the v0.3.x skeletons with working implementations.
vLLM (refract/backends/vllm.py):
- Caches LLM instances per (model, kv_dtype, max_len) across calls
so back-to-back ref+candidate KLD doesn't pay 2x weight load.
- kv_config_str translation table: f16/q8_0/turbo3/turbo4 → vLLM
kv_cache_dtype (auto, fp8_e4m3, turboquant_*).
- run_kld via SamplingParams(prompt_logprobs=K), top-K capped at 64.
- chat template via tokenizer.apply_chat_template.
SGLang (refract/backends/sglang.py):
- HTTP-based; server is launched separately, backend assumes it is
running at REFRACT_SGLANG_URL.
- run_kld requires two simultaneous servers (REFRACT_SGLANG_REF_URL
and REFRACT_SGLANG_CAND_URL) since SGLang's KV dtype is set at
server-launch time, not per-request.
- Rejects TurboQuant configs with a clear capability error (SGLang
upstream has no TurboQuant KV path).
Signed-off-by: TheTom <tturney1@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
REFRACT itself doesn't import numpy or scipy — those were turboquant's
deps but were declared as base deps in pyproject.toml, so a friend who
just wanted to run `refract score` had to pull ~100MB of unused
libraries. Move them to a `[turboquant]` extra. Base install is now
stdlib-only.
pyproject cleanups:
- dependencies = [] (REFRACT is pure stdlib)
- new [turboquant] extra for numpy/scipy
- testpaths fixed: tests + refract/tests (was just "tests")
- added classifiers, license, [project.urls], readme, keywords
- coverage>=7.0 added to dev extras
README:
- new Install section showing the extras matrix
- Platform support table covers all 3 backends (llamacpp / mlx / vllm)
per OS, not just llamacpp
- Windows status flagged as TBD with a pointer to selftest's
Windows-specific remediation hints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mlx.py was 36% — every method needed real mlx-lm + a model. Mock the
mlx + mlx_lm imports (chat template, stream_generate token IDs, native
KLD softmax math, model cache) and exercise the dispatch glue without
the actual ML runtime.
- 16 new tests, mlx.py jumps to 96%
- Total project coverage: 96% (was 92%), 398 passing tests
- Pins MX-array-vs-int token coercion and the corpus-too-short /
zero-chunks guards on the KLD path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When llama-completion fails to launch (Linux missing libllama.so on the
loader path, or Windows missing the DLL next to the .exe), the binary
emits empty stdout and a "cannot open shared object" / 0xc0000135
stderr. The old selftest read empty stdout and reported "--jinja
missing" — misleading diagnosis that sent users hunting for chat
template support when the real problem was the loader.
Now the selftest checks return code + stderr for shared-library / DLL
errors and emits OS-specific remediation (LD_LIBRARY_PATH/ldconfig on
Linux, PATH-next-to-exe on Windows). Three new tests pin both failure
modes plus the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTML output existed since v0.3.2 but only as a passing flag mention
in command examples. Friend doing first-time read wouldn't know:
- It's a single self-contained file (works offline modulo font CDN)
- What's in it (composite, diagnosis, R-NIAH heatmap, etc.)
- Browser compatibility (light-dark() needs 2024+ browsers; older
fall back to light theme cleanly)
- The sun/moon toggle exists
Now in:
- README "Reports" section right under Subcommands, mentioning
JSON + HTML as parallel outputs and what's bundled vs external
- QUICKSTART "Generating the HTML report" subsection in Step 1+2,
with full command, content list, and what works offline vs online
- Subcommand list in README adds `refract fetch` (was missing)
No code changes. Tests still 82/82.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit when figuring out R-NIAH lengths: had to
manually compute the doubling list, pass --rniah-ctx-max separately,
and remember they're related. Replaced with a single primary knob.
What changed
------------
- --rniah-up-to N (new, default 16384) — auto-generates the lengths
list as a doubling step-up from 4K up to N. So
--rniah-up-to 32768 → [4096, 8192, 16384, 32768]
--rniah-up-to 131072 → [4096, ..., 131072]
Cost scales roughly linearly with max length, called out in
QUICKSTART:
16K (default) ~10–15 min on 7B Q8
32K ~25–35 min
64K ~60–90 min
128K ~3+ hours
- --rniah-ctx-max is now optional; defaults to --rniah-up-to. Kept
as a power-user override for when the model's nominal max ctx
exceeds what it can actually do (R-NIAH should skip those cells).
- --rniah-lengths still works for explicit lists, overrides
--rniah-up-to. Documented as power-user.
- --rniah-haystack is now optional; auto-resolved from the
~/.cache/refract/ wikitext-2-raw cache (downloaded by `refract
fetch` or auto-fetched on first score run).
QUICKSTART now shows just `--rniah-up-to 16384` (or 32768/65536/131072)
instead of three coordinated R-NIAH flags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit on the M2 mini fresh-install audit: corpus and
haystack files have to be downloaded separately, and figuring out where
to get wikitext-2-raw is its own subquest. v0.3.2 fixes this.
What changed
------------
- New `refract fetch` subcommand: downloads wikitext-2-raw zip from
the ggml-org HF mirror (the same source llama.cpp's
scripts/get-wikitext-2.sh uses) into ~/.cache/refract/. ~10MB,
~30s on a typical home connection. No third-party deps; stdlib
urllib + zipfile.
- `--corpus` is now optional in `refract score`. When omitted, the
cache is auto-resolved (and auto-fetched if empty) before the
score axes run. Same for `--rniah-haystack` when `--full` is
passed.
- `--no-auto-fetch` flag for CI-friendly invocations that want a
hard error rather than a silent network fetch.
- QUICKSTART updated to call out the auto-fetch and drop the
"fetch wikitext-2 yourself from llama.cpp's repo" step.
Cost trade-off
--------------
- First-run friction: was "go find wikitext-2-raw, download, unzip,
pass --corpus". Now zero — just pass --model + --candidate.
- Network IO: silent on first run, idempotent thereafter. Off-mode
is a flag away.
- Cache directory is ~/.cache/refract/ (XDG-conformant on Linux;
OK on macOS).
Tests: 82/82 still pass. `refract fetch` smoke-tested end-to-end on
the dev box (download + extract + idempotent re-run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues from the in-browser audit:
- Axis title showed "Axis A — GTM" with no spelling-out. Now reads
"Axis A — GTM (Greedy Trajectory Match)" etc, so first-time
readers don't have to scroll for the meaning.
- Reproduce block leaked /tmp/render_example_html.py (the path of
whatever script re-rendered the JSON) instead of the original
`refract score ...` invocation. Now:
1. score-time captures sanitized argv into JSON's repro_command,
BUT only when argv looks like a refract CLI run (gated on
"refract" in argv) so regen scripts don't pollute the field
2. html_report uses raw_json.repro_command if present, else
falls back to sys.argv (sanitized), else synthesizes a clean
"python3 -m refract.cli score --model X --candidate Y ..."
from the report fields
3. /Users/<name>/ paths are scrubbed to ~/ so shared HTML
doesn't leak personal home dirs
- Added a footer link to the GitHub README so a layman opening a
shared HTML can find the framework explanation:
"What is this? See github.com/TheTom/turboquant_plus/refract for
documentation, the motivation paper, and how to interpret these
scores."
Tests: 82/82 still pass. Examples regenerated with clean synthesized
repro commands and the spelled-out axis labels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues found by running the freshly-pushed v0.3.2 through a clean clone
on toms-mac-mini.local:
- Version banner showed v0.3.1; refract/__init__.py was missing the
bump. Now reports v0.3.2 correctly.
- pyproject.toml `requires-python` was >=3.10 but mini ships 3.9.6;
REFRACT works fine on 3.9 thanks to `from __future__ import
annotations`. Lowered the floor.
- pyproject.toml lacked optional deps for the MLX and vLLM backends.
Added `refract-mlx` and `refract-vllm` extras with the right
minimum versions (mlx-lm >= 0.31 since 0.29 doesn't recognise the
qwen3_5 model type).
- pyproject.toml had no `[project.scripts]` entry point. Added
`refract = "refract.cli:main"` so installs get a proper command
instead of `python3 -m refract.cli` only.
- selftest's thinking-mode probe printed "✓ no markers detected"
even when generation failed. Gated the probe on gen_ok so the
happy-path output only appears when the model actually generated.
- selftest description leaned on the "30 minutes of a real run"
framing, scaring readers who don't realize default mode is
5-7 min. Reworded.
- QUICKSTART now leads with a runtime expectations table:
selftest: ~1s static, ~30s with --model
score (default): ~5-7 min (Trajectory + KLD)
score --full: ~25-30 min (+ R-NIAH + PLAD)
repeatability: 4× default
Default is quick. Most users should never wait 30 minutes.
- QUICKSTART preflight example now shows BOTH .gguf and MLX
directory paths so MLX users see they're supported.
- HTML report hardware card showed "137.4 GB" RAM on a 128 GB Mac
(used /1e9 SI gigabytes). Switched to /1024**3 binary GiB so
output matches Apple's marketing convention (128 GB exact).
Also fixed model file size to use binary GiB.
- Regenerated examples/*.html with the GiB fix and live hardware
metadata.
Tests: 82/82 pass. No semantic changes to scoring, only metadata
display + setup ergonomics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Section 9 to the V3 paper documenting the port to vLLM ROCm Triton
on AMD MI300X. New subsections:
9.1 Hardware (MI300X / ROCm 7.2 / Triton 3.6 / vLLM 0.1.dev1)
9.2 Implementation summary (files added, three vLLM-specific design
choices: custom-op Q capture, worker-side env-var lazy init,
mask-based eviction without compaction)
9.3 Models tested (7-row table: validated, partially validated,
blocked, algorithmically incompatible)
9.4 Paper-exact reproduction on Qwen2.5-7B-Instruct
9.4.1 PPL: vLLM deltas all within ±0.5% noise floor
9.4.2 KV cache compression: stack hits 2.96x at 90% retention,
matching the paper's "approximately 2.9x" headline
9.4.3 NIAH: V3 + stack regress vs paper at the middle position
(PARTIAL_NUMBER instead of PASS). Real gap.
9.5 Where the vLLM port falls short of the paper (3 plausible causes
ordered by likelihood: KV preset, scoring numerics, eviction
trigger frequency)
9.6 Hybrid models still untested in vLLM (3 attempts, 3 different
blockers: transformers compat, no-RoPE incompat, vLLM page-size
unifier)
9.7 Headline claim envelope - what I can and cannot claim post-port
9.8 Open work
9.9 How to Run (paper-exact PPL + NIAH commands, tunable knob table,
cross-vendor note, doc pointers)
Honest framing throughout: PPL reproduces within noise, NIAH degrades
vs paper at middle position. Not a refutation of the paper, not a
clean confirmation. The original V3 paper's point that PPL alone is
insufficient signal for V3 quality (Section 3.2) is reinforced
empirically by the vLLM port.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.
Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
(test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
corpus path/size/SHA; refuses to score against a base built from a
different corpus.
New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
--no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).
Total: 14 → 44 unit tests + 1 strengthened integration test.
Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
(detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>
Two bugs surfaced by the v0.1.1 matrix run:
1. Mistral-Small-24B failed in the KLD subprocess with UnicodeDecodeError
on llama-perplexity stderr. text=True does strict utf-8; some llama-cpp
stderr bytes aren't valid utf-8.
2. GTM mean_prefix_agreement_length was in whitespace tokens while
n_predict was in model tokens. Whitespace tokenization can over-count
(gemma-4 31B v0.1.1 produced mean_prefix=59 > n_predict=48 → score
clipped to 100 even though KLD was 49).
Patches:
- runner.py: errors="replace" on all 3 subprocess.run(text=True) calls
- runner.py: new tokenize_to_ids(model, text) helper that shells to
llama-tokenize --ids --no-bos --no-parse-special --log-disable --stdin
- axes/gtm.py: use tokenize_to_ids in the diff path; whitespace _tokenize_words
remains as a fallback if the tokenizer call fails
CHANGELOG fixed to reverse-chronological order with v0.1.2 → v0.1.1 → v0.1.
All 14 unit tests still pass. Smoke test on Qwen3.5-2B Q8 q8/turbo4 went
from GTM=54 (whitespace, 17.4/32) to GTM=94 (model tokens, 30/32).
v0.1's first 7-model matrix run revealed:
1. composite serialised as nested dict, not scalar (aggregator print bug)
2. GTM was comparing llama-cli help banners, not generations (the fork
rejects --no-conversation; runner captured help text)
3. backspace control chars in stdout broke the gen-line regex even after
the --no-conversation fix, so generations were dropped
4. matrix per-model timeout was 30 min; Mistral-Small-24B's KLD axis ran
over and the run failed mid-way
Patches in this commit:
- runner.py: drop --no-conversation, strip \\x08 first, add Loading/prompt-
echo to noise patterns, add gen-line extractor + ASCII-art block stripper
- axes/gtm.py: switch GTM score from binary full-match-rate to continuous
mean-prefix-agreement / n_predict (still report full_match_rate as
diagnostic)
- report.py: flatten composite scalar to JSON top level, full breakdown
moves to composite_detail, schema bumped to refract.report.v0.1.1
- /tmp/refract-matrix.sh (test harness, not committed): per-model timeout
bumped from 30 min to 2 hr
KLD@D axis was always correct in v0.1 — it reproduced the paper §4.3
findings exactly (gemma-4 26B-A4B at KLD 1.74, Qwen2.5-7B at 0.013, etc.).
Only GTM and the JSON layout were broken.
Adds refract/CHANGELOG.md documenting all of this with the v0.1 matrix
results table for reference. v0.1.1 matrix results to be appended when
re-run completes.
All 14 unit tests still pass.
REFRACT is meant to be a general KV-quant quality oracle, applicable
across model families (gemma, qwen, mistral, phi, llama, etc.) and not
TurboQuant-specific. Top-level placement reflects that scope and gives
a cleaner CLI: `python3 -m refract.cli` instead of nested under turboquant.
Updates:
- git mv turboquant/refract/ -> refract/
- All `from turboquant.refract` -> `from refract` imports rewritten
- pyproject.toml: include both turboquant* and refract* in setuptools
- All "turboquant/refract/" path references in README/tests updated
Verified: 14/14 unit tests pass, CLI module path works.
Two-axis composite (GTM + KLD@D corpus proxy) with bit-exact-zero floor
verification, replacing corpus PPL as the primary correctness oracle for
KV-cache quantization changes. Motivated by the §4.3/§4.5/§4.8 findings
in docs/papers/attn-rotation-and-ppl-artifact.md, where corpus PPL inverts
sign on gemma-4-it while KLD vs the fp16-KV reference ranks configs
correctly.
- Axis A (GTM): greedy trajectory match across 30 CC0 prompts, 128 tokens
each, via llama-cli --single-turn (avoids the interactive-mode hang).
- Axis B (KLD@D): mean KL divergence via llama-perplexity --kl-divergence
against an fp16-KV --kl-divergence-base reference (v0.1 corpus proxy;
v0.2 will compute true trajectory KLD).
- Composite: harmonic_mean(GTM, KLD), banded as
EXCELLENT/PASS/DEGRADED/FAIL.
- Floor: REFRACT(ref, ref) must be >= 99.5 or the run aborts.
- CLI: python3 -m turboquant.refract.cli score ...
- Reports: ANSI text card + JSON twin for ML pipelines.
- Tests: 14 fast unit tests + 1 integration test reproducing the
paper's gemma-4 26B-A4B q8/turbo4 OFF cell (must land DEGRADED/FAIL).
Strictly additive under turboquant/refract/; no upstream files touched.
No new pip dependencies (stdlib only).
Co-authored-by: tturney@psyguard.ai
Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
Native Swift/Metal vLLM backend for OpenAI-compatible serving
with TurboQuant+ support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- turbo4v2 is faster than no-turbo (not "can be"), explained by Apple
Silicon matmul favoring smaller types during dequant
- removed "Gap to Python" row — Swift is at parity or ahead now
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Active collaboration with Eric on the Swift MLX fork — turbo4v2 working,
Gemma4 fixes in progress. M-series users should use that for fastest decode.
llama.cpp repo remains the cross-platform (CUDA, ROCm, CPU, Metal) path.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated context scaling table with fresh Qwen2.5-7B benchmarks (GPU idle,
2 runs per context, best-of-2). Previous numbers (61-83%) were measured
before the delegated KVCache optimization (7ad7500).
Updated Quick Start to recommend TurboKVCache (delegated architecture)
over the older TurboKVCacheLite/make_turbo_cache pattern.
New numbers: 97-100% baseline decode at all context lengths 512-16K.
PPL: +0.51% asym, +1.70% sym.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Float32 scales fix resolved the quality issue. C++ kernel now works
at all contexts: 98-100% baseline at 128-4K, 87.5% at 32K.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compact_turbo_cache() uses custom sdpa_vector_qv C++ Metal kernel.
97-100% baseline speed with real V memory savings.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
87-98% baseline speed with ~60% KV memory savings via mx.quantized_matmul.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TurboKVCacheLite: 100% baseline decode, 74% V savings.
Both 7B dense and 35B MoE tested from 128 to 128K context.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
make_turbo_cache() moved into TheTom/mlx with auto monkey-patch.
No mlx-lm fork required.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
35B MoE: 94.2 tok/s (96% of 98.3 baseline). Only 6/10 KV layers
compressed after boundary 2+2.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previous 93% was measured on prompt that barely triggered compression.
Clean benchmark with full compression active: 81% baseline decode.
PPL improved to -0.05% (turbo slightly better than baseline).
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First/last 2 KV layers stay at FP16 to prevent NaN from extreme
V norms on boundary layers. Matches the production recommendation.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7B dense: 60.0 tok/s (93% of 64.5 baseline), +0.04% PPL, zero extra
memory. Previous best was 82% (unpatched asymmetric).
Root cause of prior NaN: extreme V norms (184x) on boundary layers.
Boundary protection resolves completely.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric turbo4 is catastrophic on dense models (KLD=6.86, NIAH 0/15).
Asymmetric K=FP16 fixes everything (KLD=0.003, NIAH 15/15, top-1 98.1%).
Tested on Qwen2.5-7B-Instruct-8bit (28 layers, all KV cache).
Quality test suite added: scripts/mlx_quality_suite.py
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Detect per-layer cache types so hybrid models (Qwen3.5 linear+attention)
get TurboKVCache only on KVCache layers, not ArraysCache layers
- Use min_compress_tokens=32 for KLD test so compression actually fires
- Extend KLD text to ~500 tokens for meaningful divergence measurement
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New standalone script that tests TurboKVCache quality using mlx-lm
without touching existing llama.cpp test scripts.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TheTom