The 4 vLLM "skeleton" tests in test_backends.py asserted that
VLLMBackend methods raise NotImplementedError; they break in v0.3.2.1
where the backend is a working implementation. Replace with:
vLLM (5 tests):
- _kv_str_to_vllm_dtype known mappings (f16 -> auto, q8_0 -> fp8_e4m3)
- TurboQuant preset round-trips (turbo3, turbo4, q8_0/turbo4)
- BackendCapabilityError on unmappable ctk/ctv
- run_completion signature has the kw-only contract args
- all 5 backend methods are present and callable
SGLang (5 tests, new):
- get_backend("sglang") returns SGLangBackend
- _validate_kv_str accepts f16/bf16/q8_0
- TurboQuant rejected loudly (no SGLang TurboQuant path)
- run_kld without REFRACT_SGLANG_REF/CAND_URL raises with actionable msg
- all 5 backend methods are present and callable
Also: MLX backend uses mlx_lm.generate.maybe_quantize_kv_cache when
mlx_lm.models.cache no longer exposes it (0.31+); fall back via
getattr(cache_mod, ...) so tests that mock the old surface still work.
Refract test count: 399 -> 404 (+10 new, -4 stale, 1 mlx fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Add a Step 0 that walks through:
- git clone TheTom/turboquant_plus + cd + pip install -e .[backend-extras]
- optional sibling clones for the patched llama.cpp fork (with cmake
build steps and LLAMA_CPP_BIN_DIR export) and the vLLM fork
- pointer to the SGLang Docker image used in the cross-engine bench
Renumber the existing Steps 0..4 -> 2..6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Documents the 3-engine apples-to-apples cross-engine benchmark:
- BF16 baseline (load, PPL, prefill, decode, KV size) across vLLM, SGLang, llama.cpp
- REFRACT --full 4-axis fidelity scoring (gtm, kld, rniah, plad) on each engine's native 8-bit KV
- Headline finding: same fp8_e4m3 dtype string produces 1.8x different KL drift between vLLM and SGLang on the same model
- 8 engine-side bugs encountered and worked around (vLLM max_num_seqs, prompt_logprobs cap, GPU mem carryover; SGLang aiter.dtypes, AITER fp8 prefill broken, KV dtype fixed at server launch)
- Full reproducibility scripts and REFRACT framework changes documented
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
PLAD axis previously interleaved ref/cand per (prompt, perturbation) cell.
On vLLM with hybrid Qwen3.6 (~120GB per LLM, two won't fit 192GB MI300X),
each ref/cand swap triggers eviction-and-reload of ~30-90s. With 30 prompts
× 4 cells (anchor + 3 perturbations) = 240 swaps, the axis runs ~3 hours
just for eviction overhead.
Now plans all cells upfront, runs all ref-side completions, then all
cand-side completions, then computes drift. Two LLM loads total.
No effect on backends that can hold both configs in cache cheaply
(llama.cpp, MLX).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
R-NIAH axis calls runner.tokenize_to_ids to estimate chars-per-token. The
runner unconditionally shelled out to llama-tokenize binary, which fails
when the host's llama.cpp checkout has drifted (Symbol not found:
_llama_memory_breakdown_print) or when running on droplets without llama.cpp
on PATH at all.
Now dispatches to _ACTIVE_BACKEND.tokenize_to_ids when the backend is not
llamacpp. MLX/vLLM/SGLang backends already implement this method.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Memory-pressured backends (vLLM on hybrid Qwen3.6-35B-A3B can't fit two
simultaneous LLM instances on 192GB) need each KV config loaded exactly
once per axis. The trajectory axis previously interleaved ref/cand calls
per prompt, which forces 2N model loads (catastrophic when each load
takes 40 min on hybrid models).
Now collect all ref trajectories, then all cand trajectories, then
diff. Two model loads total instead of 2N. No effect on llama.cpp/MLX
which keep configs cached cheaply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
vLLM backend: hybrid Qwen3.6-35B-A3B doesn't fit two simultaneous LLM
instances (~120GB each on 192GB MI300X). Cache evicts prior LLM
before loading new one. Adds max_num_seqs env knob (default 32) so
Mamba cache block count doesn't underflow vs default max_num_seqs=1024.
Default gpu_memory_utilization bumped to 0.85 since serial loading
needs the headroom.
MLX backend: mlx_lm 0.31.x moved maybe_quantize_kv_cache from
mlx_lm.models.cache to mlx_lm.generate. Import from new location.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Replaces the v0.3.x skeletons with working implementations.
vLLM (refract/backends/vllm.py):
- Caches LLM instances per (model, kv_dtype, max_len) across calls
so back-to-back ref+candidate KLD doesn't pay 2x weight load.
- kv_config_str translation table: f16/q8_0/turbo3/turbo4 → vLLM
kv_cache_dtype (auto, fp8_e4m3, turboquant_*).
- run_kld via SamplingParams(prompt_logprobs=K), top-K capped at 64.
- chat template via tokenizer.apply_chat_template.
SGLang (refract/backends/sglang.py):
- HTTP-based; server is launched separately, backend assumes it is
running at REFRACT_SGLANG_URL.
- run_kld requires two simultaneous servers (REFRACT_SGLANG_REF_URL
and REFRACT_SGLANG_CAND_URL) since SGLang's KV dtype is set at
server-launch time, not per-request.
- Rejects TurboQuant configs with a clear capability error (SGLang
upstream has no TurboQuant KV path).
Signed-off-by: TheTom <tturney1@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
REFRACT itself doesn't import numpy or scipy — those were turboquant's
deps but were declared as base deps in pyproject.toml, so a friend who
just wanted to run `refract score` had to pull ~100MB of unused
libraries. Move them to a `[turboquant]` extra. Base install is now
stdlib-only.
pyproject cleanups:
- dependencies = [] (REFRACT is pure stdlib)
- new [turboquant] extra for numpy/scipy
- testpaths fixed: tests + refract/tests (was just "tests")
- added classifiers, license, [project.urls], readme, keywords
- coverage>=7.0 added to dev extras
README:
- new Install section showing the extras matrix
- Platform support table covers all 3 backends (llamacpp / mlx / vllm)
per OS, not just llamacpp
- Windows status flagged as TBD with a pointer to selftest's
Windows-specific remediation hints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mlx.py was 36% — every method needed real mlx-lm + a model. Mock the
mlx + mlx_lm imports (chat template, stream_generate token IDs, native
KLD softmax math, model cache) and exercise the dispatch glue without
the actual ML runtime.
- 16 new tests, mlx.py jumps to 96%
- Total project coverage: 96% (was 92%), 398 passing tests
- Pins MX-array-vs-int token coercion and the corpus-too-short /
zero-chunks guards on the KLD path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When llama-completion fails to launch (Linux missing libllama.so on the
loader path, or Windows missing the DLL next to the .exe), the binary
emits empty stdout and a "cannot open shared object" / 0xc0000135
stderr. The old selftest read empty stdout and reported "--jinja
missing" — misleading diagnosis that sent users hunting for chat
template support when the real problem was the loader.
Now the selftest checks return code + stderr for shared-library / DLL
errors and emits OS-specific remediation (LD_LIBRARY_PATH/ldconfig on
Linux, PATH-next-to-exe on Windows). Three new tests pin both failure
modes plus the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTML output existed since v0.3.2 but only as a passing flag mention
in command examples. Friend doing first-time read wouldn't know:
- It's a single self-contained file (works offline modulo font CDN)
- What's in it (composite, diagnosis, R-NIAH heatmap, etc.)
- Browser compatibility (light-dark() needs 2024+ browsers; older
fall back to light theme cleanly)
- The sun/moon toggle exists
Now in:
- README "Reports" section right under Subcommands, mentioning
JSON + HTML as parallel outputs and what's bundled vs external
- QUICKSTART "Generating the HTML report" subsection in Step 1+2,
with full command, content list, and what works offline vs online
- Subcommand list in README adds `refract fetch` (was missing)
No code changes. Tests still 82/82.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit when figuring out R-NIAH lengths: had to
manually compute the doubling list, pass --rniah-ctx-max separately,
and remember they're related. Replaced with a single primary knob.
What changed
------------
- --rniah-up-to N (new, default 16384) — auto-generates the lengths
list as a doubling step-up from 4K up to N. So
--rniah-up-to 32768 → [4096, 8192, 16384, 32768]
--rniah-up-to 131072 → [4096, ..., 131072]
Cost scales roughly linearly with max length, called out in
QUICKSTART:
16K (default) ~10–15 min on 7B Q8
32K ~25–35 min
64K ~60–90 min
128K ~3+ hours
- --rniah-ctx-max is now optional; defaults to --rniah-up-to. Kept
as a power-user override for when the model's nominal max ctx
exceeds what it can actually do (R-NIAH should skip those cells).
- --rniah-lengths still works for explicit lists, overrides
--rniah-up-to. Documented as power-user.
- --rniah-haystack is now optional; auto-resolved from the
~/.cache/refract/ wikitext-2-raw cache (downloaded by `refract
fetch` or auto-fetched on first score run).
QUICKSTART now shows just `--rniah-up-to 16384` (or 32768/65536/131072)
instead of three coordinated R-NIAH flags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit on the M2 mini fresh-install audit: corpus and
haystack files have to be downloaded separately, and figuring out where
to get wikitext-2-raw is its own subquest. v0.3.2 fixes this.
What changed
------------
- New `refract fetch` subcommand: downloads wikitext-2-raw zip from
the ggml-org HF mirror (the same source llama.cpp's
scripts/get-wikitext-2.sh uses) into ~/.cache/refract/. ~10MB,
~30s on a typical home connection. No third-party deps; stdlib
urllib + zipfile.
- `--corpus` is now optional in `refract score`. When omitted, the
cache is auto-resolved (and auto-fetched if empty) before the
score axes run. Same for `--rniah-haystack` when `--full` is
passed.
- `--no-auto-fetch` flag for CI-friendly invocations that want a
hard error rather than a silent network fetch.
- QUICKSTART updated to call out the auto-fetch and drop the
"fetch wikitext-2 yourself from llama.cpp's repo" step.
Cost trade-off
--------------
- First-run friction: was "go find wikitext-2-raw, download, unzip,
pass --corpus". Now zero — just pass --model + --candidate.
- Network IO: silent on first run, idempotent thereafter. Off-mode
is a flag away.
- Cache directory is ~/.cache/refract/ (XDG-conformant on Linux;
OK on macOS).
Tests: 82/82 still pass. `refract fetch` smoke-tested end-to-end on
the dev box (download + extract + idempotent re-run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues from the in-browser audit:
- Axis title showed "Axis A — GTM" with no spelling-out. Now reads
"Axis A — GTM (Greedy Trajectory Match)" etc, so first-time
readers don't have to scroll for the meaning.
- Reproduce block leaked /tmp/render_example_html.py (the path of
whatever script re-rendered the JSON) instead of the original
`refract score ...` invocation. Now:
1. score-time captures sanitized argv into JSON's repro_command,
BUT only when argv looks like a refract CLI run (gated on
"refract" in argv) so regen scripts don't pollute the field
2. html_report uses raw_json.repro_command if present, else
falls back to sys.argv (sanitized), else synthesizes a clean
"python3 -m refract.cli score --model X --candidate Y ..."
from the report fields
3. /Users/<name>/ paths are scrubbed to ~/ so shared HTML
doesn't leak personal home dirs
- Added a footer link to the GitHub README so a layman opening a
shared HTML can find the framework explanation:
"What is this? See github.com/TheTom/turboquant_plus/refract for
documentation, the motivation paper, and how to interpret these
scores."
Tests: 82/82 still pass. Examples regenerated with clean synthesized
repro commands and the spelled-out axis labels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues found by running the freshly-pushed v0.3.2 through a clean clone
on toms-mac-mini.local:
- Version banner showed v0.3.1; refract/__init__.py was missing the
bump. Now reports v0.3.2 correctly.
- pyproject.toml `requires-python` was >=3.10 but mini ships 3.9.6;
REFRACT works fine on 3.9 thanks to `from __future__ import
annotations`. Lowered the floor.
- pyproject.toml lacked optional deps for the MLX and vLLM backends.
Added `refract-mlx` and `refract-vllm` extras with the right
minimum versions (mlx-lm >= 0.31 since 0.29 doesn't recognise the
qwen3_5 model type).
- pyproject.toml had no `[project.scripts]` entry point. Added
`refract = "refract.cli:main"` so installs get a proper command
instead of `python3 -m refract.cli` only.
- selftest's thinking-mode probe printed "✓ no markers detected"
even when generation failed. Gated the probe on gen_ok so the
happy-path output only appears when the model actually generated.
- selftest description leaned on the "30 minutes of a real run"
framing, scaring readers who don't realize default mode is
5-7 min. Reworded.
- QUICKSTART now leads with a runtime expectations table:
selftest: ~1s static, ~30s with --model
score (default): ~5-7 min (Trajectory + KLD)
score --full: ~25-30 min (+ R-NIAH + PLAD)
repeatability: 4× default
Default is quick. Most users should never wait 30 minutes.
- QUICKSTART preflight example now shows BOTH .gguf and MLX
directory paths so MLX users see they're supported.
- HTML report hardware card showed "137.4 GB" RAM on a 128 GB Mac
(used /1e9 SI gigabytes). Switched to /1024**3 binary GiB so
output matches Apple's marketing convention (128 GB exact).
Also fixed model file size to use binary GiB.
- Regenerated examples/*.html with the GiB fix and live hardware
metadata.
Tests: 82/82 pass. No semantic changes to scoring, only metadata
display + setup ergonomics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Section 9 to the V3 paper documenting the port to vLLM ROCm Triton
on AMD MI300X. New subsections:
9.1 Hardware (MI300X / ROCm 7.2 / Triton 3.6 / vLLM 0.1.dev1)
9.2 Implementation summary (files added, three vLLM-specific design
choices: custom-op Q capture, worker-side env-var lazy init,
mask-based eviction without compaction)
9.3 Models tested (7-row table: validated, partially validated,
blocked, algorithmically incompatible)
9.4 Paper-exact reproduction on Qwen2.5-7B-Instruct
9.4.1 PPL: vLLM deltas all within ±0.5% noise floor
9.4.2 KV cache compression: stack hits 2.96x at 90% retention,
matching the paper's "approximately 2.9x" headline
9.4.3 NIAH: V3 + stack regress vs paper at the middle position
(PARTIAL_NUMBER instead of PASS). Real gap.
9.5 Where the vLLM port falls short of the paper (3 plausible causes
ordered by likelihood: KV preset, scoring numerics, eviction
trigger frequency)
9.6 Hybrid models still untested in vLLM (3 attempts, 3 different
blockers: transformers compat, no-RoPE incompat, vLLM page-size
unifier)
9.7 Headline claim envelope - what I can and cannot claim post-port
9.8 Open work
9.9 How to Run (paper-exact PPL + NIAH commands, tunable knob table,
cross-vendor note, doc pointers)
Honest framing throughout: PPL reproduces within noise, NIAH degrades
vs paper at middle position. Not a refutation of the paper, not a
clean confirmation. The original V3 paper's point that PPL alone is
insufficient signal for V3 quality (Section 3.2) is reinforced
empirically by the vLLM port.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.
Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
(test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
corpus path/size/SHA; refuses to score against a base built from a
different corpus.
New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
--no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).
Total: 14 → 44 unit tests + 1 strengthened integration test.
Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
(detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>
Two bugs surfaced by the v0.1.1 matrix run:
1. Mistral-Small-24B failed in the KLD subprocess with UnicodeDecodeError
on llama-perplexity stderr. text=True does strict utf-8; some llama-cpp
stderr bytes aren't valid utf-8.
2. GTM mean_prefix_agreement_length was in whitespace tokens while
n_predict was in model tokens. Whitespace tokenization can over-count
(gemma-4 31B v0.1.1 produced mean_prefix=59 > n_predict=48 → score
clipped to 100 even though KLD was 49).
Patches:
- runner.py: errors="replace" on all 3 subprocess.run(text=True) calls
- runner.py: new tokenize_to_ids(model, text) helper that shells to
llama-tokenize --ids --no-bos --no-parse-special --log-disable --stdin
- axes/gtm.py: use tokenize_to_ids in the diff path; whitespace _tokenize_words
remains as a fallback if the tokenizer call fails
CHANGELOG fixed to reverse-chronological order with v0.1.2 → v0.1.1 → v0.1.
All 14 unit tests still pass. Smoke test on Qwen3.5-2B Q8 q8/turbo4 went
from GTM=54 (whitespace, 17.4/32) to GTM=94 (model tokens, 30/32).
v0.1's first 7-model matrix run revealed:
1. composite serialised as nested dict, not scalar (aggregator print bug)
2. GTM was comparing llama-cli help banners, not generations (the fork
rejects --no-conversation; runner captured help text)
3. backspace control chars in stdout broke the gen-line regex even after
the --no-conversation fix, so generations were dropped
4. matrix per-model timeout was 30 min; Mistral-Small-24B's KLD axis ran
over and the run failed mid-way
Patches in this commit:
- runner.py: drop --no-conversation, strip \\x08 first, add Loading/prompt-
echo to noise patterns, add gen-line extractor + ASCII-art block stripper
- axes/gtm.py: switch GTM score from binary full-match-rate to continuous
mean-prefix-agreement / n_predict (still report full_match_rate as
diagnostic)
- report.py: flatten composite scalar to JSON top level, full breakdown
moves to composite_detail, schema bumped to refract.report.v0.1.1
- /tmp/refract-matrix.sh (test harness, not committed): per-model timeout
bumped from 30 min to 2 hr
KLD@D axis was always correct in v0.1 — it reproduced the paper §4.3
findings exactly (gemma-4 26B-A4B at KLD 1.74, Qwen2.5-7B at 0.013, etc.).
Only GTM and the JSON layout were broken.
Adds refract/CHANGELOG.md documenting all of this with the v0.1 matrix
results table for reference. v0.1.1 matrix results to be appended when
re-run completes.
All 14 unit tests still pass.
REFRACT is meant to be a general KV-quant quality oracle, applicable
across model families (gemma, qwen, mistral, phi, llama, etc.) and not
TurboQuant-specific. Top-level placement reflects that scope and gives
a cleaner CLI: `python3 -m refract.cli` instead of nested under turboquant.
Updates:
- git mv turboquant/refract/ -> refract/
- All `from turboquant.refract` -> `from refract` imports rewritten
- pyproject.toml: include both turboquant* and refract* in setuptools
- All "turboquant/refract/" path references in README/tests updated
Verified: 14/14 unit tests pass, CLI module path works.
Two-axis composite (GTM + KLD@D corpus proxy) with bit-exact-zero floor
verification, replacing corpus PPL as the primary correctness oracle for
KV-cache quantization changes. Motivated by the §4.3/§4.5/§4.8 findings
in docs/papers/attn-rotation-and-ppl-artifact.md, where corpus PPL inverts
sign on gemma-4-it while KLD vs the fp16-KV reference ranks configs
correctly.
- Axis A (GTM): greedy trajectory match across 30 CC0 prompts, 128 tokens
each, via llama-cli --single-turn (avoids the interactive-mode hang).
- Axis B (KLD@D): mean KL divergence via llama-perplexity --kl-divergence
against an fp16-KV --kl-divergence-base reference (v0.1 corpus proxy;
v0.2 will compute true trajectory KLD).
- Composite: harmonic_mean(GTM, KLD), banded as
EXCELLENT/PASS/DEGRADED/FAIL.
- Floor: REFRACT(ref, ref) must be >= 99.5 or the run aborts.
- CLI: python3 -m turboquant.refract.cli score ...
- Reports: ANSI text card + JSON twin for ML pipelines.
- Tests: 14 fast unit tests + 1 integration test reproducing the
paper's gemma-4 26B-A4B q8/turbo4 OFF cell (must land DEGRADED/FAIL).
Strictly additive under turboquant/refract/; no upstream files touched.
No new pip dependencies (stdlib only).
Co-authored-by: tturney@psyguard.ai
Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
Native Swift/Metal vLLM backend for OpenAI-compatible serving
with TurboQuant+ support.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- turbo4v2 is faster than no-turbo (not "can be"), explained by Apple
Silicon matmul favoring smaller types during dequant
- removed "Gap to Python" row — Swift is at parity or ahead now
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Active collaboration with Eric on the Swift MLX fork — turbo4v2 working,
Gemma4 fixes in progress. M-series users should use that for fastest decode.
llama.cpp repo remains the cross-platform (CUDA, ROCm, CPU, Metal) path.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Updated context scaling table with fresh Qwen2.5-7B benchmarks (GPU idle,
2 runs per context, best-of-2). Previous numbers (61-83%) were measured
before the delegated KVCache optimization (7ad7500).
Updated Quick Start to recommend TurboKVCache (delegated architecture)
over the older TurboKVCacheLite/make_turbo_cache pattern.
New numbers: 97-100% baseline decode at all context lengths 512-16K.
PPL: +0.51% asym, +1.70% sym.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Float32 scales fix resolved the quality issue. C++ kernel now works
at all contexts: 98-100% baseline at 128-4K, 87.5% at 32K.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compact_turbo_cache() uses custom sdpa_vector_qv C++ Metal kernel.
97-100% baseline speed with real V memory savings.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
87-98% baseline speed with ~60% KV memory savings via mx.quantized_matmul.
Co-Authored-By: tturney1@gmail.com
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
