Atlas now ships the full TurboQuant+ KV cache upstream on both
backends: CUDA/GB10 (PR #133 — canonical Hadamard, 9 asymmetric
variants, Turbo2) and Apple Metal (PR #147 — turbo8/4/3/2 +
safer-asym + sparse-V dequant).
Expand the §QJL note to state explicitly that production drops the QJL
stage on both K and V. Name TheTom/llama-cpp-turboquant as the production
path and document the recommended config
(--cache-type-k q8_0 --cache-type-v turbo3). Keep the 5-group consensus
citation and add guidance for downstream users: use TurboQuantMSE for V
or straight PolarQuant for K. Only enable QJL classes for paper
reproducibility or K-side research below 8-bit, and validate at target
context length since QJL noise historically accumulates past ~16K.
Follow-up to #93 (QJL math fix). The reference impl is now correct, but
the production guidance hadn't been said this plainly anywhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(qjl): use orthogonal projection and sqrt(d) scale factor
The QJL stage used a Gaussian random matrix S ~ N(0,1)^(d×d) for
sign(S · x). Gaussian projections introduce variance of order ||r||^2/d
per dimension; for head_dim=128, this is large relative to signal, and
attention accumulation drives generation into word-loop degeneration
once QJL is enabled on real LLM KV vectors.
Fix:
* Replace Gaussian S with a random orthogonal matrix via QR
decomposition (sign-corrected to give a proper rotation). Orthogonal
matrices preserve norms and inner products and reduce reconstruction
variance sharply.
* Correct the dequantization scale from sqrt(pi/2)/d to sqrt(pi/2)/sqrt(d).
For d=128, the previous formula was ~11x too small, which effectively
disabled the QJL correction term even when the stage was enabled.
* Add a damping parameter (default 0.7) to dequantize(), validated as
the stable operating point in M1 Pro 16K-context needle benchmarks
on Qwen 4B with K5/V4 hybrid quantization.
Empirical impact on M1 Pro (16 GB, Qwen 4B):
* Needle retrieval at 16K tokens: 0% → 100% with these QJL fixes
combined with K5/V4 Hybrid configuration.
* Verified orthogonality: ||S S^T - I||_F < 1e-14.
* Existing tests pass (e.g., test_dequantized_has_correct_scale
reports avg norm ratio 0.877 at d=128, well within [0.5, 2.0]).
Refs: turboquant-m1pro-evaluation reproducer and post-mortem at
https://github.com/devYRPauli/turboquant-m1pro-evaluation
* refactor(qjl): address review — rename damping → shrinkage, default 1.0
Per @TheTom's review on PR #93:
* Rename `damping` kwarg to `shrinkage` on QJL.dequantize and
TurboQuant.dequantize. Default 1.0 — classical paper-faithful
unbiased estimator. Existing callers without the kwarg get
classical unbiased QJL (backward-compatible).
* Docstrings document `2/np.pi ≈ 0.6366` as the MMSE-optimal
shrinkage with the one-line derivation from E[||x̂||²] =
(π/2)·||x||² and E[⟨x̂, x⟩] = ||x||².
* Add the orthogonality contract:
- `__init__` asserts `||S Sᵀ − I||_F < 1e-10` (cheap defensive
check; required for the unbiased estimator).
- New `TestQJLProjection.test_projection_matrix_is_orthogonal`
at d ∈ {64, 128, 256, 512} with the tighter `< 1e-12`
tolerance @TheTom suggested.
Verified: orthogonality error 5e-15 to 4e-14 across all tested d.
Reconstruction norm ratio under default shrinkage=1.0 averages
1.253 = √(π/2) at every d, exactly matching the closed form.
Sprint journal documenting the in-progress block-selector sparse
attention work on Apple Silicon: companion to sparse-v-dequant.md
(post-softmax) covering the pre-attention K-skip lever.
Includes the preliminary local M5 Max bench grid (1.16-1.73× decode
across B=4-8 × ctx=16K-48K on Qwen2.5-14B-1M-4bit), the Apple-Silicon-
specific findings (MLX_SDPA_BLOCKS heuristic mis-tune for the float-mask
kernel variant, pass-2 reducer modulo bug, MetalAllocator non-determinism
cliff), the failed-paths catalogue (custom Metal kernel 14× slower at B=8,
compose-gather 13.7× slower, bool mask 25% regression), and the
per-family port pattern across 10 model families.
Framed as a sprint journal — the work is not yet merged to upstream
alpha (PRs ekryski/mlx#34, ekryski/mlx-swift#36, ekryski/mlx-swift-lm#224,
ekryski/mlx-swift-lm#225 all open). Numbers are local M5 Max only;
recommendations are current best guess rather than settled defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Two new pieces of paper-form documentation covering longctx's two
operating modes and the V3 rescue fix.
NEW: docs/papers/longctx-1m-and-triattention.md
- Long-form paper on longctx — open long-context retrieval companion
for inference engines. Two modes: (1) standalone retrieval scored
on MRCR v2 8-needle on AMD MI300X with Qwen2.5-32B-Instruct, (2)
rescue layer for our TriAttention V3 eviction policy on Apple
Silicon, validated end-to-end via 256K planted-fact NIAH on
Qwen3.5-2B-4bit.
- Headline numbers: MRCR v2 1M MultiQ Selector + bge-rerank + det
copy n=30 directional 0.688 (above SubQ 0.659; mass-val pending).
V3+longctx ✓HIT every rung 32K → 256K via ChatSession;
V3-only ✗miss every rung.
- Architecture diagrams: full-stack (clients → vllm-swift →
ChatSession → mlx-swift-lm → V3 engine + rescue → longctx-svc),
V3 turn-by-turn flow (eviction-write on turn N, rehydrate-retrieve
before turn N+1).
- Appendices: module map (mlx-swift-lm side), longctx-svc endpoint
surface, V3 environment variable reference table.
- Cites Mao et al., "TriAttention: Efficient Long Reasoning with
Trigonometric KV Compression" (arXiv:2604.04921, 2026) as the
original paper. V3 in this stack is our independent Swift port +
hybrid extension; cross-references docs/papers/triattention-v3.md.
UPDATED: docs/papers/triattention-v3.md
- Adds §10 "Addendum: Swift Port + longctx Rescue — Fixing the
Hybrid NIAH Failure" after the §9 vLLM addendum.
- Reports the Apple Silicon Swift port test results that resolve the
Section 7.1 question "why does V3 break on hybrid Mamba+Attention
models?". Same failure mode reproduces on Swift port; longctx
rescue layer flips it.
- Includes the 12-cell receipt table, the V3 turn-by-turn diagram,
the driver-gotcha section explaining why bare container.generate()
doesn't rescue (Tier-3 rehydrate is ChatSession-only per commit
fe1a3b0 in mlx-swift-lm), and what's still not measured (>256K,
larger hybrids, V3+TQ+ stack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed scripts/eden-investigation/ as a public artifact
pointer, but the directory contents are scratch and don't belong in the public
repo. Methodology in §2 is sufficient for reproduction; the file pointer was
a leftover from cleaning up the earlier /tmp/ paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed personal scratch paths under /tmp/eden-investigation/
that aren't useful to public readers and look confusing. Replaces those plus the
two private Obsidian-note pointers with a single line pointing to the
scripts/eden-investigation/ directory in this repo, where the actual experiment
scripts live (h*.py, F1–F3 KL test, universal-K candidate centroids).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two independent third-party confirmations to the appendices:
asymmetric-kv-compression.md: @jagmarques tested asymmetric K/V on
Mistral-7B with E8 lattice VQ (a different quantization family from
TurboQuant) on Cerebrium A10/A100. Confirms asymmetric advantage scales
with context length: at 1742-tok prefix K3V2 +0.82% vs K2V2 +0.91%; at
3544-tok prefix K3V2 +0.35% vs K2V2 +2.26% (6.4× improvement). Clean
GQA-amplification explanation: 8 query heads per K head means K
quantization error is effectively 8× worse than V error in output
impact. Cross-method validation that the asymmetric conclusion is not
TurboQuant-specific.
turbo4-resurrection.md: @unamedkr / quantumaikr (quant.cpp) ported the
TurboQuant paper end-to-end in single-header C and found the QJL stage
contributed byte-identical zero to attention scores regardless of
input. Disabling QJL produced bit-for-bit identical perplexity.
Theoretical explanation: the √(π/2)/m constant in the unbiased
estimator is derived for Gaussian projection rows; on Rademacher (±1)
rows the per-key contribution gets dwarfed by residual-norm scaling.
Independently took the freed bits and reinvested in 4-bit codebook
(8 → 16 centroids), same path as turbo4 resurrection. Llama 3.2 3B:
turbo_kv_4b PPL 14.28 (+5.3% vs FP32), beating llama.cpp q4_0 KV
(~14.99, +10.6%). Targets embedded/edge (iOS, Android, WASM,
microcontrollers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the empirical study covering the broader 35-hypothesis investigation
spawned by the EDEN/Mitzenmacher/Portnoy critique (#87, #89). Companion
to eden-optimal-s-revisit.md.
Headline result: a K-cache centroid table that improves per-vector
reconstruction MSE by 1–13% across five model families (Qwen3-0.6B,
Llama-3.2-1B, Mistral-7B, Phi-3-mini, Gemma-2-2b) causes 70–90% mean
KL@D regressions and 50–60% catastrophic-rate failures at the model
output. This is a sign inversion, not a small mismatch: methods that
are strictly better under MSE are systematically worse under KL@D in
deployment.
The paper is structured in three acts:
- Act 1: K cache is sub-Gaussian post-WHT (H8); V is Gaussian
- Act 2: per-prompt fitting fails (PDS), ensemble (ens_4way) reaches an
irreducible ~8% catastrophic floor at scale (N=100)
- Act 3: the universal-K cross-model counterexample (F3) demonstrates
the MSE→KL inversion at the cleanest possible level
Includes a 2-D toy attention illustration of the softmax bucket-flip
mechanism, an explicit decision boundary for when MSE proxies quality
(linear consumers) vs when it doesn't (attention softmax), an
ablation summary distinguishing first-order from second-order factors,
operational guidance for practitioners, and a "why this wasn't caught
earlier" diagnosis of the methodology blind spots.
Companion paper handles the algorithmic claim head-on. This paper is
the broader evaluation-methodology critique.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a short paragraph in Appendix A documenting the algebraic identity
sqrt(S_unb * S_bias) = ||x||/||c||, which coincides with the matched-norm
scale. Frames it as an algebraic interpolation between the two
EDEN-derived forms, notes it is not derived in DRIVE, EDEN, or the
TurboQuant/EDEN note paper (arXiv:2604.18555), and does not correspond
to an optimization objective in those works.
The empirical head-to-head numbers in the table directly above the new
paragraph already show that each form attains the optimum of its own
objective and only its own; matched-norm behaves as a distinct operating
point, not as a tuning of either EDEN scale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three findings from the Mac mini PyPI install walkthrough (2026-05-03):
1. `--prompts` was required even though wheel ships them — defaulted to
bundled refract/prompts/v0.1.jsonl via importlib.resources (v0.3.2.2)
2. macOS system python3 is 3.9, mlx-lm needs >=3.10 — documented in
QUICKSTART Step 0 with brew install hint
3. MLX run_kld returned mean_kld=NaN on Qwen2.5-1.5B-Instruct-8bit +
q8_0/q8_0 KV. Root cause: log(softmax + 1e-12) underflows on
sharply-peaked logits from already-8bit weight models. Rewritten
in terms of log-softmax (logits - logsumexp) which is stable.
Non-finite per-position KL values now filtered before aggregation;
mean computed across valid positions only. If every position is
non-finite the backend raises BackendCapabilityError with an
actionable hint instead of silently returning NaN.
Test mock updated for the new mx symbols (logsumexp, exp, isfinite,
where, zeros_like). 416 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Mac mini install walkthrough (2026-05-03) found that `refract score`
required explicit --prompts even though the wheel ships the v0.1
prompts file. After `pip install refract-llm` a user shouldn't need
to know where the bundled file lives.
Now: when --prompts is omitted, _run_score resolves the bundled
refract/prompts/v0.1.jsonl via importlib.resources. Minimum viable
score invocation drops to:
refract score --backend mlx --model /path/to/model --candidate ctk=q8_0,ctv=q8_0
README install section flipped to lead with `pip install refract-llm`
(PyPI path) and demote the source-clone path to "for hacking".
Walkthrough also surfaced two non-blocking findings (kept as separate
follow-ups):
- macOS system python3 is 3.9, mlx-lm needs 3.10+; users hit this
- KLD axis returned NaN on Qwen2.5-1.5B-Instruct-8bit MLX — separate
numerical-stability bug, not a packaging issue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
The eden.py default formula S = ||x||²/⟨c,x⟩ was loosely labeled as
"biased optimal-S" throughout. On closer inspection that's imprecise:
the strict per-vector MSE-minimum is S* = ⟨c,x⟩/||c||² (textbook
least-squares; asymptotically → 1 for Lloyd-Max optimal centroids).
The eden.py default is a different formula, satisfying the inner-
product-preserving identity ⟨Sc, x⟩ = ||x||² per vector.
Empirical comparisons unaffected (we tested what eden.py ships).
Just the labeling. Added Terminology note in §4 with worked example
and credit to @amitport for surfacing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Light-touch addition to the existing 'Additional experiments and
writeups' aggregator line. Discoverable for anyone following the
academic discussion without dominating the deployment-focused README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical response to issues #87 (Mitzenmacher) and #89 (Portnoy) on
turboquant_plus. 8 hypothesis tests across synthetic Gaussian and
Qwen3-0.6B + Llama-3.2-1B real KV cache.
Headline: EDEN's optimal scale is real but second-order. Rotation choice
(WHT vs dense Haar) is first-order and accounts for ~all of the published
gap. Production already uses the first-order fix. On real KV with WHT,
matched-norm beats EDEN-S by 0.5-9% across Qwen and Llama; tightens to
0.67-0.72% on Llama. EDEN-S genuinely wins at b=8 on synthetic Gaussian
(70%+ MSE reduction) — credibility moat showing the lever exists in its
own regime.
Concedes attribution argument (#89): DRIVE/EDEN line is real prior art.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Distribution name: turboquant -> refract-llm.
Top-level package shipped: refract* only (turboquant/ stays in source
tree for dev work but is no longer in the wheel).
Bundled via package-data:
- refract/prompts/v0.1.jsonl (the 30-prompt CC0 set)
- refract/examples/*.json + *.html (4 reference reports)
- refract/llama-cpp-patches/refract-trajectory.patch (for users with
their own llama.cpp build to apply themselves)
readme = refract/README.md (PyPI long_description focused on REFRACT).
Adds a [full] extra that pulls all backend deps in one shot. Adds dev
extras for build + twine.
Refract __version__ bumped 0.3.2 -> 0.3.2.1 to match the wheel.
Local smoke: pip install dist/refract_llm-0.3.2.1-py3-none-any.whl in
a fresh venv, `refract --help` and `refract selftest` work, bundled
data accessible via importlib.resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Audit follow-up to the earlier skip-axis fix (728cd84). The stub axis
result dataclasses still carry score=100, so without these changes:
- json_report still wrote 'axes.gtm.score: 100.0' for a skipped axis,
visible to anyone parsing the JSON or running 'refract compare'
- 'refract compare' showed 100.0 in the Traj/KLD column for skipped
reports, which made cross-report comparison falsely flatter the
skipped run
Now: when CompositeScore.gtm_score is None we write axes.gtm.score = null
in the JSON. 'refract compare' reads 'skipped' or score=None and prints
'skip' in that column instead of the stub 100.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Two reader-facing fixes from AJ's screenshot (2026-05-02 X thread):
1. --skip-gtm / --skip-kld previously fed a stub score=100 to
composite_score, so a single real KLD axis of 99.47 produced a
misleading composite of 99.74 EXCELLENT. The skipped axis now
contributes None to composite_score() and is excluded from the
harmonic mean. CompositeScore.gtm_score / .kld_score are now
Optional[float]. Text + HTML reports render "n/a" with a
"skipped" badge for skipped axes instead of fake 100/EXCELLENT.
The "harmonic mean of N axes" gloss reflects the real count.
Diagnostics blocks (GTM diagnostics, KLD diagnostics) are
replaced with a single "axis was skipped" line when the axis
wasn't measured.
2. The HTML "Run details → Environment" card now surfaces the
backend name (llama.cpp / MLX / vLLM / SGLang) plus engine
version fields (llama_cpp_commit, mlx_lm_version, vllm_version,
sglang_url, served_model_id) when the active backend populates
them. Previously a report didn't say which engine ran it, which
made cross-engine comparison reports ambiguous in the inbox.
Tests added (test_score.py):
- composite_score(gtm=None, kld=99.47) returns ~99.47 (not 99.74)
- composite_score(gtm=62.40, kld=None) returns ~62.40
- composite_score(None, None) returns 0.0 FAIL (no crash)
- 3-axis skip case computes correctly
- interpret_pattern handles None
- text_report renders "n/a" + "harmonic mean of 1 axes" on skip-gtm
Refract test count: 410 -> 416.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
6 new tests in test_command_construction.py:
- _llama_extra_flags() returns [] when env unset/empty/whitespace
- _llama_extra_flags() shlex-splits multi-word values + handles quoted args
- run_completion appends user flags AFTER REFRACT defaults so last-wins
on -ngl works (regression bound: tests both -ngl indices)
- run_perplexity_kld_base appends extra flags
- run_perplexity_kld appends extra flags
- run_completion_trajectory (the trajectory axis hot path AJ would hit)
appends extra flags
Refract test count: 404 -> 410.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
User report: AJ hit CUDA OOM running Qwen3.6-35B-A3B on a 12 GB 3060
because REFRACT hardcodes -ngl 99 with no escape hatch for the MoE
offload flag (-ncmoe 32) the user needs to fit the model.
Add REFRACT_LLAMA_EXTRA_FLAGS env var. Appended (shlex-split) to every
llama-cli, llama-completion, and llama-perplexity subprocess invocation
in refract.runner. llama.cpp uses last-wins on repeated flags, so:
REFRACT_LLAMA_EXTRA_FLAGS="-ngl 28 -ncmoe 32"
overrides REFRACT's default -ngl 99 and adds the MoE expert offload
needed for the 12 GB GPU + 35B-A3B configuration.
Docs: QUICKSTART has a "Constrained VRAM?" section before Step 2;
PITFALLS has a new entry crediting AJ's report; CHANGELOG v0.3.2.1
mentions the env var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
The 4 vLLM "skeleton" tests in test_backends.py asserted that
VLLMBackend methods raise NotImplementedError; they break in v0.3.2.1
where the backend is a working implementation. Replace with:
vLLM (5 tests):
- _kv_str_to_vllm_dtype known mappings (f16 -> auto, q8_0 -> fp8_e4m3)
- TurboQuant preset round-trips (turbo3, turbo4, q8_0/turbo4)
- BackendCapabilityError on unmappable ctk/ctv
- run_completion signature has the kw-only contract args
- all 5 backend methods are present and callable
SGLang (5 tests, new):
- get_backend("sglang") returns SGLangBackend
- _validate_kv_str accepts f16/bf16/q8_0
- TurboQuant rejected loudly (no SGLang TurboQuant path)
- run_kld without REFRACT_SGLANG_REF/CAND_URL raises with actionable msg
- all 5 backend methods are present and callable
Also: MLX backend uses mlx_lm.generate.maybe_quantize_kv_cache when
mlx_lm.models.cache no longer exposes it (0.31+); fall back via
getattr(cache_mod, ...) so tests that mock the old surface still work.
Refract test count: 399 -> 404 (+10 new, -4 stale, 1 mlx fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Add a Step 0 that walks through:
- git clone TheTom/turboquant_plus + cd + pip install -e .[backend-extras]
- optional sibling clones for the patched llama.cpp fork (with cmake
build steps and LLAMA_CPP_BIN_DIR export) and the vLLM fork
- pointer to the SGLang Docker image used in the cross-engine bench
Renumber the existing Steps 0..4 -> 2..6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Documents the 3-engine apples-to-apples cross-engine benchmark:
- BF16 baseline (load, PPL, prefill, decode, KV size) across vLLM, SGLang, llama.cpp
- REFRACT --full 4-axis fidelity scoring (gtm, kld, rniah, plad) on each engine's native 8-bit KV
- Headline finding: same fp8_e4m3 dtype string produces 1.8x different KL drift between vLLM and SGLang on the same model
- 8 engine-side bugs encountered and worked around (vLLM max_num_seqs, prompt_logprobs cap, GPU mem carryover; SGLang aiter.dtypes, AITER fp8 prefill broken, KV dtype fixed at server launch)
- Full reproducibility scripts and REFRACT framework changes documented
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
PLAD axis previously interleaved ref/cand per (prompt, perturbation) cell.
On vLLM with hybrid Qwen3.6 (~120GB per LLM, two won't fit 192GB MI300X),
each ref/cand swap triggers eviction-and-reload of ~30-90s. With 30 prompts
× 4 cells (anchor + 3 perturbations) = 240 swaps, the axis runs ~3 hours
just for eviction overhead.
Now plans all cells upfront, runs all ref-side completions, then all
cand-side completions, then computes drift. Two LLM loads total.
No effect on backends that can hold both configs in cache cheaply
(llama.cpp, MLX).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
R-NIAH axis calls runner.tokenize_to_ids to estimate chars-per-token. The
runner unconditionally shelled out to llama-tokenize binary, which fails
when the host's llama.cpp checkout has drifted (Symbol not found:
_llama_memory_breakdown_print) or when running on droplets without llama.cpp
on PATH at all.
Now dispatches to _ACTIVE_BACKEND.tokenize_to_ids when the backend is not
llamacpp. MLX/vLLM/SGLang backends already implement this method.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Memory-pressured backends (vLLM on hybrid Qwen3.6-35B-A3B can't fit two
simultaneous LLM instances on 192GB) need each KV config loaded exactly
once per axis. The trajectory axis previously interleaved ref/cand calls
per prompt, which forces 2N model loads (catastrophic when each load
takes 40 min on hybrid models).
Now collect all ref trajectories, then all cand trajectories, then
diff. Two model loads total instead of 2N. No effect on llama.cpp/MLX
which keep configs cached cheaply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
vLLM backend: hybrid Qwen3.6-35B-A3B doesn't fit two simultaneous LLM
instances (~120GB each on 192GB MI300X). Cache evicts prior LLM
before loading new one. Adds max_num_seqs env knob (default 32) so
Mamba cache block count doesn't underflow vs default max_num_seqs=1024.
Default gpu_memory_utilization bumped to 0.85 since serial loading
needs the headroom.
MLX backend: mlx_lm 0.31.x moved maybe_quantize_kv_cache from
mlx_lm.models.cache to mlx_lm.generate. Import from new location.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Replaces the v0.3.x skeletons with working implementations.
vLLM (refract/backends/vllm.py):
- Caches LLM instances per (model, kv_dtype, max_len) across calls
so back-to-back ref+candidate KLD doesn't pay 2x weight load.
- kv_config_str translation table: f16/q8_0/turbo3/turbo4 → vLLM
kv_cache_dtype (auto, fp8_e4m3, turboquant_*).
- run_kld via SamplingParams(prompt_logprobs=K), top-K capped at 64.
- chat template via tokenizer.apply_chat_template.
SGLang (refract/backends/sglang.py):
- HTTP-based; server is launched separately, backend assumes it is
running at REFRACT_SGLANG_URL.
- run_kld requires two simultaneous servers (REFRACT_SGLANG_REF_URL
and REFRACT_SGLANG_CAND_URL) since SGLang's KV dtype is set at
server-launch time, not per-request.
- Rejects TurboQuant configs with a clear capability error (SGLang
upstream has no TurboQuant KV path).
Signed-off-by: TheTom <tturney1@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
REFRACT itself doesn't import numpy or scipy — those were turboquant's
deps but were declared as base deps in pyproject.toml, so a friend who
just wanted to run `refract score` had to pull ~100MB of unused
libraries. Move them to a `[turboquant]` extra. Base install is now
stdlib-only.
pyproject cleanups:
- dependencies = [] (REFRACT is pure stdlib)
- new [turboquant] extra for numpy/scipy
- testpaths fixed: tests + refract/tests (was just "tests")
- added classifiers, license, [project.urls], readme, keywords
- coverage>=7.0 added to dev extras
README:
- new Install section showing the extras matrix
- Platform support table covers all 3 backends (llamacpp / mlx / vllm)
per OS, not just llamacpp
- Windows status flagged as TBD with a pointer to selftest's
Windows-specific remediation hints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mlx.py was 36% — every method needed real mlx-lm + a model. Mock the
mlx + mlx_lm imports (chat template, stream_generate token IDs, native
KLD softmax math, model cache) and exercise the dispatch glue without
the actual ML runtime.
- 16 new tests, mlx.py jumps to 96%
- Total project coverage: 96% (was 92%), 398 passing tests
- Pins MX-array-vs-int token coercion and the corpus-too-short /
zero-chunks guards on the KLD path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When llama-completion fails to launch (Linux missing libllama.so on the
loader path, or Windows missing the DLL next to the .exe), the binary
emits empty stdout and a "cannot open shared object" / 0xc0000135
stderr. The old selftest read empty stdout and reported "--jinja
missing" — misleading diagnosis that sent users hunting for chat
template support when the real problem was the loader.
Now the selftest checks return code + stderr for shared-library / DLL
errors and emits OS-specific remediation (LD_LIBRARY_PATH/ldconfig on
Linux, PATH-next-to-exe on Windows). Three new tests pin both failure
modes plus the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTML output existed since v0.3.2 but only as a passing flag mention
in command examples. Friend doing first-time read wouldn't know:
- It's a single self-contained file (works offline modulo font CDN)
- What's in it (composite, diagnosis, R-NIAH heatmap, etc.)
- Browser compatibility (light-dark() needs 2024+ browsers; older
fall back to light theme cleanly)
- The sun/moon toggle exists
Now in:
- README "Reports" section right under Subcommands, mentioning
JSON + HTML as parallel outputs and what's bundled vs external
- QUICKSTART "Generating the HTML report" subsection in Step 1+2,
with full command, content list, and what works offline vs online
- Subcommand list in README adds `refract fetch` (was missing)
No code changes. Tests still 82/82.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit when figuring out R-NIAH lengths: had to
manually compute the doubling list, pass --rniah-ctx-max separately,
and remember they're related. Replaced with a single primary knob.
What changed
------------
- --rniah-up-to N (new, default 16384) — auto-generates the lengths
list as a doubling step-up from 4K up to N. So
--rniah-up-to 32768 → [4096, 8192, 16384, 32768]
--rniah-up-to 131072 → [4096, ..., 131072]
Cost scales roughly linearly with max length, called out in
QUICKSTART:
16K (default) ~10–15 min on 7B Q8
32K ~25–35 min
64K ~60–90 min
128K ~3+ hours
- --rniah-ctx-max is now optional; defaults to --rniah-up-to. Kept
as a power-user override for when the model's nominal max ctx
exceeds what it can actually do (R-NIAH should skip those cells).
- --rniah-lengths still works for explicit lists, overrides
--rniah-up-to. Documented as power-user.
- --rniah-haystack is now optional; auto-resolved from the
~/.cache/refract/ wikitext-2-raw cache (downloaded by `refract
fetch` or auto-fetched on first score run).
QUICKSTART now shows just `--rniah-up-to 16384` (or 32768/65536/131072)
instead of three coordinated R-NIAH flags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit on the M2 mini fresh-install audit: corpus and
haystack files have to be downloaded separately, and figuring out where
to get wikitext-2-raw is its own subquest. v0.3.2 fixes this.
What changed
------------
- New `refract fetch` subcommand: downloads wikitext-2-raw zip from
the ggml-org HF mirror (the same source llama.cpp's
scripts/get-wikitext-2.sh uses) into ~/.cache/refract/. ~10MB,
~30s on a typical home connection. No third-party deps; stdlib
urllib + zipfile.
- `--corpus` is now optional in `refract score`. When omitted, the
cache is auto-resolved (and auto-fetched if empty) before the
score axes run. Same for `--rniah-haystack` when `--full` is
passed.
- `--no-auto-fetch` flag for CI-friendly invocations that want a
hard error rather than a silent network fetch.
- QUICKSTART updated to call out the auto-fetch and drop the
"fetch wikitext-2 yourself from llama.cpp's repo" step.
Cost trade-off
--------------
- First-run friction: was "go find wikitext-2-raw, download, unzip,
pass --corpus". Now zero — just pass --model + --candidate.
- Network IO: silent on first run, idempotent thereafter. Off-mode
is a flag away.
- Cache directory is ~/.cache/refract/ (XDG-conformant on Linux;
OK on macOS).
Tests: 82/82 still pass. `refract fetch` smoke-tested end-to-end on
the dev box (download + extract + idempotent re-run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TheTom