Three findings from the Mac mini PyPI install walkthrough (2026-05-03):
1. `--prompts` was required even though wheel ships them — defaulted to
bundled refract/prompts/v0.1.jsonl via importlib.resources (v0.3.2.2)
2. macOS system python3 is 3.9, mlx-lm needs >=3.10 — documented in
QUICKSTART Step 0 with brew install hint
3. MLX run_kld returned mean_kld=NaN on Qwen2.5-1.5B-Instruct-8bit +
q8_0/q8_0 KV. Root cause: log(softmax + 1e-12) underflows on
sharply-peaked logits from already-8bit weight models. Rewritten
in terms of log-softmax (logits - logsumexp) which is stable.
Non-finite per-position KL values now filtered before aggregation;
mean computed across valid positions only. If every position is
non-finite the backend raises BackendCapabilityError with an
actionable hint instead of silently returning NaN.
Test mock updated for the new mx symbols (logsumexp, exp, isfinite,
where, zeros_like). 416 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Mac mini install walkthrough (2026-05-03) found that `refract score`
required explicit --prompts even though the wheel ships the v0.1
prompts file. After `pip install refract-llm` a user shouldn't need
to know where the bundled file lives.
Now: when --prompts is omitted, _run_score resolves the bundled
refract/prompts/v0.1.jsonl via importlib.resources. Minimum viable
score invocation drops to:
refract score --backend mlx --model /path/to/model --candidate ctk=q8_0,ctv=q8_0
README install section flipped to lead with `pip install refract-llm`
(PyPI path) and demote the source-clone path to "for hacking".
Walkthrough also surfaced two non-blocking findings (kept as separate
follow-ups):
- macOS system python3 is 3.9, mlx-lm needs 3.10+; users hit this
- KLD axis returned NaN on Qwen2.5-1.5B-Instruct-8bit MLX — separate
numerical-stability bug, not a packaging issue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Distribution name: turboquant -> refract-llm.
Top-level package shipped: refract* only (turboquant/ stays in source
tree for dev work but is no longer in the wheel).
Bundled via package-data:
- refract/prompts/v0.1.jsonl (the 30-prompt CC0 set)
- refract/examples/*.json + *.html (4 reference reports)
- refract/llama-cpp-patches/refract-trajectory.patch (for users with
their own llama.cpp build to apply themselves)
readme = refract/README.md (PyPI long_description focused on REFRACT).
Adds a [full] extra that pulls all backend deps in one shot. Adds dev
extras for build + twine.
Refract __version__ bumped 0.3.2 -> 0.3.2.1 to match the wheel.
Local smoke: pip install dist/refract_llm-0.3.2.1-py3-none-any.whl in
a fresh venv, `refract --help` and `refract selftest` work, bundled
data accessible via importlib.resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Audit follow-up to the earlier skip-axis fix (728cd84). The stub axis
result dataclasses still carry score=100, so without these changes:
- json_report still wrote 'axes.gtm.score: 100.0' for a skipped axis,
visible to anyone parsing the JSON or running 'refract compare'
- 'refract compare' showed 100.0 in the Traj/KLD column for skipped
reports, which made cross-report comparison falsely flatter the
skipped run
Now: when CompositeScore.gtm_score is None we write axes.gtm.score = null
in the JSON. 'refract compare' reads 'skipped' or score=None and prints
'skip' in that column instead of the stub 100.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Two reader-facing fixes from AJ's screenshot (2026-05-02 X thread):
1. --skip-gtm / --skip-kld previously fed a stub score=100 to
composite_score, so a single real KLD axis of 99.47 produced a
misleading composite of 99.74 EXCELLENT. The skipped axis now
contributes None to composite_score() and is excluded from the
harmonic mean. CompositeScore.gtm_score / .kld_score are now
Optional[float]. Text + HTML reports render "n/a" with a
"skipped" badge for skipped axes instead of fake 100/EXCELLENT.
The "harmonic mean of N axes" gloss reflects the real count.
Diagnostics blocks (GTM diagnostics, KLD diagnostics) are
replaced with a single "axis was skipped" line when the axis
wasn't measured.
2. The HTML "Run details → Environment" card now surfaces the
backend name (llama.cpp / MLX / vLLM / SGLang) plus engine
version fields (llama_cpp_commit, mlx_lm_version, vllm_version,
sglang_url, served_model_id) when the active backend populates
them. Previously a report didn't say which engine ran it, which
made cross-engine comparison reports ambiguous in the inbox.
Tests added (test_score.py):
- composite_score(gtm=None, kld=99.47) returns ~99.47 (not 99.74)
- composite_score(gtm=62.40, kld=None) returns ~62.40
- composite_score(None, None) returns 0.0 FAIL (no crash)
- 3-axis skip case computes correctly
- interpret_pattern handles None
- text_report renders "n/a" + "harmonic mean of 1 axes" on skip-gtm
Refract test count: 410 -> 416.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
6 new tests in test_command_construction.py:
- _llama_extra_flags() returns [] when env unset/empty/whitespace
- _llama_extra_flags() shlex-splits multi-word values + handles quoted args
- run_completion appends user flags AFTER REFRACT defaults so last-wins
on -ngl works (regression bound: tests both -ngl indices)
- run_perplexity_kld_base appends extra flags
- run_perplexity_kld appends extra flags
- run_completion_trajectory (the trajectory axis hot path AJ would hit)
appends extra flags
Refract test count: 404 -> 410.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
User report: AJ hit CUDA OOM running Qwen3.6-35B-A3B on a 12 GB 3060
because REFRACT hardcodes -ngl 99 with no escape hatch for the MoE
offload flag (-ncmoe 32) the user needs to fit the model.
Add REFRACT_LLAMA_EXTRA_FLAGS env var. Appended (shlex-split) to every
llama-cli, llama-completion, and llama-perplexity subprocess invocation
in refract.runner. llama.cpp uses last-wins on repeated flags, so:
REFRACT_LLAMA_EXTRA_FLAGS="-ngl 28 -ncmoe 32"
overrides REFRACT's default -ngl 99 and adds the MoE expert offload
needed for the 12 GB GPU + 35B-A3B configuration.
Docs: QUICKSTART has a "Constrained VRAM?" section before Step 2;
PITFALLS has a new entry crediting AJ's report; CHANGELOG v0.3.2.1
mentions the env var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
The 4 vLLM "skeleton" tests in test_backends.py asserted that
VLLMBackend methods raise NotImplementedError; they break in v0.3.2.1
where the backend is a working implementation. Replace with:
vLLM (5 tests):
- _kv_str_to_vllm_dtype known mappings (f16 -> auto, q8_0 -> fp8_e4m3)
- TurboQuant preset round-trips (turbo3, turbo4, q8_0/turbo4)
- BackendCapabilityError on unmappable ctk/ctv
- run_completion signature has the kw-only contract args
- all 5 backend methods are present and callable
SGLang (5 tests, new):
- get_backend("sglang") returns SGLangBackend
- _validate_kv_str accepts f16/bf16/q8_0
- TurboQuant rejected loudly (no SGLang TurboQuant path)
- run_kld without REFRACT_SGLANG_REF/CAND_URL raises with actionable msg
- all 5 backend methods are present and callable
Also: MLX backend uses mlx_lm.generate.maybe_quantize_kv_cache when
mlx_lm.models.cache no longer exposes it (0.31+); fall back via
getattr(cache_mod, ...) so tests that mock the old surface still work.
Refract test count: 399 -> 404 (+10 new, -4 stale, 1 mlx fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Add a Step 0 that walks through:
- git clone TheTom/turboquant_plus + cd + pip install -e .[backend-extras]
- optional sibling clones for the patched llama.cpp fork (with cmake
build steps and LLAMA_CPP_BIN_DIR export) and the vLLM fork
- pointer to the SGLang Docker image used in the cross-engine bench
Renumber the existing Steps 0..4 -> 2..6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
PLAD axis previously interleaved ref/cand per (prompt, perturbation) cell.
On vLLM with hybrid Qwen3.6 (~120GB per LLM, two won't fit 192GB MI300X),
each ref/cand swap triggers eviction-and-reload of ~30-90s. With 30 prompts
× 4 cells (anchor + 3 perturbations) = 240 swaps, the axis runs ~3 hours
just for eviction overhead.
Now plans all cells upfront, runs all ref-side completions, then all
cand-side completions, then computes drift. Two LLM loads total.
No effect on backends that can hold both configs in cache cheaply
(llama.cpp, MLX).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
R-NIAH axis calls runner.tokenize_to_ids to estimate chars-per-token. The
runner unconditionally shelled out to llama-tokenize binary, which fails
when the host's llama.cpp checkout has drifted (Symbol not found:
_llama_memory_breakdown_print) or when running on droplets without llama.cpp
on PATH at all.
Now dispatches to _ACTIVE_BACKEND.tokenize_to_ids when the backend is not
llamacpp. MLX/vLLM/SGLang backends already implement this method.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Memory-pressured backends (vLLM on hybrid Qwen3.6-35B-A3B can't fit two
simultaneous LLM instances on 192GB) need each KV config loaded exactly
once per axis. The trajectory axis previously interleaved ref/cand calls
per prompt, which forces 2N model loads (catastrophic when each load
takes 40 min on hybrid models).
Now collect all ref trajectories, then all cand trajectories, then
diff. Two model loads total instead of 2N. No effect on llama.cpp/MLX
which keep configs cached cheaply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
vLLM backend: hybrid Qwen3.6-35B-A3B doesn't fit two simultaneous LLM
instances (~120GB each on 192GB MI300X). Cache evicts prior LLM
before loading new one. Adds max_num_seqs env knob (default 32) so
Mamba cache block count doesn't underflow vs default max_num_seqs=1024.
Default gpu_memory_utilization bumped to 0.85 since serial loading
needs the headroom.
MLX backend: mlx_lm 0.31.x moved maybe_quantize_kv_cache from
mlx_lm.models.cache to mlx_lm.generate. Import from new location.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Replaces the v0.3.x skeletons with working implementations.
vLLM (refract/backends/vllm.py):
- Caches LLM instances per (model, kv_dtype, max_len) across calls
so back-to-back ref+candidate KLD doesn't pay 2x weight load.
- kv_config_str translation table: f16/q8_0/turbo3/turbo4 → vLLM
kv_cache_dtype (auto, fp8_e4m3, turboquant_*).
- run_kld via SamplingParams(prompt_logprobs=K), top-K capped at 64.
- chat template via tokenizer.apply_chat_template.
SGLang (refract/backends/sglang.py):
- HTTP-based; server is launched separately, backend assumes it is
running at REFRACT_SGLANG_URL.
- run_kld requires two simultaneous servers (REFRACT_SGLANG_REF_URL
and REFRACT_SGLANG_CAND_URL) since SGLang's KV dtype is set at
server-launch time, not per-request.
- Rejects TurboQuant configs with a clear capability error (SGLang
upstream has no TurboQuant KV path).
Signed-off-by: TheTom <tturney1@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
REFRACT itself doesn't import numpy or scipy — those were turboquant's
deps but were declared as base deps in pyproject.toml, so a friend who
just wanted to run `refract score` had to pull ~100MB of unused
libraries. Move them to a `[turboquant]` extra. Base install is now
stdlib-only.
pyproject cleanups:
- dependencies = [] (REFRACT is pure stdlib)
- new [turboquant] extra for numpy/scipy
- testpaths fixed: tests + refract/tests (was just "tests")
- added classifiers, license, [project.urls], readme, keywords
- coverage>=7.0 added to dev extras
README:
- new Install section showing the extras matrix
- Platform support table covers all 3 backends (llamacpp / mlx / vllm)
per OS, not just llamacpp
- Windows status flagged as TBD with a pointer to selftest's
Windows-specific remediation hints
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mlx.py was 36% — every method needed real mlx-lm + a model. Mock the
mlx + mlx_lm imports (chat template, stream_generate token IDs, native
KLD softmax math, model cache) and exercise the dispatch glue without
the actual ML runtime.
- 16 new tests, mlx.py jumps to 96%
- Total project coverage: 96% (was 92%), 398 passing tests
- Pins MX-array-vs-int token coercion and the corpus-too-short /
zero-chunks guards on the KLD path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When llama-completion fails to launch (Linux missing libllama.so on the
loader path, or Windows missing the DLL next to the .exe), the binary
emits empty stdout and a "cannot open shared object" / 0xc0000135
stderr. The old selftest read empty stdout and reported "--jinja
missing" — misleading diagnosis that sent users hunting for chat
template support when the real problem was the loader.
Now the selftest checks return code + stderr for shared-library / DLL
errors and emits OS-specific remediation (LD_LIBRARY_PATH/ldconfig on
Linux, PATH-next-to-exe on Windows). Three new tests pin both failure
modes plus the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HTML output existed since v0.3.2 but only as a passing flag mention
in command examples. Friend doing first-time read wouldn't know:
- It's a single self-contained file (works offline modulo font CDN)
- What's in it (composite, diagnosis, R-NIAH heatmap, etc.)
- Browser compatibility (light-dark() needs 2024+ browsers; older
fall back to light theme cleanly)
- The sun/moon toggle exists
Now in:
- README "Reports" section right under Subcommands, mentioning
JSON + HTML as parallel outputs and what's bundled vs external
- QUICKSTART "Generating the HTML report" subsection in Step 1+2,
with full command, content list, and what works offline vs online
- Subcommand list in README adds `refract fetch` (was missing)
No code changes. Tests still 82/82.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit when figuring out R-NIAH lengths: had to
manually compute the doubling list, pass --rniah-ctx-max separately,
and remember they're related. Replaced with a single primary knob.
What changed
------------
- --rniah-up-to N (new, default 16384) — auto-generates the lengths
list as a doubling step-up from 4K up to N. So
--rniah-up-to 32768 → [4096, 8192, 16384, 32768]
--rniah-up-to 131072 → [4096, ..., 131072]
Cost scales roughly linearly with max length, called out in
QUICKSTART:
16K (default) ~10–15 min on 7B Q8
32K ~25–35 min
64K ~60–90 min
128K ~3+ hours
- --rniah-ctx-max is now optional; defaults to --rniah-up-to. Kept
as a power-user override for when the model's nominal max ctx
exceeds what it can actually do (R-NIAH should skip those cells).
- --rniah-lengths still works for explicit lists, overrides
--rniah-up-to. Documented as power-user.
- --rniah-haystack is now optional; auto-resolved from the
~/.cache/refract/ wikitext-2-raw cache (downloaded by `refract
fetch` or auto-fetched on first score run).
QUICKSTART now shows just `--rniah-up-to 16384` (or 32768/65536/131072)
instead of three coordinated R-NIAH flags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit on the M2 mini fresh-install audit: corpus and
haystack files have to be downloaded separately, and figuring out where
to get wikitext-2-raw is its own subquest. v0.3.2 fixes this.
What changed
------------
- New `refract fetch` subcommand: downloads wikitext-2-raw zip from
the ggml-org HF mirror (the same source llama.cpp's
scripts/get-wikitext-2.sh uses) into ~/.cache/refract/. ~10MB,
~30s on a typical home connection. No third-party deps; stdlib
urllib + zipfile.
- `--corpus` is now optional in `refract score`. When omitted, the
cache is auto-resolved (and auto-fetched if empty) before the
score axes run. Same for `--rniah-haystack` when `--full` is
passed.
- `--no-auto-fetch` flag for CI-friendly invocations that want a
hard error rather than a silent network fetch.
- QUICKSTART updated to call out the auto-fetch and drop the
"fetch wikitext-2 yourself from llama.cpp's repo" step.
Cost trade-off
--------------
- First-run friction: was "go find wikitext-2-raw, download, unzip,
pass --corpus". Now zero — just pass --model + --candidate.
- Network IO: silent on first run, idempotent thereafter. Off-mode
is a flag away.
- Cache directory is ~/.cache/refract/ (XDG-conformant on Linux;
OK on macOS).
Tests: 82/82 still pass. `refract fetch` smoke-tested end-to-end on
the dev box (download + extract + idempotent re-run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues from the in-browser audit:
- Axis title showed "Axis A — GTM" with no spelling-out. Now reads
"Axis A — GTM (Greedy Trajectory Match)" etc, so first-time
readers don't have to scroll for the meaning.
- Reproduce block leaked /tmp/render_example_html.py (the path of
whatever script re-rendered the JSON) instead of the original
`refract score ...` invocation. Now:
1. score-time captures sanitized argv into JSON's repro_command,
BUT only when argv looks like a refract CLI run (gated on
"refract" in argv) so regen scripts don't pollute the field
2. html_report uses raw_json.repro_command if present, else
falls back to sys.argv (sanitized), else synthesizes a clean
"python3 -m refract.cli score --model X --candidate Y ..."
from the report fields
3. /Users/<name>/ paths are scrubbed to ~/ so shared HTML
doesn't leak personal home dirs
- Added a footer link to the GitHub README so a layman opening a
shared HTML can find the framework explanation:
"What is this? See github.com/TheTom/turboquant_plus/refract for
documentation, the motivation paper, and how to interpret these
scores."
Tests: 82/82 still pass. Examples regenerated with clean synthesized
repro commands and the spelled-out axis labels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues found by running the freshly-pushed v0.3.2 through a clean clone
on toms-mac-mini.local:
- Version banner showed v0.3.1; refract/__init__.py was missing the
bump. Now reports v0.3.2 correctly.
- pyproject.toml `requires-python` was >=3.10 but mini ships 3.9.6;
REFRACT works fine on 3.9 thanks to `from __future__ import
annotations`. Lowered the floor.
- pyproject.toml lacked optional deps for the MLX and vLLM backends.
Added `refract-mlx` and `refract-vllm` extras with the right
minimum versions (mlx-lm >= 0.31 since 0.29 doesn't recognise the
qwen3_5 model type).
- pyproject.toml had no `[project.scripts]` entry point. Added
`refract = "refract.cli:main"` so installs get a proper command
instead of `python3 -m refract.cli` only.
- selftest's thinking-mode probe printed "✓ no markers detected"
even when generation failed. Gated the probe on gen_ok so the
happy-path output only appears when the model actually generated.
- selftest description leaned on the "30 minutes of a real run"
framing, scaring readers who don't realize default mode is
5-7 min. Reworded.
- QUICKSTART now leads with a runtime expectations table:
selftest: ~1s static, ~30s with --model
score (default): ~5-7 min (Trajectory + KLD)
score --full: ~25-30 min (+ R-NIAH + PLAD)
repeatability: 4× default
Default is quick. Most users should never wait 30 minutes.
- QUICKSTART preflight example now shows BOTH .gguf and MLX
directory paths so MLX users see they're supported.
- HTML report hardware card showed "137.4 GB" RAM on a 128 GB Mac
(used /1e9 SI gigabytes). Switched to /1024**3 binary GiB so
output matches Apple's marketing convention (128 GB exact).
Also fixed model file size to use binary GiB.
- Regenerated examples/*.html with the GiB fix and live hardware
metadata.
Tests: 82/82 pass. No semantic changes to scoring, only metadata
display + setup ergonomics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.
Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
(test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
corpus path/size/SHA; refuses to score against a base built from a
different corpus.
New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
--no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).
Total: 14 → 44 unit tests + 1 strengthened integration test.
Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
(detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>
Two bugs surfaced by the v0.1.1 matrix run:
1. Mistral-Small-24B failed in the KLD subprocess with UnicodeDecodeError
on llama-perplexity stderr. text=True does strict utf-8; some llama-cpp
stderr bytes aren't valid utf-8.
2. GTM mean_prefix_agreement_length was in whitespace tokens while
n_predict was in model tokens. Whitespace tokenization can over-count
(gemma-4 31B v0.1.1 produced mean_prefix=59 > n_predict=48 → score
clipped to 100 even though KLD was 49).
Patches:
- runner.py: errors="replace" on all 3 subprocess.run(text=True) calls
- runner.py: new tokenize_to_ids(model, text) helper that shells to
llama-tokenize --ids --no-bos --no-parse-special --log-disable --stdin
- axes/gtm.py: use tokenize_to_ids in the diff path; whitespace _tokenize_words
remains as a fallback if the tokenizer call fails
CHANGELOG fixed to reverse-chronological order with v0.1.2 → v0.1.1 → v0.1.
All 14 unit tests still pass. Smoke test on Qwen3.5-2B Q8 q8/turbo4 went
from GTM=54 (whitespace, 17.4/32) to GTM=94 (model tokens, 30/32).
v0.1's first 7-model matrix run revealed:
1. composite serialised as nested dict, not scalar (aggregator print bug)
2. GTM was comparing llama-cli help banners, not generations (the fork
rejects --no-conversation; runner captured help text)
3. backspace control chars in stdout broke the gen-line regex even after
the --no-conversation fix, so generations were dropped
4. matrix per-model timeout was 30 min; Mistral-Small-24B's KLD axis ran
over and the run failed mid-way
Patches in this commit:
- runner.py: drop --no-conversation, strip \\x08 first, add Loading/prompt-
echo to noise patterns, add gen-line extractor + ASCII-art block stripper
- axes/gtm.py: switch GTM score from binary full-match-rate to continuous
mean-prefix-agreement / n_predict (still report full_match_rate as
diagnostic)
- report.py: flatten composite scalar to JSON top level, full breakdown
moves to composite_detail, schema bumped to refract.report.v0.1.1
- /tmp/refract-matrix.sh (test harness, not committed): per-model timeout
bumped from 30 min to 2 hr
KLD@D axis was always correct in v0.1 — it reproduced the paper §4.3
findings exactly (gemma-4 26B-A4B at KLD 1.74, Qwen2.5-7B at 0.013, etc.).
Only GTM and the JSON layout were broken.
Adds refract/CHANGELOG.md documenting all of this with the v0.1 matrix
results table for reference. v0.1.1 matrix results to be appended when
re-run completes.
All 14 unit tests still pass.
REFRACT is meant to be a general KV-quant quality oracle, applicable
across model families (gemma, qwen, mistral, phi, llama, etc.) and not
TurboQuant-specific. Top-level placement reflects that scope and gives
a cleaner CLI: `python3 -m refract.cli` instead of nested under turboquant.
Updates:
- git mv turboquant/refract/ -> refract/
- All `from turboquant.refract` -> `from refract` imports rewritten
- pyproject.toml: include both turboquant* and refract* in setuptools
- All "turboquant/refract/" path references in README/tests updated
Verified: 14/14 unit tests pass, CLI module path works.
