Mac mini install walkthrough (2026-05-03) found that `refract score`
required explicit --prompts even though the wheel ships the v0.1
prompts file. After `pip install refract-llm` a user shouldn't need
to know where the bundled file lives.
Now: when --prompts is omitted, _run_score resolves the bundled
refract/prompts/v0.1.jsonl via importlib.resources. Minimum viable
score invocation drops to:
refract score --backend mlx --model /path/to/model --candidate ctk=q8_0,ctv=q8_0
README install section flipped to lead with `pip install refract-llm`
(PyPI path) and demote the source-clone path to "for hacking".
Walkthrough also surfaced two non-blocking findings (kept as separate
follow-ups):
- macOS system python3 is 3.9, mlx-lm needs 3.10+; users hit this
- KLD axis returned NaN on Qwen2.5-1.5B-Instruct-8bit MLX — separate
numerical-stability bug, not a packaging issue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Audit follow-up to the earlier skip-axis fix (728cd84). The stub axis
result dataclasses still carry score=100, so without these changes:
- json_report still wrote 'axes.gtm.score: 100.0' for a skipped axis,
visible to anyone parsing the JSON or running 'refract compare'
- 'refract compare' showed 100.0 in the Traj/KLD column for skipped
reports, which made cross-report comparison falsely flatter the
skipped run
Now: when CompositeScore.gtm_score is None we write axes.gtm.score = null
in the JSON. 'refract compare' reads 'skipped' or score=None and prints
'skip' in that column instead of the stub 100.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Two reader-facing fixes from AJ's screenshot (2026-05-02 X thread):
1. --skip-gtm / --skip-kld previously fed a stub score=100 to
composite_score, so a single real KLD axis of 99.47 produced a
misleading composite of 99.74 EXCELLENT. The skipped axis now
contributes None to composite_score() and is excluded from the
harmonic mean. CompositeScore.gtm_score / .kld_score are now
Optional[float]. Text + HTML reports render "n/a" with a
"skipped" badge for skipped axes instead of fake 100/EXCELLENT.
The "harmonic mean of N axes" gloss reflects the real count.
Diagnostics blocks (GTM diagnostics, KLD diagnostics) are
replaced with a single "axis was skipped" line when the axis
wasn't measured.
2. The HTML "Run details → Environment" card now surfaces the
backend name (llama.cpp / MLX / vLLM / SGLang) plus engine
version fields (llama_cpp_commit, mlx_lm_version, vllm_version,
sglang_url, served_model_id) when the active backend populates
them. Previously a report didn't say which engine ran it, which
made cross-engine comparison reports ambiguous in the inbox.
Tests added (test_score.py):
- composite_score(gtm=None, kld=99.47) returns ~99.47 (not 99.74)
- composite_score(gtm=62.40, kld=None) returns ~62.40
- composite_score(None, None) returns 0.0 FAIL (no crash)
- 3-axis skip case computes correctly
- interpret_pattern handles None
- text_report renders "n/a" + "harmonic mean of 1 axes" on skip-gtm
Refract test count: 410 -> 416.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
When llama-completion fails to launch (Linux missing libllama.so on the
loader path, or Windows missing the DLL next to the .exe), the binary
emits empty stdout and a "cannot open shared object" / 0xc0000135
stderr. The old selftest read empty stdout and reported "--jinja
missing" — misleading diagnosis that sent users hunting for chat
template support when the real problem was the loader.
Now the selftest checks return code + stderr for shared-library / DLL
errors and emits OS-specific remediation (LD_LIBRARY_PATH/ldconfig on
Linux, PATH-next-to-exe on Windows). Three new tests pin both failure
modes plus the happy path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit when figuring out R-NIAH lengths: had to
manually compute the doubling list, pass --rniah-ctx-max separately,
and remember they're related. Replaced with a single primary knob.
What changed
------------
- --rniah-up-to N (new, default 16384) — auto-generates the lengths
list as a doubling step-up from 4K up to N. So
--rniah-up-to 32768 → [4096, 8192, 16384, 32768]
--rniah-up-to 131072 → [4096, ..., 131072]
Cost scales roughly linearly with max length, called out in
QUICKSTART:
16K (default) ~10–15 min on 7B Q8
32K ~25–35 min
64K ~60–90 min
128K ~3+ hours
- --rniah-ctx-max is now optional; defaults to --rniah-up-to. Kept
as a power-user override for when the model's nominal max ctx
exceeds what it can actually do (R-NIAH should skip those cells).
- --rniah-lengths still works for explicit lists, overrides
--rniah-up-to. Documented as power-user.
- --rniah-haystack is now optional; auto-resolved from the
~/.cache/refract/ wikitext-2-raw cache (downloaded by `refract
fetch` or auto-fetched on first score run).
QUICKSTART now shows just `--rniah-up-to 16384` (or 32768/65536/131072)
instead of three coordinated R-NIAH flags.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Friction the friend hit on the M2 mini fresh-install audit: corpus and
haystack files have to be downloaded separately, and figuring out where
to get wikitext-2-raw is its own subquest. v0.3.2 fixes this.
What changed
------------
- New `refract fetch` subcommand: downloads wikitext-2-raw zip from
the ggml-org HF mirror (the same source llama.cpp's
scripts/get-wikitext-2.sh uses) into ~/.cache/refract/. ~10MB,
~30s on a typical home connection. No third-party deps; stdlib
urllib + zipfile.
- `--corpus` is now optional in `refract score`. When omitted, the
cache is auto-resolved (and auto-fetched if empty) before the
score axes run. Same for `--rniah-haystack` when `--full` is
passed.
- `--no-auto-fetch` flag for CI-friendly invocations that want a
hard error rather than a silent network fetch.
- QUICKSTART updated to call out the auto-fetch and drop the
"fetch wikitext-2 yourself from llama.cpp's repo" step.
Cost trade-off
--------------
- First-run friction: was "go find wikitext-2-raw, download, unzip,
pass --corpus". Now zero — just pass --model + --candidate.
- Network IO: silent on first run, idempotent thereafter. Off-mode
is a flag away.
- Cache directory is ~/.cache/refract/ (XDG-conformant on Linux;
OK on macOS).
Tests: 82/82 still pass. `refract fetch` smoke-tested end-to-end on
the dev box (download + extract + idempotent re-run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues found by running the freshly-pushed v0.3.2 through a clean clone
on toms-mac-mini.local:
- Version banner showed v0.3.1; refract/__init__.py was missing the
bump. Now reports v0.3.2 correctly.
- pyproject.toml `requires-python` was >=3.10 but mini ships 3.9.6;
REFRACT works fine on 3.9 thanks to `from __future__ import
annotations`. Lowered the floor.
- pyproject.toml lacked optional deps for the MLX and vLLM backends.
Added `refract-mlx` and `refract-vllm` extras with the right
minimum versions (mlx-lm >= 0.31 since 0.29 doesn't recognise the
qwen3_5 model type).
- pyproject.toml had no `[project.scripts]` entry point. Added
`refract = "refract.cli:main"` so installs get a proper command
instead of `python3 -m refract.cli` only.
- selftest's thinking-mode probe printed "✓ no markers detected"
even when generation failed. Gated the probe on gen_ok so the
happy-path output only appears when the model actually generated.
- selftest description leaned on the "30 minutes of a real run"
framing, scaring readers who don't realize default mode is
5-7 min. Reworded.
- QUICKSTART now leads with a runtime expectations table:
selftest: ~1s static, ~30s with --model
score (default): ~5-7 min (Trajectory + KLD)
score --full: ~25-30 min (+ R-NIAH + PLAD)
repeatability: 4× default
Default is quick. Most users should never wait 30 minutes.
- QUICKSTART preflight example now shows BOTH .gguf and MLX
directory paths so MLX users see they're supported.
- HTML report hardware card showed "137.4 GB" RAM on a 128 GB Mac
(used /1e9 SI gigabytes). Switched to /1024**3 binary GiB so
output matches Apple's marketing convention (128 GB exact).
Also fixed model file size to use binary GiB.
- Regenerated examples/*.html with the GiB fix and live hardware
metadata.
Tests: 82/82 pass. No semantic changes to scoring, only metadata
display + setup ergonomics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.
Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
(test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
corpus path/size/SHA; refuses to score against a base built from a
different corpus.
New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
--no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).
Total: 14 → 44 unit tests + 1 strengthened integration test.
Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
(detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>
REFRACT is meant to be a general KV-quant quality oracle, applicable
across model families (gemma, qwen, mistral, phi, llama, etc.) and not
TurboQuant-specific. Top-level placement reflects that scope and gives
a cleaner CLI: `python3 -m refract.cli` instead of nested under turboquant.
Updates:
- git mv turboquant/refract/ -> refract/
- All `from turboquant.refract` -> `from refract` imports rewritten
- pyproject.toml: include both turboquant* and refract* in setuptools
- All "turboquant/refract/" path references in README/tests updated
Verified: 14/14 unit tests pass, CLI module path works.
