- 12 new test files; 379 passing tests; 92% line coverage (was 33%)
- Bugs fixed:
- cli compare: TypeError when report JSON missing composite/band keys
- text_report: NaN PLAD per-perturbation displayed as "nan FAIL" red
(now renders "skipped n/a" matching the HTML behaviour)
- Re-ran catastrophic-symturbo example with v0.3.2 defaults (neutral
needle wording, --jinja chat template). New numbers:
Trajectory 3.93 (was 9.74), KLD 11.84 unchanged, R-NIAH 100 (real
signal now), PLAD 72.21, composite 11 (was 19). Updated CHANGELOG,
MATRIX-RESULTS, LEADERBOARD, examples/README.
- Regen of 3 other example HTML reports picks up acronym expansion
in the footer ("REFRACT — REFerence-anchored ...").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues from the in-browser audit:
- Axis title showed "Axis A — GTM" with no spelling-out. Now reads
"Axis A — GTM (Greedy Trajectory Match)" etc, so first-time
readers don't have to scroll for the meaning.
- Reproduce block leaked /tmp/render_example_html.py (the path of
whatever script re-rendered the JSON) instead of the original
`refract score ...` invocation. Now:
1. score-time captures sanitized argv into JSON's repro_command,
BUT only when argv looks like a refract CLI run (gated on
"refract" in argv) so regen scripts don't pollute the field
2. html_report uses raw_json.repro_command if present, else
falls back to sys.argv (sanitized), else synthesizes a clean
"python3 -m refract.cli score --model X --candidate Y ..."
from the report fields
3. /Users/<name>/ paths are scrubbed to ~/ so shared HTML
doesn't leak personal home dirs
- Added a footer link to the GitHub README so a layman opening a
shared HTML can find the framework explanation:
"What is this? See github.com/TheTom/turboquant_plus/refract for
documentation, the motivation paper, and how to interpret these
scores."
Tests: 82/82 still pass. Examples regenerated with clean synthesized
repro commands and the spelled-out axis labels.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issues found by running the freshly-pushed v0.3.2 through a clean clone
on toms-mac-mini.local:
- Version banner showed v0.3.1; refract/__init__.py was missing the
bump. Now reports v0.3.2 correctly.
- pyproject.toml `requires-python` was >=3.10 but mini ships 3.9.6;
REFRACT works fine on 3.9 thanks to `from __future__ import
annotations`. Lowered the floor.
- pyproject.toml lacked optional deps for the MLX and vLLM backends.
Added `refract-mlx` and `refract-vllm` extras with the right
minimum versions (mlx-lm >= 0.31 since 0.29 doesn't recognise the
qwen3_5 model type).
- pyproject.toml had no `[project.scripts]` entry point. Added
`refract = "refract.cli:main"` so installs get a proper command
instead of `python3 -m refract.cli` only.
- selftest's thinking-mode probe printed "✓ no markers detected"
even when generation failed. Gated the probe on gen_ok so the
happy-path output only appears when the model actually generated.
- selftest description leaned on the "30 minutes of a real run"
framing, scaring readers who don't realize default mode is
5-7 min. Reworded.
- QUICKSTART now leads with a runtime expectations table:
selftest: ~1s static, ~30s with --model
score (default): ~5-7 min (Trajectory + KLD)
score --full: ~25-30 min (+ R-NIAH + PLAD)
repeatability: 4× default
Default is quick. Most users should never wait 30 minutes.
- QUICKSTART preflight example now shows BOTH .gguf and MLX
directory paths so MLX users see they're supported.
- HTML report hardware card showed "137.4 GB" RAM on a 128 GB Mac
(used /1e9 SI gigabytes). Switched to /1024**3 binary GiB so
output matches Apple's marketing convention (128 GB exact).
Also fixed model file size to use binary GiB.
- Regenerated examples/*.html with the GiB fix and live hardware
metadata.
Tests: 82/82 pass. No semantic changes to scoring, only metadata
display + setup ergonomics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
