mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 02:53:13 -05:00
TheTomandClaude Opus 4.7
01c22235bc
Adopted the polished design from a friend's iteration:
- Geist + Geist Mono via Google Fonts (system fallbacks for offline)
- Semantic colour tokens (--red/amber/green/blue + bg/border variants)
- Top-of-page stats strip with composite + 4 axes in one 5-column row
- Pill badges with colored dots
- Dense per-axis table; R-NIAH heatmap and PLAD per-perturbation
table inline as detail rows
- Run details: 3-card grid (Model / Hardware / Environment)
- Reproduce: dark code block with syntax-highlighted flags / args /
placeholders
- Footer: "What is this?" + GitHub link + version + report ID
(e.g. #0430-1537)
- Responsive breakpoints for narrow viewports
Plus dark-mode toggle:
- color-scheme: light dark (follows OS by default)
- light-dark() in every theme variable (Chrome 123+ / Safari 17.5+ /
Firefox 120+ — all 2024). Older browsers see the light values
cleanly; the dark theme is best-effort progressive enhancement
- Sun/moon button top-right of masthead, manual override via
data-theme on <html>
- Tiny inline JS for the click handler (no framework, no CDN JS)
Misc polish in this pass:
- R-NIAH "0.00 / 0.00" cells now render as italic-gray "n/a" with a
tooltip explaining "fp16 baseline does not retrieve at this cell"
(was confusing to see literal zeros for the uninformative cells)
- Pretty platform string ("macOS 26.4 arm64") instead of the
full platform.platform() output
- Diagnosis summary box auto-detects which axes drove the band
Tests: 82/82 still pass. Examples regenerated (~38 KB each, still
single-file).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sample REFRACT reports
Real reports from the 2026-04-30 v0.3.0 matrix run. Each cell ships in two formats:
<name>.json— machine-readable, schemarefract.report.v0.3.1, consumable byrefract compare *.jsonor any JSON-aware tool.<name>.html— self-contained HTML page. Open in any browser (no CDN, no JS framework). Pasteable as a single file in Discord/X for visual sharing.
Use these to sanity-check your own first runs: if your composite is in the same ballpark as the matching reference for a similar model size + KV config, your framework is wired up correctly.
| File | Composite | Band | What it shows |
|---|---|---|---|
clean-q8q8-mistral24b |
~91 | EXCELLENT | Mistral-Small-24B Q4_K_M with q8/turbo4 KV. Faithful quant on a robust model. Use as a "what success looks like" anchor. |
degraded-qwen7b |
~76 | DEGRADED | Qwen2.5-7B Q8 with q8/turbo4 KV. Trajectory low (55) and PLAD low (76); KLD high (98). Per-token decode drift + brittleness, retrieval intact. |
distribution-broken-gemma26b |
~29 | FAIL | gemma-4-26B-A4B Q8 with q8/turbo4 KV. Trajectory + KLD both ~17 (1.74 nats), R-NIAH 100, PLAD 78. The "distribution wrecked but reasoning intact" pattern. |
catastrophic-symturbo |
~19 | FAIL | gemma-4-26B-A4B Q8 with symmetric turbo4/turbo4 (the deliberately-broken negative control). Trajectory 9.74, KLD 11.84 (2.13 nats). If you run this and don't get FAIL, your framework setup has a problem. |
Sanity reading
# View a JSON
python3 -m json.tool examples/clean-q8q8-mistral24b.json | less
# Open an HTML report
open examples/clean-q8q8-mistral24b.html
# Side-by-side via the compare subcommand
python3 -m refract.cli compare examples/*.json
Reproducing on your hardware
The v0.3.0+ matrix-runner script that produced these is reproducible
via refract score directly. The shapes you should see if running the
same model + same candidate:
- Trajectory + KLD reproducible to ~1 point on the same llama.cpp commit
- R-NIAH and PLAD reproducible to ~5 points (single-trial cells, some stochasticity in retrieval cell-by-cell)
- Composite reproducible to ~2 points
If your numbers diverge by more than that, suspect:
- Different llama.cpp commit (capture from your JSON's
environment.llama_cpp_commit) - Different candidate KV config than you think
- Different chat-template handling (v0.3+ vs older REFRACT)
The HTML reports embed the raw JSON in a <details> block at the
bottom — one artifact carries everything, no "did the JSON come from
the same run as the screenshot?" ambiguity.