mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 02:53:13 -05:00
TheTomandClaude Opus 4.7
a4c59a0b05
Compressed history of v0.1.4 → v0.3.2 in one alpha-ship commit so the
framework is testable end-to-end by friends. Detailed per-version
discussion lives in refract/CHANGELOG.md.
What ships
----------
Four scoring axes vs the model's own fp16-KV reference:
- Axis A (Trajectory, v0.1.4) — decode-time token IDs via patched
llama-completion, no detokenize round-trip. Replaces the buggy
v0.1.x GTM as the default.
- Axis B (KLD@D) — corpus KL divergence, bit-exact zero on Metal
when ref==cand. Native MLX implementation in addition to the
llama-perplexity subprocess path.
- Axis C (R-NIAH, v0.2.0) — long-context retrieval at multiple
(length, position) cells. Confidence guard for low base_acc.
- Axis D (PLAD, v0.2.0) — perturbation-locality drift across
typo/case/punct/paraphrase. Skipped-perturbation handling.
Composite = harmonic mean. Bands EXCELLENT/PASS/DEGRADED/FAIL.
Four CLI subcommands:
- score — run REFRACT on a candidate KV config
- selftest — 30s preflight: binaries, flags, model probe
- compare — multi-report side-by-side
- repeatability — run N times, report stdev/range health verdict
Three backends:
- llamacpp (production) subprocess wrappers around llama.cpp
- mlx (production) native via mlx-lm; verified end-to-end
on Qwen2.5-3B-Instruct-4bit
- vllm (skeleton) interface defined; plug-in pointers
in module docstring
Reporting
---------
- Text (ANSI colour, terminal)
- JSON schema refract.report.v0.3.1 with framework_version +
environment metadata (backend, llama_cpp_commit, mlx_lm_version)
- HTML (self-contained single file) with hardware info, model
params, repro command, R-NIAH heatmap, PLAD per-perturbation bars
Discoveries from the matrix runs (all logged in CHANGELOG):
- "secret password" needle wording triggers safety refusals on
instruct models. v0.2.1 switched to neutral "rare paint color"
- Thinking-mode models burn n_predict before answering. v0.3.0 added
-rea off + auto-detect at startup
- Symmetric turbo4/turbo4 is the canonical FAIL config; framework
correctly identifies (composite ~19) on gemma-4-26B-A4B-Q8
Documentation
-------------
- README.md what + alpha banner + "where do I go" navigator
- QUICKSTART.md 0 to first PASS in 30 minutes
- INTERPRETATION.md per-axis "what to do if low" guide
- LEADERBOARD.md cross-model rankings + "ranks the quant, not the
model" disclaimer
- PITFALLS.md known traps from today's matrix runs
- LIMITATIONS.md what v0.3 explicitly does not do
- CHANGELOG.md version history with discovery stories
- MATRIX-RESULTS.md the 7-model matrix log
- examples/ 4 sample JSONs + HTML reports + README
Tests
-----
80 → 82 unit tests, all green:
- 17 trajectory tests (incl. R-NIAH/PLAD impls + 4-axis composite)
- 5 strip-noise pinning tests
- 14 unit tests (math + KVConfig)
- 8 JSON schema tests (incl. v0.3.1 framework_version field)
- 6 KLD regex tests
- 5 corpus-identity tests
- 5 tokenize-to-ids tests
- 4 command-construction tests
Verified
--------
- 7-model v0.3.0 matrix end-to-end (~2h on M5 Max 128GB)
- Llama-4 Scout extras + symmetric-turbo negative control
- MLX backend on Qwen2.5-3B-Instruct-4bit
- selftest on llamacpp (no model) and mlx (with model)
- 2-run repeatability on qwen2.5-1.5b: composite stdev = 0.00 on
Trajectory + KLD (bit-exact reproducibility on cheap axes)
Open follow-ups (v0.4)
----------------------
- vLLM backend implementation (skeleton today)
- T-Call axis (tool-call fidelity) building on v0.3 chat-template
- Bundled corpus + prompts distribution
- `refract leaderboard --in dir/ --out LEADERBOARD.md` auto-gen
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sample REFRACT reports
Real reports from the 2026-04-30 v0.3.0 matrix run. Each cell ships in two formats:
<name>.json— machine-readable, schemarefract.report.v0.3.1, consumable byrefract compare *.jsonor any JSON-aware tool.<name>.html— self-contained HTML page. Open in any browser (no CDN, no JS framework). Pasteable as a single file in Discord/X for visual sharing.
Use these to sanity-check your own first runs: if your composite is in the same ballpark as the matching reference for a similar model size + KV config, your framework is wired up correctly.
| File | Composite | Band | What it shows |
|---|---|---|---|
clean-q8q8-mistral24b |
~91 | EXCELLENT | Mistral-Small-24B Q4_K_M with q8/turbo4 KV. Faithful quant on a robust model. Use as a "what success looks like" anchor. |
degraded-qwen7b |
~76 | DEGRADED | Qwen2.5-7B Q8 with q8/turbo4 KV. Trajectory low (55) and PLAD low (76); KLD high (98). Per-token decode drift + brittleness, retrieval intact. |
distribution-broken-gemma26b |
~29 | FAIL | gemma-4-26B-A4B Q8 with q8/turbo4 KV. Trajectory + KLD both ~17 (1.74 nats), R-NIAH 100, PLAD 78. The "distribution wrecked but reasoning intact" pattern. |
catastrophic-symturbo |
~19 | FAIL | gemma-4-26B-A4B Q8 with symmetric turbo4/turbo4 (the deliberately-broken negative control). Trajectory 9.74, KLD 11.84 (2.13 nats). If you run this and don't get FAIL, your framework setup has a problem. |
Sanity reading
# View a JSON
python3 -m json.tool examples/clean-q8q8-mistral24b.json | less
# Open an HTML report
open examples/clean-q8q8-mistral24b.html
# Side-by-side via the compare subcommand
python3 -m refract.cli compare examples/*.json
Reproducing on your hardware
The v0.3.0+ matrix-runner script that produced these is reproducible
via refract score directly. The shapes you should see if running the
same model + same candidate:
- Trajectory + KLD reproducible to ~1 point on the same llama.cpp commit
- R-NIAH and PLAD reproducible to ~5 points (single-trial cells, some stochasticity in retrieval cell-by-cell)
- Composite reproducible to ~2 points
If your numbers diverge by more than that, suspect:
- Different llama.cpp commit (capture from your JSON's
environment.llama_cpp_commit) - Different candidate KV config than you think
- Different chat-template handling (v0.3+ vs older REFRACT)
The HTML reports embed the raw JSON in a <details> block at the
bottom — one artifact carries everything, no "did the JSON come from
the same run as the screenshot?" ambiguity.