0 Fork 0

mirror of https://github.com/TheTom/turboquant_plus.git synced 2026-10-23 03:22:55 -05:00

Files

T

TheTomandClaude Opus 4.7 2d8a2472f7 REFRACT v0.1.3: defensive patch round (tests + fail-loud + honest docs)

v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.

Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
  using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
  n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
  the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
  on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
  composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
  (test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
  corpus path/size/SHA; refuses to score against a base built from a
  different corpus.

New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
  block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
  --no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
  Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
  mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).

Total: 14 → 44 unit tests + 1 strengthened integration test.

Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
  (detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
  single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
  at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
  the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>

2026-04-29 20:36:25 -05:00

4.2 KiB

Raw Permalink Blame History

REFRACT v0.1.3 — Known Limitations

REFRACT v0.1.3 is a defensive release. It does not add new functional axes; it fixes the bugs the v0.1.2 matrix exposed, removes a silent fallback that was producing wrong-unit scores, and adds the regression test coverage that earlier versions lacked.

The limitations below are the ones the codex review surfaced and that we are explicitly choosing to ship with. Each one has a v0.2 plan.

1. Detokenize → retokenize gap

GTM tokenization comes from re-tokenizing the decoded stdout text of llama-cli, not from the model's actual decoded token IDs.

Detokenization can collapse special tokens (BOS/EOS/role markers) and whitespace.
Retokenizing the resulting text can produce a different sequence than the model originally emitted — sometimes 2-3× as many tokens.
v0.1.3 mitigates the score impact by normalizing by mean_cand_length rather than n_predict, so the score stays bounded and meaningful even when retokenization inflates. But the underlying comparison is still text-based, not ID-based.

v0.2 plan: capture token IDs at decode time via a custom llama.cpp binary that dumps both per-step logits and the chosen ID. This also unlocks true trajectory-KLD (the v0.2 KLD axis), so the two problems get fixed together.

2. Corpus-anchored KLD

The KLD axis uses llama-perplexity --kl-divergence's corpus-KLD machinery against a reference base file built from the same corpus.

Rankings between configurations on the same corpus are reliable — this is what the paper §4.3 uses and what we re-validated.
Absolute KLD magnitudes depend on corpus choice. A 1.7-nat result on wikitext-2 is not directly comparable to a 1.7-nat result on a code corpus.
v0.1.3 records corpus identity (path, size, first-MiB SHA) in the output JSON and refuses to score against a base file built from a different corpus, so cross-corpus accidents are caught at runtime.

v0.2 plan: trajectory KLD computed at decode time on the same prompts the GTM axis uses, eliminating the corpus dependence entirely.

3. Bands are provisional

The 95 / 80 / 60 thresholds (EXCELLENT / PASS / DEGRADED / FAIL) were calibrated against a single matrix on wikitext-2 with limited model coverage (7 models, M5 Max only). Expect re-calibration in v0.2 once trajectory KLD changes the absolute KLD scale.

The harmonic-mean composite formula and the band names are stable; the numeric thresholds are not.

4. Single platform tested

REFRACT has only been validated on Apple Silicon Metal builds (M5 Max). The --measure-floor step relies on the bit-exact-zero KLD noise floor that Metal exhibits on the llama-perplexity codepath (paper §4.5). CUDA and HIP builds may have a non-zero noise floor; users on those platforms should re-measure with --measure-floor and treat any non-zero floor as their working noise threshold rather than assuming 99.5 is achievable.

5. GTM conflates divergence with EOS truncation

A model that emits EOS at token 5 and a model that diverges at token 5 both produce prefix_agreement_length = 5. GTM treats them identically even though the failure modes are completely different.

In practice this rarely matters for KV-quantization comparisons (both candidate and reference tend to either both stop short or both keep going), but it's a genuine ambiguity in the metric.

v0.2 plan: distinguish "EOS at position k" from "wrong token at position k" by recording the per-step token IDs and a per-step is_eos flag.

6. Whitespace fallback removed (v0.1.3 fail-loud change)

Earlier versions had a silent try/except that fell back to whitespace tokenization if llama-tokenize raised. This produced scores in mixed units (whitespace tokens vs model tokens) that the caller had no way to detect.

v0.1.3 removes this fallback entirely. If the tokenizer subprocess fails — for any reason: the binary is missing, the model is unsupported, or the input is malformed — the GTM axis raises a RuntimeError with the original tokenizer error attached. This is intentional: a wrong score is worse than no score.

If you actually need a whitespace-only run for debugging, call _tokenize_words directly; it remains exported as a stable utility (it just is not used by the diff path anymore).

4.2 KiB Raw Permalink Blame History Unescape Escape