6 new tests in test_command_construction.py:
- _llama_extra_flags() returns [] when env unset/empty/whitespace
- _llama_extra_flags() shlex-splits multi-word values + handles quoted args
- run_completion appends user flags AFTER REFRACT defaults so last-wins
on -ngl works (regression bound: tests both -ngl indices)
- run_perplexity_kld_base appends extra flags
- run_perplexity_kld appends extra flags
- run_completion_trajectory (the trajectory axis hot path AJ would hit)
appends extra flags
Refract test count: 404 -> 410.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
v0.1.3 ships no new functional axes — only the test coverage, fail-loud
guards, and corpus-identity machinery needed to stop the v0.1.x
patch-round-after-matrix-run cycle.
Code fixes:
- axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently
using whitespace (was producing wrong-unit scores with no detection).
- axes/gtm.py: GTM score now normalizes by mean_cand_length instead of
n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped
the score to 100 in v0.1.2).
- cli.py: --measure-floor additionally asserts mean_prefix == mean_cand
on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the
composite floor passes even when GTM is broken).
- runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form
(test surfaced that v0.1.2 only matched the dashed form).
- runner.py: new corpus_identity / sidecar machinery; KLDResult includes
corpus path/size/SHA; refuses to score against a base built from a
different corpus.
New tests (30 across 6 files; all pass in <0.05s, no subprocess):
- test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace /
block-char regressions from v0.1.1).
- test_command_construction.py: 4 — pins --single-turn presence,
--no-conversation absence, --kl-divergence-base, errors='replace'.
- test_tokenize_to_ids.py: 5 — parser + raise-on-error contract.
- test_kld_regex.py: 6 — perplexity output regexes incl. gemma
Same-top-p missing case (this test caught the regex bug).
- test_report_json_layout.py: 5 — pins schema + composite-is-scalar.
- test_corpus_identity.py: 5 — sidecar machinery.
- test_validation.py: +4 GTM assertions (full_match_rate < 1.0,
mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref).
Total: 14 → 44 unit tests + 1 strengthened integration test.
Docs:
- LIMITATIONS.md: NEW — documents the limitations we ship with
(detokenize→retokenize gap, corpus-anchored KLD, provisional bands,
single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal).
- README.md: GTM scoring formula updated; new test files listed; points
at LIMITATIONS.md.
- axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3.
- CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and
the file-by-file change summary.
- report.py: schema bumped refract.report.v0.1.1 → v0.1.3.
- __init__.py: __version__ = "0.1.3".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney <tturney1@gmail.com>
