mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-23 11:32:33 -05:00
TheTomandClaude Opus 4.7
2d8a2472f7
v0.1.3 ships no new functional axes — only the test coverage, fail-loud guards, and corpus-identity machinery needed to stop the v0.1.x patch-round-after-matrix-run cycle. Code fixes: - axes/gtm.py: tokenizer-failure fallback now RAISES instead of silently using whitespace (was producing wrong-unit scores with no detection). - axes/gtm.py: GTM score now normalizes by mean_cand_length instead of n_predict (fixes gemma-31B 2.87× retokenize inflation that re-clipped the score to 100 in v0.1.2). - cli.py: --measure-floor additionally asserts mean_prefix == mean_cand on ref-vs-ref greedy (KLD ref-vs-ref is bit-exact zero on Metal so the composite floor passes even when GTM is broken). - runner.py: _TOPP_RE regex now tolerates 'Same top p' (space) form (test surfaced that v0.1.2 only matched the dashed form). - runner.py: new corpus_identity / sidecar machinery; KLDResult includes corpus path/size/SHA; refuses to score against a base built from a different corpus. New tests (30 across 6 files; all pass in <0.05s, no subprocess): - test_strip_noise.py: 5 — pins runner._strip_noise (banner / backspace / block-char regressions from v0.1.1). - test_command_construction.py: 4 — pins --single-turn presence, --no-conversation absence, --kl-divergence-base, errors='replace'. - test_tokenize_to_ids.py: 5 — parser + raise-on-error contract. - test_kld_regex.py: 6 — perplexity output regexes incl. gemma Same-top-p missing case (this test caught the regex bug). - test_report_json_layout.py: 5 — pins schema + composite-is-scalar. - test_corpus_identity.py: 5 — sidecar machinery. - test_validation.py: +4 GTM assertions (full_match_rate < 1.0, mean_prefix > 0, ≥3 distinct refs, ratio == 1.0 on ref-vs-ref). Total: 14 → 44 unit tests + 1 strengthened integration test. Docs: - LIMITATIONS.md: NEW — documents the limitations we ship with (detokenize→retokenize gap, corpus-anchored KLD, provisional bands, single-platform tested, GTM/EOS conflation, v0.1.3 fail-loud removal). - README.md: GTM scoring formula updated; new test files listed; points at LIMITATIONS.md. - axes/gtm.py module docstring: refreshed to reflect v0.1.2 + v0.1.3. - CHANGELOG.md: v0.1.3 entry at top with the v0.1.2 matrix table and the file-by-file change summary. - report.py: schema bumped refract.report.v0.1.1 → v0.1.3. - __init__.py: __version__ = "0.1.3". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney <tturney1@gmail.com>
20 lines
697 B
Python
20 lines
697 B
Python
"""REFRACT v0.1 — REFerence-anchored Robust Acid-test for Compressed Transformers.
|
|
|
|
A benchmaxx-resistant alternative to corpus PPL for evaluating KV-cache
|
|
quantization quality. Replaces "lower PPL = better" with a composite of:
|
|
|
|
- Axis A (GTM): Greedy Trajectory Match — do quantized + reference produce
|
|
the same generated tokens?
|
|
- Axis B (KLD@D): KL Divergence vs the fp16-KV reference distribution
|
|
(v0.1 uses corpus KLD via llama-perplexity as a proxy for trajectory KLD).
|
|
|
|
See:
|
|
- refract/README.md for usage.
|
|
- docs/papers/attn-rotation-and-ppl-artifact.md for the motivating paper.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
__version__ = "0.1.3"
|
|
__all__ = ["__version__"]
|