turboquant_plus/refract/INTERPRETATION.md

REFRACT INTERPRETATION GUIDE

How to read a REFRACT report card. Targets a reader who knows quants generally but is new to REFRACT specifically.

Composite score

A single 0–100 number. Higher is better. Computed as the harmonic mean of every axis that ran (so any single broken axis tanks the composite — that's intentional, the framework is fail-loud).

Band	Range	Meaning
EXCELLENT	90–100	Within reference noise. Indistinguishable.
PASS	80–90	Minor drift; safe to deploy in most uses.
DEGRADED	60–80	Visible drift; audit on your workload first.
FAIL	0–60	Material quality loss; treat as broken.

Per-axis interpretation

Each axis has its own 0–100 score and its own band. The Diagnosis block in the report card translates the per-axis pattern into plain English. Below is the deeper "what to do about it" guide per axis.

Axis A: Trajectory (formerly GTM)

Measures: Token-level agreement with fp16. Greedy-decodes N tokens under both reference and candidate KV configs, captures token IDs at decode time, computes prefix agreement length / candidate length.

• EXCELLENT (90–100): the model decodes essentially the same tokens under both configs. KV quantization is doing nothing visible in the short-context greedy regime.
• PASS (80–90): tokens diverge after 5–10 steps but the leading context matches. Common for fine-grained quants. Safe.
• DEGRADED (60–80): divergence within first few tokens on most prompts. Suspect a softmax-shift class of error: e.g. K rotation collisions, V rotation breaking with insufficient bits.
• FAIL (0–60): model decodes essentially different text under cand. The KV scheme is wrecking next-token distributions.

Remediation when low: try a higher-bit V cache; switch from symmetric to asymmetric turbo; try non-rotating V. If cand and ref diverge at step 1 (median_first_divergence near 0), the problem is purely at the per-token distribution level.

Axis B: KLD@D (corpus KL divergence)

Measures: Distribution-level divergence from fp16 on natural-text inputs, averaged per token across a corpus. Bit-exact zero on Metal when cand == ref, so any non-zero value is real signal not float jitter.

• EXCELLENT (90–100, KLD < 0.05 nats): distributions essentially match. The candidate doesn't measurably move the model.
• PASS (80–90, KLD ~0.05–0.2 nats): small but measurable drift. Most quants land here.
• DEGRADED (60–80, KLD ~0.2–0.5 nats): material distribution shift. Argmax may still match (so trajectory could be high) but the second/third choices have moved.
• FAIL (< 60, KLD > 0.5 nats): large divergence; the per-token predictive distributions are fundamentally different.

Remediation when low: measure the noise floor first (--measure-floor) to confirm the reference itself is deterministic on your build. If floor passes, the candidate is genuinely lossy at the distribution level. Try a higher-bit baseline (q8 instead of q4) or a non-rotating V scheme.

Note: KLD@D is the best-disciplined axis. If KLD says broken, trust it. If KLD says clean while Trajectory says broken, the bug is likely in chat-template handling or in the test workload not the quant itself.

Axis C: R-NIAH (Retrieval Needle-In-A-Haystack)

Measures: Long-context retrieval quality vs fp16. Inserts a sentinel fact into a long context at fractional positions; checks whether the candidate retrieves it as well as the reference does. Score per (length, position) cell, aggregated.

• EXCELLENT (100): candidate retrieves wherever the reference does. Either the model is robust at long context OR the model can't retrieve at all under either config (check confidence field).
• PASS / DEGRADED / FAIL (< 100): candidate loses retrieval at cells where the reference still finds the needle. Inspect the per-cell breakdown for the (length, position) where it breaks.

Remediation when low: revert to a higher-bit KV at long contexts; or add a runtime guard that switches KV schemes by sequence length.

Confidence: the JSON includes confidence: low when base_acc_avg < 0.2 — the model isn't engaging retrieval at all and R-NIAH = 100 is then a noise-floor reading. Re-check on a stronger model or with a different needle.

Axis D: PLAD (Perturbation-Locality Aware Drift)

Measures: Robustness to small prompt changes vs fp16. Generates anchor completions, perturbs each prompt minimally (typo, casing, punctuation, paraphrase), measures how much the candidate's output drifts vs how much the reference's drifts.

Per-perturbation breakdown is the actionable signal:

• typo low → candidate is brittle to typos. Consider input sanitization or higher-bit fallback when typos are detected.
• case low → candidate is sensitive to capitalization changes.
• punct low → candidate's output flips on terminal punctuation (sometimes acts as an EOS-like signal).
• paraphrase low → candidate is sensitive to lexical substitution. Less common; usually the most worrying signal.

Confidence: if a perturbation can't apply to your prompts (e.g., no ≥4-char words → typo skipped, no synonyms in table → paraphrase skipped), it's listed in skipped_perturbations and excluded from the band.

Pattern recognition (Diagnosis block)

The Diagnosis block fires plain-English sentences for the most common patterns. Some examples and what they imply:

• "Per-token distribution is broken but high-level surfaces are intact" → Trajectory and KLD low, R-NIAH and PLAD intact. The quant is producing different tokens at decode time but reasoning and retrieval still work. Common for aggressive turbo schemes. Likely safe if you don't depend on exact text reproduction.

• "Long-context retrieval degraded" → R-NIAH low, others high. Candidate fails specifically at long context. Affects RAG, summarization, code understanding. Unsafe for those workloads.

• "Brittleness to small input changes" → PLAD low, others high. Candidate flips its output under input variations. Affects real-user deployments where prompts have typos or slight paraphrasing. Stress-test before shipping.

• "Catastrophic" → every measured axis < 60. Treat as non-functional; revert to higher-bit config.

Composite vs per-axis: which to trust

If composite says PASS but one axis is DEGRADED, read the per-axis band first. Composite is a summary; per-axis tells you what kind of failure you have. A 95 composite hides a 99/99/99/85 reality where the model is actually brittle to perturbations.

For shipping decisions:

• All 4 axes ≥ 80 → safe
• One axis 60–80 → audit the specific surface that's degraded; decide based on whether that surface matters to your workload
• One axis < 60 → ship only if you understand the failure and have mitigations
• Multiple axes < 60 → don't ship

Cross-version comparisons

Reports embed framework_version. Comparing scores across versions is fine for the same model+candidate, but a "+1.5 composite" between v0.2.0 and v0.3.0 may be partly attributable to methodology changes (e.g., chat-template handling). Apples-to-apples requires same version on both sides.