diff --git a/docs/cross-model-validation.md b/docs/cross-model-validation.md index b2d1b75..7273509 100644 --- a/docs/cross-model-validation.md +++ b/docs/cross-model-validation.md @@ -37,7 +37,7 @@ The root cause was found by reading all 5 overloads of `build_attn` in `llama-gr | Qwen 27B Dense | 128 | 6.89 | 6.94 | +0.72% | 7.01 | 0.99x | ✅ | | Phi-4 | 128 | 6.00 | 6.10 | +1.68% | 6.23 | 0.91x | ✅ | | Mistral Small 24B | 128 | 6.09 | 6.12 | +0.46% | 6.28 | 0.86x | ✅ | -| Gemma 2 27B (ISWA) | 128 | 3.75 | 3.79 | +0.9% | 3.80 | — | — | +| Gemma 2 27B (ISWA) | 128 | 7.06 | 7.18 | +1.7% | 7.15 | 0.88x | ✅ | | Llama 3.1 70B | 128 | 2.44 | 2.64 | +8.3% | 2.79 | 0.94x | ✅ | | Mixtral 8x7B MoE | 128 | — | — | — | — | — | skipped (bad GGUF) | @@ -78,11 +78,13 @@ Already validated extensively. See README and turbo4-resurrection.md. **Note:** Gemma 2 K/V head_dim IS 128 (not 256 as initially assumed — GGUF metadata confirms `key_length=128`). -| Cache | PPL | vs q8_0 | Decode tok/s | NIAH | -|-------|-----|---------|-------------|------| -| q8_0 | 3.7504 | — | 28.44 | 3/3 | -| turbo4 | 3.7852 | +0.9% | 26.12 | — | -| turbo3 | 3.7957 | +1.2% | 25.36 | — | +| Cache | PPL (8-chunk) | vs q8_0 | Decode tok/s | NIAH | +|-------|--------------|---------|-------------|------| +| q8_0 | 7.0590 | — | 28.28 | 3/3 | +| turbo3 | 7.1489 | +1.3% | 24.08 | — | +| turbo4 | 7.1794 | +1.7% | 24.79 | — | + +Note: turbo4 slightly worse than turbo3 on Gemma 2 — the only model showing this pattern. May be related to ISWA dual-cache interaction or Gemma's logit softcapping. Earlier 1-chunk estimates (3.75-3.80) were unreliable — use 8-chunk numbers above. This fix also applies to any other ISWA model (Cohere2, OLMo2, Gemma3N). diff --git a/docs/threshold-ablation-logs/gemma2_turbo4_outlier.txt b/docs/threshold-ablation-logs/gemma2_turbo4_outlier.txt new file mode 100644 index 0000000..ca2a6b7 --- /dev/null +++ b/docs/threshold-ablation-logs/gemma2_turbo4_outlier.txt @@ -0,0 +1,26 @@ +Gemma 2 27B: turbo4 slightly worse than turbo3 (outlier) +========================================================= + +Date: 2026-03-28 +Model: gemma-2-27b-it-Q4_K_M +Architecture: ISWA, logit softcapping (50.0), hd128 + +PPL (8-chunk, c=512): + q8_0: 7.0590 + turbo3: 7.1489 (+1.3%) + turbo4: 7.1794 (+1.7%) + +Every other model: turbo4 beats turbo3. +Gemma 2: turbo4 is 0.4% worse than turbo3. + +Possible causes to investigate: +1. ISWA dual-cache interaction with 4-bit centroids +2. Logit softcapping (tanh) amplifies 4-bit vs 3-bit centroid spacing differently +3. Gemma 2's GQA ratio (32Q/16KV) interacts differently with more centroids +4. The 8-entry constant LUT (turbo3) may have better cache behavior than + 16-entry LUT (turbo4) on ISWA's alternating cache access pattern +5. Statistical noise — 8 chunks at c=512 may not be enough to distinguish + a 0.04 PPL difference + +Priority: Low — both are within 2% of q8_0, and the difference is small. +Investigate if the pattern persists on other ISWA models (Cohere2, OLMo2).