diff --git a/docs/experiment-decode-fp16-lut-results.md b/docs/experiment-decode-fp16-lut-results.md index db15617..a4a9b77 100644 --- a/docs/experiment-decode-fp16-lut-results.md +++ b/docs/experiment-decode-fp16-lut-results.md @@ -41,11 +41,34 @@ One change to the Metal shader: the vec flash attention dequant (`dequantize_tur The non-vec dequant (`dequantize_turbo3_0`) is unchanged — fp16 hurts the non-vec path due to half→float conversion overhead in the 16-element-per-call pattern. +## Variant A: Half LUT + Float Norm Broadcast (NEW BEST) + +Further optimization: centroid lookup in half (2 bytes, small cache footprint), but norm as `float4(centroids) * float_norm` (single broadcast multiply instead of 4 half multiplies). + +| Context | Main | fp16 LUT | **Half LUT + float norm** | q8_0 | +|---------|------|----------|--------------------------|------| +| Short | 75.3 | 77.2 | **78.7** | 85.2 | +| 8K | 59.2 | 67.3 | **68.8** | 77.7 | +| 48K PDF | 36.7 | 39.0 | **39.9** | 55.6 | +| vs q8_0 @ 48K | 0.66x | 0.70x | **0.72x** | — | + +PPL: 6.211 (unchanged across all variants). + +## All Decode Experiments Summary + +| # | Approach | Short | 8K | 48K | vs main | Verdict | +|---|---------|-------|-----|-----|---------|---------| +| — | Main (float LUT) | 75.3 | 59.2 | 36.7 | baseline | — | +| — | No LUT (ceiling) | ~78 | 70.1 | — | theoretical max | — | +| 1 | fp16 LUT (vec only) | 77.2 | 67.3 | 39.0 | +6% | good | +| **2** | **Half LUT + float norm** | **78.7** | **68.8** | **39.9** | **+8.7%** | **BEST** | +| 3 | Register float4 select | — | — | — | -4% | worse | +| 4 | Inline switch | — | — | — | -13% | much worse | +| 5 | Split lo/hi + select | — | — | — | -10% | worse | +| 6 | Precomputed centroid*norm | — | — | — | +2% | marginal | +| 7 | fp16 LUT both paths | — | — | — | -10% | worse | +| 8 | Custom GGML_OP_TURBO_WHT | — | — | — | -7% | net negative | + ## Recommendation -**This change is a net positive and should be merged to main.** It: -- Improves decode speed at all context depths (biggest win at long context) -- Has zero quality regression -- Has no prefill regression -- Is a 2-line change with minimal risk -- Directly addresses the tester-reported decode slowdown at long context +**Merge the "half LUT + float norm broadcast" to main.** It's the best decode speed achievable within the current dequant function interface. +4.5% to +16.2% vs main, zero regressions.