From fdedbfe039ce9878d32ebecb209ef53e60b14869 Mon Sep 17 00:00:00 2001
From: TheTom <tturney1@gmail.com>
Date: Wed, 25 Mar 2026 22:51:44 -0500
Subject: [PATCH] =?UTF-8?q?docs:=20variant=20A=20(half=20LUT=20+=20float?=
 =?UTF-8?q?=20norm)=20is=20new=20best=20=E2=80=94=20+8.7%=20at=2048K?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

8 decode experiments tested, 2 winners, 6 dead.
Half LUT + float norm broadcast: 78.7/68.8/39.9 tok/s (short/8K/48K)
vs main: 75.3/59.2/36.7 (+4.5%/+16.2%/+8.7%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
---
 docs/experiment-decode-fp16-lut-results.md | 35 ++++++++++++++++++----
 1 file changed, 29 insertions(+), 6 deletions(-)

diff --git a/docs/experiment-decode-fp16-lut-results.md b/docs/experiment-decode-fp16-lut-results.md
index db15617..a4a9b77 100644
--- a/docs/experiment-decode-fp16-lut-results.md
+++ b/docs/experiment-decode-fp16-lut-results.md
@@ -41,11 +41,34 @@ One change to the Metal shader: the vec flash attention dequant (`dequantize_tur
 
 The non-vec dequant (`dequantize_turbo3_0`) is unchanged — fp16 hurts the non-vec path due to half→float conversion overhead in the 16-element-per-call pattern.
 
+## Variant A: Half LUT + Float Norm Broadcast (NEW BEST)
+
+Further optimization: centroid lookup in half (2 bytes, small cache footprint), but norm as `float4(centroids) * float_norm` (single broadcast multiply instead of 4 half multiplies).
+
+| Context | Main | fp16 LUT | **Half LUT + float norm** | q8_0 |
+|---------|------|----------|--------------------------|------|
+| Short | 75.3 | 77.2 | **78.7** | 85.2 |
+| 8K | 59.2 | 67.3 | **68.8** | 77.7 |
+| 48K PDF | 36.7 | 39.0 | **39.9** | 55.6 |
+| vs q8_0 @ 48K | 0.66x | 0.70x | **0.72x** | — |
+
+PPL: 6.211 (unchanged across all variants).
+
+## All Decode Experiments Summary
+
+| # | Approach | Short | 8K | 48K | vs main | Verdict |
+|---|---------|-------|-----|-----|---------|---------|
+| — | Main (float LUT) | 75.3 | 59.2 | 36.7 | baseline | — |
+| — | No LUT (ceiling) | ~78 | 70.1 | — | theoretical max | — |
+| 1 | fp16 LUT (vec only) | 77.2 | 67.3 | 39.0 | +6% | good |
+| **2** | **Half LUT + float norm** | **78.7** | **68.8** | **39.9** | **+8.7%** | **BEST** |
+| 3 | Register float4 select | — | — | — | -4% | worse |
+| 4 | Inline switch | — | — | — | -13% | much worse |
+| 5 | Split lo/hi + select | — | — | — | -10% | worse |
+| 6 | Precomputed centroid*norm | — | — | — | +2% | marginal |
+| 7 | fp16 LUT both paths | — | — | — | -10% | worse |
+| 8 | Custom GGML_OP_TURBO_WHT | — | — | — | -7% | net negative |
+
 ## Recommendation
 
-**This change is a net positive and should be merged to main.** It:
-- Improves decode speed at all context depths (biggest win at long context)
-- Has zero quality regression
-- Has no prefill regression
-- Is a 2-line change with minimal risk
-- Directly addresses the tester-reported decode slowdown at long context
+**Merge the "half LUT + float norm broadcast" to main.** It's the best decode speed achievable within the current dequant function interface. +4.5% to +16.2% vs main, zero regressions.