mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 11:02:42 -05:00
TheTom
0d2c723079
Investigation of master llama.cpp #21038 attention rotation in our fork, ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE / LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork (rotation OFF on both sides); the contribution is per-side control plus a matrix documenting which models want which knob. Headline finding while running the matrix: corpus PPL on wikitext-2 is unreliable for KV-quantization evaluation on gemma-class instruct models. Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it GGUFs; KLD against the fp16-KV reference points the opposite direction. Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512 and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best on row). The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B, large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against fp16-KV logits as the oracle for KV-quant evaluation on this model class. Independently observed prior: vektorprime in #21394, AesSedai's tables in #21038, ggerganov's "track KLD rather than PPL" comment in same PR, localbench. This paper adds the controlled per-side measurement, the cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise floor on Metal, and the resulting engineering policy.