mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-19 01:30:50 -05:00
661bc7fb47
Complete turbo4 4-bit PolarQuant benchmark suite: - PPL: +0.23% vs q8_0 (best after q8_0, better than q4_0) - Prefill: 101-105% of q8_0 across 2K-32K - Decode: 79.87 tok/s (93% q8_0, faster than turbo3) - NIAH: 7/9 (matches q8_0) - KLD: 0.0096 (40% lower than turbo3, matches q4_0 same-top-p) - PDF 24K: 63.7 tok/s decode (20% faster than turbo3) Also includes turbo4-resurrection.md paper documenting the full journey from PPL 679 to beating q4_0 quality. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
38 lines
1.6 KiB
Plaintext
38 lines
1.6 KiB
Plaintext
Turbo4 2+1 Bit Packing Fix — +33% Decode Speed
|
|
=================================================
|
|
|
|
Date: 2026-03-28
|
|
Branch: experiment/turbo4-quality-investigation
|
|
Hardware: Apple M5 Max 128GB
|
|
Model: Qwen3.5-35B-A3B Q8_0 (MoE)
|
|
|
|
Change: Replaced byte-spanning 3-bit packed index extraction with
|
|
turbo3-style 2+1 bit scheme (low 2 bits in qs[], high 1 bit in signs[]).
|
|
Also dropped QJL entirely (signs[] now stores index high bit, not QJL signs).
|
|
|
|
Results:
|
|
Config | PPL | Decode tok/s | vs q8_0
|
|
q8_0 | 6.1109 | 86.01 | 1.00x
|
|
turbo3 | 6.1756 | 78.24 | 0.91x
|
|
turbo4 2+1 bit (NEW) | 6.1756 | 63.79 | 0.74x
|
|
turbo4 no QJL (old 3-bit pack) | 6.1756 | 48.05 | 0.56x
|
|
turbo4 full (old, with QJL) | 6.1894 | 44.43 | 0.52x
|
|
|
|
Improvement: 48.05 → 63.79 tok/s (+33% from packing fix alone)
|
|
PPL: identical to turbo3 (6.1756)
|
|
|
|
Remaining gap vs turbo3 (63.79 vs 78.24 = 18% slower):
|
|
Entirely due to 128-block FA tiling overhead (nl=8 vs nl=2).
|
|
turbo4 block is 128 elements → dequant called 4x more per position.
|
|
The per-thread block cache mitigates this but doesn't eliminate it.
|
|
|
|
Updated component breakdown:
|
|
Component | tok/s lost | % of gap
|
|
128-block FA tiling overhead | 13.7 | 62%
|
|
2+1 bit unpack + centroid LUT | 8.4 | 38%
|
|
QJL (removed) | 0.0 | 0%
|
|
TOTAL gap vs q8_0 | 22.2 | 100%
|
|
|
|
The 3-bit byte-spanning unpack was the single biggest fixable bottleneck.
|
|
Remaining gap is structural (block size in FA kernel).
|