mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-21 18:42:44 -05:00
661bc7fb47
Complete turbo4 4-bit PolarQuant benchmark suite: - PPL: +0.23% vs q8_0 (best after q8_0, better than q4_0) - Prefill: 101-105% of q8_0 across 2K-32K - Decode: 79.87 tok/s (93% q8_0, faster than turbo3) - NIAH: 7/9 (matches q8_0) - KLD: 0.0096 (40% lower than turbo3, matches q4_0 same-top-p) - PDF 24K: 63.7 tok/s decode (20% faster than turbo3) Also includes turbo4-resurrection.md paper documenting the full journey from PPL 679 to beating q4_0 quality. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1.9 KiB
1.9 KiB
Sparse V Upstream Validation Note
What changed
A single conditional added to the Metal flash attention vec kernel's V accumulation loop. When the post-softmax attention weight for a position is below 1e-6, the V dequantization and accumulation for that position is skipped.
Gated by compile-time preprocessor define GGML_METAL_FA_SKIP_V, controlled via environment variable. Default: off.
Why safe
- The conditional only fires in the quantized V path (dequant branch)
- Threshold 1e-6 is far below meaningful contribution to output
- When disabled (
GGML_METAL_FA_SKIP_V=0), code path is identical to unpatched - No changes to attention scores, softmax, normalization, or K computation
What was tested
- PPL: 32K context, 50 chunks, wikitext-103, q8_0/q4_0/turbo3 — ON/OFF delta 0.0000 in all formats
- PPL: 8K/16K/32K multi-context sweep — ON/OFF delta 0.0000 at all depths
- Decode speed: q8_0 +5% short, turbo3 +22.8% at 32K, q4_0 neutral
- NIAH retrieval: identical or improved across formats
- KL divergence: measured, consistent with underlying quantization
- Threshold ablation: 1e-4 through 1e-8, PPL identical at all values
- Dense model: no regression
- Hardware: Apple M5 Max 128GB
Supported claims (safe for PR)
- No measurable PPL change when enabled (across tested formats and contexts)
- Decode speed improvement scales with V dequant cost
- Safe to enable for quantized KV cache formats on Metal
Unsupported claims (do NOT put in PR)
- Universal improvement across all hardware
- CUDA validation (not tested on upstream CUDA path)
- "Improves quality" (NIAH improvement is secondary signal, not primary claim)
- Specific percentage claims beyond tested setup
Limitations
- Validated on Apple Metal only (M5 Max)
- Threshold is hardcoded (not adaptive)
- Skip rate directly measured on smaller model (1.7B), estimated on larger
- Short context benefit is minimal (attention is dense)
- Dense models see negligible benefit (FFN dominates)