Models with head_dim=64 (e.g., GPT-OSS-120B) may crash or produce
degraded output with turbo compression. WHT requires sufficient
dimensionality for CLT convergence. Reported by NigelTufnel12345.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
The turbo4-K PPL explosions were caused by kernel bugs (our fork) and
head_dim misdetection (AmesianX fork), not an IQ4_XS-specific issue.
AmesianX independently confirmed turbo4-K works correctly after fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Add compression reference table and config warning to getting-started.md
- Add independent validation from sztlink, Madreag, and mudler (APEX/LocalAI) across asymmetric, layer-aware, MoE, and sparse V papers
- Update sparse-v-dequant.md with Madreag 4-GPU cross-validation
- Minor README wording fix (branch -> repository)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
New paper: q8_0/turbo2 with Boundary V on Qwen3.5-35B MoE achieves
7.53x V compression with PPL within 1% of q8_0 at 512c/8K/32K.
Quality matches or beats q8_0/turbo3 at every tested length. 32K
decode ~2-3% faster than turbo3-V (n=2, M5 Max). Scoped to one
tested MoE model. Includes decode investigation negative results
(20+ approaches exhausted) and dense-vs-MoE bottleneck analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Replace parquet link with direct raw text download. Add HF CLI as
alternative. Credit: @henkiespenkie44
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Users were building from main instead of feature/turboquant-kv-cache,
resulting in "Unsupported cache type: turbo4" errors.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Corrections from verification review:
- Fix HyperionMS2040 dates (03-30 not 03-31) and numbers (Qwen2.5
symmetric was catastrophic PPL 3472, not +3.6%)
- Fix Aaryan-Kapoor attribution (was TheTom's words, not his)
- Remove seanrasch block_size=128 claim (he reported a crash, not
confirmation). Replace with HyperionMS2040 PR #32 fix
- Fix Arclabs001 framing (QJL works with independent rotations,
not "QJL hurts")
- Reword elusznik (scoped out QJL for PR size, not as validation)
- Fix Madreag date (03-26), AmesianX dates (03-30), dusterbloom
dates (03-30)
- Mark AMD HIP as author's own testing, not independent validation
- Mark treblewoe and Corianas_ as X collaborators
- Mark spiritbuun prefill claim as X collaborator with approximate date
- Add jesusmb1995 Vulkan mixed K/V data to asymmetric paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each paper now documents who confirmed the findings, on what hardware,
with what data, and when. Validations span Metal (M5 Max), CUDA (RTX
3090, 4090, 5090, Blackwell), and AMD HIP (RX 9070 XT).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Why V compression is free and K precision is everything. Validated
across 7 models (1.5B-104B), 3 weight quants, 3 GPU backends, 2
Apple Silicon generations. Independently confirmed by sztlink (RTX
4090), HyperionMS2040 (RTX 3090), and AMD HIP testing.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document built-in optimizations: Sparse V dequant (Metal, auto-enabled),
Boundary V (auto-enabled with turbo2-V), and maximum V compression config.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Standalone doc with build instructions (Metal, CUDA, HIP, Windows),
config selection table, benchmarking steps, and how to share results.
Community requested.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community feedback: setting iogpu.wired_limit_mb above 90% of physical
RAM risks kernel panics under sustained inference load. 90% is now the
recommended safe default. Credit: @treblewoe
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Isolation experiments confirmed GGML_METAL_NO_RESIDENCY=1 has no
measurable effect. The sysctl alone fixes the 49K+ context hang.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
104B Q4_K_M on M5 Max 128GB: turbo3/turbo3 +3.6% PPL, turbo4/turbo4
+1.9%. 128K full native context achieved (PPL 4.024) with Metal
settings iogpu.wired_limit_mb + GGML_METAL_NO_RESIDENCY. NIAH 10/10
at 4K/8K. turbo3 prefill faster than q8_0 at 32K. Context wall fix
confirmed working at 48K/64K/96K/128K. Biggest model tested.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
RotorQuant/IsoQuant modules (PR #34, not merged) and draft docs
were unintentionally bundled into the rename commit. Fixes CI
coverage failure (64% -> 95%+).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
m5-max-stress-test.md — generic name for ongoing large model validation.
Will add Command-R+ 104B and future models.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Llama-70B Q4_K_M results (symmetric turbo works, asymmetric rescue confirmed)
- Add CUDA community data (seanrasch, dusterbloom, Char__Bob)
- Add AMD RDNA 4 tested configs
- Update Boundary V from experimental to auto-enabled for turbo2-V
- Note Sparse V as default on all Metal
- Update block size section to reflect 128 as shipped default
- Update backend scope from Metal-only to multi-backend validated
- Add 70B M5 Max stress test paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First AMD GPU validation: RX 9070 XT (gfx1201) on Windows 11 with HIP SDK 7.1.
Asymmetric q8_0-K + turbo4-V confirmed working (+1.0% PPL, no speed penalty).
9 Windows/HIP gotchas documented. Community count updated to 30+.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Block_size=128 achieves 5.12x compression (vs 4.57x at 32) with
identical PPL across all tested Metal paths: asymmetric q8_0-K +
turbo{2,3}-V, symmetric turbo3/turbo3, and Boundary V (LA-V7).
Validated on 3 architectures (dense, Qwen, hybrid MoE), 3 context
lengths (512, 8K, 32K), and 2 hardware platforms (M5 Max, M2 Pro).
M2 Pro showed +3-7% decode improvement on tested 1.5B model.
No speed regression on M5 Max.
Also fixes pre-existing flaky d=192 turbo4 test threshold (0.6 -> 0.7).
Credit to @AmesianX whose block_size=256 implementation prompted
the investigation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tested f16 vs q8_0 boundary layers at 2 and 4 layer widths.
Finding: width matters more than precision. Wide q8_0 (first4+last4)
beats narrow f16 (first2+last2) on both phi-4 and Qwen3.5-27B.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New: docs/papers/layer-aware-v-compression.md — full research writeup
- Updated: docs/turboquant-recommendations.md — Boundary V as advanced experimental option
- New: proof/niah/niah_boundary_v/ — NIAH retrieval evidence
Boundary V protects first 2 + last 2 layers with q8_0-V while compressing
remaining layers with turbo2-V. Quality between turbo3 and turbo2, effective
compression between the two. Validated across 4 models on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Move all 20 NIAH result directories from repo root to proof/niah/.
Move NIAH summary docs from docs/ to proof/niah/.
No doc references needed updating (all were text mentions, not path links).
Prepares proof/ as the standard location for validation artifacts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Correct phi-4 PPL numbers to final 4-chunk validated values
- Correct 35B MoE turbo4 to 5.078 (was ~4.8)
- Add Mistral-24B Q4_K_M result (symmetric turbo works at this size)
- Note turbo × turbo asymmetric work and pipeline naming change
- Soften language: "sensitive" not "broken", model-specific not universal
- Link to recommendations doc
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- README: add asymmetric K/V section with practical guidance and backend scope note
- New: turboquant-recommendations.md — validated good/risky/experimental configs
- New: asymmetric-kv-discovery.md — full investigation narrative
- Updated: m2-ppl-investigation.md — concise final status
Key finding: K precision is the dominant quality factor on low-bit models.
q8_0-K + turbo-V rescues quality where symmetric turbo fails (validated on
Qwen2.5-7B Q4_K_M). Not all Q4_K_M models are sensitive — Mistral-24B works
fine with symmetric turbo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
turbo3/turbo4 completely flat across context depths. q8_0 scales with
context (FFN dominates at short, attention at long). No crossover.
Zero-dequant gap only 7% on 1.5B. Asymmetric K/V not yet supported.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Metal compiler already fully specializes templates. -1% regression from
duplicated inline code. Generic template IS the optimized kernel.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Initial -77% was env var bug (not set at runtime). Corrected: -1 to -3%.
SMEM still dead — not worth shipping for any config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
M5's constant cache (Apple10) is already fast — SMEM overhead dominates.
Opposite of M2 where it helped +4% on small models.
20 approaches tested, 4-mag LUT is the ceiling on both M2 and M5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Direct encode (no LUT, pure ALU) = same speed — proves constant memory is free.
Async prefetch = -2% regression — GPU already interleaves optimally.
Bottleneck is device memory bandwidth, not constant cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Half register LUT and threadgroup centroid cache both noise on M2 Pro.
Tested across 7B and 1.5B models with turbo3 and turbo4.
Revised theory: bottleneck is full WHT dequant pipeline, not just centroid lookup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Resurrection doc is about turbo4 journey. ISWA discovery is a cross-model
testing finding. Added "Why Cross-Model Testing Matters" section to
cross-model-validation.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added Section 10 to resurrection doc: how cross-model smoke testing
caught the ISWA bug that single-model testing never would have found.
PPL 13.7 trillion → 3.80 on Gemma 2. Updated daily note with full
investigation timeline and cross-model results table.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
QC precompute: 10.10 tok/s at 8K (same as SMEM, -33% vs 4-mag).
Select() chains are catastrophically expensive on Apple8.
Key discovery: the 4-mag LUT works because of ILP — interleaving
constant reads with dot product ALU lets the GPU overlap memory
and compute. Separating phases (SMEM, deferred norm, QC precompute)
destroys this overlap and runs at 2x latency.
The remaining 38% gap cannot be closed by rearranging reads and ALU.
Requires block format change or fundamentally different computation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Threadgroup memory pre-dequant tested on M2 Pro: -51% at 8K context.
SMEM store/load overhead exceeds constant cache savings when each
dequanted value is used exactly once by its producer thread.
4-mag LUT remains the dequant-level ceiling (62% of no-dequant ceiling).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
f16 passes 8K/100%, q8_0 and turbo4 both fail. Shared failure is
quantization noise on recently-written KV entries at end of context.
turbo4 recovers at 16K while q8_0 doesn't.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
