- Asymmetric paper: Sjoerd 122B on 2x L40S at 82K. q8_0/turbo3 100%
decode recovery. q8_0/turbo2 also 100% recovery (V free to 2-bit).
- MoE paper: turbo2 no-crossover at 82K on hybrid MoE (12 attn layers),
architecture-dependent finding.
- KLD results: new context-length scaling data (2K/4K/8K on M2 Pro).
Mean KLD stable/improving with context, does not drift.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
The turbo4-K PPL explosions were caused by kernel bugs (our fork) and
head_dim misdetection (AmesianX fork), not an IQ4_XS-specific issue.
AmesianX independently confirmed turbo4-K works correctly after fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Add compression reference table and config warning to getting-started.md
- Add independent validation from sztlink, Madreag, and mudler (APEX/LocalAI) across asymmetric, layer-aware, MoE, and sparse V papers
- Update sparse-v-dequant.md with Madreag 4-GPU cross-validation
- Minor README wording fix (branch -> repository)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
New paper: q8_0/turbo2 with Boundary V on Qwen3.5-35B MoE achieves
7.53x V compression with PPL within 1% of q8_0 at 512c/8K/32K.
Quality matches or beats q8_0/turbo3 at every tested length. 32K
decode ~2-3% faster than turbo3-V (n=2, M5 Max). Scoped to one
tested MoE model. Includes decode investigation negative results
(20+ approaches exhausted) and dense-vs-MoE bottleneck analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Corrections from verification review:
- Fix HyperionMS2040 dates (03-30 not 03-31) and numbers (Qwen2.5
symmetric was catastrophic PPL 3472, not +3.6%)
- Fix Aaryan-Kapoor attribution (was TheTom's words, not his)
- Remove seanrasch block_size=128 claim (he reported a crash, not
confirmation). Replace with HyperionMS2040 PR #32 fix
- Fix Arclabs001 framing (QJL works with independent rotations,
not "QJL hurts")
- Reword elusznik (scoped out QJL for PR size, not as validation)
- Fix Madreag date (03-26), AmesianX dates (03-30), dusterbloom
dates (03-30)
- Mark AMD HIP as author's own testing, not independent validation
- Mark treblewoe and Corianas_ as X collaborators
- Mark spiritbuun prefill claim as X collaborator with approximate date
- Add jesusmb1995 Vulkan mixed K/V data to asymmetric paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each paper now documents who confirmed the findings, on what hardware,
with what data, and when. Validations span Metal (M5 Max), CUDA (RTX
3090, 4090, 5090, Blackwell), and AMD HIP (RX 9070 XT).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Why V compression is free and K precision is everything. Validated
across 7 models (1.5B-104B), 3 weight quants, 3 GPU backends, 2
Apple Silicon generations. Independently confirmed by sztlink (RTX
4090), HyperionMS2040 (RTX 3090), and AMD HIP testing.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community feedback: setting iogpu.wired_limit_mb above 90% of physical
RAM risks kernel panics under sustained inference load. 90% is now the
recommended safe default. Credit: @treblewoe
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
104B Q4_K_M on M5 Max 128GB: turbo3/turbo3 +3.6% PPL, turbo4/turbo4
+1.9%. 128K full native context achieved (PPL 4.024) with Metal
settings iogpu.wired_limit_mb + GGML_METAL_NO_RESIDENCY. NIAH 10/10
at 4K/8K. turbo3 prefill faster than q8_0 at 32K. Context wall fix
confirmed working at 48K/64K/96K/128K. Biggest model tested.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
RotorQuant/IsoQuant modules (PR #34, not merged) and draft docs
were unintentionally bundled into the rename commit. Fixes CI
coverage failure (64% -> 95%+).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
m5-max-stress-test.md — generic name for ongoing large model validation.
Will add Command-R+ 104B and future models.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Llama-70B Q4_K_M results (symmetric turbo works, asymmetric rescue confirmed)
- Add CUDA community data (seanrasch, dusterbloom, Char__Bob)
- Add AMD RDNA 4 tested configs
- Update Boundary V from experimental to auto-enabled for turbo2-V
- Note Sparse V as default on all Metal
- Update block size section to reflect 128 as shipped default
- Update backend scope from Metal-only to multi-backend validated
- Add 70B M5 Max stress test paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Block_size=128 achieves 5.12x compression (vs 4.57x at 32) with
identical PPL across all tested Metal paths: asymmetric q8_0-K +
turbo{2,3}-V, symmetric turbo3/turbo3, and Boundary V (LA-V7).
Validated on 3 architectures (dense, Qwen, hybrid MoE), 3 context
lengths (512, 8K, 32K), and 2 hardware platforms (M5 Max, M2 Pro).
M2 Pro showed +3-7% decode improvement on tested 1.5B model.
No speed regression on M5 Max.
Also fixes pre-existing flaky d=192 turbo4 test threshold (0.6 -> 0.7).
Credit to @AmesianX whose block_size=256 implementation prompted
the investigation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tested f16 vs q8_0 boundary layers at 2 and 4 layer widths.
Finding: width matters more than precision. Wide q8_0 (first4+last4)
beats narrow f16 (first2+last2) on both phi-4 and Qwen3.5-27B.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New: docs/papers/layer-aware-v-compression.md — full research writeup
- Updated: docs/turboquant-recommendations.md — Boundary V as advanced experimental option
- New: proof/niah/niah_boundary_v/ — NIAH retrieval evidence
Boundary V protects first 2 + last 2 layers with q8_0-V while compressing
remaining layers with turbo2-V. Quality between turbo3 and turbo2, effective
compression between the two. Validated across 4 models on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Resurrection doc is about turbo4 journey. ISWA discovery is a cross-model
testing finding. Added "Why Cross-Model Testing Matters" section to
cross-model-validation.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added Section 10 to resurrection doc: how cross-model smoke testing
caught the ISWA bug that single-model testing never would have found.
PPL 13.7 trillion → 3.80 on Gemma 2. Updated daily note with full
investigation timeline and cross-model results table.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q8_0 fails at 16K/30%, 8K/100%, 16K/100%.
turbo4 fails at 4K/0%, 8K/100%.
Shared failure at 8K/100% = model weakness, not quantization.
turbo4 has better retrieval than q8_0 baseline.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete turbo4 4-bit PolarQuant benchmark suite:
- PPL: +0.23% vs q8_0 (best after q8_0, better than q4_0)
- Prefill: 101-105% of q8_0 across 2K-32K
- Decode: 79.87 tok/s (93% q8_0, faster than turbo3)
- NIAH: 7/9 (matches q8_0)
- KLD: 0.0096 (40% lower than turbo3, matches q4_0 same-top-p)
- PDF 24K: 63.7 tok/s decode (20% faster than turbo3)
Also includes turbo4-resurrection.md paper documenting the full journey
from PPL 679 to beating q4_0 quality.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q4_0 tested across PPL, NIAH, decode — all neutral (0.0000 PPL delta,
identical NIAH, decode within noise). Added Section 7.4 (q4_0) and
Section 7.5 (cross-format summary) to paper. Updated conclusion with
format-independence finding.
Three formats validated: q8_0, q4_0, turbo3. No measurable impact on
any metric in any format. Attention-weight magnitude is a reliable proxy
for computational relevance, independent of KV representation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q4_0 (7.0857) included as reference baseline alongside q8_0 (7.0638).
turbo3 (7.1796) uses fewer bits (3.5 vs 4.0), slightly higher PPL expected.
Sparse V delta still 0.0000. q4_0 note: no optimization applied, included
for comparison only.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
32K context, 50 chunks, CI ±0.021:
q8_0 baseline: 7.0638
turbo3 WITHOUT sparse V: 7.1796
turbo3 WITH sparse V: 7.1796
Sparse V delta: 0.0000
10x statistical power vs prior wikitext-2 runs. Updated paper Section 4.3,
long-context validation doc, and quality benchmarks. All raw logs saved
with exact commands, methodology, and corpus details.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Direct skip-rate measurement via output_attentions (9% at 512 → 28% at 4K)
- PPL validation with q8_0 baselines: 8K/20ch, 16K/10ch, 32K/5ch
- ON/OFF delta 0.0000 at all context lengths on clean branch
- turbo3 +1.1-1.4% vs q8_0, consistent and unaffected by sparse V
- Note: PR #4 was reverted (broke turbo3 PPL 6.18 → 181.6)
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Long-context PPL validation (512-32K) shows numerically identical results
with and without sparse V. Scoped all claims to "in this setup", replaced
absolute language with measured observations.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MoE turbo3: KLD 0.016, 94.3% same top-p (2x q4_0, expected at 3.5 bits)
Dense turbo3: KLD 0.010, 96.0% same top-p
Added Section 4.4 to paper, filled quality-benchmarks.md tables,
created kl-divergence-results.md with full analysis.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sparse V on dense: +0.7% short, +1.3% 8K, +1.8% 16K. No downside.
Earlier -5.4% was system noise from stale processes (A/B/A confirmed).
Added Section 7.3 to paper + dense decode table to README.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Abstract: generality driven by attention distribution, not dequant mechanism
Section 1: explicit shift from instruction-level to operation elimination
Section 6: attention-aware computation paragraph
Section 7: strengthened opening reasoning
Section 9: broader implication grounded as observation, not overclaim
No new experiments, tables, or references.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combined K+V dequant optimizations on M2 Pro: 62% improvement from
unoptimized baseline. Prefill blended at 90% of q8_0. Moved from
future work to results in paper Section 7.2.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPL 6.1109 both ways. NIAH 7/9 both ways (same failures).
Sparse V is a pure compute win on q8_0: +5% decode, zero quality impact.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q8_0 with sparse V: 80.7 → 84.7 tok/s (+5.0%) on M5 Max.
Not TurboQuant-specific — any quantized KV cache benefits.
Added TURBO_SPARSE_V=0 disable override for A/B testing.
Raw logs included.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPL identical across all 5 thresholds. Short-context decode within noise.
Confirms τ=1e-6 is safe with significant headroom to be more aggressive.
Raw benchmark logs included.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
70-page PDF via llama-server: 0.78x q8_0 decode at 24K context.
Documents the gap between synthetic bench (0.93x) and real server workloads.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3-line sparse V dequant yields +22.8% decode at 32K with zero quality
loss. NIAH 9/9 (100%), PPL identical. Documents 14 failed dequant
approaches and why sparse V succeeds where they don't.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
