Resurrection doc is about turbo4 journey. ISWA discovery is a cross-model
testing finding. Added "Why Cross-Model Testing Matters" section to
cross-model-validation.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added Section 10 to resurrection doc: how cross-model smoke testing
caught the ISWA bug that single-model testing never would have found.
PPL 13.7 trillion → 3.80 on Gemma 2. Updated daily note with full
investigation timeline and cross-model results table.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
QC precompute: 10.10 tok/s at 8K (same as SMEM, -33% vs 4-mag).
Select() chains are catastrophically expensive on Apple8.
Key discovery: the 4-mag LUT works because of ILP — interleaving
constant reads with dot product ALU lets the GPU overlap memory
and compute. Separating phases (SMEM, deferred norm, QC precompute)
destroys this overlap and runs at 2x latency.
The remaining 38% gap cannot be closed by rearranging reads and ALU.
Requires block format change or fundamentally different computation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Threadgroup memory pre-dequant tested on M2 Pro: -51% at 8K context.
SMEM store/load overhead exceeds constant cache savings when each
dequanted value is used exactly once by its producer thread.
4-mag LUT remains the dequant-level ceiling (62% of no-dequant ceiling).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
f16 passes 8K/100%, q8_0 and turbo4 both fail. Shared failure is
quantization noise on recently-written KV entries at end of context.
turbo4 recovers at 16K while q8_0 doesn't.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q8_0 fails at 16K/30%, 8K/100%, 16K/100%.
turbo4 fails at 4K/0%, 8K/100%.
Shared failure at 8K/100% = model weakness, not quantization.
turbo4 has better retrieval than q8_0 baseline.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expanded from 9 to 33 tests (11 depths × 3 contexts).
Two consistent failures: 4K/0% and 8K/100% (same in both runs).
All other positions pass. 93.9% aggregate.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete turbo4 4-bit PolarQuant benchmark suite:
- PPL: +0.23% vs q8_0 (best after q8_0, better than q4_0)
- Prefill: 101-105% of q8_0 across 2K-32K
- Decode: 79.87 tok/s (93% q8_0, faster than turbo3)
- NIAH: 7/9 (matches q8_0)
- KLD: 0.0096 (40% lower than turbo3, matches q4_0 same-top-p)
- PDF 24K: 63.7 tok/s decode (20% faster than turbo3)
Also includes turbo4-resurrection.md paper documenting the full journey
from PPL 679 to beating q4_0 quality.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Multi-context PPL (8K/16K/32K): all 0.0000 delta ON/OFF
Prefill speed: matches q8_0 across all context lengths
Dense model: neutral (within noise)
Coverage matrix now complete for all 3 formats (q8_0, q4_0, turbo3)
across all test suite categories.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q4_0 tested across PPL, NIAH, decode — all neutral (0.0000 PPL delta,
identical NIAH, decode within noise). Added Section 7.4 (q4_0) and
Section 7.5 (cross-format summary) to paper. Updated conclusion with
format-independence finding.
Three formats validated: q8_0, q4_0, turbo3. No measurable impact on
any metric in any format. Attention-weight magnitude is a reliable proxy
for computational relevance, independent of KV representation.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q4_0 (7.0857) included as reference baseline alongside q8_0 (7.0638).
turbo3 (7.1796) uses fewer bits (3.5 vs 4.0), slightly higher PPL expected.
Sparse V delta still 0.0000. q4_0 note: no optimization applied, included
for comparison only.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
32K context, 50 chunks, CI ±0.021:
q8_0 baseline: 7.0638
turbo3 WITHOUT sparse V: 7.1796
turbo3 WITH sparse V: 7.1796
Sparse V delta: 0.0000
10x statistical power vs prior wikitext-2 runs. Updated paper Section 4.3,
long-context validation doc, and quality benchmarks. All raw logs saved
with exact commands, methodology, and corpus details.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Direct skip-rate measurement via output_attentions (9% at 512 → 28% at 4K)
- PPL validation with q8_0 baselines: 8K/20ch, 16K/10ch, 32K/5ch
- ON/OFF delta 0.0000 at all context lengths on clean branch
- turbo3 +1.1-1.4% vs q8_0, consistent and unaffected by sparse V
- Note: PR #4 was reverted (broke turbo3 PPL 6.18 → 181.6)
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Long-context PPL validation (512-32K) shows numerically identical results
with and without sparse V. Scoped all claims to "in this setup", replaced
absolute language with measured observations.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bit-for-bit identical PPL at 512, 8K, 16K, 32K with/without sparse V
on both MoE and dense. Even at 32K (~90% skip rate), no quality impact.
Includes skip-rate estimates, methodology notes, and raw commands.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
f16 path too cheap for branch to pay off. -0.3% short, +1.3% long.
Logged and closed.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MoE +0.6-0.9%, dense -0.8-1.1%. Tile max scan overhead exceeds savings.
Per-position sparse V remains the optimal approach.
Logged results and closed in optimization tracker.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MoE turbo3: KLD 0.016, 94.3% same top-p (2x q4_0, expected at 3.5 bits)
Dense turbo3: KLD 0.010, 96.0% same top-p
Added Section 4.4 to paper, filled quality-benchmarks.md tables,
created kl-divergence-results.md with full analysis.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sparse V on dense: +0.7% short, +1.3% 8K, +1.8% 16K. No downside.
Earlier -5.4% was system noise from stale processes (A/B/A confirmed).
Added Section 7.3 to paper + dense decode table to README.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Abstract: generality driven by attention distribution, not dequant mechanism
Section 1: explicit shift from instruction-level to operation elimination
Section 6: attention-aware computation paragraph
Section 7: strengthened opening reasoning
Section 9: broader implication grounded as observation, not overclaim
No new experiments, tables, or references.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Combined K+V dequant optimizations on M2 Pro: 62% improvement from
unoptimized baseline. Prefill blended at 90% of q8_0. Moved from
future work to results in paper Section 7.2.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPL 6.1109 both ways. NIAH 7/9 both ways (same failures).
Sparse V is a pure compute win on q8_0: +5% decode, zero quality impact.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
q8_0 with sparse V: 80.7 → 84.7 tok/s (+5.0%) on M5 Max.
Not TurboQuant-specific — any quantized KV cache benefits.
Added TURBO_SPARSE_V=0 disable override for A/B testing.
Raw logs included.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PPL identical across all 5 thresholds. Short-context decode within noise.
Confirms τ=1e-6 is safe with significant headroom to be more aggressive.
Raw benchmark logs included.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
70-page PDF via llama-server: 0.78x q8_0 decode at 24K context.
Documents the gap between synthetic bench (0.93x) and real server workloads.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3-line sparse V dequant yields +22.8% decode at 32K with zero quality
loss. NIAH 9/9 (100%), PPL identical. Documents 14 failed dequant
approaches and why sparse V succeeds where they don't.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LUT cost grows from 20% to 34% on M5 as context increases.
4-mag helps at 16K (+2.4%) but not 32K (-7.3%).
Context-adaptive dispatch planned.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
