Sprint journal documenting the in-progress block-selector sparse
attention work on Apple Silicon: companion to sparse-v-dequant.md
(post-softmax) covering the pre-attention K-skip lever.
Includes the preliminary local M5 Max bench grid (1.16-1.73× decode
across B=4-8 × ctx=16K-48K on Qwen2.5-14B-1M-4bit), the Apple-Silicon-
specific findings (MLX_SDPA_BLOCKS heuristic mis-tune for the float-mask
kernel variant, pass-2 reducer modulo bug, MetalAllocator non-determinism
cliff), the failed-paths catalogue (custom Metal kernel 14× slower at B=8,
compose-gather 13.7× slower, bool mask 25% regression), and the
per-family port pattern across 10 model families.
Framed as a sprint journal — the work is not yet merged to upstream
alpha (PRs ekryski/mlx#34, ekryski/mlx-swift#36, ekryski/mlx-swift-lm#224,
ekryski/mlx-swift-lm#225 all open). Numbers are local M5 Max only;
recommendations are current best guess rather than settled defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Two new pieces of paper-form documentation covering longctx's two
operating modes and the V3 rescue fix.
NEW: docs/papers/longctx-1m-and-triattention.md
- Long-form paper on longctx — open long-context retrieval companion
for inference engines. Two modes: (1) standalone retrieval scored
on MRCR v2 8-needle on AMD MI300X with Qwen2.5-32B-Instruct, (2)
rescue layer for our TriAttention V3 eviction policy on Apple
Silicon, validated end-to-end via 256K planted-fact NIAH on
Qwen3.5-2B-4bit.
- Headline numbers: MRCR v2 1M MultiQ Selector + bge-rerank + det
copy n=30 directional 0.688 (above SubQ 0.659; mass-val pending).
V3+longctx ✓HIT every rung 32K → 256K via ChatSession;
V3-only ✗miss every rung.
- Architecture diagrams: full-stack (clients → vllm-swift →
ChatSession → mlx-swift-lm → V3 engine + rescue → longctx-svc),
V3 turn-by-turn flow (eviction-write on turn N, rehydrate-retrieve
before turn N+1).
- Appendices: module map (mlx-swift-lm side), longctx-svc endpoint
surface, V3 environment variable reference table.
- Cites Mao et al., "TriAttention: Efficient Long Reasoning with
Trigonometric KV Compression" (arXiv:2604.04921, 2026) as the
original paper. V3 in this stack is our independent Swift port +
hybrid extension; cross-references docs/papers/triattention-v3.md.
UPDATED: docs/papers/triattention-v3.md
- Adds §10 "Addendum: Swift Port + longctx Rescue — Fixing the
Hybrid NIAH Failure" after the §9 vLLM addendum.
- Reports the Apple Silicon Swift port test results that resolve the
Section 7.1 question "why does V3 break on hybrid Mamba+Attention
models?". Same failure mode reproduces on Swift port; longctx
rescue layer flips it.
- Includes the 12-cell receipt table, the V3 turn-by-turn diagram,
the driver-gotcha section explaining why bare container.generate()
doesn't rescue (Tier-3 rehydrate is ChatSession-only per commit
fe1a3b0 in mlx-swift-lm), and what's still not measured (>256K,
larger hybrids, V3+TQ+ stack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed scripts/eden-investigation/ as a public artifact
pointer, but the directory contents are scratch and don't belong in the public
repo. Methodology in §2 is sufficient for reproduction; the file pointer was
a leftover from cleaning up the earlier /tmp/ paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed personal scratch paths under /tmp/eden-investigation/
that aren't useful to public readers and look confusing. Replaces those plus the
two private Obsidian-note pointers with a single line pointing to the
scripts/eden-investigation/ directory in this repo, where the actual experiment
scripts live (h*.py, F1–F3 KL test, universal-K candidate centroids).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two independent third-party confirmations to the appendices:
asymmetric-kv-compression.md: @jagmarques tested asymmetric K/V on
Mistral-7B with E8 lattice VQ (a different quantization family from
TurboQuant) on Cerebrium A10/A100. Confirms asymmetric advantage scales
with context length: at 1742-tok prefix K3V2 +0.82% vs K2V2 +0.91%; at
3544-tok prefix K3V2 +0.35% vs K2V2 +2.26% (6.4× improvement). Clean
GQA-amplification explanation: 8 query heads per K head means K
quantization error is effectively 8× worse than V error in output
impact. Cross-method validation that the asymmetric conclusion is not
TurboQuant-specific.
turbo4-resurrection.md: @unamedkr / quantumaikr (quant.cpp) ported the
TurboQuant paper end-to-end in single-header C and found the QJL stage
contributed byte-identical zero to attention scores regardless of
input. Disabling QJL produced bit-for-bit identical perplexity.
Theoretical explanation: the √(π/2)/m constant in the unbiased
estimator is derived for Gaussian projection rows; on Rademacher (±1)
rows the per-key contribution gets dwarfed by residual-norm scaling.
Independently took the freed bits and reinvested in 4-bit codebook
(8 → 16 centroids), same path as turbo4 resurrection. Llama 3.2 3B:
turbo_kv_4b PPL 14.28 (+5.3% vs FP32), beating llama.cpp q4_0 KV
(~14.99, +10.6%). Targets embedded/edge (iOS, Android, WASM,
microcontrollers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the empirical study covering the broader 35-hypothesis investigation
spawned by the EDEN/Mitzenmacher/Portnoy critique (#87, #89). Companion
to eden-optimal-s-revisit.md.
Headline result: a K-cache centroid table that improves per-vector
reconstruction MSE by 1–13% across five model families (Qwen3-0.6B,
Llama-3.2-1B, Mistral-7B, Phi-3-mini, Gemma-2-2b) causes 70–90% mean
KL@D regressions and 50–60% catastrophic-rate failures at the model
output. This is a sign inversion, not a small mismatch: methods that
are strictly better under MSE are systematically worse under KL@D in
deployment.
The paper is structured in three acts:
- Act 1: K cache is sub-Gaussian post-WHT (H8); V is Gaussian
- Act 2: per-prompt fitting fails (PDS), ensemble (ens_4way) reaches an
irreducible ~8% catastrophic floor at scale (N=100)
- Act 3: the universal-K cross-model counterexample (F3) demonstrates
the MSE→KL inversion at the cleanest possible level
Includes a 2-D toy attention illustration of the softmax bucket-flip
mechanism, an explicit decision boundary for when MSE proxies quality
(linear consumers) vs when it doesn't (attention softmax), an
ablation summary distinguishing first-order from second-order factors,
operational guidance for practitioners, and a "why this wasn't caught
earlier" diagnosis of the methodology blind spots.
Companion paper handles the algorithmic claim head-on. This paper is
the broader evaluation-methodology critique.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a short paragraph in Appendix A documenting the algebraic identity
sqrt(S_unb * S_bias) = ||x||/||c||, which coincides with the matched-norm
scale. Frames it as an algebraic interpolation between the two
EDEN-derived forms, notes it is not derived in DRIVE, EDEN, or the
TurboQuant/EDEN note paper (arXiv:2604.18555), and does not correspond
to an optimization objective in those works.
The empirical head-to-head numbers in the table directly above the new
paragraph already show that each form attains the optimum of its own
objective and only its own; matched-norm behaves as a distinct operating
point, not as a tuning of either EDEN scale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The eden.py default formula S = ||x||²/⟨c,x⟩ was loosely labeled as
"biased optimal-S" throughout. On closer inspection that's imprecise:
the strict per-vector MSE-minimum is S* = ⟨c,x⟩/||c||² (textbook
least-squares; asymptotically → 1 for Lloyd-Max optimal centroids).
The eden.py default is a different formula, satisfying the inner-
product-preserving identity ⟨Sc, x⟩ = ||x||² per vector.
Empirical comparisons unaffected (we tested what eden.py ships).
Just the labeling. Added Terminology note in §4 with worked example
and credit to @amitport for surfacing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical response to issues #87 (Mitzenmacher) and #89 (Portnoy) on
turboquant_plus. 8 hypothesis tests across synthetic Gaussian and
Qwen3-0.6B + Llama-3.2-1B real KV cache.
Headline: EDEN's optimal scale is real but second-order. Rotation choice
(WHT vs dense Haar) is first-order and accounts for ~all of the published
gap. Production already uses the first-order fix. On real KV with WHT,
matched-norm beats EDEN-S by 0.5-9% across Qwen and Llama; tightens to
0.67-0.72% on Llama. EDEN-S genuinely wins at b=8 on synthetic Gaussian
(70%+ MSE reduction) — credibility moat showing the lever exists in its
own regime.
Concedes attribution argument (#89): DRIVE/EDEN line is real prior art.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the 3-engine apples-to-apples cross-engine benchmark:
- BF16 baseline (load, PPL, prefill, decode, KV size) across vLLM, SGLang, llama.cpp
- REFRACT --full 4-axis fidelity scoring (gtm, kld, rniah, plad) on each engine's native 8-bit KV
- Headline finding: same fp8_e4m3 dtype string produces 1.8x different KL drift between vLLM and SGLang on the same model
- 8 engine-side bugs encountered and worked around (vLLM max_num_seqs, prompt_logprobs cap, GPU mem carryover; SGLang aiter.dtypes, AITER fp8 prefill broken, KV dtype fixed at server launch)
- Full reproducibility scripts and REFRACT framework changes documented
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Adds Section 9 to the V3 paper documenting the port to vLLM ROCm Triton
on AMD MI300X. New subsections:
9.1 Hardware (MI300X / ROCm 7.2 / Triton 3.6 / vLLM 0.1.dev1)
9.2 Implementation summary (files added, three vLLM-specific design
choices: custom-op Q capture, worker-side env-var lazy init,
mask-based eviction without compaction)
9.3 Models tested (7-row table: validated, partially validated,
blocked, algorithmically incompatible)
9.4 Paper-exact reproduction on Qwen2.5-7B-Instruct
9.4.1 PPL: vLLM deltas all within ±0.5% noise floor
9.4.2 KV cache compression: stack hits 2.96x at 90% retention,
matching the paper's "approximately 2.9x" headline
9.4.3 NIAH: V3 + stack regress vs paper at the middle position
(PARTIAL_NUMBER instead of PASS). Real gap.
9.5 Where the vLLM port falls short of the paper (3 plausible causes
ordered by likelihood: KV preset, scoring numerics, eviction
trigger frequency)
9.6 Hybrid models still untested in vLLM (3 attempts, 3 different
blockers: transformers compat, no-RoPE incompat, vLLM page-size
unifier)
9.7 Headline claim envelope - what I can and cannot claim post-port
9.8 Open work
9.9 How to Run (paper-exact PPL + NIAH commands, tunable knob table,
cross-vendor note, doc pointers)
Honest framing throughout: PPL reproduces within noise, NIAH degrades
vs paper at middle position. Not a refutation of the paper, not a
clean confirmation. The original V3 paper's point that PPL alone is
insufficient signal for V3 quality (Section 3.2) is reinforced
empirically by the vLLM port.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Blackwell decode penalty drops from 25-40% to 7-9% with signalnine's
fused shmem mmvq kernel. 3.7-5.4x improvement. Prefill flips from
penalty to bonus. RTX 5090 now matches RTX 4090 penalty levels.
signalnine authored the kernel; TheTom collaborated on integration.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primoco diagnosed AmesianX's 30.8% F16 baseline as RoPE frequency
scaling from ctx_size < native context. At native 32768: all configs
(f16, tbq4_1, q8_0/tbq4_1) hit 100% accuracy. Zero cost from TBQ.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-user confirmation on Blackwell consumer hardware. All 6 K/V
combinations tested with philosophical, mathematical, and 23K-token
inputs. Originally hit corrupt output on Madreag's fork.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric tbqp3/tbq3 matches f16/f16 exactly (20/65, 30.8%) at temp=0.
Confirms no TBQ degradation on head_dim=128 models. Notable that
symmetric turbo3 works on Qwen3-14B Q4_K_M unlike Qwen2.5-7B Q4_K_M.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hannu Varjoranta's PR #1 on vibhavagarwal5/vllm changed default
value_quant_bits from 4 to 8 (FP8), fixing 0% gsm8k. Cites @TheTom
as source for "value precision is the quality bottleneck." Independent
vLLM validation of asymmetric thesis and turbo4-resurrection findings.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Independent validation of asymmetric KV compression on Ada (SM 89)
and Blackwell (SM 120). Asymmetric q8_0-K/turbo4-V confirmed best
config on Ada (+8.4% prefill, -6.2% decode). Blackwell shows larger
decode regression (-25.2%), likely dp4a vs fp8/fp4 tensor core mismatch.
LA=1 boundary layers improve decode speed in addition to quality.
Source: https://github.com/ggml-org/llama.cpp/discussions/20969
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vilenkin basis decomposition on Phi-2 suggests K and V occupy disjoint
spectral bands, providing potential mathematical basis for asymmetric
compression. Caveat: single model, needs validation across architectures.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
14-config Metal benchmark independently confirms asymmetric K/V as
default recommendation. BitNet compound case is new finding. NIAH
survives catastrophic PPL (retrieval degrades independently).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vLLM TurboQuant integration explicitly cites turboquant_plus for
boundary layers and asymmetric KV. Community ablations on A100/H100
confirm our findings.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Weight compression is now on feature/turboquant-kv-cache (main).
No separate branch needed. PR #45 remains open for community logs.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 4.4: TurboQuantDC (662 tests, MIT) independently confirmed
stacking, boundary layer protection, and QJL removal on RTX 4090.
Links to community results page for full hardware coverage.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
19-version kernel optimization log proves CUDA TQ4_1S decode gap is
fundamental (float32 bandwidth + FMA density), not an optimization gap.
Q4_K_M is the practical CUDA runtime answer. TQ4_1S optimal for Metal,
storage/distribution format on CUDA.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add TQ3_4S mention and reframe contribution to include follow-on work.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- asymmetric paper: correct Sjoerd's q8_0/turbo3 results (corrupt output
on head_dim=256, speed data accurate but content garbage)
- recommendations: add MTMD_BACKEND_DEVICE=CUDA1 for multi-GPU vision
- community count: 30+ to 50+ testers
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CUDA port now available via signalnine (PR #45)
- Add CUDA GPUs to tested hardware (1080 Ti, RTX 5090)
- Replace "Metal only" limitation with CUDA decode gap analysis
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Asymmetric paper: Sjoerd 122B on 2x L40S at 82K. q8_0/turbo3 100%
decode recovery. q8_0/turbo2 also 100% recovery (V free to 2-bit).
- MoE paper: turbo2 no-crossover at 82K on hybrid MoE (12 attn layers),
architecture-dependent finding.
- KLD results: new context-length scaling data (2K/4K/8K on M2 Pro).
Mean KLD stable/improving with context, does not drift.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
The turbo4-K PPL explosions were caused by kernel bugs (our fork) and
head_dim misdetection (AmesianX fork), not an IQ4_XS-specific issue.
AmesianX independently confirmed turbo4-K works correctly after fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Add compression reference table and config warning to getting-started.md
- Add independent validation from sztlink, Madreag, and mudler (APEX/LocalAI) across asymmetric, layer-aware, MoE, and sparse V papers
- Update sparse-v-dequant.md with Madreag 4-GPU cross-validation
- Minor README wording fix (branch -> repository)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
New paper: q8_0/turbo2 with Boundary V on Qwen3.5-35B MoE achieves
7.53x V compression with PPL within 1% of q8_0 at 512c/8K/32K.
Quality matches or beats q8_0/turbo3 at every tested length. 32K
decode ~2-3% faster than turbo3-V (n=2, M5 Max). Scoped to one
tested MoE model. Includes decode investigation negative results
(20+ approaches exhausted) and dense-vs-MoE bottleneck analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Corrections from verification review:
- Fix HyperionMS2040 dates (03-30 not 03-31) and numbers (Qwen2.5
symmetric was catastrophic PPL 3472, not +3.6%)
- Fix Aaryan-Kapoor attribution (was TheTom's words, not his)
- Remove seanrasch block_size=128 claim (he reported a crash, not
confirmation). Replace with HyperionMS2040 PR #32 fix
- Fix Arclabs001 framing (QJL works with independent rotations,
not "QJL hurts")
- Reword elusznik (scoped out QJL for PR size, not as validation)
- Fix Madreag date (03-26), AmesianX dates (03-30), dusterbloom
dates (03-30)
- Mark AMD HIP as author's own testing, not independent validation
- Mark treblewoe and Corianas_ as X collaborators
- Mark spiritbuun prefill claim as X collaborator with approximate date
- Add jesusmb1995 Vulkan mixed K/V data to asymmetric paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each paper now documents who confirmed the findings, on what hardware,
with what data, and when. Validations span Metal (M5 Max), CUDA (RTX
3090, 4090, 5090, Blackwell), and AMD HIP (RX 9070 XT).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Why V compression is free and K precision is everything. Validated
across 7 models (1.5B-104B), 3 weight quants, 3 GPU backends, 2
Apple Silicon generations. Independently confirmed by sztlink (RTX
4090), HyperionMS2040 (RTX 3090), and AMD HIP testing.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community feedback: setting iogpu.wired_limit_mb above 90% of physical
RAM risks kernel panics under sustained inference load. 90% is now the
recommended safe default. Credit: @treblewoe
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
104B Q4_K_M on M5 Max 128GB: turbo3/turbo3 +3.6% PPL, turbo4/turbo4
+1.9%. 128K full native context achieved (PPL 4.024) with Metal
settings iogpu.wired_limit_mb + GGML_METAL_NO_RESIDENCY. NIAH 10/10
at 4K/8K. turbo3 prefill faster than q8_0 at 32K. Context wall fix
confirmed working at 48K/64K/96K/128K. Biggest model tested.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
RotorQuant/IsoQuant modules (PR #34, not merged) and draft docs
were unintentionally bundled into the rename commit. Fixes CI
coverage failure (64% -> 95%+).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
m5-max-stress-test.md — generic name for ongoing large model validation.
Will add Command-R+ 104B and future models.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Llama-70B Q4_K_M results (symmetric turbo works, asymmetric rescue confirmed)
- Add CUDA community data (seanrasch, dusterbloom, Char__Bob)
- Add AMD RDNA 4 tested configs
- Update Boundary V from experimental to auto-enabled for turbo2-V
- Note Sparse V as default on all Metal
- Update block size section to reflect 128 as shipped default
- Update backend scope from Metal-only to multi-backend validated
- Add 70B M5 Max stress test paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
