Blackwell decode penalty drops from 25-40% to 7-9% with signalnine's
fused shmem mmvq kernel. 3.7-5.4x improvement. Prefill flips from
penalty to bonus. RTX 5090 now matches RTX 4090 penalty levels.
signalnine authored the kernel; TheTom collaborated on integration.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primoco diagnosed AmesianX's 30.8% F16 baseline as RoPE frequency
scaling from ctx_size < native context. At native 32768: all configs
(f16, tbq4_1, q8_0/tbq4_1) hit 100% accuracy. Zero cost from TBQ.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-user confirmation on Blackwell consumer hardware. All 6 K/V
combinations tested with philosophical, mathematical, and 23K-token
inputs. Originally hit corrupt output on Madreag's fork.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric tbqp3/tbq3 matches f16/f16 exactly (20/65, 30.8%) at temp=0.
Confirms no TBQ degradation on head_dim=128 models. Notable that
symmetric turbo3 works on Qwen3-14B Q4_K_M unlike Qwen2.5-7B Q4_K_M.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hannu Varjoranta's PR #1 on vibhavagarwal5/vllm changed default
value_quant_bits from 4 to 8 (FP8), fixing 0% gsm8k. Cites @TheTom
as source for "value precision is the quality bottleneck." Independent
vLLM validation of asymmetric thesis and turbo4-resurrection findings.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Independent validation of asymmetric KV compression on Ada (SM 89)
and Blackwell (SM 120). Asymmetric q8_0-K/turbo4-V confirmed best
config on Ada (+8.4% prefill, -6.2% decode). Blackwell shows larger
decode regression (-25.2%), likely dp4a vs fp8/fp4 tensor core mismatch.
LA=1 boundary layers improve decode speed in addition to quality.
Source: https://github.com/ggml-org/llama.cpp/discussions/20969
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I on Gemma 4 26B MoE: -2.3% size, neutral/better PPL, 100% speed.
Minimal weight compression benefit on MoE (few dense layers). KV compression
is the real win for this architecture.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new community reports:
- theaaronhughes: RTX 5090 Gemma 4 26B KV at 64k/128k
- tryingET: RTX PRO 6000 Config I+turbo3 beats Q8_0+turbo3 by 7% decode
- pjsgsy: RTX 3050 APEX-I-Quality pre-quantized models with CPU offload
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First Volta (V100, SM 7.0) validation: Config I +8.7% faster decode.
First Pascal (1080 Ti, SM 6.1) validation: Config I +6-11% faster.
Config I is now faster than Q8_0 baseline on every CUDA architecture
tested (Pascal, Volta, Ada, Blackwell).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vilenkin basis decomposition on Phi-2 suggests K and V occupy disjoint
spectral bands, providing potential mathematical basis for asymmetric
compression. Caveat: single model, needs validation across architectures.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I 30% faster than Q8_0 decode on RDNA 4. First successful AMD
HIP weight compression result. PPL +1.8%.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
14-config Metal benchmark independently confirms asymmetric K/V as
default recommendation. BitNet compound case is new finding. NIAH
survives catastrophic PPL (retrieval degrades independently).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@tryingET: Qwen3.5-27B, 96GB VRAM. Config I is faster than Q8_0 on
Blackwell Workstation (+11.9% prefill, +8.5% decode). PPL +0.048%.
Config I + turbo3 beats Q8_0 + turbo3/turbo4 at 32K.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vLLM TurboQuant integration explicitly cites turboquant_plus for
boundary layers and asymmetric KV. Community ablations on A100/H100
confirm our findings.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Weight compression is now on feature/turboquant-kv-cache (main).
No separate branch needed. PR #45 remains open for community logs.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 4.4: TurboQuantDC (662 tests, MIT) independently confirmed
stacking, boundary layer protection, and QJL removal on RTX 4090.
Links to community results page for full hardware coverage.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12+ GPUs, 11+ models, 6 families, 10+ independent testers. All data
from PR #45 community testing. Includes GPU compatibility matrix,
model results by family, stacking validation, decode speed comparison,
known issues, confidence ratings, and contribution template.
Linked from getting-started.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
19-version kernel optimization log proves CUDA TQ4_1S decode gap is
fundamental (float32 bandwidth + FMA density), not an optimization gap.
Q4_K_M is the practical CUDA runtime answer. TQ4_1S optimal for Metal,
storage/distribution format on CUDA.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
clang on Windows does not inherit CMake Release flags. Without -O3 -DNDEBUG
the build runs ~50x slower. Tested on RX 9070 XT (gfx1201, RDNA 4).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add TQ3_4S mention and reframe contribution to include follow-on work.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- asymmetric paper: correct Sjoerd's q8_0/turbo3 results (corrupt output
on head_dim=256, speed data accurate but content garbage)
- recommendations: add MTMD_BACKEND_DEVICE=CUDA1 for multi-GPU vision
- community count: 30+ to 50+ testers
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CUDA port now available via signalnine (PR #45)
- Add CUDA GPUs to tested hardware (1080 Ti, RTX 5090)
- Replace "Metal only" limitation with CUDA decode gap analysis
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update backend support: Metal + CUDA (was Metal only)
- Add copy-paste Quick Test section: clone, build, compress, bench
- Recommend Qwen3.5-27B as test model (known good, fits 32GB)
- Explicit instructions to paste results back in PR #45
- Note Windows MSVC support
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Asymmetric paper: Sjoerd 122B on 2x L40S at 82K. q8_0/turbo3 100%
decode recovery. q8_0/turbo2 also 100% recovery (V free to 2-bit).
- MoE paper: turbo2 no-crossover at 82K on hybrid MoE (12 attn layers),
architecture-dependent finding.
- KLD results: new context-length scaling data (2K/4K/8K on M2 Pro).
Mean KLD stable/improving with context, does not drift.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Models with head_dim=64 (e.g., GPT-OSS-120B) may crash or produce
degraded output with turbo compression. WHT requires sufficient
dimensionality for CLT convergence. Reported by NigelTufnel12345.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
The turbo4-K PPL explosions were caused by kernel bugs (our fork) and
head_dim misdetection (AmesianX fork), not an IQ4_XS-specific issue.
AmesianX independently confirmed turbo4-K works correctly after fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
- Add compression reference table and config warning to getting-started.md
- Add independent validation from sztlink, Madreag, and mudler (APEX/LocalAI) across asymmetric, layer-aware, MoE, and sparse V papers
- Update sparse-v-dequant.md with Madreag 4-GPU cross-validation
- Minor README wording fix (branch -> repository)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
New paper: q8_0/turbo2 with Boundary V on Qwen3.5-35B MoE achieves
7.53x V compression with PPL within 1% of q8_0 at 512c/8K/32K.
Quality matches or beats q8_0/turbo3 at every tested length. 32K
decode ~2-3% faster than turbo3-V (n=2, M5 Max). Scoped to one
tested MoE model. Includes decode investigation negative results
(20+ approaches exhausted) and dense-vs-MoE bottleneck analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Replace parquet link with direct raw text download. Add HF CLI as
alternative. Credit: @henkiespenkie44
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Users were building from main instead of feature/turboquant-kv-cache,
resulting in "Unsupported cache type: turbo4" errors.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Corrections from verification review:
- Fix HyperionMS2040 dates (03-30 not 03-31) and numbers (Qwen2.5
symmetric was catastrophic PPL 3472, not +3.6%)
- Fix Aaryan-Kapoor attribution (was TheTom's words, not his)
- Remove seanrasch block_size=128 claim (he reported a crash, not
confirmation). Replace with HyperionMS2040 PR #32 fix
- Fix Arclabs001 framing (QJL works with independent rotations,
not "QJL hurts")
- Reword elusznik (scoped out QJL for PR size, not as validation)
- Fix Madreag date (03-26), AmesianX dates (03-30), dusterbloom
dates (03-30)
- Mark AMD HIP as author's own testing, not independent validation
- Mark treblewoe and Corianas_ as X collaborators
- Mark spiritbuun prefill claim as X collaborator with approximate date
- Add jesusmb1995 Vulkan mixed K/V data to asymmetric paper
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each paper now documents who confirmed the findings, on what hardware,
with what data, and when. Validations span Metal (M5 Max), CUDA (RTX
3090, 4090, 5090, Blackwell), and AMD HIP (RX 9070 XT).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Why V compression is free and K precision is everything. Validated
across 7 models (1.5B-104B), 3 weight quants, 3 GPU backends, 2
Apple Silicon generations. Independently confirmed by sztlink (RTX
4090), HyperionMS2040 (RTX 3090), and AMD HIP testing.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document built-in optimizations: Sparse V dequant (Metal, auto-enabled),
Boundary V (auto-enabled with turbo2-V), and maximum V compression config.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Standalone doc with build instructions (Metal, CUDA, HIP, Windows),
config selection table, benchmarking steps, and how to share results.
Community requested.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community feedback: setting iogpu.wired_limit_mb above 90% of physical
RAM risks kernel panics under sustained inference load. 90% is now the
recommended safe default. Credit: @treblewoe
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Isolation experiments confirmed GGML_METAL_NO_RESIDENCY=1 has no
measurable effect. The sysctl alone fixes the 49K+ context hang.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
