Sprint journal documenting the in-progress block-selector sparse
attention work on Apple Silicon: companion to sparse-v-dequant.md
(post-softmax) covering the pre-attention K-skip lever.
Includes the preliminary local M5 Max bench grid (1.16-1.73× decode
across B=4-8 × ctx=16K-48K on Qwen2.5-14B-1M-4bit), the Apple-Silicon-
specific findings (MLX_SDPA_BLOCKS heuristic mis-tune for the float-mask
kernel variant, pass-2 reducer modulo bug, MetalAllocator non-determinism
cliff), the failed-paths catalogue (custom Metal kernel 14× slower at B=8,
compose-gather 13.7× slower, bool mask 25% regression), and the
per-family port pattern across 10 model families.
Framed as a sprint journal — the work is not yet merged to upstream
alpha (PRs ekryski/mlx#34, ekryski/mlx-swift#36, ekryski/mlx-swift-lm#224,
ekryski/mlx-swift-lm#225 all open). Numbers are local M5 Max only;
recommendations are current best guess rather than settled defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Two new pieces of paper-form documentation covering longctx's two
operating modes and the V3 rescue fix.
NEW: docs/papers/longctx-1m-and-triattention.md
- Long-form paper on longctx — open long-context retrieval companion
for inference engines. Two modes: (1) standalone retrieval scored
on MRCR v2 8-needle on AMD MI300X with Qwen2.5-32B-Instruct, (2)
rescue layer for our TriAttention V3 eviction policy on Apple
Silicon, validated end-to-end via 256K planted-fact NIAH on
Qwen3.5-2B-4bit.
- Headline numbers: MRCR v2 1M MultiQ Selector + bge-rerank + det
copy n=30 directional 0.688 (above SubQ 0.659; mass-val pending).
V3+longctx ✓HIT every rung 32K → 256K via ChatSession;
V3-only ✗miss every rung.
- Architecture diagrams: full-stack (clients → vllm-swift →
ChatSession → mlx-swift-lm → V3 engine + rescue → longctx-svc),
V3 turn-by-turn flow (eviction-write on turn N, rehydrate-retrieve
before turn N+1).
- Appendices: module map (mlx-swift-lm side), longctx-svc endpoint
surface, V3 environment variable reference table.
- Cites Mao et al., "TriAttention: Efficient Long Reasoning with
Trigonometric KV Compression" (arXiv:2604.04921, 2026) as the
original paper. V3 in this stack is our independent Swift port +
hybrid extension; cross-references docs/papers/triattention-v3.md.
UPDATED: docs/papers/triattention-v3.md
- Adds §10 "Addendum: Swift Port + longctx Rescue — Fixing the
Hybrid NIAH Failure" after the §9 vLLM addendum.
- Reports the Apple Silicon Swift port test results that resolve the
Section 7.1 question "why does V3 break on hybrid Mamba+Attention
models?". Same failure mode reproduces on Swift port; longctx
rescue layer flips it.
- Includes the 12-cell receipt table, the V3 turn-by-turn diagram,
the driver-gotcha section explaining why bare container.generate()
doesn't rescue (Tier-3 rehydrate is ChatSession-only per commit
fe1a3b0 in mlx-swift-lm), and what's still not measured (>256K,
larger hybrids, V3+TQ+ stack).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed scripts/eden-investigation/ as a public artifact
pointer, but the directory contents are scratch and don't belong in the public
repo. Methodology in §2 is sufficient for reproduction; the file pointer was
a leftover from cleaning up the earlier /tmp/ paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The references block listed personal scratch paths under /tmp/eden-investigation/
that aren't useful to public readers and look confusing. Replaces those plus the
two private Obsidian-note pointers with a single line pointing to the
scripts/eden-investigation/ directory in this repo, where the actual experiment
scripts live (h*.py, F1–F3 KL test, universal-K candidate centroids).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds two independent third-party confirmations to the appendices:
asymmetric-kv-compression.md: @jagmarques tested asymmetric K/V on
Mistral-7B with E8 lattice VQ (a different quantization family from
TurboQuant) on Cerebrium A10/A100. Confirms asymmetric advantage scales
with context length: at 1742-tok prefix K3V2 +0.82% vs K2V2 +0.91%; at
3544-tok prefix K3V2 +0.35% vs K2V2 +2.26% (6.4× improvement). Clean
GQA-amplification explanation: 8 query heads per K head means K
quantization error is effectively 8× worse than V error in output
impact. Cross-method validation that the asymmetric conclusion is not
TurboQuant-specific.
turbo4-resurrection.md: @unamedkr / quantumaikr (quant.cpp) ported the
TurboQuant paper end-to-end in single-header C and found the QJL stage
contributed byte-identical zero to attention scores regardless of
input. Disabling QJL produced bit-for-bit identical perplexity.
Theoretical explanation: the √(π/2)/m constant in the unbiased
estimator is derived for Gaussian projection rows; on Rademacher (±1)
rows the per-key contribution gets dwarfed by residual-norm scaling.
Independently took the freed bits and reinvested in 4-bit codebook
(8 → 16 centroids), same path as turbo4 resurrection. Llama 3.2 3B:
turbo_kv_4b PPL 14.28 (+5.3% vs FP32), beating llama.cpp q4_0 KV
(~14.99, +10.6%). Targets embedded/edge (iOS, Android, WASM,
microcontrollers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the empirical study covering the broader 35-hypothesis investigation
spawned by the EDEN/Mitzenmacher/Portnoy critique (#87, #89). Companion
to eden-optimal-s-revisit.md.
Headline result: a K-cache centroid table that improves per-vector
reconstruction MSE by 1–13% across five model families (Qwen3-0.6B,
Llama-3.2-1B, Mistral-7B, Phi-3-mini, Gemma-2-2b) causes 70–90% mean
KL@D regressions and 50–60% catastrophic-rate failures at the model
output. This is a sign inversion, not a small mismatch: methods that
are strictly better under MSE are systematically worse under KL@D in
deployment.
The paper is structured in three acts:
- Act 1: K cache is sub-Gaussian post-WHT (H8); V is Gaussian
- Act 2: per-prompt fitting fails (PDS), ensemble (ens_4way) reaches an
irreducible ~8% catastrophic floor at scale (N=100)
- Act 3: the universal-K cross-model counterexample (F3) demonstrates
the MSE→KL inversion at the cleanest possible level
Includes a 2-D toy attention illustration of the softmax bucket-flip
mechanism, an explicit decision boundary for when MSE proxies quality
(linear consumers) vs when it doesn't (attention softmax), an
ablation summary distinguishing first-order from second-order factors,
operational guidance for practitioners, and a "why this wasn't caught
earlier" diagnosis of the methodology blind spots.
Companion paper handles the algorithmic claim head-on. This paper is
the broader evaluation-methodology critique.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a short paragraph in Appendix A documenting the algebraic identity
sqrt(S_unb * S_bias) = ||x||/||c||, which coincides with the matched-norm
scale. Frames it as an algebraic interpolation between the two
EDEN-derived forms, notes it is not derived in DRIVE, EDEN, or the
TurboQuant/EDEN note paper (arXiv:2604.18555), and does not correspond
to an optimization objective in those works.
The empirical head-to-head numbers in the table directly above the new
paragraph already show that each form attains the optimum of its own
objective and only its own; matched-norm behaves as a distinct operating
point, not as a tuning of either EDEN scale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The eden.py default formula S = ||x||²/⟨c,x⟩ was loosely labeled as
"biased optimal-S" throughout. On closer inspection that's imprecise:
the strict per-vector MSE-minimum is S* = ⟨c,x⟩/||c||² (textbook
least-squares; asymptotically → 1 for Lloyd-Max optimal centroids).
The eden.py default is a different formula, satisfying the inner-
product-preserving identity ⟨Sc, x⟩ = ||x||² per vector.
Empirical comparisons unaffected (we tested what eden.py ships).
Just the labeling. Added Terminology note in §4 with worked example
and credit to @amitport for surfacing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical response to issues #87 (Mitzenmacher) and #89 (Portnoy) on
turboquant_plus. 8 hypothesis tests across synthetic Gaussian and
Qwen3-0.6B + Llama-3.2-1B real KV cache.
Headline: EDEN's optimal scale is real but second-order. Rotation choice
(WHT vs dense Haar) is first-order and accounts for ~all of the published
gap. Production already uses the first-order fix. On real KV with WHT,
matched-norm beats EDEN-S by 0.5-9% across Qwen and Llama; tightens to
0.67-0.72% on Llama. EDEN-S genuinely wins at b=8 on synthetic Gaussian
(70%+ MSE reduction) — credibility moat showing the lever exists in its
own regime.
Concedes attribution argument (#89): DRIVE/EDEN line is real prior art.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the 3-engine apples-to-apples cross-engine benchmark:
- BF16 baseline (load, PPL, prefill, decode, KV size) across vLLM, SGLang, llama.cpp
- REFRACT --full 4-axis fidelity scoring (gtm, kld, rniah, plad) on each engine's native 8-bit KV
- Headline finding: same fp8_e4m3 dtype string produces 1.8x different KL drift between vLLM and SGLang on the same model
- 8 engine-side bugs encountered and worked around (vLLM max_num_seqs, prompt_logprobs cap, GPU mem carryover; SGLang aiter.dtypes, AITER fp8 prefill broken, KV dtype fixed at server launch)
- Full reproducibility scripts and REFRACT framework changes documented
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney1@gmail.com <tturney1@gmail.com>
Adds Section 9 to the V3 paper documenting the port to vLLM ROCm Triton
on AMD MI300X. New subsections:
9.1 Hardware (MI300X / ROCm 7.2 / Triton 3.6 / vLLM 0.1.dev1)
9.2 Implementation summary (files added, three vLLM-specific design
choices: custom-op Q capture, worker-side env-var lazy init,
mask-based eviction without compaction)
9.3 Models tested (7-row table: validated, partially validated,
blocked, algorithmically incompatible)
9.4 Paper-exact reproduction on Qwen2.5-7B-Instruct
9.4.1 PPL: vLLM deltas all within ±0.5% noise floor
9.4.2 KV cache compression: stack hits 2.96x at 90% retention,
matching the paper's "approximately 2.9x" headline
9.4.3 NIAH: V3 + stack regress vs paper at the middle position
(PARTIAL_NUMBER instead of PASS). Real gap.
9.5 Where the vLLM port falls short of the paper (3 plausible causes
ordered by likelihood: KV preset, scoring numerics, eviction
trigger frequency)
9.6 Hybrid models still untested in vLLM (3 attempts, 3 different
blockers: transformers compat, no-RoPE incompat, vLLM page-size
unifier)
9.7 Headline claim envelope - what I can and cannot claim post-port
9.8 Open work
9.9 How to Run (paper-exact PPL + NIAH commands, tunable knob table,
cross-vendor note, doc pointers)
Honest framing throughout: PPL reproduces within noise, NIAH degrades
vs paper at middle position. Not a refutation of the paper, not a
clean confirmation. The original V3 paper's point that PPL alone is
insufficient signal for V3 quality (Section 3.2) is reinforced
empirically by the vLLM port.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Investigation of master llama.cpp #21038 attention rotation in our fork,
ending in a per-side env-knob policy (LLAMA_ATTN_ROT_K_OVERRIDE /
LLAMA_ATTN_ROT_V_OVERRIDE). Default unchanged from the original fork
(rotation OFF on both sides); the contribution is per-side control plus a
matrix documenting which models want which knob.
Headline finding while running the matrix: corpus PPL on wikitext-2 is
unreliable for KV-quantization evaluation on gemma-class instruct models.
Quantized KV scores 7-42% BELOW the fp16-KV baseline on three gemma-4-it
GGUFs; KLD against the fp16-KV reference points the opposite direction.
Reproduced at q8/q8 and q8/turbo4, on wikitext-2 and wikitext-103, ctx=512
and ctx=2048. Cleanest case: t4/t4 K-only PPL +52.7% but KLD -4.9% (best
on row).
The PPL/KLD ranking inversion is a continuum (small magnitude on Qwen2.5-7B,
large on gemma-4 26B-A4B), not gemma-only. Recommendation: use KLD against
fp16-KV logits as the oracle for KV-quant evaluation on this model class.
Independently observed prior: vektorprime in #21394, AesSedai's tables in
#21038, ggerganov's "track KLD rather than PPL" comment in same PR,
localbench. This paper adds the controlled per-side measurement, the
cross-format / cross-corpus reproduction, the bit-exact-zero KLD noise
floor on Metal, and the resulting engineering policy.
- turbo4v2 is faster than no-turbo (not "can be"), explained by Apple
Silicon matmul favoring smaller types during dequant
- removed "Gap to Python" row — Swift is at parity or ahead now
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Adds a new subsection documenting the boundary layer skip experiment
motivated by Sharpe 2026's coupled-oscillator framework and by the
boundary protection pattern already used on the weight-quant side of
TurboQuant+. The experiment did not pan out.
Results:
Qwen2.5-7B @ 32K, V3 prefix=128:
skip=0 (control): PPL 6.8508 (+0.006%)
skip=1: PPL 6.9150 (+0.94%)
skip=2: PPL 6.9166 (+0.97%)
Qwen2.5-7B @ 64K, V3 prefix=128:
skip=0 (control): PPL 6.2530 (-0.002%)
skip=2: PPL 6.2940 (+0.65%)
Qwen3.5-27B @ 32K, V3 prefix=128:
skip=0 (control): PPL 7.4853 (+0.29%), NIAH middle/end FAIL
skip=4: PPL 7.4779 (+0.19%), NIAH middle/end FAIL
skip=8: PPL 7.4817 (+0.24%), NIAH middle/end FAIL
Three takeaways documented in Section 4.8.5:
1. PPL and NIAH are decoupled on this failure mode. The marginal PPL
improvement on qwen35 does not translate into any NIAH recovery.
2. Boundary protection is the right pattern for the weight-quant path,
not the eviction-scoring path. Same layer set, different sensitivity.
3. Sharpe's framework correctly predicts that sparser attention coupling
makes eviction harder, but boundary skip is not the fix. The likely
candidate is replacing V3's phase-alignment scoring with a direct
coupling measurement, which is a larger change left as follow-on work.
Section 7.1 (why V3 breaks on hybrid models) updated to explicitly
record that the "layer 0 is a transducer" hypothesis has been tested
and ruled out. The partial-RoPE and scoring-formula hypotheses remain
viable.
Appendix A extended with the new PPL rows for completeness.
The corresponding code change on the llama.cpp fork flips the
--triatt-boundary-skip default from 2 to 0 so the flag is strictly
opt-in and nobody hits the PPL regression by accident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 3 is now dedicated to the testing protocol with five
subsections: hardware and build, perplexity protocol, NIAH protocol,
wall time protocol, and how to reproduce each table. Documents the
traps we hit along the way (single-chunk noise, batch size = context
causing eviction to not fire, --no-display-prompt false-pass on NIAH,
-n 32 breaking reasoning models, case-sensitive grep edge case at 64K).
Existing Results, Production Reality, How to Run, Missing Pieces, and
References sections are renumbered accordingly. No numeric changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A minimal hybrid memory policy for long-context KV cache eviction on
top of the paper-faithful TriAttention formula (Mao et al., arXiv:2604.04921).
V3 adds two structural constraints to the paper's scoring:
- Hard prefix window: first N tokens never evicted
- Per-segment quota: eviction spreads evenly across position buckets
Validated envelope (full data in the paper):
- Qwen2.5-7B at 32K: PPL +0.006% vs baseline, NIAH all positions PASS
- Qwen2.5-7B at 64K: PPL -0.002% vs baseline, NIAH all positions PASS
- TurboQuant+ + V3 stack at 32K: +0.84% PPL, 2.9x KV memory reduction
- Wall time overhead: within baseline noise after four optimization passes
Where it does not work:
- Qwen3.5-27B and Qwen3.5-35B-A3B (hybrid Mamba+Attention): PPL transfers
cleanly but NIAH silently fails at middle and end positions. The failure
mode suggests the scoring is evicting semantically important tokens in
these architectures, not that the implementation is broken. Open question
documented in Section 6.
Paper includes: full implementation journey (V1 paper-faithful, V2
per-segment quota only, V3 prefix-protect + quota), how-to-run section
with concrete CLI commands for the new --triatt-hybrid and --triatt-prefix
flags, a dedicated section asking the community for input on the missing
pieces, and appendices with complete numeric data and the commit-level
rollback ladder on the llama.cpp fork.
This paper accompanies the draft pull request on the
experiment/triattention-integration branch at TheTom/llama-cpp-turboquant
(not for merge yet; community review and comparison notes welcome).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Blackwell decode penalty drops from 25-40% to 7-9% with signalnine's
fused shmem mmvq kernel. 3.7-5.4x improvement. Prefill flips from
penalty to bonus. RTX 5090 now matches RTX 4090 penalty levels.
signalnine authored the kernel; TheTom collaborated on integration.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primoco diagnosed AmesianX's 30.8% F16 baseline as RoPE frequency
scaling from ctx_size < native context. At native 32768: all configs
(f16, tbq4_1, q8_0/tbq4_1) hit 100% accuracy. Zero cost from TBQ.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First end-user confirmation on Blackwell consumer hardware. All 6 K/V
combinations tested with philosophical, mathematical, and 23K-token
inputs. Originally hit corrupt output on Madreag's fork.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric tbqp3/tbq3 matches f16/f16 exactly (20/65, 30.8%) at temp=0.
Confirms no TBQ degradation on head_dim=128 models. Notable that
symmetric turbo3 works on Qwen3-14B Q4_K_M unlike Qwen2.5-7B Q4_K_M.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hannu Varjoranta's PR #1 on vibhavagarwal5/vllm changed default
value_quant_bits from 4 to 8 (FP8), fixing 0% gsm8k. Cites @TheTom
as source for "value precision is the quality bottleneck." Independent
vLLM validation of asymmetric thesis and turbo4-resurrection findings.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Independent validation of asymmetric KV compression on Ada (SM 89)
and Blackwell (SM 120). Asymmetric q8_0-K/turbo4-V confirmed best
config on Ada (+8.4% prefill, -6.2% decode). Blackwell shows larger
decode regression (-25.2%), likely dp4a vs fp8/fp4 tensor core mismatch.
LA=1 boundary layers improve decode speed in addition to quality.
Source: https://github.com/ggml-org/llama.cpp/discussions/20969
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I on Gemma 4 26B MoE: -2.3% size, neutral/better PPL, 100% speed.
Minimal weight compression benefit on MoE (few dense layers). KV compression
is the real win for this architecture.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new community reports:
- theaaronhughes: RTX 5090 Gemma 4 26B KV at 64k/128k
- tryingET: RTX PRO 6000 Config I+turbo3 beats Q8_0+turbo3 by 7% decode
- pjsgsy: RTX 3050 APEX-I-Quality pre-quantized models with CPU offload
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First Volta (V100, SM 7.0) validation: Config I +8.7% faster decode.
First Pascal (1080 Ti, SM 6.1) validation: Config I +6-11% faster.
Config I is now faster than Q8_0 baseline on every CUDA architecture
tested (Pascal, Volta, Ada, Blackwell).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vilenkin basis decomposition on Phi-2 suggests K and V occupy disjoint
spectral bands, providing potential mathematical basis for asymmetric
compression. Caveat: single model, needs validation across architectures.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config I 30% faster than Q8_0 decode on RDNA 4. First successful AMD
HIP weight compression result. PPL +1.8%.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
14-config Metal benchmark independently confirms asymmetric K/V as
default recommendation. BitNet compound case is new finding. NIAH
survives catastrophic PPL (retrieval degrades independently).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@tryingET: Qwen3.5-27B, 96GB VRAM. Config I is faster than Q8_0 on
Blackwell Workstation (+11.9% prefill, +8.5% decode). PPL +0.048%.
Config I + turbo3 beats Q8_0 + turbo3/turbo4 at 32K.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vLLM TurboQuant integration explicitly cites turboquant_plus for
boundary layers and asymmetric KV. Community ablations on A100/H100
confirm our findings.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Weight compression is now on feature/turboquant-kv-cache (main).
No separate branch needed. PR #45 remains open for community logs.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Section 4.4: TurboQuantDC (662 tests, MIT) independently confirmed
stacking, boundary layer protection, and QJL removal on RTX 4090.
Links to community results page for full hardware coverage.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12+ GPUs, 11+ models, 6 families, 10+ independent testers. All data
from PR #45 community testing. Includes GPU compatibility matrix,
model results by family, stacking validation, decode speed comparison,
known issues, confidence ratings, and contribution template.
Linked from getting-started.md.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
19-version kernel optimization log proves CUDA TQ4_1S decode gap is
fundamental (float32 bandwidth + FMA density), not an optimization gap.
Q4_K_M is the practical CUDA runtime answer. TQ4_1S optimal for Metal,
storage/distribution format on CUDA.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
clang on Windows does not inherit CMake Release flags. Without -O3 -DNDEBUG
the build runs ~50x slower. Tested on RX 9070 XT (gfx1201, RDNA 4).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add TQ3_4S mention and reframe contribution to include follow-on work.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- asymmetric paper: correct Sjoerd's q8_0/turbo3 results (corrupt output
on head_dim=256, speed data accurate but content garbage)
- recommendations: add MTMD_BACKEND_DEVICE=CUDA1 for multi-GPU vision
- community count: 30+ to 50+ testers
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CUDA port now available via signalnine (PR #45)
- Add CUDA GPUs to tested hardware (1080 Ti, RTX 5090)
- Replace "Metal only" limitation with CUDA decode gap analysis
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
