mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 02:53:13 -05:00
TheTomandClaude Opus 4.7
baebe5745f
Adds Section 9 to the V3 paper documenting the port to vLLM ROCm Triton
on AMD MI300X. New subsections:
9.1 Hardware (MI300X / ROCm 7.2 / Triton 3.6 / vLLM 0.1.dev1)
9.2 Implementation summary (files added, three vLLM-specific design
choices: custom-op Q capture, worker-side env-var lazy init,
mask-based eviction without compaction)
9.3 Models tested (7-row table: validated, partially validated,
blocked, algorithmically incompatible)
9.4 Paper-exact reproduction on Qwen2.5-7B-Instruct
9.4.1 PPL: vLLM deltas all within ±0.5% noise floor
9.4.2 KV cache compression: stack hits 2.96x at 90% retention,
matching the paper's "approximately 2.9x" headline
9.4.3 NIAH: V3 + stack regress vs paper at the middle position
(PARTIAL_NUMBER instead of PASS). Real gap.
9.5 Where the vLLM port falls short of the paper (3 plausible causes
ordered by likelihood: KV preset, scoring numerics, eviction
trigger frequency)
9.6 Hybrid models still untested in vLLM (3 attempts, 3 different
blockers: transformers compat, no-RoPE incompat, vLLM page-size
unifier)
9.7 Headline claim envelope - what I can and cannot claim post-port
9.8 Open work
9.9 How to Run (paper-exact PPL + NIAH commands, tunable knob table,
cross-vendor note, doc pointers)
Honest framing throughout: PPL reproduces within noise, NIAH degrades
vs paper at middle position. Not a refutation of the paper, not a
clean confirmation. The original V3 paper's point that PPL alone is
insufficient signal for V3 quality (Section 3.2) is reinforced
empirically by the vLLM port.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>