mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-22 11:02:42 -05:00
TheTomandClaude Opus 4.7
1224fef358
Two new pieces of paper-form documentation covering longctx's two operating modes and the V3 rescue fix. NEW: docs/papers/longctx-1m-and-triattention.md - Long-form paper on longctx — open long-context retrieval companion for inference engines. Two modes: (1) standalone retrieval scored on MRCR v2 8-needle on AMD MI300X with Qwen2.5-32B-Instruct, (2) rescue layer for our TriAttention V3 eviction policy on Apple Silicon, validated end-to-end via 256K planted-fact NIAH on Qwen3.5-2B-4bit. - Headline numbers: MRCR v2 1M MultiQ Selector + bge-rerank + det copy n=30 directional 0.688 (above SubQ 0.659; mass-val pending). V3+longctx ✓HIT every rung 32K → 256K via ChatSession; V3-only ✗miss every rung. - Architecture diagrams: full-stack (clients → vllm-swift → ChatSession → mlx-swift-lm → V3 engine + rescue → longctx-svc), V3 turn-by-turn flow (eviction-write on turn N, rehydrate-retrieve before turn N+1). - Appendices: module map (mlx-swift-lm side), longctx-svc endpoint surface, V3 environment variable reference table. - Cites Mao et al., "TriAttention: Efficient Long Reasoning with Trigonometric KV Compression" (arXiv:2604.04921, 2026) as the original paper. V3 in this stack is our independent Swift port + hybrid extension; cross-references docs/papers/triattention-v3.md. UPDATED: docs/papers/triattention-v3.md - Adds §10 "Addendum: Swift Port + longctx Rescue — Fixing the Hybrid NIAH Failure" after the §9 vLLM addendum. - Reports the Apple Silicon Swift port test results that resolve the Section 7.1 question "why does V3 break on hybrid Mamba+Attention models?". Same failure mode reproduces on Swift port; longctx rescue layer flips it. - Includes the 12-cell receipt table, the V3 turn-by-turn diagram, the driver-gotcha section explaining why bare container.generate() doesn't rescue (Tier-3 rehydrate is ChatSession-only per commit fe1a3b0 in mlx-swift-lm), and what's still not measured (>256K, larger hybrids, V3+TQ+ stack). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>