mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-23 03:22:55 -05:00
f1c2397680
Simulates KV cache compression at Qwen 3.5 27B and 35B-A3B dimensions: 28 layers × 8 KV heads × 128 head_dim × up to 8192 tokens. Results: - 3-bit K+V: 2.6× compression, K cosine 0.91, attention cosine 0.98 - 4-bit K+V: 1.9× compression, K cosine 0.96, attention cosine 0.99 - Prince Canuma gets 4.9× at 2.5-bit (uses outlier channel strategy) Next: implement outlier channel strategy for non-integer bit rates (2.5-bit, 3.5-bit) to match Prince's compression ratios. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>