mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-21 18:42:44 -05:00
Add head_dim=64 limitation warning to getting-started guide
Models with head_dim=64 (e.g., GPT-OSS-120B) may crash or produce degraded output with turbo compression. WHT requires sufficient dimensionality for CLT convergence. Reported by NigelTufnel12345. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai
This commit is contained in:
1 parent
e93a0027e1
commit
05fde211c0
1 file changed
+2
@@ -54,6 +54,8 @@ Run a model with TurboQuant+ KV cache compression:
|
||||
|
||||
Asymmetric: full precision K, compressed V. Safe on all tested models including sensitive ones like Qwen2.5.
|
||||
|
||||
> **Known limitation:** Models with `head_dim=64` (e.g., GPT-OSS-120B) may crash or produce degraded output with turbo V compression. The WHT kernel requires sufficient dimensionality for CLT convergence, and d=64 is at the lower boundary. If you hit issues on a head_dim=64 model, fall back to `-ctk q8_0 -ctv q8_0` (no turbo compression). The paper validated at head_dim=128 and 256.
|
||||
|
||||
### More compression (works on most models)
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user