starred/turboquant_plus
1
0 Fork 0
mirror of https://github.com/TheTom/turboquant_plus.git synced 2026-10-22 02:53:13 -05:00
Code Issues Packages Projects Releases Wiki Activity

docs: add decode speed table with context scaling to README

Browse Source 
Decode at 87-91% of q8_0 for typical context, 70% at 48K (PDF test).
Prefill remains at 99% parity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
This commit is contained in:
TheTomClaude Opus 4.6
 committed 2026-03-25 22:43:51 -05:00
1 parent 7fa354b7e8
commit dd440be4b8
1 file changed
+11 -1
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11 -1
View File
@@ -41,7 +41,17 @@ Compresses transformer KV cache **4.6x** using PolarQuant + Walsh-Hadamard rotat
| 16K | 1737 | 1757 | 0.989x |
| 32K | 1211 | 1217 | 0.995x |
**Flat 99% of q8_0 speed regardless of context length.** See [Context Scaling Deep Dive](docs/context-scaling-deep-dive.md) for the full investigation.
**Prefill: flat 99% of q8_0 speed regardless of context length.**
### Decode Speed (Server, Real-World)
| Context | turbo3 decode | q8_0 decode | turbo3/q8_0 |
|---------|-------------|-----------|-------------|
| Short (~12 tok) | 77.2 | 85.2 | 0.91x |
| 8K | 67.3 | 77.7 | 0.87x |
| 48K (70-page PDF) | 39.0 | 55.6 | 0.70x |
Decode is 87-91% of q8_0 at typical context. At very long context (48K+) it's 70% due to per-position dequant cost. See [Context Scaling Deep Dive](docs/context-scaling-deep-dive.md) and [Decode Speed Investigation](docs/experiment-decode-speed.md).
### Speed Optimization Journey