mirror of
https://github.com/TheTom/turboquant_plus.git
synced 2026-10-19 01:30:50 -05:00
docs: fix relative links in benchmarks.md after move
This commit is contained in:
1 parent
38aa2ae567
commit
b77dbe4ecb
1 file changed
+8
-8
+8
-8
@@ -38,7 +38,7 @@ Validated at 512 and 8K context. NIAH retrieval passed. Benefit scales with mode
|
||||
llama-server -m model.gguf -ctk q8_0 -ctv turbo2 -fa 1
|
||||
```
|
||||
|
||||
See [full paper](docs/papers/layer-aware-v-compression.md).
|
||||
See [full paper](papers/layer-aware-v-compression.md).
|
||||
|
||||
## Prefill Context Scaling (Verified 2K-32K)
|
||||
|
||||
@@ -76,7 +76,7 @@ turbo4 decode is faster than turbo3 due to simpler nibble packing and direct-ext
|
||||
|------|------|--------|-------------------|
|
||||
| Single needle (33 positions) | 30/33 (90.9%) | **31/33 (93.9%)** | 9/9 (3-pos) |
|
||||
|
||||
turbo4 beats q8_0 on retrieval (31/33 vs 30/33). Shared failure at 8K/100% is a model weakness, not quantization. See [turbo4 resurrection](docs/papers/turbo4-resurrection.md) for the full investigation.
|
||||
turbo4 beats q8_0 on retrieval (31/33 vs 30/33). Shared failure at 8K/100% is a model weakness, not quantization. See [turbo4 resurrection](papers/turbo4-resurrection.md) for the full investigation.
|
||||
|
||||
## Large Model Stress Tests (M5 Max 128GB)
|
||||
|
||||
@@ -89,9 +89,9 @@ turbo4 beats q8_0 on retrieval (31/33 vs 30/33). Shared failure at 8K/100% is a
|
||||
|
||||
turbo3 prefill is faster than q8_0 at 32K on both models (70B: 80.8 vs 75.2 t/s, 104B: 64.5 vs 62.3 t/s). Smaller KV cache = less memory bandwidth during attention.
|
||||
|
||||
104B at 128K requires raising macOS GPU memory cap: `sudo sysctl iogpu.wired_limit_mb=117964` (90% of 128GB). Without this, Metal stalls at ~49K context on 70B+ models. See [Getting Started Guide](docs/getting-started.md) for per-RAM values.
|
||||
104B at 128K requires raising macOS GPU memory cap: `sudo sysctl iogpu.wired_limit_mb=117964` (90% of 128GB). Without this, Metal stalls at ~49K context on 70B+ models. See [Getting Started Guide](getting-started.md) for per-RAM values.
|
||||
|
||||
See [M5 Max stress test](docs/papers/m5-max-stress-test.md) for the full data.
|
||||
See [M5 Max stress test](papers/m5-max-stress-test.md) for the full data.
|
||||
|
||||
## KL Divergence vs f16
|
||||
|
||||
@@ -116,9 +116,9 @@ Dense models see smaller gains (attention is <5% of decode — FFN dominates). N
|
||||
|
||||
**Sparse V dequant** skips V dequantization for positions where softmax attention weight < 1e-6. At long context, most attention weights are negligible — this saves approximately half the total dequant cost. +22.8% decode at 32K vs turbo3 without sparse V, pushing the ratio from 0.76x to 0.93x of q8_0. Sparse V introduces no additional PPL degradation beyond the underlying compression (validated at 32K with 50 chunks on wikitext-103, CI ±0.021). Benefit scales with context length. This is implemented as a minimal kernel modification.
|
||||
|
||||
Sparse V is not TurboQuant-specific: on q8_0 KV cache it yields a +5% decode speedup with identical PPL and NIAH, confirming this is a general attention-aware optimization rather than a compression-specific trick. See the [full paper](docs/papers/sparse-v-dequant.md).
|
||||
Sparse V is not TurboQuant-specific: on q8_0 KV cache it yields a +5% decode speedup with identical PPL and NIAH, confirming this is a general attention-aware optimization rather than a compression-specific trick. See the [full paper](papers/sparse-v-dequant.md).
|
||||
|
||||
On M2/M1 (pre-M5), the auto-detected 4-mag LUT gives an additional +38-45% decode improvement at long context, and is additive with sparse V. See [Decode Speed Hardware Analysis](docs/decode-speed-hardware-analysis.md) for the full 14-approach experiment log, and [Context Scaling Deep Dive](docs/context-scaling-deep-dive.md) for the M5 Max optimization journey.
|
||||
On M2/M1 (pre-M5), the auto-detected 4-mag LUT gives an additional +38-45% decode improvement at long context, and is additive with sparse V. See [Decode Speed Hardware Analysis](decode-speed-hardware-analysis.md) for the full 14-approach experiment log, and [Context Scaling Deep Dive](context-scaling-deep-dive.md) for the M5 Max optimization journey.
|
||||
|
||||
## Community Hardware: CUDA (RTX 3090)
|
||||
|
||||
@@ -188,7 +188,7 @@ First AMD GPU validation. First attempt — no debugging, no analysis, just raw
|
||||
- Speed flat across configs (~85 t/s decode, ~590 t/s prefill at pp512)
|
||||
- Context scaling: 0.96-0.99x vs q8_0 at pp2048-8192
|
||||
|
||||
See [Windows RDNA 4 Setup Guide](docs/windows-rdna4-setup.md) for build instructions and 9 gotchas.
|
||||
See [Windows RDNA 4 Setup Guide](windows-rdna4-setup.md) for build instructions and 9 gotchas.
|
||||
|
||||
## Speed Optimization Journey
|
||||
|
||||
@@ -201,7 +201,7 @@ See [Windows RDNA 4 Setup Guide](docs/windows-rdna4-setup.md) for build instruct
|
||||
| + block-32 storage | 2747 | 1.02x |
|
||||
| **+ optimized dequant** | **2524** | **0.98x** |
|
||||
|
||||
> The final number (2524 at 4K) is lower than the peak (2747 at 512) because longer context is naturally slower. The key metric is the **ratio** vs q8_0, which stays flat at 0.99x. See [Speed Experiments](docs/speed-experiments.md) for the full journey.
|
||||
> The final number (2524 at 4K) is lower than the peak (2747 at 512) because longer context is naturally slower. The key metric is the **ratio** vs q8_0, which stays flat at 0.99x. See [Speed Experiments](speed-experiments.md) for the full journey.
|
||||
|
||||
## Compression Quality (Python Prototype)
|
||||
|
||||
|
||||
Reference in new issue
Block a user