From a2dce218a3d75b19c389d1901ac118efa5d14e34 Mon Sep 17 00:00:00 2001 From: TheTom Date: Sun, 5 Apr 2026 07:49:58 -0500 Subject: [PATCH] docs: expand MLX results with prefill, PPL, 35B data, fix branch link Co-Authored-By: Tom Turney Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 24da6ed..a6ab887 100644 --- a/README.md +++ b/README.md @@ -546,15 +546,28 @@ Detailed debugging logs, gotchas, and benchmarks from the llama.cpp port: TurboQuant KV cache compression is being ported to Apple's [MLX framework](https://github.com/ml-explore/mlx) for native Python/Swift inference on Apple Silicon. -**Fork:** [TheTom/mlx](https://github.com/TheTom/mlx) — branch `feature/turboquant-plus` +**Fork:** [TheTom/mlx `feature/turboquant-plus`](https://github.com/TheTom/mlx/tree/feature/turboquant-plus) ### Preliminary Results (Qwen3.5-2B 8bit, M5 Max) -| Config | Decode | vs Baseline | PPL Delta | KV Savings | -|--------|--------|-------------|-----------|------------| -| Baseline (f16 KV) | 204 tok/s | — | — | — | -| turbo4 asymmetric (K=FP16, V=4bit) | 177 tok/s | 87% | +0.18% | 73% | -| turbo3 | 170 tok/s | 83% | -0.10% | 80% | +**Speed + Quality:** + +| Config | Prefill | Decode | vs Baseline | PPL | PPL Delta | KV Savings | +|--------|---------|--------|-------------|-----|-----------|------------| +| Baseline (f16 KV) | 584 | **204** | 100% | 3.0732 | — | — | +| turbo4 all fused | 1,153 (+97%) | 168 | 83% | 3.0890 | +0.51% | 73% | +| turbo4 asymmetric fused | 984 (+68%) | **177** | **87%** | 3.0787 | **+0.18%** | 73% | +| turbo3 | 264 | 170 | 83% | 3.0702 | -0.10% | 80% | + +**Quality:** Output text indistinguishable from baseline. KL divergence < 0.001, cosine similarity > 0.989. + +**35B MoE (Qwen3.5-35B-A3B 8bit):** + +| Config | Prefill | Decode | vs Baseline | +|--------|---------|--------|-------------| +| Baseline | 11.4 | 95.7 | 100% | +| turbo4 fused | 254.7 (+22x) | 53.9 | 56% | +| turbo4 asymmetric | 241.1 (+21x) | 55.2 | 58% | ### What's implemented - TurboQuant encode/decode using `mx.hadamard_transform` (MLX built-in)