From 7d69ed88e114055231663add92f717681f8d7152 Mon Sep 17 00:00:00 2001 From: TheTom Date: Fri, 12 Jun 2026 08:27:19 -0500 Subject: [PATCH] README: link lablup/mlxcel TurboQuant+ KV cache port --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e924afd..bf9a0d4 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,11 @@ # TurboQuant+ -> ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [llama.cpp Fork](https://github.com/TheTom/llama-cpp-turboquant) | [Atlas](https://github.com/Avarok-Cybersecurity/atlas) | [Swift MLX Fork](https://github.com/ekryski/mlx-swift-lm/tree/alpha) | [vllm-swift](https://github.com/TheTom/vllm-swift) | [Commercial Support](https://x.com/no_stp_on_snek) +> ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [llama.cpp Fork](https://github.com/TheTom/llama-cpp-turboquant) | [Atlas](https://github.com/Avarok-Cybersecurity/atlas) | [Swift MLX Fork](https://github.com/ekryski/mlx-swift-lm/tree/alpha) | [vllm-swift](https://github.com/TheTom/vllm-swift) | [mlxcel Port](https://github.com/lablup/mlxcel) | [Commercial Support](https://x.com/no_stp_on_snek) > **🍎 On Apple Silicon and want the fastest path?** Use [ekryski/mlx-swift-lm](https://github.com/ekryski/mlx-swift-lm/tree/alpha) — Eric Kryski's Swift MLX implementation that I've been actively collaborating on. Native Swift, ~2.5x faster decode than Python mlx-lm, full TurboQuant+ support including turbo4v2 (4-bit K + 2-bit V). 144 tok/s on Qwen3.5-35B-A3B MoE at 4K on M5 Max. For OpenAI-compatible serving, use [vllm-swift](https://github.com/TheTom/vllm-swift) — a native Swift/Metal backend for vLLM built on mlx-swift-lm. No Python in the inference hot path, works with Hermes, OpenCode, and any OpenAI client. This llama.cpp repo is for cross-platform deployment (CUDA, ROCm, CPU, Metal). +> **🦀 mlxcel port:** [lablup/mlxcel](https://github.com/lablup/mlxcel), an MLX-based experimental inference engine in Rust, ports the TurboQuant+ KV cache compression (see their [turbo KV cache docs](https://github.com/lablup/mlxcel/blob/main/docs/turbo-kv-cache.md)). Thanks to [@inureyes](https://github.com/inureyes) and the Lablup team for the careful port and upstream attribution. + ## Prebuilt Binaries Download ready-to-run builds from [Releases](https://github.com/TheTom/llama-cpp-turboquant/releases). No build tools needed.