starred/turboquant_plus
1
0 Fork 0
mirror of https://github.com/TheTom/turboquant_plus.git synced 2026-10-22 02:53:13 -05:00
Code Issues Packages Projects Releases Wiki Activity

README: link lablup/mlxcel TurboQuant+ KV cache port

Browse Source
This commit is contained in:
TheTom committed 2026-06-12 08:27:19 -05:00
1 parent 9eb0341091
commit 7d69ed88e1
1 file changed
+3 -1
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -1
View File
@@ -1,9 +1,11 @@
# TurboQuant+
> ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [llama.cpp Fork](https://github.com/TheTom/llama-cpp-turboquant) | [Atlas](https://github.com/Avarok-Cybersecurity/atlas) | [Swift MLX Fork](https://github.com/ekryski/mlx-swift-lm/tree/alpha) | [vllm-swift](https://github.com/TheTom/vllm-swift) | [Commercial Support](https://x.com/no_stp_on_snek)
> ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [llama.cpp Fork](https://github.com/TheTom/llama-cpp-turboquant) | [Atlas](https://github.com/Avarok-Cybersecurity/atlas) | [Swift MLX Fork](https://github.com/ekryski/mlx-swift-lm/tree/alpha) | [vllm-swift](https://github.com/TheTom/vllm-swift) | [mlxcel Port](https://github.com/lablup/mlxcel) | [Commercial Support](https://x.com/no_stp_on_snek)
> **🍎 On Apple Silicon and want the fastest path?** Use [ekryski/mlx-swift-lm](https://github.com/ekryski/mlx-swift-lm/tree/alpha) — Eric Kryski's Swift MLX implementation that I've been actively collaborating on. Native Swift, ~2.5x faster decode than Python mlx-lm, full TurboQuant+ support including turbo4v2 (4-bit K + 2-bit V). 144 tok/s on Qwen3.5-35B-A3B MoE at 4K on M5 Max. For OpenAI-compatible serving, use [vllm-swift](https://github.com/TheTom/vllm-swift) — a native Swift/Metal backend for vLLM built on mlx-swift-lm. No Python in the inference hot path, works with Hermes, OpenCode, and any OpenAI client. This llama.cpp repo is for cross-platform deployment (CUDA, ROCm, CPU, Metal).
> **🦀 mlxcel port:** [lablup/mlxcel](https://github.com/lablup/mlxcel), an MLX-based experimental inference engine in Rust, ports the TurboQuant+ KV cache compression (see their [turbo KV cache docs](https://github.com/lablup/mlxcel/blob/main/docs/turbo-kv-cache.md)). Thanks to [@inureyes](https://github.com/inureyes) and the Lablup team for the careful port and upstream attribution.
## Prebuilt Binaries
Download ready-to-run builds from [Releases](https://github.com/TheTom/llama-cpp-turboquant/releases). No build tools needed.