First/last 2 KV layers stay at FP16 to prevent NaN from extreme
V norms on boundary layers. Matches the production recommendation.
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Symmetric turbo4 is catastrophic on dense models (KLD=6.86, NIAH 0/15).
Asymmetric K=FP16 fixes everything (KLD=0.003, NIAH 15/15, top-1 98.1%).
Tested on Qwen2.5-7B-Instruct-8bit (28 layers, all KV cache).
Quality test suite added: scripts/mlx_quality_suite.py
Co-Authored-By: Tom Turney <tturney1@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Detect per-layer cache types so hybrid models (Qwen3.5 linear+attention)
get TurboKVCache only on KVCache layers, not ArraysCache layers
- Use min_compress_tokens=32 for KLD test so compression actually fires
- Extend KLD text to ~500 tokens for meaningful divergence measurement
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New standalone script that tests TurboKVCache quality using mlx-lm
without touching existing llama.cpp test scripts.
Co-Authored-By: Tom Turney <tturney@psyguard.ai>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merges MLX LoRA adapters directly into HF base weights, bypassing
mlx_lm.fuse and its sanitize transforms that break MoE/hybrid
architectures. Supports expert gate_up fusion, SwitchLinear 3D
deltas, optional TQ4_1S Config I compression.
Tested on Qwen3.5-35B-A3B (256-expert MoE + GatedDeltaNet).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reports numbers without pass/fail judgment when reference values
don't apply. Use for testing new models where baseline PPL differs.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Direct skip-rate measurement via output_attentions (9% at 512 → 28% at 4K)
- PPL validation with q8_0 baselines: 8K/20ch, 16K/10ch, 32K/5ch
- ON/OFF delta 0.0000 at all context lengths on clean branch
- turbo3 +1.1-1.4% vs q8_0, consistent and unaffected by sparse V
- Note: PR #4 was reverted (broke turbo3 PPL 6.18 → 181.6)
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Top-level at-a-glance table with PPL, NIAH, speed ratios across all
tested hardware, compression ratios, and link to decode regression
analysis doc.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First community diagnostic from @Ambisphaeric (M1 Max 64GB):
- Prefill: 93-98% of q8_0 (excellent, flat curve)
- Decode: 84% short → 39% at 32K (M1 constant LUT issue)
M2 Pro Mac Mini partial results (in progress, noisy from workloads).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces /Users/username (or /home/username) with ~ in all output
files (log, JSON profile, monitor CSV) before packaging into the
zip. Prevents accidental PII leak when users share diagnostic results.
Reported by community tester who noticed their Mac username in
command paths logged by llama-bench output.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community testers need to know which fork to build. Apple Silicon
uses our Metal fork, NVIDIA/AMD uses spiritbuun's CUDA fork.
Also added flash attention requirement warning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Qwen3.5 thinking mode puts the answer in content field, but
reasoning_content mentions ALL needle codes from the haystack.
Scoring from reasoning gave false positives. Now only score
from content — if model ran out of tokens during thinking,
it's a legitimate miss.
Also: increased max_tokens to 2048 (model needs room to finish
thinking) and server context headroom to +4096.
TODO: Proper rewrite based on Kamradt/RULER methodology — need
depth sweep, diverse filler, distractor needles.
Qwen3.5 with --jinja uses thinking mode by default, returning empty
content field (thinking goes in a separate field). Fix:
1. Send enable_thinking: false in the API request
2. Strip <think>...</think> tags from response as fallback
3. Increase max_tokens to 64 (thinking models need more room)
Also fixed test assertions for new needle positions (5%-95%)
and server timeout (120s).
niah_test.py — standalone NIAH benchmark for TurboQuant:
- Spins up llama-server per config, inserts 6-digit "secret code" needles
into filler text, queries via /v1/chat/completions, scores exact match
- Tests all combos of depths x needles x cache types
- Needle positions span 5%-95% of context (not just first half)
- Regex-based scoring (exact 6-digit word boundary match, no false positives)
- Configurable: --depths, --needles, --cache-types, --server-timeout,
--query-timeout, --server-bin, --chars-per-token, --verbose
- Server lifecycle: atexit + signal handlers, stderr captured to temp file
- Deterministic: seed=42, positions logged for reproducibility
- No personal paths in defaults, cross-platform (macOS + Linux)
Hardened per Codex + Roast review:
- Scoring: regex r'\b(\d{6})\b' not substring match (Codex critical)
- Positions: 5%-95% spread not 10%-50% (Roast critical)
- 20 cities (was 10), validates --needles <= city count
- Server timeout 120s (was 60), query timeout 300s (was 120)
- --server-bin flag for ROCm/custom builds
- Server stderr always captured for remote debugging
- Token estimation documented + --chars-per-token override
74 tests, all passing. No /Users/tom/ in any file.
Reference: @jtdavies found TQ 3.5-bit drops to 17-45% of bf16 NIAH
at 128K on MLX. This script lets us measure where we stand on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. Zip failure no longer deletes raw artifacts — preserves data if zip fails
2. main() returns 1 on unhandled exception (was always returning 0)
3. --skip-ppl no longer requires llama-perplexity binary
4. Multi-GPU nvidia-smi: sum memory/util, max temp across all GPUs
5. Model path logged as basename only (was leaking full path with username)
6. Build instructions now platform-aware (Metal vs CUDA)
7. Sharing instructions: GitHub issues + @no_stp_on_snek DM
8. Removed legacy bash diagnostic script (superseded by Python v5)
253 tests, 100% coverage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Coverage: 66% → 97% (127 new tests)
Failure logging: 7 → 60 log points — every except block now describes
what it was trying to do and includes the exception detail.
New features:
- detect_storage_type() — SSD/HDD detection (macOS diskutil + Linux rotational flag)
- [STORAGE] tag in diagnostic output
- HDD warning if model is on spinning disk
New test classes: TestDetectStorageType, TestFindModel, TestDiagLogExtended,
TestBackgroundMonitorExtended, TestLiveDisplayExtended, TestAnomalyDetectorExtended,
TestCaptureLoad, TestSectionsExtended, TestBuildJsonProfileExtended,
TestPackageResultsExtended, TestMainFunction, TestRunCmd, TestSubprocessRunner,
TestLinuxHardwareDetection, TestExceptPaths
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. BackgroundMonitor: add threading.Lock for _samples/_sample_count
2. Fix UnboundLocalError: initialize hw={} and gpu_init="" before try block
3. SIGINT handler: use _interrupted flag to prevent writing to closed log
4. Monitor errors tracked instead of silently swallowed
5. Guard finally block against interrupted state
Also added outlandish detection (both directions):
- turbo3 faster than q8_0 → [INVESTIGATE]
- Excellent decode ratio at long context → [NOTABLE]
- Decode improving at deeper context → [INVESTIGATE]
- PPL better than q8_0 → [INVESTIGATE]
- 10 new tests for notable/investigate flags
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Tests decode speed at realistic context depths with actual document content.
Compares turbo3 vs q8_0 using llama-server API.
Usage: bash scripts/turbo-realworld-bench.sh [path-to-pdf]
PDF not included in repo — testers provide their own.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
These belong in our repo, not in a fork of someone else's project.
Added:
- docs/quality-benchmarks.md — perplexity validation, bisection log, top-of-tree table
- docs/turbo-speed-investigation.md — Metal gotchas, fp16 WHT, optimization history
- docs/pre-rotate-queries-investigation.md — WHT/RoPE commutativity issue
- docs/pre-rotate-queries-plan.md — original optimization plan
- scripts/turbo-quality-gate.sh — pre-push perplexity check
README links updated to point locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
