Top-level at-a-glance table with PPL, NIAH, speed ratios across all
tested hardware, compression ratios, and link to decode regression
analysis doc.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First community diagnostic from @Ambisphaeric (M1 Max 64GB):
- Prefill: 93-98% of q8_0 (excellent, flat curve)
- Decode: 84% short → 39% at 32K (M1 constant LUT issue)
M2 Pro Mac Mini partial results (in progress, noisy from workloads).
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces /Users/username (or /home/username) with ~ in all output
files (log, JSON profile, monitor CSV) before packaging into the
zip. Prevents accidental PII leak when users share diagnostic results.
Reported by community tester who noticed their Mac username in
command paths logged by llama-bench output.
Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Community testers need to know which fork to build. Apple Silicon
uses our Metal fork, NVIDIA/AMD uses spiritbuun's CUDA fork.
Also added flash attention requirement warning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Qwen3.5 thinking mode puts the answer in content field, but
reasoning_content mentions ALL needle codes from the haystack.
Scoring from reasoning gave false positives. Now only score
from content — if model ran out of tokens during thinking,
it's a legitimate miss.
Also: increased max_tokens to 2048 (model needs room to finish
thinking) and server context headroom to +4096.
TODO: Proper rewrite based on Kamradt/RULER methodology — need
depth sweep, diverse filler, distractor needles.
Qwen3.5 with --jinja uses thinking mode by default, returning empty
content field (thinking goes in a separate field). Fix:
1. Send enable_thinking: false in the API request
2. Strip <think>...</think> tags from response as fallback
3. Increase max_tokens to 64 (thinking models need more room)
Also fixed test assertions for new needle positions (5%-95%)
and server timeout (120s).
niah_test.py — standalone NIAH benchmark for TurboQuant:
- Spins up llama-server per config, inserts 6-digit "secret code" needles
into filler text, queries via /v1/chat/completions, scores exact match
- Tests all combos of depths x needles x cache types
- Needle positions span 5%-95% of context (not just first half)
- Regex-based scoring (exact 6-digit word boundary match, no false positives)
- Configurable: --depths, --needles, --cache-types, --server-timeout,
--query-timeout, --server-bin, --chars-per-token, --verbose
- Server lifecycle: atexit + signal handlers, stderr captured to temp file
- Deterministic: seed=42, positions logged for reproducibility
- No personal paths in defaults, cross-platform (macOS + Linux)
Hardened per Codex + Roast review:
- Scoring: regex r'\b(\d{6})\b' not substring match (Codex critical)
- Positions: 5%-95% spread not 10%-50% (Roast critical)
- 20 cities (was 10), validates --needles <= city count
- Server timeout 120s (was 60), query timeout 300s (was 120)
- --server-bin flag for ROCm/custom builds
- Server stderr always captured for remote debugging
- Token estimation documented + --chars-per-token override
74 tests, all passing. No /Users/tom/ in any file.
Reference: @jtdavies found TQ 3.5-bit drops to 17-45% of bf16 NIAH
at 128K on MLX. This script lets us measure where we stand on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. Zip failure no longer deletes raw artifacts — preserves data if zip fails
2. main() returns 1 on unhandled exception (was always returning 0)
3. --skip-ppl no longer requires llama-perplexity binary
4. Multi-GPU nvidia-smi: sum memory/util, max temp across all GPUs
5. Model path logged as basename only (was leaking full path with username)
6. Build instructions now platform-aware (Metal vs CUDA)
7. Sharing instructions: GitHub issues + @no_stp_on_snek DM
8. Removed legacy bash diagnostic script (superseded by Python v5)
253 tests, 100% coverage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Coverage: 66% → 97% (127 new tests)
Failure logging: 7 → 60 log points — every except block now describes
what it was trying to do and includes the exception detail.
New features:
- detect_storage_type() — SSD/HDD detection (macOS diskutil + Linux rotational flag)
- [STORAGE] tag in diagnostic output
- HDD warning if model is on spinning disk
New test classes: TestDetectStorageType, TestFindModel, TestDiagLogExtended,
TestBackgroundMonitorExtended, TestLiveDisplayExtended, TestAnomalyDetectorExtended,
TestCaptureLoad, TestSectionsExtended, TestBuildJsonProfileExtended,
TestPackageResultsExtended, TestMainFunction, TestRunCmd, TestSubprocessRunner,
TestLinuxHardwareDetection, TestExceptPaths
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. BackgroundMonitor: add threading.Lock for _samples/_sample_count
2. Fix UnboundLocalError: initialize hw={} and gpu_init="" before try block
3. SIGINT handler: use _interrupted flag to prevent writing to closed log
4. Monitor errors tracked instead of silently swallowed
5. Guard finally block against interrupted state
Also added outlandish detection (both directions):
- turbo3 faster than q8_0 → [INVESTIGATE]
- Excellent decode ratio at long context → [NOTABLE]
- Decode improving at deeper context → [INVESTIGATE]
- PPL better than q8_0 → [INVESTIGATE]
- 10 new tests for notable/investigate flags
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Tests decode speed at realistic context depths with actual document content.
Compares turbo3 vs q8_0 using llama-server API.
Usage: bash scripts/turbo-realworld-bench.sh [path-to-pdf]
PDF not included in repo — testers provide their own.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
These belong in our repo, not in a fork of someone else's project.
Added:
- docs/quality-benchmarks.md — perplexity validation, bisection log, top-of-tree table
- docs/turbo-speed-investigation.md — Metal gotchas, fp16 WHT, optimization history
- docs/pre-rotate-queries-investigation.md — WHT/RoPE commutativity issue
- docs/pre-rotate-queries-plan.md — original optimization plan
- scripts/turbo-quality-gate.sh — pre-push perplexity check
README links updated to point locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
