niah_test.py — standalone NIAH benchmark for TurboQuant:
- Spins up llama-server per config, inserts 6-digit "secret code" needles
into filler text, queries via /v1/chat/completions, scores exact match
- Tests all combos of depths x needles x cache types
- Needle positions span 5%-95% of context (not just first half)
- Regex-based scoring (exact 6-digit word boundary match, no false positives)
- Configurable: --depths, --needles, --cache-types, --server-timeout,
--query-timeout, --server-bin, --chars-per-token, --verbose
- Server lifecycle: atexit + signal handlers, stderr captured to temp file
- Deterministic: seed=42, positions logged for reproducibility
- No personal paths in defaults, cross-platform (macOS + Linux)
Hardened per Codex + Roast review:
- Scoring: regex r'\b(\d{6})\b' not substring match (Codex critical)
- Positions: 5%-95% spread not 10%-50% (Roast critical)
- 20 cities (was 10), validates --needles <= city count
- Server timeout 120s (was 60), query timeout 300s (was 120)
- --server-bin flag for ROCm/custom builds
- Server stderr always captured for remote debugging
- Token estimation documented + --chars-per-token override
74 tests, all passing. No /Users/tom/ in any file.
Reference: @jtdavies found TQ 3.5-bit drops to 17-45% of bf16 NIAH
at 128K on MLX. This script lets us measure where we stand on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. Zip failure no longer deletes raw artifacts — preserves data if zip fails
2. main() returns 1 on unhandled exception (was always returning 0)
3. --skip-ppl no longer requires llama-perplexity binary
4. Multi-GPU nvidia-smi: sum memory/util, max temp across all GPUs
5. Model path logged as basename only (was leaking full path with username)
6. Build instructions now platform-aware (Metal vs CUDA)
7. Sharing instructions: GitHub issues + @no_stp_on_snek DM
8. Removed legacy bash diagnostic script (superseded by Python v5)
253 tests, 100% coverage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Coverage: 66% → 97% (127 new tests)
Failure logging: 7 → 60 log points — every except block now describes
what it was trying to do and includes the exception detail.
New features:
- detect_storage_type() — SSD/HDD detection (macOS diskutil + Linux rotational flag)
- [STORAGE] tag in diagnostic output
- HDD warning if model is on spinning disk
New test classes: TestDetectStorageType, TestFindModel, TestDiagLogExtended,
TestBackgroundMonitorExtended, TestLiveDisplayExtended, TestAnomalyDetectorExtended,
TestCaptureLoad, TestSectionsExtended, TestBuildJsonProfileExtended,
TestPackageResultsExtended, TestMainFunction, TestRunCmd, TestSubprocessRunner,
TestLinuxHardwareDetection, TestExceptPaths
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
1. BackgroundMonitor: add threading.Lock for _samples/_sample_count
2. Fix UnboundLocalError: initialize hw={} and gpu_init="" before try block
3. SIGINT handler: use _interrupted flag to prevent writing to closed log
4. Monitor errors tracked instead of silently swallowed
5. Guard finally block against interrupted state
Also added outlandish detection (both directions):
- turbo3 faster than q8_0 → [INVESTIGATE]
- Excellent decode ratio at long context → [NOTABLE]
- Decode improving at deeper context → [INVESTIGATE]
- PPL better than q8_0 → [INVESTIGATE]
- 10 new tests for notable/investigate flags
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
Tests decode speed at realistic context depths with actual document content.
Compares turbo3 vs q8_0 using llama-server API.
Usage: bash scripts/turbo-realworld-bench.sh [path-to-pdf]
PDF not included in repo — testers provide their own.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
These belong in our repo, not in a fork of someone else's project.
Added:
- docs/quality-benchmarks.md — perplexity validation, bisection log, top-of-tree table
- docs/turbo-speed-investigation.md — Metal gotchas, fp16 WHT, optimization history
- docs/pre-rotate-queries-investigation.md — WHT/RoPE commutativity issue
- docs/pre-rotate-queries-plan.md — original optimization plan
- scripts/turbo-quality-gate.sh — pre-push perplexity check
README links updated to point locally.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
TheTom