Qwen3.5 thinking mode puts the answer in content field, but
reasoning_content mentions ALL needle codes from the haystack.
Scoring from reasoning gave false positives. Now only score
from content — if model ran out of tokens during thinking,
it's a legitimate miss.
Also: increased max_tokens to 2048 (model needs room to finish
thinking) and server context headroom to +4096.
TODO: Proper rewrite based on Kamradt/RULER methodology — need
depth sweep, diverse filler, distractor needles.
Qwen3.5 with --jinja uses thinking mode by default, returning empty
content field (thinking goes in a separate field). Fix:
1. Send enable_thinking: false in the API request
2. Strip <think>...</think> tags from response as fallback
3. Increase max_tokens to 64 (thinking models need more room)
Also fixed test assertions for new needle positions (5%-95%)
and server timeout (120s).
niah_test.py — standalone NIAH benchmark for TurboQuant:
- Spins up llama-server per config, inserts 6-digit "secret code" needles
into filler text, queries via /v1/chat/completions, scores exact match
- Tests all combos of depths x needles x cache types
- Needle positions span 5%-95% of context (not just first half)
- Regex-based scoring (exact 6-digit word boundary match, no false positives)
- Configurable: --depths, --needles, --cache-types, --server-timeout,
--query-timeout, --server-bin, --chars-per-token, --verbose
- Server lifecycle: atexit + signal handlers, stderr captured to temp file
- Deterministic: seed=42, positions logged for reproducibility
- No personal paths in defaults, cross-platform (macOS + Linux)
Hardened per Codex + Roast review:
- Scoring: regex r'\b(\d{6})\b' not substring match (Codex critical)
- Positions: 5%-95% spread not 10%-50% (Roast critical)
- 20 cities (was 10), validates --needles <= city count
- Server timeout 120s (was 60), query timeout 300s (was 120)
- --server-bin flag for ROCm/custom builds
- Server stderr always captured for remote debugging
- Token estimation documented + --chars-per-token override
74 tests, all passing. No /Users/tom/ in any file.
Reference: @jtdavies found TQ 3.5-bit drops to 17-45% of bf16 NIAH
at 128K on MLX. This script lets us measure where we stand on Metal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
