rea/experiments/analysis-efficiency/model-utf8-validation.md

Omitted GGUF string validation

Issue #5774 follows the existing real-model and completed-task audit in #5603. The coverage reconciliation explains why neither model inspection nor IL2CPP, firmware extraction or IDA should be described as wholly unaudited. This experiment changes only one model producer hot path.

Owner and preserved semantics

GGUF arrays retain eight preview values. Previously, each omitted string still constructed a fatal UTF-8 decoder and materialized discarded text. GgufHeader now validates an omitted value of at most one existing 65,536-byte chunk with Node's isUtf8. Longer values retain the streaming fatal decoder, including multi-byte characters split across chunks. Retained values use their original decoder, preserving BOMs and exact text.

Every admitted byte is still read, hashed and validated. Header bounds, representation admission, cancellation checks, array-loop fairness, source offsets, partial-progress semantics, preview omission and complete retained Evidence are unchanged. There is no cache, new contract, selector or default. CLI and MCP share this producer through the existing application workflow.

Corpus and correctness

The curated receipts pin source commits, runtime file tables, driver/resource sources, immutable public URLs, artifact digests, complete report/answer fingerprints and all final samples. Baseline is main 264862bf2e3612c9cb6fcfb861843cbba4e1893f; measured candidate code is c2550343173c5abd8f34e925eac1bcc12b7ee6a2. Later documentation/integration commits do not relabel that cohort. That measured local commit predates the rebase: its production file is byte-identical to published 2f091ccc734d7bd945bd61432d41cf5ba68be12b, with both source hashes recorded.

Artifact	Bytes	Metadata / tensors	Independent reader
ggml-org stories15M Q4_0 GGUF	19,077,344	20 / 57	gguf 0.19.0
ggml-org stories260K GGUF	1,185,376	19 / 48	gguf 0.19.0
hf-internal-testing tiny-random-bert safetensors	520,212	2 / 100	safetensors 0.8.0

Official independent readers match all exposed metadata values/array previews and counts, tensor names, types, shapes and parameter counts. GGUF reader offsets/lengths match every tensor; safetensors reader geometry and independent Python JSON offsets match its file ranges. Python byte hashing checks every header digest. No weights are executed or loaded into a model framework. The main GGUF has 32,000 vocabulary entries; the smaller GGUF and safetensors are controls, not evidence for larger vocabularies or other GGUF versions.

Completed public tasks

Linux x64, Node 24.18.0/npm 11.16.0. Three alternating baseline/candidate, candidate/baseline, baseline/candidate pairs per artifact run serially. Each fresh production stdio MCP task discovers all 212 tools, directly selects two metadata records, retrieves two exact tensors from retained Evidence, repeats both calls, exports complete Evidence, validates answers and closes the transport. OS caches are unflushed; the controller reads inputs for hashing. Fresh process does not mean cold disk or cold model-engine analysis.

Medians of three whole tasks per side, including SDK/catalog/schema validation, all five calls, source checks, export and teardown:

Artifact	Wall seconds, before → after	Family CPU seconds, before → after	Peak family MiB, before → after
stories15M	6.667 → 6.567	6.498 → 6.386	771.0 → 816.3
stories260K	6.565 → 6.566	6.324 → 6.338	743.9 → 744.9
tiny BERT	6.667 → 6.569	6.461 → 6.388	741.6 → 741.3

The primary task's observed wall reduction is about 1.5%, but the unchanged safetensors control has similar variation. No stable whole-task speedup is established. Main-model peak memory rises: ranges 769.3–799.0 → 813.6–818.9 MiB; its cause is unmeasured. No peak-memory improvement is claimed.

Separate fresh-process provider cohorts perform one first inspection and six repeated inspections, comparing the complete report after every operation:

Artifact	First / repeat median ms, before	First / repeat median ms, after
stories15M	154.03 / 53.58	121.41 / 37.19
stories260K	33.63 / 8.77	30.51 / 8.17
tiny BERT	27.65 / 8.11	28.01 / 8.08

For stories15M this is about 21% faster first inspection and 31% faster repeated inspection. Median CPU for the seven-inspection process is 0.855 → 0.729 s; peak family memory is 75.6 → 77.9 MiB. Phase gains do not include discovery or substitute for the complete-task results above.

One catalog plus all named request/reply JSON chunks estimates median 895,583 / 895,421 / 893,755 cl100k_base tokens for the three artifacts in both implementations, using tiktoken 0.12.0. This includes serialized output schemas, not merely named input schemas. It is an SDK serialization proxy; actual model usage, repeated host injection, negotiation and framing are unmeasured. No token saving is claimed.

All 18 public tasks and 18 provider processes completed under one CPU, 4 GiB hard / 3 GiB early-stop family memory, no swap and 192 tasks. Payload exit, counters and owned cleanup are verified; no limit/OOM events occurred. Kernel cgroup peak includes file cache. Terminal whole-scope counters after scope removal remain unknown. Raw inputs, exports, profiles and transcripts stay outside Git.

Reproduction and validation boundaries

Build the baseline and the same baseline with the published GGUF source file using the pinned toolchain; this reconstructs the measured local candidate. Capture a complete baseline normalized report from existing CLI/MCP Evidence using the same absolute input path. Run model-utf8-validation.mjs with absolute --runtime, --artifact, --expected and a new --output directory under resource-run.py --observe-payload-exit. Add --phase for the seven-inspection provider probe. The driver validates advertised named schemas through the actual SDK and compares full exported parents, preview facts and tensor selections. A failing receipt or cleanup invalidates a task; it is not silently retried as success.

Five new producer regressions cover empty omitted strings, exact-chunk strings, split multi-byte characters, both byte orders, retained BOMs and rejection of short/long malformed omitted values. Existing malformed-header, admission, cancellation, partial-progress and public CLI/MCP tests remain required. An initial new-test assertion incorrectly hashed alignment padding; the corrected oracle uses the actual declared header, matching existing semantics. The first test launcher refused built projects; the exact built-project gate was then used. Neither attempt is a passing check. The initial resource-wrapper copy failed source-hash lookup before launching a build; corrected guarded builds qualified before measurement.

Integrated onto main 71b696582: all 68 tests pass again, and three real public CLI complete reports plus three production SDK tasks match the original normalized reports. Typecheck, lint/module boundaries, formatting, metadata, runtime build, docs checks (five tests) and separate architecture guards pass. npm run check does not pass: Knip reports an unlisted g++ invocation in the Windows test added by main's #5772. An independent main source checkout, with matching compiled worker artifacts and an uncached Knip run, reproduces only that finding. It is outside this change and is left unchanged. Initial control attempts lacked resolvable worker artifacts and are not clean controls. Development builds/checks used separate 6/8 GiB hard guards; runtime experiments retain their 4 GiB limit. No guard stop or OOM occurred.

Greptile identified a receipt defect in the phase driver, not the model parser: failed inspections/assertions were not recorded before cleanup, and a close failure could suppress publication. Each attempted phase now records its cost and primary failure before cleanup. Phase and MCP paths share one atomic receipt publisher; cleanup failures are independent facts and cannot replace the primary failure. Three new fault-injection tests fail on the original driver and pass after the correction; all 11 existing shared receipt tests pass. These stand-ins qualify failure accounting only. Real stories15M MCP and seven-inspection phase reruns also pass with verified counters/cleanup. The original 36 paired samples keep their measured driver hash; the receipt records the post-review driver and publication-owner hashes separately. No timing cohort is relabeled.