rea/experiments/analysis-efficiency/signal-selection-workflows.md

Sigrok selection and repeated-page checkpoint (#3131)

This experiment preserves production APIs, defaults, caller selections and Evidence semantics. Existing annotation selectors substantially reduce replies, but this small real-model cohort does not establish a consistent reduction in total input tokens or elapsed task time. No decoder cache or new production preset is introduced. Complete necessary output stays available through existing explicit null limits; original defaults remain 50 display lines.

Immutable corpus and independent questions

The runtime is pinned to ca7450beffff59325ce3bddc78e181693ce77177, with Node 24.18.0, SDK 2.3.1 and Linux x86-64. Experiment driver code for model trials is 23317a58d; each SDK result separately records workflow/oracle source hashes, runtime-file and lockfile digests, explicit provider settings and original bytes. Later integration checks must remain separate from this cohort.

Two official sigrok-dumps recordings are pinned to upstream commit 0ad13477abc959d37fc9a5acbd23901c371c9c76 in corpus.json. Original Git blob IDs, SHA-256, sizes and immutable reference/download URLs are recorded there. No sample binaries or raw output are committed. The upstream repository declares dumps public domain unless otherwise noted; the inspected sample READMEs state no exception. Decoder software licensing is separate.

• MAX3232E UART: 12,742 compressed bytes, 10,000,000 logic bytes, 5,000,000 samples at the recorded nominal 100 MHz. Stored DOUT1 is bit 1; DIN1 is bit 2. The exact question requests all RX octets in order, including repetitions. Independent 57,600-baud 8N1 stop-bit sampling with DOUT1 inversion yields 65 bytes: five copies of Hello world\r\n. The source metadata and physical probe numbers remain distinct from stored bit positions.
• Wii Nunchuk I2C: 20,373 compressed bytes, 20,000,000 samples at nominal 1 MHz. The source README's physical probes 3/7 are repacked as stored named channels SCL/SDA at bits 0/1. The question asks every read payload as consecutive six-byte arrays, excluding address/write bytes and hardware/button inference. Independent START/STOP, rising-clock, byte and ACK/NACK reading identifies seven transactions and three read payloads, matching the upstream reference: 75 7F 77 4F 82 3B, 75 7F 75 44 82 34, 75 7F 77 43 83 5D.

The independent reader accepts the scoped single-device classic-ZIP profile, checks directory representation before object allocation, bounds compressed and expanded input to 32 MiB, and limits selected transitions to 65,536. It preserves source metadata/digest, chunk ordering and channel identity; incomplete samples, ambiguous names, partial bytes and unsupported transactions fail explicitly. Receiver rearming follows the verified UART stop sample, accommodating the recording's slightly faster sender. I2C STOP preparation is distinct from data. These checks do not establish other formats, decoders, OSes or physical acquisition.

An initial DOUT1 decode without inversion exited successfully but produced wrong bytes. Its cost/output is retained privately as configuration exploration, outside eligible measurements. An independent sample oracle prevents equal-but-wrong producer results from passing. Correct inversion was established before the cohort.

Actual CLI and SDK evidence

sigrok-cli 0.7.2 reports libsigrok 0.5.2 and libsigrokdecode 0.5.3. Its executable SHA-256 is e1b1eb2f25edc17d24a78acbab16ff7e1dfdee74bb2cccb7ad00497ccc7a0c44. Selected installed UART/I2C decoder module bytes are fingerprinted privately. The banner and executable digest do not fingerprint the complete installation; modules and libraries remain trusted. Both actual CLI and MCP results match raw sample bytes; data intervals correlate with the independently examined samples.

Fourteen actual SDK workflows comprise two interleaved repetitions per recording and arm (baseline/focused/discovery), plus first/next/repeated pages. All 22 calls preserve source bytes and necessary decoded facts. Baseline returns all necessary annotation output. Focused selects uart=rx-data or i2c=data-read, with explicit null line limits because the task needs the entire selected payload. Discovery first queries the installed decoder help for the already specified class; it does not establish discovery benefits for unknown protocols.

Recording	Baseline bytes	Focused bytes	Discovery + focused bytes	First + next + repeat bytes
UART	49,417	12,441	18,839	31,779
I2C	26,829	9,928	16,274	27,623

Focused reply reductions are 74.8% and 63.0%. These are parsed SDK JSON bytes, not wire bytes or model tokens. Baseline/focused RPC times (ms, repetitions 1/2) are UART 2010/582 versus 1459/911, and I2C 2851/676 versus 2330/685; n=2, with order/cold/warm variation, does not establish a general speedup. Pages retain input/class/window coverage separately, rather than equating display exhaustion with a complete protocol decode. Repeated first pages match their original text and scope; reused content-based Evidence identities are valid.

Complete real-model task costs

Eight actual Codex CLI 0.161.0 tasks requested gpt-5.6-terra / high, in isolated profiles using this runtime's REA MCP. Both tools were enabled and preapproved identically in all arms; shell, browser and extra agents were disabled. The model alias is a requested identifier, not independently resolved backend identity. Questions contain no expected payloads. Trial order is baseline 1, focused 1, focused 2, baseline 2, with UART then I2C in each step.

Trial	Total input	Cached input	Uncached input (derived)	Output	Seconds	Completed tool calls	Facts / fixed plan
uart baseline 1	92,399	49,408	42,991	1,295	86.411	1	passed / passed
uart focused 1	130,029	98,048	31,981	780	49.623	1	passed / passed
uart focused 2	98,894	78,592	20,302	878	73.861	1	passed / passed
uart baseline 2	90,395	48,384	42,011	926	43.974	1	passed / passed
i2c baseline 1	172,162	121,344	50,818	1,194	36.330	2	passed / failed
i2c focused 1	74,815	49,408	25,407	937	114.515	1	passed / passed
i2c focused 2	74,684	48,384	26,300	668	73.877	1	passed / passed
i2c baseline 2	86,241	49,408	36,833	778	41.084	1	passed / passed

All eight answers match independent raw samples and cite authenticated decoder observations. I2C baseline 1 also queried generic MCP resources before its single required decode. The predeclared strict single-call grader marks its plan failed; its actual 172,162 input tokens and 36.330 seconds remain recorded, separately from the eligible fixed-plan pairs. It is not retried into a better result.

UART eligible pairs increase total input by 40.7% and 9.4%; elapsed changes are -42.6% and +68.0%. The one eligible I2C pair reduces total input by 13.4% but increases elapsed time by 79.8%. Reduced uncached input does not replace total input as the primary token measure. No reliable whole-task token/time benefit is established across these two questions; reply projection alone is insufficient.

Producer, window and resource scope

Eight separately bounded direct producer runs preserve source bytes and render correct stdout: UART baseline 159/131 ms versus focused 163/130 ms; I2C baseline 355/298 ms versus focused 317/290 ms. This isolates the producer process from REA RPC, version probing and snapshot work, but startup/decoding/annotation formatting remain combined. Decoding-only versus formatting-only time is unmeasured.

REA literal-window extraction and JSON encoding were measured separately from retained producer text (one cold observation and 200 warm iterations per arm). Warm mean extraction ranges from 0.0040 to 0.1794 ms; JSON encoding from 0.0021 to 0.0121 ms. These exclude file reads, RPC and decoding.

Every page still invokes decoding. Retained output might help repeated inspection, but this fixed cohort does not demonstrate a completed-task improvement warranting a new cache. Any future work should own bounded output explicitly by original Evidence identity and lifetime, with cancellation/cleanup, rather than use a CLI-digest-only generic cache of an incompletely fingerprinted installation.

Heavy work ran serially under one CPU, 3 GiB early stop / 4 GiB hard memory, no swap and host-availability checks. Verified payload-exit peak bytes were SDK 587,513,856; model 889,323,520; direct producer 26,288,128. No OOM/OOM-kill or resource abort occurred in that verified phase; task descendants were cleaned. Terminal whole-scope counters are unavailable and remain unknown.

Verification and replay

Eleven independent-oracle regressions and 12 comparator/sample-interval tests pass. Twenty-four private replays using actual SDK Evidence cover valid answers, failed calls before/after recovery, wrong bytes, unknown citations and missing limitations. These are grading checks with injected state, not additional model/provider trials. The grader retains authenticated observations by identity, rejects conflicts, combines cited pages in sample order and keeps failed calls/plan deviations separate from supported facts. Relevant typecheck/lint and pre-push checks pass.

Review hardening ties the oracle to the named corpus digest, bounds each I2C interval against neighboring byte clocks/transaction STOP, and counts actual classic-ZIP directory records on the same bounded snapshot before allocating ZipFile objects; ZIP64 is rejected. Forged-count and ZIP64 admission regressions fail on the original implementation. Replaying all eight original model trials and 24 injected grading cases with these checks preserves their factual/plan grades and recorded usage/time; these replays are not new model measurements.

Metadata reads cap inflater output even when ZIP sizes are forged. A streamed 8 MiB malformed metadata member declaring 8 KiB is rejected with traced Python allocation below 1 MiB; the same regression fails on the preceding implementation. Compressed snapshots use actual file size plus one growth-detection byte rather than preallocating the entire admission budget. Both genuine waveform oracles retain identical bytes, metadata, sample intervals and source digests.

UART interval checks also require coverage of the first and last independently sampled data-bit centers within the owning frame. Shortened, shifted and single-sample intervals are rejected even when the decoded byte is correct. The added oracle coordinates retain the original byte/frame facts; replaying saved real Evidence and all eight model transcripts preserves the original grades and costs. This is stronger retrospective verification, not a new cohort.

After rebasing onto main 44d98de2367b99808d78db732ec7d72533266233, npm run build:cached and the 16 focused comparator/sample-interval/MCP-boundary tests passed. Two additional real focused MCP workflows independently verified UART/I2C bytes and intervals on that base. These integration checks are separate from the original cost cohort and do not replace or extend its measurements.

A subsequent rebase onto main 3b79af28e80785afce9f5998d8f03289402a1847 passed the 17 focused tests, including the upstream-fixed Hopper doctor case, and two additional real UART/I2C MCP workflows. Original cohort costs and grades remain unchanged; these checks verify integration on the updated main only.

Use measure.mjs --signal-selection baseline|focused|discovery|pages, preserving runtime/source fingerprints and explicit provider JSON. Generate the exact task with signal-model-prompt.mjs --artifact ID --selection baseline|focused, run the isolated agent-trial.py --mode direct --capture-tool-context, and grade with review-signal.mjs and the independently produced oracle. Keep original failures, resources and raw observations private. For migration, existing optional annotation selection is sufficient for scoped reply reduction; no default/API change follows from this mixed whole-task result.