mirror of
https://github.com/morluto/rea.git
synced 2026-10-22 05:42:31 -05:00
Ysopwnableandfactory-droid[bot]
ee87111009
Add optional owned Codex tool/usage capture and exact-name discovery experiments while preserving default ephemeral trials and all REA public APIs. Retain scoped managed and actual Android results, mixed outcomes, source identities and resource unknowns. Six semantic Python tests, full checked pre-push, byte-identical own-file rebase verification, manual review and all 39 exact-head CI checks pass. Historical main fixture failures and their independent repair remain documented; measured source/runtime pins are unchanged. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
210 lines
13 KiB
Markdown
210 lines
13 KiB
Markdown
# Observe the client boundary before claiming token savings
|
||
|
||
Continues [#2016](https://github.com/morluto/rea/issues/2016) after the managed
|
||
summary [PR #2413](https://github.com/morluto/rea/pull/2413) merged as
|
||
`82cfcee19f18b0eeae9a330ba04e0411cdf85225`. That opt-in view has verified smaller
|
||
delivery and complete-default migration, while its original real-model cohort
|
||
does not establish token savings. This follow-up measures why raw reply bytes
|
||
and aggregate model usage cannot be treated as interchangeable observations.
|
||
|
||
## Concrete diagnostic observation
|
||
|
||
An individual real Codex CLI 0.161.0 control uses frozen runtime
|
||
`ae8051302b5834322fa00cfa31c968ae74ab6ed9`, the unchanged ShareX identity question,
|
||
requested `gpt-5.6-terra` high and the same two enabled REA tools. A private
|
||
diagnostic copy removes the original runner's explicit `--ephemeral` flag.
|
||
Authentication/profile cleanup still runs; only selected tool/usage items are
|
||
retained after filtering the owned rollout. Its source rollout SHA-256 is
|
||
`d4df41d6187880d177f00b7533d85ad032c9c4ec502a39e7d5a566180e10461b` and the retained
|
||
filtered-item SHA-256 is
|
||
`0d5b4fb67c5674d1ba14ac32ae37e9a72b0368f4d9f5f9f81eb4d13193cda260`.
|
||
The original whole-copy is removed; account metadata, world state, messages and
|
||
internal passthrough are not retained in the filtered receipt.
|
||
|
||
Despite the requested `direct` configuration label, the observed client uses
|
||
three JavaScript `exec` calls. First it filters `ALL_TOOLS` with a broad
|
||
`/REA|reverse|artifact|managed|assembly|metadata/i` expression and prints matching
|
||
metadata. The client reports truncation of a 164,005-token original output.
|
||
This is the client's output measurement, not billed input or a complete inventory.
|
||
Next it invokes complete managed inspection but selects requested normalized
|
||
identity/classification fields before printing. Finally it invokes native-boundary
|
||
inspection and prints selected fields. The actual original complete MCP replies
|
||
remain available in the separate public-event transcript.
|
||
|
||
The persisted custom-tool output values are 41,704 / 2,154 / 13,489 UTF-8 bytes.
|
||
These are the serialized recorded output **values**, not raw wire bytes or exact
|
||
submitted model requests. The trace reports four per-response usage observations:
|
||
|
||
| Response | Input | Cached input | Output |
|
||
| -------- | -----: | -----------: | -----: |
|
||
| 1 | 10,400 | 0 | 132 |
|
||
| 2 | 19,992 | 9,984 | 456 |
|
||
| 3 | 21,242 | 19,200 | 251 |
|
||
| 4 | 24,799 | 20,224 | 1,535 |
|
||
|
||
Their totals match public turn usage: 76,433 input, 49,408 cached, 27,025 uncached
|
||
and 2,374 output, with 61.436 seconds observed wall time. Two MCP calls alone do
|
||
not describe the client's discovery/execution rounds. This proves the observed
|
||
control can project a complete result before printing it; it does not establish
|
||
that every earlier control/candidate did the same, or attribute every token
|
||
difference to discovery. The earlier `direct` label specifies requested client
|
||
configuration, not a verified absence of an executor.
|
||
|
||
Initial transcript-capture attempts retained empty source lists because the
|
||
original runner intentionally passed `--ephemeral`. A first private flag-removal
|
||
edit missed its newline and failed; that candidate remains an unsuccessful
|
||
capture attempt. The later corrected control is an individual diagnostic,
|
||
not a comparable pair with that candidate. Other exploratory aggregates remain
|
||
private and are not silently folded into the original four-trial cohort.
|
||
|
||
## Reproducible optional capture
|
||
|
||
`agent-trial.py --capture-tool-context` omits `--ephemeral` only for that explicitly
|
||
selected temporary trial. Its default is unchanged. The profile still disappears
|
||
in `finally`, including when capture fails. The capture reads only its owned
|
||
`sessions/*.jsonl` records, rejects symlink sources, and retains:
|
||
|
||
- Custom/function tool calls and replies, preserving call IDs and exact values.
|
||
- Per-response token counts and cumulative counts, with original response IDs.
|
||
- Client token-count events, retaining explicit unknown `info: null`.
|
||
|
||
It excludes session/account metadata, world state, user/developer messages,
|
||
encrypted reasoning, internal passthrough and unknown nested usage fields. It
|
||
does not copy authentication, configuration or the ambient environment. Receipts
|
||
record source/retained digests, event counts, malformed-line failures and missing
|
||
rollouts as unavailable. Capture is an observation of persisted client items,
|
||
not proof of the exact request accepted by a remote model service.
|
||
|
||
Run the same frozen runtime, question, tool selection and capture mode on both
|
||
sides. Preserve source/settings hashes and separate requested versus observed
|
||
behavior. Use the existing resource wrapper with unchanged one CPU, 4 GiB hard
|
||
memory, 3 GiB early-stop, no swap and task/host safeguards. Final physical counters
|
||
remain unknown when transient cgroups disappear; this instrumentation does not
|
||
repair that measurement limitation.
|
||
|
||
The focused Python tests run through the existing CI `test:python` gate.
|
||
They cover account/ambient exclusion while preserving caller data, malformed and
|
||
missing records, foreign symlinks, and capture failure without copied-credential
|
||
leakage. The cleanup test simulates a CLI and establishes only cleanup semantics;
|
||
it is not real-model/provider evidence.
|
||
|
||
## An explicit discovery-guidance experiment
|
||
|
||
`--guide-enabled-tools` is another opt-in experiment flag. It requires a nonempty
|
||
`--enabled-tools` list and adds the exact corresponding `mcp__rea__...` function
|
||
names to the temporary developer instructions, requesting only exact-name
|
||
metadata discovery. It preserves the analyst question, operation order, tool
|
||
selection and provider inputs. The default instruction is unchanged. Missing or
|
||
invalid selected tools fail before creating an output/profile. The test gate
|
||
also verifies the new prerequisite without acquiring a profile.
|
||
|
||
Compare guided and unguided runs on the **same** frozen REA runtime with matching
|
||
capture mode and requested facts. A smaller discovery response or additional
|
||
hint is not itself a task/token success: grade answers, count all response usage,
|
||
preserve failed/truncated runs and repeat both orders before recommending this
|
||
guidance. This is a caller-side experiment, not a new REA tool or provider preset.
|
||
|
||
## First committed capture pair
|
||
|
||
Checkpoint `9ebdcc458c399b9bdecaaca2ef47422888231181` runs the unchanged question
|
||
against frozen control `ae8051302b5834322fa00cfa31c968ae74ab6ed9` and candidate
|
||
`8bc1601ca51e417ca472d90ea0367e9418a1a0f2`, both with capture enabled and no tool
|
||
guidance. Both captures succeed with 10 retained events, unchanged source bytes,
|
||
removed profiles, two successful MCP calls and passing selected factual checks.
|
||
Full grader gates remain false because final physical resource counters are
|
||
unavailable; factual subsets do not override that limitation.
|
||
|
||
| Side | Input | Cached | Uncached | Output | Wall seconds |
|
||
| ----------------- | -----: | -----: | -------: | -----: | -----------: |
|
||
| Complete control | 52,656 | 27,136 | 25,520 | 2,015 | 57.736 |
|
||
| Summary candidate | 56,655 | 29,184 | 27,471 | 1,534 | 55.055 |
|
||
|
||
Both clients make two executor calls and three response-usage observations.
|
||
First they print broad discovered metadata, with truncation. Then each executor
|
||
sequentially performs both MCP operations, preserving selected-target dependency
|
||
order. The control prints a 2,086-byte output value; the candidate prints 22,770
|
||
bytes. Their discovery output values are 41,723 and 41,675 bytes. These are
|
||
recorded serialized values and remain distinct from raw MCP envelopes, billed
|
||
tokens and exact submitted requests. Both requested modes are `direct`; the
|
||
actual executor use is separately observed.
|
||
|
||
The candidate's smaller raw metadata reply does not result in a smaller printed
|
||
client value in this pair. Candidate input/uncached tokens remain higher. The
|
||
observed projection and round counts are useful constraints for the next
|
||
experiment; this one pair does not attribute all usage differences to either
|
||
formatting or discovery, or establish a reliable speedup.
|
||
|
||
## Same-runtime guided discovery, repeated in both orders
|
||
|
||
Checkpoint `e20bd0e2a905f06bc8c56fe04b0e61f3da88636a` runs four actual Codex trials
|
||
in unguided/guided/guided/unguided order. All use the **same** frozen candidate
|
||
runtime, artifact, original question, requested model/reasoning, two selected
|
||
tools, provider settings and capture mode. Only the explicit discovery-guidance
|
||
treatment changes. Every trial selects artifact summary and makes the same two
|
||
successful MCP calls in dependency order; source bytes stay unchanged, captures
|
||
succeed and profiles are removed.
|
||
|
||
| Trial | Input | Cached | Uncached | Output | Seconds | Observed responses |
|
||
| ---------- | ------: | -----: | -------: | -----: | ------: | -----------------: |
|
||
| Unguided 1 | 77,251 | 50,432 | 26,819 | 1,931 | 62.066 | 4 |
|
||
| Guided 1 | 39,175 | 23,040 | 16,135 | 1,763 | 55.658 | 3 |
|
||
| Guided 2 | 58,336 | 38,144 | 20,192 | 1,626 | 50.685 | 4 |
|
||
| Unguided 2 | 103,301 | 72,704 | 30,597 | 1,983 | 60.336 | 5 |
|
||
|
||
Both guided discovery output values repeat at 12,336 bytes without truncation,
|
||
versus 41,709 bytes with truncation on both unguided trials. The recorded
|
||
executor/output usage explains observable discovery and round differences; it
|
||
does not certify exact remote request contents or attribute every token to the
|
||
hint. CPU samples are 6.303–6.554 seconds and peak family memory 684,519,424–
|
||
712,028,160 bytes, with no guard abort. Final resource counters remain unknown.
|
||
These samples do not establish a memory or CPU improvement.
|
||
|
||
Unguided 1 omits `classification.container` and fails the selected factual gate;
|
||
it is not a fully acceptable comparison answer. Guided 1 and Unguided 2 pass
|
||
that gate. Guided 2 initially fails exact coverage-object equality because it
|
||
adds `module_ref_count: 0` and `pinvoke_import_count: 0` to the explicitly named
|
||
native scope. Both numbers exactly match the producer's returned arrays. The
|
||
grader now explicitly verifies only these two known additional counts, including
|
||
integer type, while still requiring all original coverage facts. Unknown extra
|
||
claims, conflicting aliases, wrong counts, boolean/string counts and omitted
|
||
issues continue to fail. Source coverage remains unchanged; returned-row counts
|
||
do not establish global absence. Original failure receipts remain preserved;
|
||
the regrade separately records its new source hash. Guided 2 then passes selected
|
||
facts, but every **full** gate still fails unknown final physical resource checks.
|
||
|
||
The second comparison has two factually acceptable answers: observed guided input
|
||
falls about 43.5%, uncached input about 34.0%, and wall time about 16.0%. The first
|
||
comparison has an incomplete unguided answer and is reported without counting it
|
||
as equivalent completed work. Two trials per treatment on one question/application
|
||
remain a preliminary signal, not a general default recommendation. The hint stays
|
||
opt-in; wider questions/domains, count/coverage representation and client round
|
||
behavior need further verification before promoting it into general guidance.
|
||
|
||
## Real Android follow-up: the primary outcomes are mixed
|
||
|
||
Checkpoint `22834c7d73056e2480b426848f58c31af0c307bc` checks the pinned real
|
||
NewPipe 0.29.1 APK using actual configured JADX 0.7.1/Java, the original Android
|
||
question and the same frozen candidate runtime. First a fresh SDK workflow
|
||
validates three live advertised operations and produces the comparison facts:
|
||
package/version identity, every case-sensitive MainActivity match and all
|
||
reported onCreate entries, preserving signatures, overload indices and null DEX
|
||
descriptors. It takes 26.961 seconds; the owned scope samples 1,234,735,104 bytes.
|
||
|
||
Both subsequent actual Codex trials use the same three selected tool names,
|
||
capture mode, model/reasoning, question and provider settings. Only discovery
|
||
guidance changes. Each makes three successful MCP calls and five observed model
|
||
responses, passes selected facts against the fresh actual producer, preserves
|
||
source bytes and removes its temporary profile. No APK execution is requested.
|
||
|
||
| Trial | Input | Cached | Uncached | Output | Wall seconds | Sampled peak bytes |
|
||
| -------- | -----: | -----: | -------: | -----: | -----------: | -----------------: |
|
||
| Unguided | 99,913 | 80,640 | 19,273 | 1,527 | 66.797 | 1,405,829,120 |
|
||
| Guided | 89,220 | 62,464 | 26,756 | 2,142 | 76.936 | 1,345,576,960 |
|
||
|
||
Total input falls about 10.7%, but uncached input increases and wall time is
|
||
longer. One pair does not demonstrate improved Android performance or memory.
|
||
No guard abort occurs; final counters remain unknown and full resource gates
|
||
remain false. This second domain is a concrete counterexample to generalizing
|
||
the favorable managed comparison. Keep both flags optional and continue
|
||
task-specific experiments before changing installed/default agent guidance.
|