Files
RuView/docs/validation/ruforecast-requirements-evidence.md
rUv e5bf0d4892 feat(forecast): add independent Rust multivariate training stack (#1766)
* feat(forecast): add independent Rust training stack

Signed-off-by: Codex <codex@openai.com>

* docs(forecast): record informal HPO exploration note (unaccepted)

Records an exploratory Darwin Mode numeric-genome hyperparameter search
against the tiny_ci synthetic accuracy protocol: WQL improved 0.257 -> 0.161
-> 0.153 over two search rounds, staying ahead of both baselines throughout.
Explicitly scoped as informal/exploratory (not a frozen leakage-free report,
not eligible for the evidence ledger, not validated for large_linux or any
real dataset) per this doc's own evidence-tagging discipline.

* feat(forecast-core): add per-horizon weighted quantile loss

weighted_quantile_loss collapses the whole horizon into one aggregate
number, hiding whether error grows with lead time. Add a sibling
function using the identical per-cell pinball formula and domain
checks, just reduced per horizon step instead of globally. A unit
test reconstructs the aggregate from the per-step numerators/
denominators to prove the two functions agree exactly, not just
approximately.

* feat(forecast-model): add evaluation-only local activation

Adds activate_for_evaluation and build_eval_input, gated behind the
cpu feature. activate_for_evaluation self-signs an unsigned candidate
with a fixed, publicly-known, non-secret Ed25519 key so a CLI operator
can run inference against their own just-trained candidate without a
real release signature -- explicitly never a production trust path,
documented as such in the module doc comment. The schema-digest check
in ArtifactActivationPolicy still applies in full; only signing is
relaxed. build_eval_input constructs a single-window ModelInput for
CPU inference from raw context-major values/mask arrays, matching the
training batch builders exact time-feature/descriptor encoding.

Six unit tests cover: malformed-candidate rejection, digest-mismatch
rejection (proving the schema check has teeth even on this relaxed
path), getting past the policy gate with a matching digest, and
eval-input shape validation.

* feat(forecast-train): add evaluate and prepare-synthetic-dataset CLI

Turns this sessions throwaway HPO scripts into real, tested CLI
surfaces, replacing two example binaries that bypassed the CLI/TOML
training path entirely.

prepare-synthetic-dataset generates a larger, configurable synthetic
training shard (default 24 windows) plus a matching train-local.toml
with the given OptimizerSpec hyperparameters baked in, and a separate
held-out test.jsonl -- the same synthetic-only, local-validation-only
posture as prepare-local-example, just bigger and configurable.

evaluate scores a trained (unsigned) candidate against held-out
windows and the LastValue/SeasonalNaive baselines: overall weighted
quantile loss, a per-horizon breakdown, 80% interval coverage, and
missingness, for the model and both baselines. Explicitly lists what
docs/benchmarks/ruforecast.mds full accuracy protocol additionally
asks for that this does not cover (abstention coverage, selective
risk, site/device slices, interference regime, RuVector-retrieval
ablation) rather than silently omitting them -- each needs
infrastructure a single-entity synthetic fixture does not have.

Together the three real commands (prepare-synthetic-dataset,
train-local, evaluate) reproduce the exact WQL numbers the old
library-bypassing example scripts produced for the same
hyperparameters, confirmed by re-running the harness/ruview HPO
dry-run end to end against the new CLI path.

New integration test trains a real candidate via smoke, evaluates it
against held-out synthetic windows, and asserts the full report shape
and an empty-input rejection.

* feat(forecast-train): add prepare-synthetic-dataset --seed + Autogenous bridge

--seed (default 0, backward compatible) makes prepare-synthetic-dataset's
generator deterministic per offset: same seed -> byte-identical corpus
across runs (needed to train two genomes on the SAME corpus for a fair
comparison); different seeds -> genuinely independent corpora (needed for
honest multi-seed/multi-judge evaluation upstream, both in
harness/ruview/flywheel/ruforecast/gate.mjs's fitness function and in the
new v2/crates/ruforecast-autogenous-bridge crate added here).

ruforecast-autogenous-bridge is a LOCAL-DEV-ONLY (excluded from the v2
workspace, publish = false) bridge from this crate's real evaluate CLI
into ruvnet/autogenous's new regression-candidate promotion path (separate,
unpushed branch feat/regression-candidate-kind): runs N independent
train+evaluate judges on genuinely distinct synthetic corpora, signs
receipts, and gets a real cryptographically-verified PROMOTE/REJECT
decision -- defense in depth on top of, not a replacement for, Darwin's own
promotion gate. Path-depends on a sibling autogenous checkout that does not
exist in CI; build/run directly via `cargo build --manifest-path
crates/ruforecast-autogenous-bridge/Cargo.toml`.

cargo test -p ruview-forecast-train --no-default-features --features
cpu,cli: all green (unaffected existing tests + this backward-compatible
addition).

* docs(forecast): retract round-2 HPO result, record honest multi-seed finding

The earlier "Informal HPO exploration note" claimed a real improvement
(WQL 0.257 -> 0.153) from a 3-round hyperparameter search. That search
evaluated every candidate against one fixed synthetic corpus (seed 0) for
every round -- textbook overfitting. Independent verification against two
fresh corpora (via the new ruforecast-autogenous-bridge crate) showed the
"winner" losing to the baseline on both.

Fixed the root cause in harness/ruview/flywheel/ruforecast/gate.mjs:
candidates are now scored against three independent corpora, worst-case
across them, not one fixed corpus.

Re-ran the search under the fix. It found a new winner that genuinely beat
baseline on all three of its own search seeds -- and that winner ALSO lost
independent verification on fresh seeds. Two independent search rounds,
pre- and post-fix, both produced an illusory "winner." The honest reading:
at this dataset scale (24 synthetic windows) held-out WQL varies enormously
by which corpus is drawn, regardless of hyperparameters -- confirmed
directly by the baseline genome's own primary swinging from 0.83 to a full
regression across the three fixed search seeds with unchanged
hyperparameters. No RuForecast hyperparameter configuration has been shown
to reliably beat the trivial baselines out-of-sample at this scale.

Also fixes a real design inconsistency surfaced by this exploration: the
Autogenous regression-candidate promotion verifier
(envelope::regression::verify_regression_promotion, separate unpushed
ruvnet/autogenous branch feat/regression-candidate-kind) required all
judges to share one corpus_id, which conflicts with this kind's
intentional cross-corpus judge design. Corrected there (commit bfa4c48);
did not change either REJECT verdict, which were already driven by the
real NotBetterThanParent signal on their own.

Append-only: the original round-2 row is kept, read together with this
new amendment section, per this doc's own evidence-ledger discipline.

* refactor(forecast): extract RuForecast to ruvnet/RuForecast submodule

Mirrors the v2/crates/worldgraph pattern: RuForecast becomes its own
independent public repo/workspace (ruforecast-core/model/train), mounted
at v2/crates/ruforecast as a git submodule, with v2/Cargo.toml path-depping
into its sub-crates and excluding the submodule from the v2 workspace.

Full v2 --workspace check passes; the ruforecast CLI builds and runs
correctly from its new location. Real git history for the extracted
crates (7 commits) was preserved via git-filter-repo, not squashed.

* feat(autogenous-bridge): real-data independent verification tool

real_data_verify.rs: signs two real, temporally-independent judge
measurements (different train/test split boundaries on the same real
household vitals corpus, not synthetic seeds) through the real
Autogenous regression-candidate promotion path, and gets a genuine
cryptographically-checked PROMOTE/REJECT verdict.

Result recorded (see docs/benchmarks/ruforecast.md): REJECT. One judge
nominally beat the trivial baseline but by less than the 0.01
non-inferiority margin; the other judge lost outright. Consistent with
every synthetic-data search this session -- no configuration has yet
been shown to reliably beat trivial baselines out-of-sample, now
including a real 6390-sample household corpus.

Bumps the ruforecast submodule to pick up the evaluate real-gap fix.

* docs(forecast): record real-household-data result (REJECT, signed)

6390 real vital-signs samples, two independent temporal splits,
independently verified via the real Autogenous regression-candidate
promotion path. Signed verdict: REJECT -- neither split cleared the
non-inferiority margin. Same conclusion as every synthetic search this
session, now confirmed with real data too: more real data is the
credible next lever, not further search on this scale of fixture.

* chore(forecast): point ruforecast submodule at published main (post gap-tolerance fix)

* ci(forecast): fix ruforecast-ci.yml for the new v2/crates/ruforecast submodule layout

The RuForecast crates were extracted into a standalone submodule (v2/crates/ruforecast, mirroring the existing v2/crates/worldgraph pattern) and are no longer members of the v2 workspace. The CI workflow still referenced the old in-tree package names/paths (ruview-forecast-core/model/train under v2/), which broke every forecast job with "cannot specify features for packages outside of workspace".

Fixed every job to target the submodule: working-directory -> v2/crates/ruforecast, -p ruview-forecast-* -> -p ruforecast-*, all hardcoded Cargo.toml/source paths in the Python assertion and clean-room-scan blocks, the Swatinem/rust-cache workspaces input, the artifact-tree output paths, and the two levels of relative path (../ -> ../../) in the informational-benchmark steps whose working directory moved one level deeper.

Also collapsed the paths: trigger filters internal-crate globs (v2/crates/ruview-forecast-*/**) down to the single v2/crates/ruforecast path, since GitHub Actions path filters only ever see a submodule gitlink change as one entry in the parent tree, never its internal file paths -- the old globs could never have matched anything.

Includes a real cargo fmt fix (submodule commit d9902f9) for formatting drift left over from the earlier real-data gap-tolerance fix, which the contract job's fmt --check step would otherwise have failed on.

All four forecast CI jobs' real commands verified locally before push: cargo test/check for CPU, CUDA/WGPU compile-check, the contract job's Python assertion block, the hosted-boundaries clean-room scan, and cargo fmt --check.

* docs(forecast): record real BIDMC cross-entity holdout result (REJECT)

Two independent real-patient partitions (34/16 and 24/26 splits, 53 total
ICU patients from PhysioNet BIDMC, Open Data Commons Attribution License
v1.0) both show the model losing to trivial baselines by a decisive margin
(+69% and +601% worse WQL). First genuine cross-entity real-data test this
project has run; same conclusion as every prior synthetic and single-household
test. Full Autogenous signed verification was not run for this entry -
flagged explicitly in the note.

* chore(forecast): point ruforecast submodule at published main (example feature-gate fix)

* chore(forecast): point ruforecast submodule at published main (clippy fix)

---------

Signed-off-by: Codex <codex@openai.com>
2026-09-01 23:35:42 -04:00

16 KiB

RuForecast requirement to evidence matrix

Authority and current state

This matrix operationalizes ADR-348 and its ADR-349/350 children without changing them. The ADRs own the requirements and gates. Rust tests, immutable reports, signed receipts, and explicit human approvals supply evidence. A checked box or green build is not permission to deploy, publish a checkpoint, upload customer-derived data, or advance the rollout mode.

Current state: all release and operational gates are open. No benchmark result, trained checkpoint, external training receipt, or production authority is recorded by this document.

Focused verification commands

The feature-off contract remains on the workspace Rust 1.89 line:

cd v2
cargo +1.89.0 test --locked -p ruview-forecast-core --no-default-features --lib --tests
cargo +1.89.0 test --locked -p ruview-forecast-model --no-default-features --lib --tests
cargo +1.89.0 check --locked -p ruview-forecast-model -p ruview-forecast-train --no-default-features --all-targets
cargo +1.89.0 test --locked -p ruview-forecast-model --no-default-features --features ruvector --lib --tests
cargo +1.89.0 clippy --locked -p ruview-forecast-core -p ruview-forecast-model -p ruview-forecast-train --no-default-features --all-targets -- -D warnings
cargo +1.89.0 clippy --locked -p ruview-forecast-model --no-default-features --features ruvector --all-targets -- -D warnings

Burn 0.21 CPU activation uses the explicitly separate Rust 1.92 line:

cd v2
cargo +1.92.0 test --locked -p ruview-forecast-model --no-default-features --features cpu --lib --tests
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli --lib --bins
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu --test local_jsonl_smoke -- --exact local_hash_addressed_jsonl_executes_one_real_optimizer_step
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli --test cli_smoke -- --exact cli_smoke_trains_and_writes_the_complete_candidate_set
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli,server,fal-client --lib --bins
cargo +1.92.0 clippy --locked -p ruview-forecast-model --no-default-features --features cpu --all-targets -- -D warnings
cargo +1.92.0 clippy --locked -p ruview-forecast-train --no-default-features --features cpu,cli,server,fal-client --all-targets -- -D warnings

The CUDA line is compile-only. It proves that the explicitly gated types build on Rust 1.92; it is not a GPU execution, training, latency, or compatibility claim:

cd v2
cargo +1.92.0 check --locked -p ruview-forecast-model --no-default-features --features cuda --lib
cargo +1.92.0 check --locked -p ruview-forecast-train --no-default-features --features cuda,cli,server --lib --bins

The hosted boundary stays backend-free and is exercised on Rust 1.89 with no provider credential. Tests must use local mocks; compiling these features does not authorize network use or a hosted training run:

cd v2
cargo +1.89.0 tree --locked -e normal,build -p ruview-forecast-train --no-default-features --features cli,server,fal-client > forecast-hosted-feature-tree.txt
! grep -Eiq '(^|[[:space:]])(burn|cubecl)(-|[[:space:]])' forecast-hosted-feature-tree.txt
FAL_KEY='' cargo +1.89.0 test --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client --lib --bins --tests
FAL_KEY='' cargo +1.89.0 test --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client privacy_external_dataset_payload_is_denied
FAL_KEY='' cargo +1.89.0 clippy --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client --all-targets -- -D warnings

The broader repository gate remains:

cd v2
cargo +1.89.0 test --locked --workspace --no-default-features
cargo +1.89.0 bench --locked --workspace --no-default-features --no-run

No CUDA runtime result is implied by these commands. CUDA execution needs a separately identified GPU, driver, toolkit, source commit, container digest, and signed training or inference receipt.

The repository-wide security workflow separately hard-gates the checked-in lockfile with cargo audit --file v2/Cargo.lock --json. The forecast workflow also runs both modes of scripts/csi-data-policy-check.sh, rejects forbidden clean-room source/import names and model blobs, checks the hosted DTO field surface, and blocks common literal private-key, FAL-key, JWT, cloud-key and presigned-URL patterns. These focused checks do not close dependency-license or complete secret-scanning evidence: the repository has no committed v2/deny.toml, inherited advisory/yanked-version debt needs an owner/date/expiry baseline, and the existing general secret scanners are non-blocking. Until dedicated retained reports exist, those parts of G0, G1, and G5 remain open.

Supply-chain and secret release prerequisites

Check Reproducer Present authority
Advisory database cargo audit --file v2/Cargo.lock --json Existing blocking workflow; report retained
Advisory warnings cargo audit --file v2/Cargo.lock --deny warnings Not green by assertion; inherited warning debt needs an expiring reviewed baseline
Rust sources/licenses/bans cargo deny --manifest-path v2/Cargo.toml --config v2/deny.toml check advisories bans licenses sources Blocked until a reviewed v2/deny.toml is committed and CI pins cargo-deny
Forecast-path secrets gitleaks detect --no-banner --redact --source . Focused CI blocks common literal patterns; the broader GitLeaks job/report is still non-blocking
Tracked sensing data bash scripts/csi-data-policy-check.sh --self-test && bash scripts/csi-data-policy-check.sh --tracked Blocking in the focused hosted-boundary job

cargo vet check becomes a release prerequisite only if the project commits a vet policy and review process; listing it without that policy would be a false supply-chain claim.

Existing baseline debt and scope

  • ../benchmarks/physics-pose-refinement.md records unrelated workspace rustfmt/warning debt, an incomplete full-workspace test run on that authoring host, existing RustSec debt, and yanked spin 0.9.8 in the optional Burn graph. Forecast CI therefore uses focused warnings-denied gates and does not describe the whole workspace as green.
  • .github/workflows/bench-regression.yml records an upstream stable-Rust failure in optional ruvector-crv; the focused forecast benchmarks neither enable nor inherit a CRV claim.
  • The current ADR corpus contains pre-existing duplicate ADR-263 and ADR-264 numbers. The focused contract rejects a duplicate ADR-348, ADR-349 or ADR-350 without falsely asserting that the historic corpus already passes a global uniqueness lint.
  • Shared GitHub-hosted timing is noisy by repository policy. Only benchmark compilation gates the PR; timing logs are informational until repeated on a named, controlled host.

Requirements

Requirement Machine evidence Human or external evidence Acceptance authority Current state
RF-001 Forecast-path source and artifact scan; forbidden model/blob fixture scan; clean-room manifest schema and tamper tests Current contributor attestations and clean-room custodian adjudication ADR-348 G0 OPEN
RF-002 Canonical receipt round trip; digest/signature tamper negatives; source, config, dataset and checkpoint identifiers required Reviewer verifies every dependency, reference, job and checkpoint lineage ADR-348 G0 and G1 OPEN
RF-003 Dataset manifest rejects missing license, incompatible use, unknown privacy class and unresolved bytes Data steward and legal approval for every immutable dataset digest ADR-348 G1 OPEN
RF-004 Feature-off dependency boundary excludes Burn/CubeCL; fixed artifact/input CPU replay; offline load tests; bounded property and fuzz corpus Named platform-class review and 24-hour accelerated replay ADR-348 G2 OPEN
RF-005 Shape/product overflow, finite-value, timestamp, mask, schema, horizon, quantile and resource-limit tests Review of production input caps and abstention thresholds ADR-348 G2 OPEN
RF-006 Forecast evidence label is immutable; observation hash round trips; confidence cannot increase; derived values cannot overwrite observations Evidence-engine and downstream schema review ADR-348 G2 and G4 OPEN
RF-007 Frozen split manifest; train-only normalization/calibration; quantile order; interval coverage and loss by horizon/domain Independent evaluation report on untouched site/session/device holdouts ADR-348 G3 OPEN
RF-008 Per-split RuVector index isolation; no overlapping horizon or holdout neighbour; paired retrieval-off/retrieval-on report Reviewer verifies index manifest and ablation comparability ADR-348 G3 OPEN
RF-009 Capability tests prove the forecast and RuVLLM explanation have no actuator, spending, access-control, emergency or model-promotion authority Downstream policy owner approves any advisory consumer ADR-348 G4 OPEN
RF-010 Local and hosted receipts compare source, lock, container, data, config and initial-weight digests; provider result is untrusted until verified Provider retention, log, credential and data-processing review ADR-348 G1 and G5 OPEN
RF-011 Benchmark report binds commit, lock, toolchain, host, backend, config, corpus, command and evidence label Maintainer adjudicates claim label and model card language ADR-348 G3 through G5 OPEN
RF-012 Mode transition and rollback state-machine tests; failed activation retains the prior artifact; raw sensing continues Signed rollback drill and authorized mode-transition record ADR-348 G5 OPEN

Governed training requirements

These rows trace ADR-349. A mock-backed test can close a software subcondition, but cannot close the hosted operational evidence named in the ADR.

Requirement Machine evidence Human or external evidence Acceptance authority Current state
FT-001 Synthetic hosted DTO and reconstructed local request bind the same generator, model, optimizer and build identities; external manifests are rejected Reviewer compares immutable local and provider receipts ADR-349 and G1 OPEN
FT-002 Unknown-field and arbitrary command/image/URL/environment/path negatives Endpoint capability and worker-image review ADR-349 and G2 OPEN
FT-003 Concurrent/retry/lost-response idempotency state-machine tests Real provider ambiguous-retry drill and cost reconciliation ADR-349 and G1/G5 OPEN
FT-004 Budget boundary and over-budget rejection tests Provider price/bill fixture review and bounded real-run receipt ADR-349 and G5 OPEN
FT-005 Authenticated cancellation, checkpoint and terminal-state property tests Real hosted cancellation and orphan-job drill ADR-349 and G2/G5 OPEN
FT-006 Missing/duplicate/truncated/tampered fixed-kind export matrix Quarantine/export receipt review ADR-349 and G1/G2 OPEN
FT-007 Hosted-signing denial, local verification and atomic promotion/rollback tests Release-key isolation approval and rollback drill ADR-349 and G1/G5 OPEN
FT-008 Captured egress and log-redaction tests for credentials/privacy/retention fields Provider DPA, region, retention, reuse and deletion review ADR-349 and G1/G5 OPEN

Predictive-memory and explanation requirements

These rows trace ADR-350. This PR creates no sensing-server, RuVector or RuVLLM runtime authority; the rows remain open until a separately reviewed bridge provides the evidence.

Requirement Machine evidence Human or external evidence Acceptance authority Current state
PM-001 Cross-tenant/split/version/time-scope query negatives Signed index-manifest and tenancy review ADR-350 and G3/G5 OPEN
PM-002 Property tests reject holdout identities and overlapping contexts/horizons Frozen leakage-report review ADR-350 and G3 OPEN
PM-003 Bounded zero/error/success retrieval receipts and tamper tests Retrieval receipt schema approval ADR-350 and G2/G3 OPEN
PM-004 Same-example, same-artifact paired retrieval-off/on evaluator Frozen paired ablation and overhead report ADR-350 and G3 OPEN
PM-005 Envelope signature mutation, expiry, tenant, key and replay negatives Local signing/key-rotation receipt ADR-350 and G2/G5 OPEN
PM-006 Evidence-monotonicity property tests reject measured-observation promotion Evidence-engine owner review ADR-350 and G2/G4 OPEN
PM-007 Numeric/unit/provenance mutation corpus fails closed RuVLLM adapter review ADR-350 and G4 OPEN
PM-008 Default-deny capability and attempted-escalation tests Downstream policy/capability review ADR-350 and G4/G5 OPEN
PM-009 Tenant deletion, retention and access-audit tests Privacy and membership/extraction-risk approval ADR-350 and G5 OPEN

Threat-model contract coverage

The IDs below are defined by ../security/ruview-forecast-threat-model.md. The feature job compiles and runs the current mock-backed suite, but an ID stays open until the named negative test and its retained CI report exist.

Contract IDs Required machine evidence Current state
PRIV-001 through PRIV-009 Egress-denial, tenant-isolation, log-redaction, nonce-isolation and tracked-data-policy negatives OPEN
AUTH-001 through AUTH-005; STATE-001 through STATE-004 Fail-closed identity/scope/service-auth tests and property-tested idempotent transition machine OPEN
BOUND-001 through BOUND-007 Boundary/property cases, parser fuzzing, cumulative resource limits and bounded accelerated replay OPEN
RVEC-001 through RVEC-006 Scope-authority construction denial, identifier/privacy bounds, isolation, cancellation and non-persistence evidence OPEN
FAL-001 through FAL-012; SSRF-001; PATH-001 through PATH-003 Mock request capture, exact-body webhook/replay cases, app/build/expiry binding, URL/DNS/redirect denial and filesystem escape negatives OPEN
ART-001 through ART-005; DE-001; EVID-001 through EVID-002 Artifact tamper/rollback/crash matrix, trusted-type construction denial and evidence-authority invariants OPEN

Gate evidence bundles

Gate Minimum bundle before review Status
G0 Clean-room manifest, contributor attestations, repository/artifact scan report, adjudication log OPEN
G1 Dataset rights manifests, byte-level lineage report, random-initialization receipt, local/hosted digest comparison OPEN
G2 Focused tests, property/fuzz reports, deterministic replay hash, dependency-boundary tree, 24-hour replay report OPEN
G3 Frozen split manifest, baseline/model/retrieval rows, weighted quantile loss, interval calibration, reproducer and report digests OPEN
G4 Fourteen-day shadow report, empty-room/occupied-room confusion matrices, drift/abstention slices, authority audit OPEN
G5 Dedicated-host latency and process-RSS report, signed model card, SBOM/provenance, rollback drill, security/privacy/legal approvals OPEN

Evidence record template

Copy one row per immutable evidence artifact. Never replace a failed result.

Evidence ID Requirement/gate Commit Lock/config/data digest Environment Command Result Evidence label Artifact digest Reviewer/date

No evidence artifacts have been accepted.