* feat(forecast): add independent Rust training stack Signed-off-by: Codex <codex@openai.com> * docs(forecast): record informal HPO exploration note (unaccepted) Records an exploratory Darwin Mode numeric-genome hyperparameter search against the tiny_ci synthetic accuracy protocol: WQL improved 0.257 -> 0.161 -> 0.153 over two search rounds, staying ahead of both baselines throughout. Explicitly scoped as informal/exploratory (not a frozen leakage-free report, not eligible for the evidence ledger, not validated for large_linux or any real dataset) per this doc's own evidence-tagging discipline. * feat(forecast-core): add per-horizon weighted quantile loss weighted_quantile_loss collapses the whole horizon into one aggregate number, hiding whether error grows with lead time. Add a sibling function using the identical per-cell pinball formula and domain checks, just reduced per horizon step instead of globally. A unit test reconstructs the aggregate from the per-step numerators/ denominators to prove the two functions agree exactly, not just approximately. * feat(forecast-model): add evaluation-only local activation Adds activate_for_evaluation and build_eval_input, gated behind the cpu feature. activate_for_evaluation self-signs an unsigned candidate with a fixed, publicly-known, non-secret Ed25519 key so a CLI operator can run inference against their own just-trained candidate without a real release signature -- explicitly never a production trust path, documented as such in the module doc comment. The schema-digest check in ArtifactActivationPolicy still applies in full; only signing is relaxed. build_eval_input constructs a single-window ModelInput for CPU inference from raw context-major values/mask arrays, matching the training batch builders exact time-feature/descriptor encoding. Six unit tests cover: malformed-candidate rejection, digest-mismatch rejection (proving the schema check has teeth even on this relaxed path), getting past the policy gate with a matching digest, and eval-input shape validation. * feat(forecast-train): add evaluate and prepare-synthetic-dataset CLI Turns this sessions throwaway HPO scripts into real, tested CLI surfaces, replacing two example binaries that bypassed the CLI/TOML training path entirely. prepare-synthetic-dataset generates a larger, configurable synthetic training shard (default 24 windows) plus a matching train-local.toml with the given OptimizerSpec hyperparameters baked in, and a separate held-out test.jsonl -- the same synthetic-only, local-validation-only posture as prepare-local-example, just bigger and configurable. evaluate scores a trained (unsigned) candidate against held-out windows and the LastValue/SeasonalNaive baselines: overall weighted quantile loss, a per-horizon breakdown, 80% interval coverage, and missingness, for the model and both baselines. Explicitly lists what docs/benchmarks/ruforecast.mds full accuracy protocol additionally asks for that this does not cover (abstention coverage, selective risk, site/device slices, interference regime, RuVector-retrieval ablation) rather than silently omitting them -- each needs infrastructure a single-entity synthetic fixture does not have. Together the three real commands (prepare-synthetic-dataset, train-local, evaluate) reproduce the exact WQL numbers the old library-bypassing example scripts produced for the same hyperparameters, confirmed by re-running the harness/ruview HPO dry-run end to end against the new CLI path. New integration test trains a real candidate via smoke, evaluates it against held-out synthetic windows, and asserts the full report shape and an empty-input rejection. * feat(forecast-train): add prepare-synthetic-dataset --seed + Autogenous bridge --seed (default 0, backward compatible) makes prepare-synthetic-dataset's generator deterministic per offset: same seed -> byte-identical corpus across runs (needed to train two genomes on the SAME corpus for a fair comparison); different seeds -> genuinely independent corpora (needed for honest multi-seed/multi-judge evaluation upstream, both in harness/ruview/flywheel/ruforecast/gate.mjs's fitness function and in the new v2/crates/ruforecast-autogenous-bridge crate added here). ruforecast-autogenous-bridge is a LOCAL-DEV-ONLY (excluded from the v2 workspace, publish = false) bridge from this crate's real evaluate CLI into ruvnet/autogenous's new regression-candidate promotion path (separate, unpushed branch feat/regression-candidate-kind): runs N independent train+evaluate judges on genuinely distinct synthetic corpora, signs receipts, and gets a real cryptographically-verified PROMOTE/REJECT decision -- defense in depth on top of, not a replacement for, Darwin's own promotion gate. Path-depends on a sibling autogenous checkout that does not exist in CI; build/run directly via `cargo build --manifest-path crates/ruforecast-autogenous-bridge/Cargo.toml`. cargo test -p ruview-forecast-train --no-default-features --features cpu,cli: all green (unaffected existing tests + this backward-compatible addition). * docs(forecast): retract round-2 HPO result, record honest multi-seed finding The earlier "Informal HPO exploration note" claimed a real improvement (WQL 0.257 -> 0.153) from a 3-round hyperparameter search. That search evaluated every candidate against one fixed synthetic corpus (seed 0) for every round -- textbook overfitting. Independent verification against two fresh corpora (via the new ruforecast-autogenous-bridge crate) showed the "winner" losing to the baseline on both. Fixed the root cause in harness/ruview/flywheel/ruforecast/gate.mjs: candidates are now scored against three independent corpora, worst-case across them, not one fixed corpus. Re-ran the search under the fix. It found a new winner that genuinely beat baseline on all three of its own search seeds -- and that winner ALSO lost independent verification on fresh seeds. Two independent search rounds, pre- and post-fix, both produced an illusory "winner." The honest reading: at this dataset scale (24 synthetic windows) held-out WQL varies enormously by which corpus is drawn, regardless of hyperparameters -- confirmed directly by the baseline genome's own primary swinging from 0.83 to a full regression across the three fixed search seeds with unchanged hyperparameters. No RuForecast hyperparameter configuration has been shown to reliably beat the trivial baselines out-of-sample at this scale. Also fixes a real design inconsistency surfaced by this exploration: the Autogenous regression-candidate promotion verifier (envelope::regression::verify_regression_promotion, separate unpushed ruvnet/autogenous branch feat/regression-candidate-kind) required all judges to share one corpus_id, which conflicts with this kind's intentional cross-corpus judge design. Corrected there (commit bfa4c48); did not change either REJECT verdict, which were already driven by the real NotBetterThanParent signal on their own. Append-only: the original round-2 row is kept, read together with this new amendment section, per this doc's own evidence-ledger discipline. * refactor(forecast): extract RuForecast to ruvnet/RuForecast submodule Mirrors the v2/crates/worldgraph pattern: RuForecast becomes its own independent public repo/workspace (ruforecast-core/model/train), mounted at v2/crates/ruforecast as a git submodule, with v2/Cargo.toml path-depping into its sub-crates and excluding the submodule from the v2 workspace. Full v2 --workspace check passes; the ruforecast CLI builds and runs correctly from its new location. Real git history for the extracted crates (7 commits) was preserved via git-filter-repo, not squashed. * feat(autogenous-bridge): real-data independent verification tool real_data_verify.rs: signs two real, temporally-independent judge measurements (different train/test split boundaries on the same real household vitals corpus, not synthetic seeds) through the real Autogenous regression-candidate promotion path, and gets a genuine cryptographically-checked PROMOTE/REJECT verdict. Result recorded (see docs/benchmarks/ruforecast.md): REJECT. One judge nominally beat the trivial baseline but by less than the 0.01 non-inferiority margin; the other judge lost outright. Consistent with every synthetic-data search this session -- no configuration has yet been shown to reliably beat trivial baselines out-of-sample, now including a real 6390-sample household corpus. Bumps the ruforecast submodule to pick up the evaluate real-gap fix. * docs(forecast): record real-household-data result (REJECT, signed) 6390 real vital-signs samples, two independent temporal splits, independently verified via the real Autogenous regression-candidate promotion path. Signed verdict: REJECT -- neither split cleared the non-inferiority margin. Same conclusion as every synthetic search this session, now confirmed with real data too: more real data is the credible next lever, not further search on this scale of fixture. * chore(forecast): point ruforecast submodule at published main (post gap-tolerance fix) * ci(forecast): fix ruforecast-ci.yml for the new v2/crates/ruforecast submodule layout The RuForecast crates were extracted into a standalone submodule (v2/crates/ruforecast, mirroring the existing v2/crates/worldgraph pattern) and are no longer members of the v2 workspace. The CI workflow still referenced the old in-tree package names/paths (ruview-forecast-core/model/train under v2/), which broke every forecast job with "cannot specify features for packages outside of workspace". Fixed every job to target the submodule: working-directory -> v2/crates/ruforecast, -p ruview-forecast-* -> -p ruforecast-*, all hardcoded Cargo.toml/source paths in the Python assertion and clean-room-scan blocks, the Swatinem/rust-cache workspaces input, the artifact-tree output paths, and the two levels of relative path (../ -> ../../) in the informational-benchmark steps whose working directory moved one level deeper. Also collapsed the paths: trigger filters internal-crate globs (v2/crates/ruview-forecast-*/**) down to the single v2/crates/ruforecast path, since GitHub Actions path filters only ever see a submodule gitlink change as one entry in the parent tree, never its internal file paths -- the old globs could never have matched anything. Includes a real cargo fmt fix (submodule commit d9902f9) for formatting drift left over from the earlier real-data gap-tolerance fix, which the contract job's fmt --check step would otherwise have failed on. All four forecast CI jobs' real commands verified locally before push: cargo test/check for CPU, CUDA/WGPU compile-check, the contract job's Python assertion block, the hosted-boundaries clean-room scan, and cargo fmt --check. * docs(forecast): record real BIDMC cross-entity holdout result (REJECT) Two independent real-patient partitions (34/16 and 24/26 splits, 53 total ICU patients from PhysioNet BIDMC, Open Data Commons Attribution License v1.0) both show the model losing to trivial baselines by a decisive margin (+69% and +601% worse WQL). First genuine cross-entity real-data test this project has run; same conclusion as every prior synthetic and single-household test. Full Autogenous signed verification was not run for this entry - flagged explicitly in the note. * chore(forecast): point ruforecast submodule at published main (example feature-gate fix) * chore(forecast): point ruforecast submodule at published main (clippy fix) --------- Signed-off-by: Codex <codex@openai.com>
16 KiB
RuForecast requirement to evidence matrix
Authority and current state
This matrix operationalizes ADR-348 and its ADR-349/350 children without changing them. The ADRs own the requirements and gates. Rust tests, immutable reports, signed receipts, and explicit human approvals supply evidence. A checked box or green build is not permission to deploy, publish a checkpoint, upload customer-derived data, or advance the rollout mode.
Current state: all release and operational gates are open. No benchmark result, trained checkpoint, external training receipt, or production authority is recorded by this document.
Focused verification commands
The feature-off contract remains on the workspace Rust 1.89 line:
cd v2
cargo +1.89.0 test --locked -p ruview-forecast-core --no-default-features --lib --tests
cargo +1.89.0 test --locked -p ruview-forecast-model --no-default-features --lib --tests
cargo +1.89.0 check --locked -p ruview-forecast-model -p ruview-forecast-train --no-default-features --all-targets
cargo +1.89.0 test --locked -p ruview-forecast-model --no-default-features --features ruvector --lib --tests
cargo +1.89.0 clippy --locked -p ruview-forecast-core -p ruview-forecast-model -p ruview-forecast-train --no-default-features --all-targets -- -D warnings
cargo +1.89.0 clippy --locked -p ruview-forecast-model --no-default-features --features ruvector --all-targets -- -D warnings
Burn 0.21 CPU activation uses the explicitly separate Rust 1.92 line:
cd v2
cargo +1.92.0 test --locked -p ruview-forecast-model --no-default-features --features cpu --lib --tests
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli --lib --bins
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu --test local_jsonl_smoke -- --exact local_hash_addressed_jsonl_executes_one_real_optimizer_step
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli --test cli_smoke -- --exact cli_smoke_trains_and_writes_the_complete_candidate_set
cargo +1.92.0 test --locked -p ruview-forecast-train --no-default-features --features cpu,cli,server,fal-client --lib --bins
cargo +1.92.0 clippy --locked -p ruview-forecast-model --no-default-features --features cpu --all-targets -- -D warnings
cargo +1.92.0 clippy --locked -p ruview-forecast-train --no-default-features --features cpu,cli,server,fal-client --all-targets -- -D warnings
The CUDA line is compile-only. It proves that the explicitly gated types build on Rust 1.92; it is not a GPU execution, training, latency, or compatibility claim:
cd v2
cargo +1.92.0 check --locked -p ruview-forecast-model --no-default-features --features cuda --lib
cargo +1.92.0 check --locked -p ruview-forecast-train --no-default-features --features cuda,cli,server --lib --bins
The hosted boundary stays backend-free and is exercised on Rust 1.89 with no provider credential. Tests must use local mocks; compiling these features does not authorize network use or a hosted training run:
cd v2
cargo +1.89.0 tree --locked -e normal,build -p ruview-forecast-train --no-default-features --features cli,server,fal-client > forecast-hosted-feature-tree.txt
! grep -Eiq '(^|[[:space:]])(burn|cubecl)(-|[[:space:]])' forecast-hosted-feature-tree.txt
FAL_KEY='' cargo +1.89.0 test --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client --lib --bins --tests
FAL_KEY='' cargo +1.89.0 test --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client privacy_external_dataset_payload_is_denied
FAL_KEY='' cargo +1.89.0 clippy --locked -p ruview-forecast-train --no-default-features --features cli,server,fal-client --all-targets -- -D warnings
The broader repository gate remains:
cd v2
cargo +1.89.0 test --locked --workspace --no-default-features
cargo +1.89.0 bench --locked --workspace --no-default-features --no-run
No CUDA runtime result is implied by these commands. CUDA execution needs a separately identified GPU, driver, toolkit, source commit, container digest, and signed training or inference receipt.
The repository-wide security workflow separately hard-gates the checked-in
lockfile with cargo audit --file v2/Cargo.lock --json. The forecast workflow
also runs both modes of scripts/csi-data-policy-check.sh, rejects forbidden
clean-room source/import names and model blobs, checks the hosted DTO field
surface, and blocks common literal private-key, FAL-key, JWT, cloud-key and
presigned-URL patterns. These focused checks do not close dependency-license or
complete secret-scanning evidence: the repository has no committed
v2/deny.toml, inherited advisory/yanked-version debt needs an
owner/date/expiry baseline, and the existing general secret scanners are
non-blocking. Until dedicated retained reports exist, those parts of G0, G1,
and G5 remain open.
Supply-chain and secret release prerequisites
| Check | Reproducer | Present authority |
|---|---|---|
| Advisory database | cargo audit --file v2/Cargo.lock --json |
Existing blocking workflow; report retained |
| Advisory warnings | cargo audit --file v2/Cargo.lock --deny warnings |
Not green by assertion; inherited warning debt needs an expiring reviewed baseline |
| Rust sources/licenses/bans | cargo deny --manifest-path v2/Cargo.toml --config v2/deny.toml check advisories bans licenses sources |
Blocked until a reviewed v2/deny.toml is committed and CI pins cargo-deny |
| Forecast-path secrets | gitleaks detect --no-banner --redact --source . |
Focused CI blocks common literal patterns; the broader GitLeaks job/report is still non-blocking |
| Tracked sensing data | bash scripts/csi-data-policy-check.sh --self-test && bash scripts/csi-data-policy-check.sh --tracked |
Blocking in the focused hosted-boundary job |
cargo vet check becomes a release prerequisite only if the project commits a
vet policy and review process; listing it without that policy would be a false
supply-chain claim.
Existing baseline debt and scope
../benchmarks/physics-pose-refinement.mdrecords unrelated workspace rustfmt/warning debt, an incomplete full-workspace test run on that authoring host, existing RustSec debt, and yankedspin 0.9.8in the optional Burn graph. Forecast CI therefore uses focused warnings-denied gates and does not describe the whole workspace as green..github/workflows/bench-regression.ymlrecords an upstream stable-Rust failure in optionalruvector-crv; the focused forecast benchmarks neither enable nor inherit a CRV claim.- The current ADR corpus contains pre-existing duplicate ADR-263 and ADR-264 numbers. The focused contract rejects a duplicate ADR-348, ADR-349 or ADR-350 without falsely asserting that the historic corpus already passes a global uniqueness lint.
- Shared GitHub-hosted timing is noisy by repository policy. Only benchmark compilation gates the PR; timing logs are informational until repeated on a named, controlled host.
Requirements
| Requirement | Machine evidence | Human or external evidence | Acceptance authority | Current state |
|---|---|---|---|---|
| RF-001 | Forecast-path source and artifact scan; forbidden model/blob fixture scan; clean-room manifest schema and tamper tests | Current contributor attestations and clean-room custodian adjudication | ADR-348 G0 | OPEN |
| RF-002 | Canonical receipt round trip; digest/signature tamper negatives; source, config, dataset and checkpoint identifiers required | Reviewer verifies every dependency, reference, job and checkpoint lineage | ADR-348 G0 and G1 | OPEN |
| RF-003 | Dataset manifest rejects missing license, incompatible use, unknown privacy class and unresolved bytes | Data steward and legal approval for every immutable dataset digest | ADR-348 G1 | OPEN |
| RF-004 | Feature-off dependency boundary excludes Burn/CubeCL; fixed artifact/input CPU replay; offline load tests; bounded property and fuzz corpus | Named platform-class review and 24-hour accelerated replay | ADR-348 G2 | OPEN |
| RF-005 | Shape/product overflow, finite-value, timestamp, mask, schema, horizon, quantile and resource-limit tests | Review of production input caps and abstention thresholds | ADR-348 G2 | OPEN |
| RF-006 | Forecast evidence label is immutable; observation hash round trips; confidence cannot increase; derived values cannot overwrite observations | Evidence-engine and downstream schema review | ADR-348 G2 and G4 | OPEN |
| RF-007 | Frozen split manifest; train-only normalization/calibration; quantile order; interval coverage and loss by horizon/domain | Independent evaluation report on untouched site/session/device holdouts | ADR-348 G3 | OPEN |
| RF-008 | Per-split RuVector index isolation; no overlapping horizon or holdout neighbour; paired retrieval-off/retrieval-on report | Reviewer verifies index manifest and ablation comparability | ADR-348 G3 | OPEN |
| RF-009 | Capability tests prove the forecast and RuVLLM explanation have no actuator, spending, access-control, emergency or model-promotion authority | Downstream policy owner approves any advisory consumer | ADR-348 G4 | OPEN |
| RF-010 | Local and hosted receipts compare source, lock, container, data, config and initial-weight digests; provider result is untrusted until verified | Provider retention, log, credential and data-processing review | ADR-348 G1 and G5 | OPEN |
| RF-011 | Benchmark report binds commit, lock, toolchain, host, backend, config, corpus, command and evidence label | Maintainer adjudicates claim label and model card language | ADR-348 G3 through G5 | OPEN |
| RF-012 | Mode transition and rollback state-machine tests; failed activation retains the prior artifact; raw sensing continues | Signed rollback drill and authorized mode-transition record | ADR-348 G5 | OPEN |
Governed training requirements
These rows trace ADR-349. A mock-backed test can close a software subcondition, but cannot close the hosted operational evidence named in the ADR.
| Requirement | Machine evidence | Human or external evidence | Acceptance authority | Current state |
|---|---|---|---|---|
| FT-001 | Synthetic hosted DTO and reconstructed local request bind the same generator, model, optimizer and build identities; external manifests are rejected | Reviewer compares immutable local and provider receipts | ADR-349 and G1 | OPEN |
| FT-002 | Unknown-field and arbitrary command/image/URL/environment/path negatives | Endpoint capability and worker-image review | ADR-349 and G2 | OPEN |
| FT-003 | Concurrent/retry/lost-response idempotency state-machine tests | Real provider ambiguous-retry drill and cost reconciliation | ADR-349 and G1/G5 | OPEN |
| FT-004 | Budget boundary and over-budget rejection tests | Provider price/bill fixture review and bounded real-run receipt | ADR-349 and G5 | OPEN |
| FT-005 | Authenticated cancellation, checkpoint and terminal-state property tests | Real hosted cancellation and orphan-job drill | ADR-349 and G2/G5 | OPEN |
| FT-006 | Missing/duplicate/truncated/tampered fixed-kind export matrix | Quarantine/export receipt review | ADR-349 and G1/G2 | OPEN |
| FT-007 | Hosted-signing denial, local verification and atomic promotion/rollback tests | Release-key isolation approval and rollback drill | ADR-349 and G1/G5 | OPEN |
| FT-008 | Captured egress and log-redaction tests for credentials/privacy/retention fields | Provider DPA, region, retention, reuse and deletion review | ADR-349 and G1/G5 | OPEN |
Predictive-memory and explanation requirements
These rows trace ADR-350. This PR creates no sensing-server, RuVector or RuVLLM runtime authority; the rows remain open until a separately reviewed bridge provides the evidence.
| Requirement | Machine evidence | Human or external evidence | Acceptance authority | Current state |
|---|---|---|---|---|
| PM-001 | Cross-tenant/split/version/time-scope query negatives | Signed index-manifest and tenancy review | ADR-350 and G3/G5 | OPEN |
| PM-002 | Property tests reject holdout identities and overlapping contexts/horizons | Frozen leakage-report review | ADR-350 and G3 | OPEN |
| PM-003 | Bounded zero/error/success retrieval receipts and tamper tests | Retrieval receipt schema approval | ADR-350 and G2/G3 | OPEN |
| PM-004 | Same-example, same-artifact paired retrieval-off/on evaluator | Frozen paired ablation and overhead report | ADR-350 and G3 | OPEN |
| PM-005 | Envelope signature mutation, expiry, tenant, key and replay negatives | Local signing/key-rotation receipt | ADR-350 and G2/G5 | OPEN |
| PM-006 | Evidence-monotonicity property tests reject measured-observation promotion | Evidence-engine owner review | ADR-350 and G2/G4 | OPEN |
| PM-007 | Numeric/unit/provenance mutation corpus fails closed | RuVLLM adapter review | ADR-350 and G4 | OPEN |
| PM-008 | Default-deny capability and attempted-escalation tests | Downstream policy/capability review | ADR-350 and G4/G5 | OPEN |
| PM-009 | Tenant deletion, retention and access-audit tests | Privacy and membership/extraction-risk approval | ADR-350 and G5 | OPEN |
Threat-model contract coverage
The IDs below are defined by
../security/ruview-forecast-threat-model.md.
The feature job compiles and runs the current mock-backed suite, but an ID stays
open until the named negative test and its retained CI report exist.
| Contract IDs | Required machine evidence | Current state |
|---|---|---|
| PRIV-001 through PRIV-009 | Egress-denial, tenant-isolation, log-redaction, nonce-isolation and tracked-data-policy negatives | OPEN |
| AUTH-001 through AUTH-005; STATE-001 through STATE-004 | Fail-closed identity/scope/service-auth tests and property-tested idempotent transition machine | OPEN |
| BOUND-001 through BOUND-007 | Boundary/property cases, parser fuzzing, cumulative resource limits and bounded accelerated replay | OPEN |
| RVEC-001 through RVEC-006 | Scope-authority construction denial, identifier/privacy bounds, isolation, cancellation and non-persistence evidence | OPEN |
| FAL-001 through FAL-012; SSRF-001; PATH-001 through PATH-003 | Mock request capture, exact-body webhook/replay cases, app/build/expiry binding, URL/DNS/redirect denial and filesystem escape negatives | OPEN |
| ART-001 through ART-005; DE-001; EVID-001 through EVID-002 | Artifact tamper/rollback/crash matrix, trusted-type construction denial and evidence-authority invariants | OPEN |
Gate evidence bundles
| Gate | Minimum bundle before review | Status |
|---|---|---|
| G0 | Clean-room manifest, contributor attestations, repository/artifact scan report, adjudication log | OPEN |
| G1 | Dataset rights manifests, byte-level lineage report, random-initialization receipt, local/hosted digest comparison | OPEN |
| G2 | Focused tests, property/fuzz reports, deterministic replay hash, dependency-boundary tree, 24-hour replay report | OPEN |
| G3 | Frozen split manifest, baseline/model/retrieval rows, weighted quantile loss, interval calibration, reproducer and report digests | OPEN |
| G4 | Fourteen-day shadow report, empty-room/occupied-room confusion matrices, drift/abstention slices, authority audit | OPEN |
| G5 | Dedicated-host latency and process-RSS report, signed model card, SBOM/provenance, rollback drill, security/privacy/legal approvals | OPEN |
Evidence record template
Copy one row per immutable evidence artifact. Never replace a failed result.
| Evidence ID | Requirement/gate | Commit | Lock/config/data digest | Environment | Command | Result | Evidence label | Artifact digest | Reviewer/date |
|---|
No evidence artifacts have been accepted.