rUv
e5bf0d4892
* feat(forecast): add independent Rust training stack Signed-off-by: Codex <codex@openai.com> * docs(forecast): record informal HPO exploration note (unaccepted) Records an exploratory Darwin Mode numeric-genome hyperparameter search against the tiny_ci synthetic accuracy protocol: WQL improved 0.257 -> 0.161 -> 0.153 over two search rounds, staying ahead of both baselines throughout. Explicitly scoped as informal/exploratory (not a frozen leakage-free report, not eligible for the evidence ledger, not validated for large_linux or any real dataset) per this doc's own evidence-tagging discipline. * feat(forecast-core): add per-horizon weighted quantile loss weighted_quantile_loss collapses the whole horizon into one aggregate number, hiding whether error grows with lead time. Add a sibling function using the identical per-cell pinball formula and domain checks, just reduced per horizon step instead of globally. A unit test reconstructs the aggregate from the per-step numerators/ denominators to prove the two functions agree exactly, not just approximately. * feat(forecast-model): add evaluation-only local activation Adds activate_for_evaluation and build_eval_input, gated behind the cpu feature. activate_for_evaluation self-signs an unsigned candidate with a fixed, publicly-known, non-secret Ed25519 key so a CLI operator can run inference against their own just-trained candidate without a real release signature -- explicitly never a production trust path, documented as such in the module doc comment. The schema-digest check in ArtifactActivationPolicy still applies in full; only signing is relaxed. build_eval_input constructs a single-window ModelInput for CPU inference from raw context-major values/mask arrays, matching the training batch builders exact time-feature/descriptor encoding. Six unit tests cover: malformed-candidate rejection, digest-mismatch rejection (proving the schema check has teeth even on this relaxed path), getting past the policy gate with a matching digest, and eval-input shape validation. * feat(forecast-train): add evaluate and prepare-synthetic-dataset CLI Turns this sessions throwaway HPO scripts into real, tested CLI surfaces, replacing two example binaries that bypassed the CLI/TOML training path entirely. prepare-synthetic-dataset generates a larger, configurable synthetic training shard (default 24 windows) plus a matching train-local.toml with the given OptimizerSpec hyperparameters baked in, and a separate held-out test.jsonl -- the same synthetic-only, local-validation-only posture as prepare-local-example, just bigger and configurable. evaluate scores a trained (unsigned) candidate against held-out windows and the LastValue/SeasonalNaive baselines: overall weighted quantile loss, a per-horizon breakdown, 80% interval coverage, and missingness, for the model and both baselines. Explicitly lists what docs/benchmarks/ruforecast.mds full accuracy protocol additionally asks for that this does not cover (abstention coverage, selective risk, site/device slices, interference regime, RuVector-retrieval ablation) rather than silently omitting them -- each needs infrastructure a single-entity synthetic fixture does not have. Together the three real commands (prepare-synthetic-dataset, train-local, evaluate) reproduce the exact WQL numbers the old library-bypassing example scripts produced for the same hyperparameters, confirmed by re-running the harness/ruview HPO dry-run end to end against the new CLI path. New integration test trains a real candidate via smoke, evaluates it against held-out synthetic windows, and asserts the full report shape and an empty-input rejection. * feat(forecast-train): add prepare-synthetic-dataset --seed + Autogenous bridge --seed (default 0, backward compatible) makes prepare-synthetic-dataset's generator deterministic per offset: same seed -> byte-identical corpus across runs (needed to train two genomes on the SAME corpus for a fair comparison); different seeds -> genuinely independent corpora (needed for honest multi-seed/multi-judge evaluation upstream, both in harness/ruview/flywheel/ruforecast/gate.mjs's fitness function and in the new v2/crates/ruforecast-autogenous-bridge crate added here). ruforecast-autogenous-bridge is a LOCAL-DEV-ONLY (excluded from the v2 workspace, publish = false) bridge from this crate's real evaluate CLI into ruvnet/autogenous's new regression-candidate promotion path (separate, unpushed branch feat/regression-candidate-kind): runs N independent train+evaluate judges on genuinely distinct synthetic corpora, signs receipts, and gets a real cryptographically-verified PROMOTE/REJECT decision -- defense in depth on top of, not a replacement for, Darwin's own promotion gate. Path-depends on a sibling autogenous checkout that does not exist in CI; build/run directly via `cargo build --manifest-path crates/ruforecast-autogenous-bridge/Cargo.toml`. cargo test -p ruview-forecast-train --no-default-features --features cpu,cli: all green (unaffected existing tests + this backward-compatible addition). * docs(forecast): retract round-2 HPO result, record honest multi-seed finding The earlier "Informal HPO exploration note" claimed a real improvement (WQL 0.257 -> 0.153) from a 3-round hyperparameter search. That search evaluated every candidate against one fixed synthetic corpus (seed 0) for every round -- textbook overfitting. Independent verification against two fresh corpora (via the new ruforecast-autogenous-bridge crate) showed the "winner" losing to the baseline on both. Fixed the root cause in harness/ruview/flywheel/ruforecast/gate.mjs: candidates are now scored against three independent corpora, worst-case across them, not one fixed corpus. Re-ran the search under the fix. It found a new winner that genuinely beat baseline on all three of its own search seeds -- and that winner ALSO lost independent verification on fresh seeds. Two independent search rounds, pre- and post-fix, both produced an illusory "winner." The honest reading: at this dataset scale (24 synthetic windows) held-out WQL varies enormously by which corpus is drawn, regardless of hyperparameters -- confirmed directly by the baseline genome's own primary swinging from 0.83 to a full regression across the three fixed search seeds with unchanged hyperparameters. No RuForecast hyperparameter configuration has been shown to reliably beat the trivial baselines out-of-sample at this scale. Also fixes a real design inconsistency surfaced by this exploration: the Autogenous regression-candidate promotion verifier (envelope::regression::verify_regression_promotion, separate unpushed ruvnet/autogenous branch feat/regression-candidate-kind) required all judges to share one corpus_id, which conflicts with this kind's intentional cross-corpus judge design. Corrected there (commit bfa4c48); did not change either REJECT verdict, which were already driven by the real NotBetterThanParent signal on their own. Append-only: the original round-2 row is kept, read together with this new amendment section, per this doc's own evidence-ledger discipline. * refactor(forecast): extract RuForecast to ruvnet/RuForecast submodule Mirrors the v2/crates/worldgraph pattern: RuForecast becomes its own independent public repo/workspace (ruforecast-core/model/train), mounted at v2/crates/ruforecast as a git submodule, with v2/Cargo.toml path-depping into its sub-crates and excluding the submodule from the v2 workspace. Full v2 --workspace check passes; the ruforecast CLI builds and runs correctly from its new location. Real git history for the extracted crates (7 commits) was preserved via git-filter-repo, not squashed. * feat(autogenous-bridge): real-data independent verification tool real_data_verify.rs: signs two real, temporally-independent judge measurements (different train/test split boundaries on the same real household vitals corpus, not synthetic seeds) through the real Autogenous regression-candidate promotion path, and gets a genuine cryptographically-checked PROMOTE/REJECT verdict. Result recorded (see docs/benchmarks/ruforecast.md): REJECT. One judge nominally beat the trivial baseline but by less than the 0.01 non-inferiority margin; the other judge lost outright. Consistent with every synthetic-data search this session -- no configuration has yet been shown to reliably beat trivial baselines out-of-sample, now including a real 6390-sample household corpus. Bumps the ruforecast submodule to pick up the evaluate real-gap fix. * docs(forecast): record real-household-data result (REJECT, signed) 6390 real vital-signs samples, two independent temporal splits, independently verified via the real Autogenous regression-candidate promotion path. Signed verdict: REJECT -- neither split cleared the non-inferiority margin. Same conclusion as every synthetic search this session, now confirmed with real data too: more real data is the credible next lever, not further search on this scale of fixture. * chore(forecast): point ruforecast submodule at published main (post gap-tolerance fix) * ci(forecast): fix ruforecast-ci.yml for the new v2/crates/ruforecast submodule layout The RuForecast crates were extracted into a standalone submodule (v2/crates/ruforecast, mirroring the existing v2/crates/worldgraph pattern) and are no longer members of the v2 workspace. The CI workflow still referenced the old in-tree package names/paths (ruview-forecast-core/model/train under v2/), which broke every forecast job with "cannot specify features for packages outside of workspace". Fixed every job to target the submodule: working-directory -> v2/crates/ruforecast, -p ruview-forecast-* -> -p ruforecast-*, all hardcoded Cargo.toml/source paths in the Python assertion and clean-room-scan blocks, the Swatinem/rust-cache workspaces input, the artifact-tree output paths, and the two levels of relative path (../ -> ../../) in the informational-benchmark steps whose working directory moved one level deeper. Also collapsed the paths: trigger filters internal-crate globs (v2/crates/ruview-forecast-*/**) down to the single v2/crates/ruforecast path, since GitHub Actions path filters only ever see a submodule gitlink change as one entry in the parent tree, never its internal file paths -- the old globs could never have matched anything. Includes a real cargo fmt fix (submodule commit d9902f9) for formatting drift left over from the earlier real-data gap-tolerance fix, which the contract job's fmt --check step would otherwise have failed on. All four forecast CI jobs' real commands verified locally before push: cargo test/check for CPU, CUDA/WGPU compile-check, the contract job's Python assertion block, the hosted-boundaries clean-room scan, and cargo fmt --check. * docs(forecast): record real BIDMC cross-entity holdout result (REJECT) Two independent real-patient partitions (34/16 and 24/26 splits, 53 total ICU patients from PhysioNet BIDMC, Open Data Commons Attribution License v1.0) both show the model losing to trivial baselines by a decisive margin (+69% and +601% worse WQL). First genuine cross-entity real-data test this project has run; same conclusion as every prior synthetic and single-household test. Full Autogenous signed verification was not run for this entry - flagged explicitly in the note. * chore(forecast): point ruforecast submodule at published main (example feature-gate fix) * chore(forecast): point ruforecast submodule at published main (clippy fix) --------- Signed-off-by: Codex <codex@openai.com>
12 KiB
ADR-350: RuVector predictive memory and RuVLLM authority boundary
- Status: Proposed
- Date: 2026-09-01
- Deciders: ruv
- Owners: RuView forecast, RuVector, evidence, and RuVLLM integration maintainers
- Tags: forecasting, ruvector, ruvllm, retrieval, memory, receipts, authority
- Parent: ADR-348
- Extends: ADR-004, ADR-010, ADR-016, ADR-145, ADR-261, ADR-295, ADR-304, ADR-319, ADR-348
- Supersedes: None
Decision
RuVector may augment RuView Forecast by retrieving analogous historical feature/forecast states from a tenant- and split-scoped predictive-memory index. Retrieval is optional context, never ground truth. Every release must report the same frozen examples with retrieval disabled and enabled so its incremental value and leakage risk remain visible.
Each forecast is published in an immutable, locally signed envelope binding the model artifact, request, output, source evidence, and retrieval receipt. RuVLLM may read that envelope to explain uncertainty and analogous history. It may not rewrite numeric forecasts, create a stronger evidence class, select a model, promote an artifact, spend money, invoke an actuator, or make medical, emergency, access-control, or life-safety decisions.
This child ADR delegates forecasting, clean-room, data, and general production gates to ADR-348. It does not authorize a sensing-server bridge.
Evidence status: canonical forecast/receipt types and an indexable latent state exist in source. A RuVector adapter, retrieval receipt, signature wrapper, outcome reconciler, RuVLLM explanation adapter, paired retrieval evaluation, latency, storage cost, accuracy lift, and operational value are all UNMEASURED and unapproved.
Existing implementation boundary
The current crates establish only the preconditions:
| Module | Implemented surface used by this ADR |
|---|---|
ruview-forecast-core/src/forecast.rs |
validated ForecastRequest, backend-neutral Forecaster, ordered finite Forecast, canonical output digest, and receipt verification |
ruview-forecast-core/src/receipt.rs |
SourceState, ArtifactReceipt, and ForecastReceipt; derived forecasts cannot retain MEASURED merely because measured input existed |
ruview-forecast-core/src/series.rs |
bounded feature schema, masked time series, canonical series digest, and training-only scaler fit |
ruview-forecast-model/src/network.rs |
ForecastModelOutput.state with shape [batch, variates, d_model], intended as a bounded representation candidate for indexing |
ForecastReceipt is content-addressed but not itself a digital signature.
ForecastModelOutput.state is indexable but not automatically safe, private,
stable across model versions, or useful. Those distinctions remain release
gates.
Predictive-memory record
RuVector stores a versioned PredictiveMemoryRecord containing:
- tenant and index namespace;
- model/artifact/config/feature-schema digests;
- source series and forecast-request digests;
- split, site, subject-class, session, device, calibration, and bounded time scope using pseudonymous identifiers;
- bounded latent-state or engineered-feature digest plus the approved vector;
- forecast envelope digest and horizon;
- observed validity mask and evidence class;
- optional reconciled outcome digest added only after the forecast horizon;
- retention/deletion class and creation/expiry time.
Raw CSI, unrestricted feature windows, precise room coordinates, persistent person identity, RuVLLM prompts, and explanation prose are excluded by default. The vector is still potentially sensitive because it can encode routines or location. Tenant isolation, encryption, access control, retention, deletion, and membership-inference review apply.
An outcome is appended through a new immutable record linked to the original; the historical forecast is never rewritten. Outcome reconciliation can measure forecast quality but cannot retroactively turn the forecast into a measured observation.
Split-scoped analogue retrieval
Every index is bound to an immutable corpus manifest, model version, feature schema, preprocessing configuration, and split policy. These rules are mandatory:
- Tenants never share an index or query result without a separately authorized privacy-preserving federation protocol.
- Training, validation, calibration, and test records have distinct namespaces. A test query may retrieve approved training analogues only; it may not retrieve test examples or validation/calibration records used to set thresholds.
- The same site, subject, session, device, calibration episode, overlapping context, or overlapping target horizon is excluded when that dimension is a holdout.
- Index construction, distance metric, normalization, filter policy,
k, and score threshold are fitted on training data and frozen before test. - Missing, stale, mismatched, unauthorized, or unverifiable indexes disable retrieval and return the no-retrieval forecast or abstention according to the model card. They never trigger an implicit global-index fallback.
- Every query returns a bounded
RetrievalReceipteven when zero neighbours qualify.
The retrieval receipt binds query digest, index/corpus/policy digests,
namespace, exclusion filter, neighbour record IDs and distances, k, latency,
and disposition. It excludes raw neighbour payloads from ordinary logs.
Paired ablation
Retrieval has no deployment authority without a paired evaluation on identical frozen requests:
| Row | Model and examples | Retrieval |
|---|---|---|
| A | exact candidate artifact and frozen examples | disabled |
| B | exact candidate artifact and frozen examples | enabled with frozen index/policy |
The report includes weighted quantile loss, interval coverage/width, abstention, per-horizon error, per-site/device/interference slices, retrieval hit/filtered/empty rates, latency, and memory/storage overhead. It reports both aggregate and paired deltas with uncertainty. A gain on pooled error cannot hide a failed deployment domain, calibration regression, or neighbour leakage.
The no-retrieval row remains a supported fallback. Retrieval is removed when its lower confidence bound does not show useful improvement, when it weakens calibration beyond the ADR-348 gate, or when its privacy/latency/storage cost exceeds its measured value. No uplift is claimed today.
Immutable signed forecast envelope
The signed envelope covers:
envelope schema/version
+ ArtifactReceipt canonical digest
+ ForecastRequest canonical digest
+ Forecast payload/output digest
+ ForecastReceipt canonical digest
+ RetrievalReceipt digest or explicit retrieval-disabled marker
+ policy/calibration/index digests
+ creation/expiry time and tenant namespace
Signing occurs at the trusted local boundary after artifact and retrieval verification. The wrapper records algorithm, public key ID, signer capability, and signature. Private keys never enter model artifacts, RuVector, RuVLLM, or a hosted training worker.
Consumers verify signature, expiry, tenant, schemas, every nested digest, and
the Forecast::verify_receipt invariant before using numeric values. Any
failure yields unavailable/abstain. Unsigned content-addressed receipts remain
useful for local tests but are not described as signed and cannot cross the
production trust boundary.
RuVLLM explanation boundary
RuVLLM receives a read-only projection of the verified envelope:
- exact point and quantile values as structured fields;
- units, horizons, calibration/OOD/abstention state;
- bounded analogue summaries and retrieval receipt references;
- evidence labels, model/version, and envelope digest;
- approved explanation policy and audience.
Explanation prose is stored separately and linked to the envelope digest. It
is CLAIMED_EXPLANATION, not a replacement forecast. Numeric API fields are
copied from the verified envelope after generation, never parsed back from LLM
text. If prose contradicts a structured number, evidence state, unit, or
disposition, the response fails validation and the numeric envelope remains
authoritative.
The explanation capability exposes no tools for artifact activation, model promotion, retraining, spending, messaging, emergency dispatch, access control, or actuation. A separate downstream policy may consume a forecast only under its own ADR, capabilities, approvals, and observed-evidence requirements. The LLM cannot grant itself that authority or lower an approval threshold.
Requirements and acceptance
| ID | Requirement | ADR-348 gate | Acceptance evidence | Current state |
|---|---|---|---|---|
| PM-001 | Every predictive-memory query is tenant-, model-, schema-, corpus-, split-, and time-scope bound. | G3, G5 | Cross-tenant/split/version negative tests and signed index manifest | OPEN / UNMEASURED |
| PM-002 | Holdout identities, overlapping contexts, and target horizons cannot appear as neighbours. | G3 | Property tests plus frozen leakage report | OPEN / UNMEASURED |
| PM-003 | Every query, including disabled/empty/error, produces a bounded retrieval receipt. | G2, G3 | Receipt round-trip, tamper, bound, and zero-result tests | OPEN / UNMEASURED |
| PM-004 | Retrieval-disabled and retrieval-enabled rows use identical artifact/examples and report paired metrics and overhead. | G3 | Frozen paired ablation with reproducer | OPEN / UNMEASURED |
| PM-005 | Forecast envelope signatures bind all nested forecast/retrieval identities and fail closed on mutation, expiry, tenant, or key error. | G2, G5 | Signature/tamper/replay/cross-tenant tests and local signing receipt | OPEN / UNMEASURED |
| PM-006 | Derived forecast and explanation can never become MEASURED_OBSERVATION or exceed source evidence. |
G2, G4 | Evidence-monotonicity schema/property tests | OPEN / UNMEASURED |
| PM-007 | RuVLLM cannot mutate structured numbers; contradictions fail validation and explanations remain separately labelled. | G4 | Numeric/unit/evidence mutation corpus and fail-closed integration tests | OPEN / UNMEASURED |
| PM-008 | RuVLLM explanation has no spending, promotion, messaging, safety, access-control, or actuator capability. | G4, G5 | Default-deny capability and attempted-escalation tests | OPEN / UNMEASURED |
| PM-009 | Vector/explanation privacy, retention, deletion, extraction, and membership risks are approved and operationally tested. | G5 | Privacy review, tenant deletion drill, and access audit | OPEN / UNMEASURED |
G3 additionally requires the paired retrieval ablation to satisfy ADR-348's forecast and calibration gates. G4 requires shadow evidence that retrieval and explanation improve operator comprehension or forecast value without changing numeric/action authority. G5 requires signed-envelope verification, rollback, tenant deletion, and key-rotation drills. None has passed.
Consequences
RuVector can evolve from retrospective search into outcome-linked predictive memory while preserving a measurable no-retrieval baseline. RuVLLM can make forecasts understandable without becoming the numeric or action authority. The cost is additional index isolation, storage, signature, deletion, evaluation, and policy complexity. If paired evidence does not justify that cost, the correct release keeps retrieval and explanation disabled.