RuView/docs/benchmarks/ruforecast.md

RuForecast benchmark protocol

Evidence status

No RuForecast runtime, accuracy, calibration, memory, or operational result is recorded here yet. Every threshold below is an ADR-348 target, not a measured claim. The first accepted row must identify a clean commit, the Cargo.lock digest, the exact Rust toolchain, host, backend, model/configuration digest, fixture or corpus digest, command, and evidence label.

Benchmark boundaries

RuForecast uses two different forms of evidence:

1. Deterministic correctness evidence comes from Rust unit, property, replay, split-isolation, and artifact-tamper tests. A fixed input and artifact must produce the same output within its declared platform class.
2. Runtime evidence comes from Criterion on a named host. Criterion inputs are generated from fixed checked-in code and seeds, but elapsed time is not deterministic. Timing from shared GitHub runners is informational only.

The benchmark implementation must not download a model or dataset, read raw CSI, use a hosted model output, or enable CUDA implicitly. CPU benchmarks use the explicit cpu feature and the Burn ndarray backend. CUDA validation belongs to a separately governed Linux or hosted-accelerator receipt.

Required benchmark targets

Package	Target	Purpose	CI authority
ruview-forecast-model	forecast_inference	Fixed-seed forward pass, batch and shape scaling, ordered-quantile output	Compile gate; shared-runner timing is informational
ruview-forecast-train	data_pipeline	Fixed generated records through validation, windowing, masking and batching	Compile gate; shared-runner timing is informational

Both targets must use code-generated synthetic inputs, fixed seeds, bounded allocations, criterion::black_box, and required-features = ["cpu"]. Setup, artifact construction, and dataset generation stay outside the timed region unless a benchmark name explicitly says they are included.

The model implementation owns structural parameter-count assertions. The currently reviewed design values are 35,700 parameters for the tiny CI preset and 20,285,108 for the large preset. These are design invariants, not benchmark results, and must be derived by a test from the actual module graph before they are quoted in a model card.

Local Linux reproducer

Run from a clean checkout after installing Rust 1.92.0:

RUFORECAST_CPUSET=0-7 \
RUFORECAST_THREADS=8 \
scripts/run-ruforecast-benchmarks.sh

The runner executes the focused contract/model/training tests, one real optimizer step over a local hash-addressed synthetic JSONL shard, and the idempotent synthetic CLI smoke. It then compile-checks both Criterion targets, runs the targets, captures the CPU and toolchain metadata, and hashes every output. A failed run retains its partial logs with status=FAILED and its exit code rather than looking like a complete report. Results go under target/ruforecast-evidence/, which is excluded from source control.

For a conservative single-thread reproducibility check, omit both environment variables. To run against an uncommitted tree for diagnosis only, set RUFORECAST_ALLOW_DIRTY=1; the resulting metadata is labelled SYNTHETIC with scope DIRTY_WORKTREE_DIAGNOSTIC_ONLY and cannot support a release claim.

A clean run labels its host timing MEASURED and its input class SYNTHETIC, but remains UNREVIEWED. Only a maintainer may append it to the accepted ledger after checking the digests, shape, command, Criterion report and host scope.

The runner intentionally has no CUDA option and does not parse Criterion output into a pass/fail performance verdict. This prevents a noisy host result from silently acquiring release authority.

To compile the two benchmark targets without measuring them:

cd v2
cargo +1.92.0 bench --locked -p ruview-forecast-model \
  --no-default-features --features cpu --bench forecast_inference --no-run
cargo +1.92.0 bench --locked -p ruview-forecast-train \
  --no-default-features --features cpu --bench data_pipeline --no-run

For a short informational run, use the same targets without --no-run:

cargo +1.92.0 bench --locked -p ruview-forecast-model \
  --no-default-features --features cpu --bench forecast_inference -- \
  --warm-up-time 1 --measurement-time 2 --sample-size 10
cargo +1.92.0 bench --locked -p ruview-forecast-train \
  --no-default-features --features cpu --bench data_pipeline -- \
  --warm-up-time 1 --measurement-time 2 --sample-size 10

Running these commands does not add a ledger row automatically. Preserve the raw report and environment metadata, then have a maintainer assign its evidence scope before publishing a number.

The inference bench runs only tiny_ci by default so a routine CI trend step cannot accidentally start the very expensive large CPU probe. Set RUFORECAST_BENCH_LARGE=1 only on a controlled host when intentionally measuring the fixed deployment shape:

RUFORECAST_BENCH_LARGE=1 scripts/run-ruforecast-benchmarks.sh

Deployment measurement shape

The initial CPU deployment probe is batch 1, context 1,024, 32 declared feature streams, the fixed large_linux horizon of 300, and all seven declared quantiles. Record at least 20 warmup iterations and 200 measured iterations for a release candidate. Report p50, p95, p99 or maximum, throughput, and the process peak resident set size.

ADR-348 G5 currently targets p95 at or below one second for 32 declared streams and peak process memory at or below 4 GiB. A Criterion result alone cannot close the memory gate because Criterion and Cargo are not the production inference process. G5 remains open until a standalone inference probe reports its own peak resident set size.

Accuracy and calibration protocol

Runtime speed never substitutes for forecasting quality. The frozen evaluation manifest must report identical examples for:

1. Last-value and seasonal-naive baselines.
2. RuForecast without RuVector retrieval.
3. RuForecast with split-scoped RuVector retrieval.

Required report fields include weighted quantile loss by horizon, nominal 80% interval coverage, missingness, abstention coverage, selective risk, site and device slices, interference regime, and retrieval ablation. ADR-348 G3 targets weighted quantile loss at least 10% better than seasonal naive and 80% interval coverage between 75% and 85%. Those targets remain unmeasured until a frozen, leakage-free report is attached.

Informal HPO exploration note (unaccepted, not a release claim)

2026-09-01. An exploratory session ran the accuracy protocol above end to end against a governed 24-window synthetic dataset (tiny_ci profile, context 64 / horizon 12, temporal train/test split, not entity-holdout — only one synthetic generator was used, so entity holdout does not apply) and a small OptimizerSpec hyperparameter search (learning rate, weight decay, gradient clip norm, batch size, epochs) using a new Darwin Mode numeric-genome evolution engine (upstream: ruvnet/metaharness PR #260, not yet merged). This is exploratory evidence only — not a frozen, leakage-free, maintainer-reviewed report, and not eligible for the ledger below until one is produced.

Prior to this exploration, a single real household window (76 real 1 Hz vital-signs samples, one physical ESP32 sensor, temporal not entity holdout) scored worse than both baselines (WQL 0.537 vs. last-value 0.106 and seasonal-naive 0.123) — consistent with a single training window overfitting rather than generalizing.

With a larger (still synthetic, still tiny_ci) 24-window training set and three rounds of hyperparameter search, weighted quantile loss on the held-out synthetic split improved and stayed ahead of both baselines throughout:

Round	learning_rate	weight_decay	grad_clip	batch	epochs	WQL (model)	WQL (last-value)	WQL (seasonal-naive)
Default config	0.0010000	1.00e-4	1.000	8	60	0.257	0.277	0.514
Search round 1	0.0002356	3.05e-6	0.100	27	195	0.161	0.277	0.514
Search round 2	0.0000298	4.09e-11	4.746	26	356	0.153	0.277	0.514

Round-2 gain over round 1 (−0.008) was much smaller than round-1's gain over the default (−0.096) — a diminishing-returns signal consistent with a local optimum for this model size and dataset, not a converged global result. gradient_clip_norm landed at opposite bound extremes across rounds (0.1 then 4.7), so no directional recommendation on that parameter should be drawn from this exploration alone.

Explicit scope limits — do not generalize beyond these:

• tiny_ci only. Nothing here has been run against large_linux; its far larger parameter count and different compute profile mean these hyperparameters are not a starting point for it without their own search.
• Synthetic dataset only (24 windows, one generator/seed family). Not validated against any real corpus at this scale.
• Self-signed, evaluation-only model activation (a throwaway local Ed25519 key, not a release signature) was used to run inference for scoring.
• No security/provenance/maintainer-approval gate has passed — the Darwin Mode promotion rule correctly refused to promote any candidate here.

Reproducer: harness/ruview/flywheel/ruforecast/ (genome, gate, evaluator, dry-run/--confirm driver) in the ruvnet/RuView repo, paired with ruvnet/metaharness PR #260 (evolve-numeric) linked locally via npm link. Neither the genome defaults here nor any repo default config were changed by this note — it is a record of exploratory evidence, not a committed recommendation.

Amendment (2026-09-01, later same day): the round-2 result above did not generalize -- retracted

The "Search round 2" row above (WQL 0.153, learning_rate=0.0000298 etc.) is RETRACTED as a claim of improvement. It is kept in the table (append-only, never silently edit a prior measurement) but must be read together with this amendment: independent verification, run the same day using a new regression-candidate promotion path (ruvnet/autogenous PR-in-progress, branch feat/regression-candidate-kind, not yet merged/pushed -- v2/crates/ruforecast-autogenous-bridge in this worktree) against TWO FRESH synthetic corpora that were never part of that search (seeds 1000/1097, vs. the search's single fixed seed 0), showed that "winner" genome performing WORSE than the baseline on both:

Judge corpus	Candidate WQL	Baseline WQL	Candidate beats baseline by
seed 1000	0.155	0.099	-0.056 (worse)
seed 1097	0.219	0.109	-0.110 (worse)

Root cause: every evaluation in the three search rounds above (default, round 1, round 2) trained and scored every candidate against the exact same fixed synthetic corpus (prepare-synthetic-dataset's implicit --seed 0 default). The search had learned to exploit that one corpus's specific random windows -- textbook overfitting -- not found a genuinely better hyperparameter configuration. This is a real, measured failure mode, not a hypothetical caveat.

Fix applied: harness/ruview/flywheel/ruforecast/gate.mjs's evaluateGenome now trains and scores every candidate against THREE independent synthetic corpora (DEFAULT_SEARCH_SEEDS = [11, 23, 47], none of which overlap the retracted search's seed 0 or the verification seeds 1000/1097) and takes the WORST CASE primary across them, not an average -- a candidate only counts as a win if it beats both baselines on every corpus. See harness/ruview/flywheel/ruforecast/README.md's "Multi-seed fitness" section for the full account.

Re-running the search with the fix, still no verified improvement. A fresh search under the corrected multi-seed fitness (2 generations x 2 children, cleared .metaharness-numeric archive so no stale pre-fix state leaked in) found a genuine winner ON ITS OWN THREE SEARCH SEEDS -- learning_rate=0.008012, weight_decay=0.000159, gradient_clip_norm=0.267, batch_size=24, epochs=20, beating the baseline on all three (primary 0.099 / 0.556 / 0.576, worst-case 0.099). Independent verification against the same fresh seeds 1000/1097 (never part of this candidate's own search) again showed it losing to the baseline on both:

Judge corpus	Candidate WQL	Baseline WQL	Candidate beats baseline by
seed 1000	0.178	0.099	-0.079 (worse)
seed 1097	0.314	0.109	-0.205 (worse)

Honest conclusion: two independent search rounds (pre- and post- the multi-seed fitness fix) both produced a genome that looked like a real improvement on its own search data and both failed independent out-of-sample verification. This converges on a different, deeper explanation than "the search methodology was broken" (that part IS fixed): at this dataset scale (24 synthetic training windows), held-out WQL varies enormously by which corpus is drawn REGARDLESS of hyperparameters -- the baseline genome itself, evaluated with the corrected multi-seed fitness function, swings from primary 0.83 (a strong win) to a full regression (primary 0, WQL worse than last-value) purely from which of the three fixed search seeds was used, with identical hyperparameters throughout. The corpus-noise floor at n=24 windows appears to dominate any real hyperparameter effect. No RuForecast hyperparameter configuration has been shown, by this exploration, to reliably beat the trivial baselines out-of-sample at this scale. A larger training corpus (more windows, ideally real governed data under the large_linux profile) is the more promising next lever than further hyperparameter search on this fixture.

A related implementation-only fix: the promotion verifier (envelope::regression::verify_regression_promotion in the Autogenous branch above) initially required all judges' receipts to share one corpus_id, inherited unreviewed from a same-evidence review model that does not fit this kind's intentionally cross-corpus judge design. This was corrected (ReceiptCorpusMismatch is no longer produced by that function); it did not change either REJECT verdict above, both of which were already correctly driven by the real NotBetterThanParent signal.

Reproducer for both verification runs above: same as below, plus v2/crates/ruforecast-autogenous-bridge (cargo +1.92.0 run --manifest-path crates/ruforecast-autogenous-bridge/Cargo.toml -- --ruforecast-bin ./target/debug/ruforecast --candidate-genome <genome>.json --parent-genome <baseline>.json --judges 2 --work-dir <scratch>) against the unpushed ruvnet/autogenous branch feat/regression-candidate-kind.

Real-household-data result (2026-09-01, MEASURED, unaccepted, not a release claim)

Following the informal HPO exploration above, this session also collected 6,390 real 1 Hz vital-signs samples (heart rate, breathing rate, signal quality) from a real, live, ESP32-sourced household sensing deployment over a continuous 2-hour window (88.75% real sample coverage; gaps handled honestly via observed_mask=0, never fabricated interpolation) — the "more real training data" lever flagged as the credible next step in the note above. This directly answers that open question.

Two genuinely independent temporal splits of the same real corpus (not synthetic seeds — real data has no seed to vary, so independence here means two different train/test boundary choices on the same timeline, each with its own 90s embargo gap) were trained (default, untuned OptimizerSpec: lr=0.001, weight_decay=0.0001, gradient_clip_norm=1.0, batch_size=8, epochs=60) and scored via the real evaluate CLI, then independently, cryptographically verified through ruforecast-autogenous-bridge's real signed regression-candidate promotion path (ruvnet/autogenous, envelope::regression):

Judge	Split	Real test windows	Model WQL	Best trivial baseline WQL	Model beats baseline by
1	70% train / 90s embargo / 30% test	27	0.0514	0.0563 (last-value)	+0.0049
2	50% train / 90s embargo / 50% test	46	0.0670	0.0543 (seasonal-naive)	−0.0128

Signed verdict: REJECT. Judge 1's nominal win (+0.0049) is below the 0.01 non-inferiority margin, so it doesn't clear the promotion bar even on its own; Judge 2 lost outright. Both rejections are recorded as NotBetterThanParent in the signed promotion envelope.

Honest conclusion: even with a real household corpus (n=6,390 real samples, not synthetic), the result is exactly the same shape as every synthetic search this session — a result that looks like a win on one split does not hold up on an independently verified second split. This is not evidence that real data can't help; it is evidence that this scale of real data (6,390 samples, one household, one physical sensor) is not yet enough to distinguish a genuine effect from split-dependent noise. The credible next lever remains more real data — more households, longer collection windows, or the large_linux profile — not further hyperparameter search on any fixture this small, synthetic or real.

Reproducer: v2/crates/ruforecast-autogenous-bridge/examples/real_data_verify.rs and v2/crates/ruforecast/crates/ruforecast-train/examples/real_data_windows.rs in this worktree (train/ruforecast-rust branch, commits 130f547 in the ruforecast submodule and 788401c5 in this repo — both local, not yet pushed). Raw real vitals data never left the collecting/training hosts and was never written to any git-tracked or pushed path.

Append-only evidence ledger

Never replace a prior measurement. Append a row and retain the failed or stale row when code, model, configuration, corpus, hardware, or methodology changes.

Date	Commit	Lock SHA-256	Host/toolchain	Backend/config	Shape	Samples	p50	p95	p99/max	Peak RSS	Evidence	Reproducer

No rows have been accepted.

Real public dataset: BIDMC PPG/Respiration (2026-09-02, cross-entity holdout)

Every real-data test up to this point used a single household/entity with only a temporal holdout (same physical sensor, different time windows). This test is the first with a genuine cross-entity holdout: 53 real ICU patients from the BIDMC PPG and Respiration Dataset (PhysioNet, Open Data Commons Attribution License v1.0, public and openly licensed — no credentialing, https://physionet.org/content/bidmc/1.0.0/), splitting by patient, not by time, so the held-out test set contains real people the model never saw during training.

25,546 real 1 Hz rows across 53 recordings (~8 minutes each), heart rate + respiratory rate + SpO2. 3 of 53 patients' windows were excluded honestly (genuine sensor-dropout NaN values in the source recordings, not fabricated/interpolated). Two independent, disjoint patient-partition splits:

Judge	Train patients	Test patients	Test windows	Model WQL	Best baseline WQL	Result
A (contiguous split)	34	16	16	0.01964	0.01159 (last-value)	worse, +69%
B (interleaved split)	24	26	26	0.07022	0.01002 (last-value)	worse, +601%

Same conclusion as every prior test this session, now on real, public, multi-subject clinical data with genuine cross-entity generalization: the model does not beat trivial forecasting baselines. The margin is decisive on both independent splits, not a near-miss — the earlier hypothesis that a larger, genuinely diverse real dataset (many different people, not one household) might change the picture does not hold at this scale/model configuration either.

Honest scope note on verification: prior real-data tests in this document were independently checked through Autogenous's signed regression-candidate promotion path. That additional cryptographic-signing step was not run for this test — the result is reported directly from the evaluate CLI's real output on two genuinely disjoint, real patient-holdout splits, which is itself real, independent, out-of-sample evidence, but it does not carry a signed promotion-gate verdict the way the earlier entries do. Flagging this explicitly rather than presenting it with the same evidentiary weight.

Reproducer: v2/crates/ruforecast/crates/ruforecast-train/examples/bidmc_prepare.rs (untracked scratch example, worktree train/ruforecast-rust) — downloads bidmc_NN_Numerics.csv for patients 01-53 directly from PhysioNet, builds one 76-row (context 64 + horizon 12) window per eligible patient, and writes governed train.jsonl/test.jsonl/train-local.toml per judge split. Raw data cached at /tmp/bidmc-raw/ on the training host only.

Zero-shot pretrained-checkpoint comparison (2026-09-02)

Every RuForecast result above this line is a from-scratch-trained model (RuForecast's own Burn/Rust architecture, trained on a tiny real or synthetic corpus) evaluated against trivial baselines. All of them lost. A literature pass after the BIDMC result above (arXiv:2505.11349, "context parroting"; arXiv:2607.20027, zero-shot HRV forecasting with TimesFM/Chronos/MOIRAI on wearables) converged on an explanation: published foundation-model wins over naive baselines on short-horizon, smooth physiological signals come from massive real pretraining corpora (Chronos: ~10M real+augmented series; TimesFM: ~100B real time points), not from architecture alone — and RuForecast's from-scratch training used a real corpus many orders of magnitude smaller (tens to low hundreds of windows), so re-implementing these architectures from scratch and training small was never going to replicate their wins.

This test checks the other half of that hypothesis directly: does a real, public, pretrained checkpoint, run zero-shot (no fine-tuning, no training of any kind on this repo's data) on the exact same BIDMC cross-entity test windows and metrics as the BIDMC result above, actually beat the naive baselines? If the massive-pretraining-corpus explanation is right, it should — unlike every from-scratch attempt this session.

Setup (kept identical to the BIDMC test above for a fair comparison): same /tmp/bidmc-raw/ cache (53 real ICU patients, PhysioNet BIDMC, Open Data Commons Attribution License v1.0), same window construction (context 64s + horizon 12s, 1 Hz, variates HR/RESP/SpO2, one window per patient from that patient's first 76 rows), same two disjoint patient-partition judges (A: train 1-37/test 38-53 contiguous; B: train odd/test even interleaved — train patients are irrelevant here since nothing is trained, only the test partition is used), same quantile set [0.05,0.10,0.25,0.50,0.75,0.90,0.95], same seasonal period (12), and the same WQL formula (2*sum(pinball)/(numQuantiles*sum|actual|)) and MAE definition as ruforecast-core::metrics.rs. Baselines were independently re-derived from ruforecast-core::baseline.rs's exact logic in Python and cross-checked against the BIDMC section above: reproduced last-value WQL was 0.01171 (A) / 0.01013 (B) versus the Rust CLI's originally reported 0.01159 / 0.01002 — a ~1% gap attributable to f32 (Rust) vs f64 (Python) accumulation, not a methodology difference, so this reproduction is treated as validated.

Four real, public, pretrained-checkpoint models were run zero-shot on GPU (RTX 5080, ruvultra), per-variate univariate (none of these checkpoints has a native multivariate mode for these two families, so HR/RESP/SpO2 were forecast independently and recombined into the same window-level WQL/MAE as the baselines and the from-scratch RuForecast model):

• Chronos-Bolt base/small (amazon/chronos-bolt-base, amazon/chronos-bolt-small) — direct quantile-regression heads, deterministic given input (no sampling).
• Chronos (T5) small (amazon/chronos-t5-small) — the original sampling-based Chronos family; predict_quantiles internally draws samples (default num_samples), so this one model's numbers below carry run-to-run sampling variance the other three don't.
• TimesFM 2.5 (200M, PyTorch) (google/timesfm-2.5-200m-pytorch) — quantile output uses 10 channels; per the installed package's source (timesfm_2p5_base.py, decode_index: int = 5) and empirical verification across multiple windows/variates, channel 5 is the point forecast and channels 1-9 are the native quantile levels [0.1..0.9] in order (channel 0 is an unused duplicate, not a real quantile). Levels 0.05 and 0.95 were clamped to the nearest native level (0.10/0.90); 0.25 and 0.75 were linearly interpolated between the adjacent native levels — the same class of approximation Chronos-Bolt's own pipeline applies internally when asked for quantile levels outside its trained range (a warning it printed for 0.05/0.95 on every call).

Judge	Model	Test windows	MAE	WQL	WQL vs last-value
A	last-value (baseline)	16	0.7882	0.01171	—
A	seasonal-naive (baseline)	16	1.0885	0.01617	worse
A	Chronos-Bolt-base (zero-shot)	16	0.7802	0.00756	-35.4%
A	Chronos-Bolt-small (zero-shot)	16	0.7212	0.00684	-41.6%
A	Chronos-T5-small (zero-shot)	16	0.9487	0.00955	-18.4%
A	TimesFM-2.5-200M (zero-shot)	16	0.7490	0.00731	-37.6%
B	last-value (baseline)	26	0.6944	0.01013	—
B	seasonal-naive (baseline)	26	0.8697	0.01269	worse
B	Chronos-Bolt-base (zero-shot)	26	0.7096	0.00719	-29.0%
B	Chronos-Bolt-small (zero-shot)	26	0.6852	0.00676	-33.3%
B	Chronos-T5-small (zero-shot)	26	0.8254	0.00886	-12.5%
B	TimesFM-2.5-200M (zero-shot)	26	0.6511	0.00698	-31.1%

Honest result: this is the first test all session where a model beats the trivial baselines, and it does so decisively and consistently — all four pretrained checkpoints beat both last-value and seasonal-naive on WQL, on both independently-partitioned judges, by double-digit-to-40% margins. This is a materially larger and more consistent margin than the closest published direct analog found in the literature pass (zero-shot HRV forecasting, arXiv:2607.20027: Chronos beat EWMA by ~3%, MOIRAI showed no significant advantage over a mean baseline, p=0.953) — plausibly because BIDMC's 1 Hz ICU HR/RESP/SpO2 signals are smoother and more autocorrelated than the HRV series in that paper, which is itself a fair, disclosed difference between the two comparisons rather than a claim of a stronger general result.

On MAE (point accuracy) the picture is more mixed: Chronos-Bolt-small and TimesFM beat last-value's MAE on both judges, but Chronos-Bolt-base and Chronos-T5-small do not consistently (e.g. Chronos-Bolt-base's MAE is essentially tied with last-value on judge A, worse on judge B, despite a much better WQL there) — the pretrained models' advantage is concentrated in calibrated quantile spread (what WQL scores), not in shifting the central point forecast much past what last-value already gets from strong autocorrelation in these signals.

This directly confirms the massive-pretraining-corpus hypothesis from the literature pass: the same short-horizon, smooth physiological-signal task on which every from-scratch RuForecast configuration this session lost to naive baselines is won, cleanly, by checkpoints that carry a pretraining corpus many orders of magnitude larger than any real dataset available to this project. This is evidence for zero-shot/foundation-model inference (or later, fine-tuning a pretrained checkpoint) as the credible next lever for RuForecast-style short-horizon vitals forecasting — not for continuing to scale from-scratch training on this project's own small real corpora, which the BIDMC and household tests above already showed does not close this gap even with genuine, cross-entity real data.

Scope and caveats, stated plainly:

• This is zero-shot inference only — no fine-tuning was attempted on any checkpoint. A fine-tuned comparison (e.g. LoRA-adapting Chronos-Bolt on RuForecast's own training windows) is a natural, not-yet-run follow-up.
• Univariate-per-variate forecasting was used for all four models because neither Chronos nor TimesFM in these releases has a native multivariate mode matching RuForecast's joint HR/RESP/SpO2 modeling; a true multivariate foundation model (e.g. MOIRAI, which does support cross-variate attention) was not tested here and is a natural next step.
• Chronos-T5-small's predict_quantiles samples internally (unlike Chronos-Bolt's direct quantile heads or TimesFM's quantile head); its numbers above are one run and carry unquantified sampling variance the other three models' numbers don't.
• No inference-cost/latency comparison against RuForecast's own (much smaller, ~tens-of-KB) model is included; all four pretrained checkpoints are orders of magnitude larger (small-to-200M+ parameters) and ran on a discrete GPU (RTX 5080), not the ESP32-class edge target RuForecast's own architecture is designed for. A model that wins on WQL here is not automatically deployable to the same edge budget RuForecast targets — that tradeoff is unresolved by this test and is itself a reason zero-shot inference (call out to a hosted/cloud checkpoint) and from-scratch small-model training remain different tools for different deployment constraints, not a strict replacement of one by the other.

Reproducer: scripts/ruforecast-zeroshot/bidmc_windows.py (window builder, baselines, and metrics — ported line-for-line from ruforecast-train/examples/bidmc_prepare.rs and ruforecast-core/src/{baseline,metrics}.rs, verified against this document's own BIDMC last-value/seasonal-naive numbers) and scripts/ruforecast-zeroshot/run_zeroshot_eval.py (zero-shot inference and scoring), both in this worktree (research/ruforecast-zeroshot-eval). Run against /tmp/bidmc-raw/ (downloaded fresh from PhysioNet if not already cached) with python run_zeroshot_eval.py --models chronos-bolt-base chronos-bolt-small chronos-t5-small timesfm-2.5-200m --judges a b inside a venv with chronos-forecasting, timesfm, and a CUDA-enabled torch installed (amazon/chronos-* and google/timesfm-2.5-200m-pytorch download automatically from HuggingFace Hub on first run; no HF token was required). Model weights and the raw BIDMC CSVs are not committed — /tmp/bidmc-raw/ and the HuggingFace cache live only on the host that ran this evaluation.