docs: document the minSamples regression skip (#606)

The baseline section of docs/evaluation-ci.md enumerates every condition
that makes a regression comparison warn and skip, and it was missing the
one a threshold introduces: when it sets `minSamples` and either the
current or the baseline aggregate is short, the comparison does not run.
Record that condition and the consequence a gate author has to plan for,
namely that a short baseline now yields a warning where it previously
yielded a `regression` failure, so the guard loosens one check while it
tightens another.

Record the guard and `passSampleCount` in the changelog, set `minSamples`
on the runnable offline-regression example so the option appears where a
gate is actually configured, and credit the contributor.

Follow-up to #602.

CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## Unreleased
 
+### Added
+
+- Added an optional positive-integer `GateThreshold.minSamples` so an
+  evaluation gate only accepts a threshold result that is backed by enough
+  evidence. Score metrics compare it against the selected aggregate's
+  `scoredCount` and `passRate` compares it against `passSampleCount`.
+  Tag-scoped thresholds use the tag aggregate's own counts. A count below the
+  minimum reports the new `insufficient_samples` failure kind carrying the
+  observed count and the configured limit, and a report written before
+  `passSampleCount` existed fails closed at zero rather than skipping the
+  guard. Baseline regression checks honor the same minimum: when either side
+  holds fewer samples, that comparison is skipped with a warning, so adding
+  `minSamples` turns a regression failure computed against a small baseline
+  into a warning. Omitting `minSamples` preserves the previous behavior and the
+  report schema version is unchanged.
+- Added `ScorerAggregate.passSampleCount`, the count of scored records that
+  define `pass`. It is emitted whenever `passRate` is, including on tag
+  aggregates, and gives `passRate` guards the denominator that `scoredCount`
+  does not provide.
+
 ## 1.19.0 - 2026-09-11
 
 ### Added

CONTRIBUTORS.md

@@ -28,6 +28,7 @@ Opening your first PR gets you added here. See the [contribution guide](.github/
 - [@LambIessz](https://github.com/LambIessz) (orchestrator cost budget, MessageBus persistence in checkpoints, retryable route fallback)
 - [@Bobuyoucrypto](https://github.com/Bobuyoucrypto) (Windows bash timeout process-tree kill)
 - [@green3sf](https://github.com/green3sf) (structured output refresh after an accepted verify revision)
+- [@Zhongghaoming](https://github.com/Zhongghaoming) (evaluation gate minimum-samples guard)
 
 ## Provider integrations
 

docs/evaluation-ci.md

@@ -123,6 +123,16 @@ does not produce a comparable score. Threshold and health checks still run.
 If baseline rules are configured but no baseline report is supplied, OMA warns
 and skips regression checks.
 
+A threshold that sets `minSamples` applies it to the regression comparison as
+well. When either the current or the baseline aggregate holds fewer samples
+than the minimum, OMA warns and skips that comparison instead of reporting a
+regression computed from a sample set the gate already called too small to
+judge. This loosens one check while it tightens another: a short baseline that
+previously produced a `regression` failure now produces only a warning, while a
+short current report still fails its own threshold with `insufficient_samples`.
+Commit a baseline large enough to satisfy every `minSamples` it serves, and
+read these warnings as a prompt to rerun the baseline rather than as noise.
+
 Use `oma eval gate` when report generation and quality enforcement are separate
 CI stages. It prints the exact verdict JSON to stdout:
 

packages/core/examples/patterns/eval-offline-regression.ts

@@ -116,8 +116,10 @@ if (baseline === undefined || candidate === undefined) throw new Error('Expected
 const verdict = evaluateGate(candidate, {
   schemaVersion: 1,
   thresholds: [
-    { scorer: 'exact_match', metric: 'passRate', min: 1 },
-    { scorer: 'answer_relevancy', metric: 'avg', min: 0.9 },
+    // minSamples is the fixture's own size: 2 cases at 2 repeats. Require far
+    // more in production, or a perfect score over a handful of samples passes.
+    { scorer: 'exact_match', metric: 'passRate', min: 1, minSamples: 4 },
+    { scorer: 'answer_relevancy', metric: 'avg', min: 0.9, minSamples: 4 },
   ],
   maxScorerErrorRate: 0,
   maxTargetErrorRate: 0,