From b67fd39fda8ad66d2ba135503196e2fca62490ff Mon Sep 17 00:00:00 2001
From: Jack Chen <jack@yuanasi.com>
Date: Mon, 14 Sep 2026 18:45:12 +0800
Subject: [PATCH] docs: document the minSamples regression skip (#606)

The baseline section of docs/evaluation-ci.md enumerates every condition
that makes a regression comparison warn and skip, and it was missing the
one a threshold introduces: when it sets `minSamples` and either the
current or the baseline aggregate is short, the comparison does not run.
Record that condition and the consequence a gate author has to plan for,
namely that a short baseline now yields a warning where it previously
yielded a `regression` failure, so the guard loosens one check while it
tightens another.

Record the guard and `passSampleCount` in the changelog, set `minSamples`
on the runnable offline-regression example so the option appears where a
gate is actually configured, and credit the contributor.

Follow-up to #602.
---
 CHANGELOG.md                                  | 20 +++++++++++++++++++
 CONTRIBUTORS.md                               |  1 +
 docs/evaluation-ci.md                         | 10 ++++++++++
 .../patterns/eval-offline-regression.ts       |  6 ++++--
 4 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8de62cd..7e5a397 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,26 @@
 
 ## Unreleased
 
+### Added
+
+- Added an optional positive-integer `GateThreshold.minSamples` so an
+  evaluation gate only accepts a threshold result that is backed by enough
+  evidence. Score metrics compare it against the selected aggregate's
+  `scoredCount` and `passRate` compares it against `passSampleCount`.
+  Tag-scoped thresholds use the tag aggregate's own counts. A count below the
+  minimum reports the new `insufficient_samples` failure kind carrying the
+  observed count and the configured limit, and a report written before
+  `passSampleCount` existed fails closed at zero rather than skipping the
+  guard. Baseline regression checks honor the same minimum: when either side
+  holds fewer samples, that comparison is skipped with a warning, so adding
+  `minSamples` turns a regression failure computed against a small baseline
+  into a warning. Omitting `minSamples` preserves the previous behavior and the
+  report schema version is unchanged.
+- Added `ScorerAggregate.passSampleCount`, the count of scored records that
+  define `pass`. It is emitted whenever `passRate` is, including on tag
+  aggregates, and gives `passRate` guards the denominator that `scoredCount`
+  does not provide.
+
 ## 1.19.0 - 2026-09-11
 
 ### Added
diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
index 8be0b6d..e25ed0c 100644
--- a/CONTRIBUTORS.md
+++ b/CONTRIBUTORS.md
@@ -28,6 +28,7 @@ Opening your first PR gets you added here. See the [contribution guide](.github/
 - [@LambIessz](https://github.com/LambIessz) (orchestrator cost budget, MessageBus persistence in checkpoints, retryable route fallback)
 - [@Bobuyoucrypto](https://github.com/Bobuyoucrypto) (Windows bash timeout process-tree kill)
 - [@green3sf](https://github.com/green3sf) (structured output refresh after an accepted verify revision)
+- [@Zhongghaoming](https://github.com/Zhongghaoming) (evaluation gate minimum-samples guard)
 
 ## Provider integrations
 
diff --git a/docs/evaluation-ci.md b/docs/evaluation-ci.md
index 9d406fd..30fa7d4 100644
--- a/docs/evaluation-ci.md
+++ b/docs/evaluation-ci.md
@@ -123,6 +123,16 @@ does not produce a comparable score. Threshold and health checks still run.
 If baseline rules are configured but no baseline report is supplied, OMA warns
 and skips regression checks.
 
+A threshold that sets `minSamples` applies it to the regression comparison as
+well. When either the current or the baseline aggregate holds fewer samples
+than the minimum, OMA warns and skips that comparison instead of reporting a
+regression computed from a sample set the gate already called too small to
+judge. This loosens one check while it tightens another: a short baseline that
+previously produced a `regression` failure now produces only a warning, while a
+short current report still fails its own threshold with `insufficient_samples`.
+Commit a baseline large enough to satisfy every `minSamples` it serves, and
+read these warnings as a prompt to rerun the baseline rather than as noise.
+
 Use `oma eval gate` when report generation and quality enforcement are separate
 CI stages. It prints the exact verdict JSON to stdout:
 
diff --git a/packages/core/examples/patterns/eval-offline-regression.ts b/packages/core/examples/patterns/eval-offline-regression.ts
index b7e830d..081b577 100644
--- a/packages/core/examples/patterns/eval-offline-regression.ts
+++ b/packages/core/examples/patterns/eval-offline-regression.ts
@@ -116,8 +116,10 @@ if (baseline === undefined || candidate === undefined) throw new Error('Expected
 const verdict = evaluateGate(candidate, {
   schemaVersion: 1,
   thresholds: [
-    { scorer: 'exact_match', metric: 'passRate', min: 1 },
-    { scorer: 'answer_relevancy', metric: 'avg', min: 0.9 },
+    // minSamples is the fixture's own size: 2 cases at 2 repeats. Require far
+    // more in production, or a perfect score over a handful of samples passes.
+    { scorer: 'exact_match', metric: 'passRate', min: 1, minSamples: 4 },
+    { scorer: 'answer_relevancy', metric: 'avg', min: 0.9, minSamples: 4 },
   ],
   maxScorerErrorRate: 0,
   maxTargetErrorRate: 0,