bit 1d40455844 feat(core): enforce minimum eval gate samples (#602) ...

Add an optional positive-integer `minSamples` guard to `GateThreshold` so a threshold only counts when the selected aggregate carries enough evidence. Score metrics compare it against `scoredCount`; `passRate` compares it against the new `ScorerAggregate.passSampleCount`, which records the pass denominator whenever `passRate` is emitted. Tag-scoped thresholds use the tag aggregate's own counts. A count below the minimum reports a new `insufficient_samples` failure carrying the observed count and the configured limit, and a report written before `passSampleCount` existed fails closed at zero rather than skipping the guard.

Baseline regression checks honor the same minimum. When either the current or the baseline aggregate is below it, the comparison is skipped with a warning instead of producing a regression number from a sample set the gate already called too small to judge.

Omitting `minSamples` preserves the previous behavior, so the report schema version is unchanged and no migration is required.

Fixes #573

2026-09-14 18:19:56 +08:00

..

e2e

fix: name the current DeepSeek Flash model instead of a retired alias (#598)

2026-09-12 01:04:31 +08:00

fixtures

feat(core): add hybrid semantic execution routing

2026-07-26 13:56:28 +08:00

helpers

feat(core): add authoritative run store and execution leases (#587)

2026-09-07 02:35:32 +08:00

abort-signal-propagation.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

abort-signal.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

acp-backend.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

adaptive-recovery.test.ts

feat(core): add adaptive plan recovery

2026-07-26 11:51:57 +08:00

advanced-sampling.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

agent-backend-config-types.test.ts

feat(agent): add generic process backend (#378)

2026-07-16 14:40:26 +08:00

agent-credentials.test.ts

feat(core): add hybrid semantic execution routing

2026-07-26 13:56:28 +08:00

agent-hooks.test.ts

feat(agent): restore persisted conversation history (#470)

2026-08-07 16:10:48 +08:00

agent-pool.test.ts

feat(core): pipeline DAG tasks as dependencies complete (#434)

2026-07-23 19:52:48 +08:00

agent-selector.test.ts

fix(core): enforce task requirements globally

2026-07-26 11:21:17 +08:00

agent-structured-input.test.ts

feat(core): accept structured agent input (#476)

2026-08-08 11:28:39 +08:00

ai-sdk-adapter.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

anthropic-adapter.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

approval.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

azure-openai-adapter.test.ts

feat(core): add framework-owned LLM egress policy (#478)

2026-08-10 15:54:43 +08:00

bash-risk-classifier.test.ts

feat(classifiers): add optional classifyBashCommand behind /classifiers subpath (#382)

2026-07-16 16:15:48 +08:00

bedrock-adapter.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

built-in-tools.test.ts

fix: kill full process tree on Windows bash timeout and settle without waiting for orphaned pipes (#357)

2026-07-06 18:51:41 +08:00

call-timeout.test.ts

feat(runner): OMA-owned per-call LLM timeout, uniform across adapters (#344)

2026-07-01 14:08:16 +08:00

checkpoint-v5.test.ts

feat(core): auditable context rewrites and journal-watermarked checkpoint v5 (#527) (#555)

2026-08-28 23:42:58 +08:00

checkpoint.test.ts

feat(core): support rich tool results (#479)

2026-08-08 11:49:55 +08:00

cli.test.ts

feat(core): add task-scoped results and handoffs (#435)

2026-07-23 23:50:00 +08:00

commission-reconciliation-recovery.test.ts

feat(examples): add repairable commission recovery cookbook (#565)

2026-09-03 11:59:54 +08:00

consensus.test.ts

fix(orchestrator): keep parsed consensus output out of the public result (#537)

2026-08-20 11:26:18 +08:00

consequential-confirmation.test.ts

feat(core): add hybrid semantic execution routing

2026-07-26 13:56:28 +08:00

context-strategy.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

contract-suite-types.test.ts

test(core): add a MemoryStore contract suite and type-check the store contracts (#572)

2026-09-04 01:24:39 +08:00

coordinator-dependency-contract.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

coordinator-manifest.test.ts

feat(core): add composite scheduling and strict assignees (#432)

2026-07-23 16:35:02 +08:00

copilot-adapter.test.ts

feat(core): add framework-owned LLM egress policy (#478)

2026-08-10 15:54:43 +08:00

dashboard-layout-tasks.test.ts

feat(core): add offline run viewer for result and trace data (#394)

2026-07-18 11:06:27 +08:00

dashboard-render-large.test.ts

feat(core): add offline run viewer for result and trace data (#394)

2026-07-18 11:06:27 +08:00

dashboard-render.test.ts

feat(core): trace execution routing decisions (#430)

2026-07-23 15:25:32 +08:00

deepseek-adapter.test.ts

feat(core): support DeepSeek V4 Flash thinking controls (#455)

2026-07-31 16:35:43 +08:00

default-deny-tools.test.ts

feat(core): add hybrid semantic execution routing

2026-07-26 13:56:28 +08:00

delegation-budget.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

delegation-concurrency.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

dependency-payload.test.ts

feat(core): add task-scoped results and handoffs (#435)

2026-07-23 23:50:00 +08:00

doubao-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

durable-approval-validation.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

durable-approval.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

egress-policy.test.ts

feat(core): add framework-owned LLM egress policy (#478)

2026-08-10 15:54:43 +08:00

error-classification.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

errors.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

eval-cli.test.ts

test: resolve tsx CLI through Node module resolution

2026-07-26 20:19:54 +08:00

eval-evalset.test.ts

feat(core): add EvalSet offline runner (#404)

2026-07-18 23:26:28 +08:00

eval-file-store.test.ts

test(core): add a MemoryStore contract suite and type-check the store contracts (#572)

2026-09-04 01:24:39 +08:00

eval-file.test.ts

feat(core): enforce minimum eval gate samples (#602)

2026-09-14 18:19:56 +08:00

eval-gate-cli.test.ts

test: resolve tsx CLI through Node module resolution

2026-07-26 20:19:54 +08:00

eval-gate.test.ts

feat(core): enforce minimum eval gate samples (#602)

2026-09-14 18:19:56 +08:00

eval-in-memory-store.test.ts

test(core): add a MemoryStore contract suite and type-check the store contracts (#572)

2026-09-04 01:24:39 +08:00

eval-judge.test.ts

feat(eval): let judgePrompt build per-judge structured input (#563)

2026-09-04 17:48:24 +08:00

eval-online-lifecycle.test.ts

feat(core): add online evaluation sampling (#408)

2026-07-19 10:21:54 +08:00

eval-online.test.ts

feat(core): add online evaluation sampling (#408)

2026-07-19 10:21:54 +08:00

eval-run-metadata.test.ts

feat(core): add per-run metadata channel (#396)

2026-07-18 15:39:32 +08:00

eval-runner.test.ts

feat(core): enforce minimum eval gate samples (#602)

2026-09-14 18:19:56 +08:00

eval-scorer.test.ts

feat(core): add evaluation scorer contracts (#403)

2026-07-18 22:53:52 +08:00

eval-scorers.test.ts

feat(core): add reference evaluation scorers (#409)

2026-07-19 10:59:17 +08:00

eval-target.test.ts

feat(core): add hybrid semantic execution routing

2026-07-26 13:56:28 +08:00

execution-receipt.test.ts

feat(core): add task-scoped results and handoffs (#435)

2026-07-23 23:50:00 +08:00

execution-router.test.ts

fix(core): harden hybrid routing failure handling

2026-07-26 20:23:37 +08:00

file-store.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

file-trace-store.test.ts

feat(observability): add FileTraceStore reference implementation (#384)

2026-07-16 16:50:59 +08:00

fs-walk.test.ts

fix: kill full process tree on Windows bash timeout and settle without waiting for orphaned pipes (#357)

2026-07-06 18:51:41 +08:00

gemini-adapter-contract.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

gemini-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

governance-floor.test.ts

feat(orchestrator): add declared governance and post-execution floor (#420)

2026-07-23 10:28:48 +08:00

governance-intent.test.ts

fix(core): honor planOnly for governed team runs (#428)

2026-07-23 14:35:43 +08:00

grok-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

hunyuan-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

journal-backends.test.ts

test(core): drive the journal batching window with fake timers (#560)

2026-08-29 09:49:15 +08:00

journal-context-replace.test.ts

feat(core): auditable context rewrites and journal-watermarked checkpoint v5 (#527) (#555)

2026-08-28 23:42:58 +08:00

journal-emission.test.ts

feat(core): auditable context rewrites and journal-watermarked checkpoint v5 (#527) (#555)

2026-08-28 23:42:58 +08:00

journal-lineage.test.ts

feat(core): auditable context rewrites and journal-watermarked checkpoint v5 (#527) (#555)

2026-08-28 23:42:58 +08:00

keywords.test.ts

feat(core): add Japanese and Korean routing support (#438)

2026-07-24 00:53:33 +08:00

llm-adapter-capabilities.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

llm-adapters.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

llm-message-validation.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

loop-detection.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

mcp-tools.test.ts

feat(core): support rich tool results (#479)

2026-08-08 11:49:55 +08:00

memory-store-contract.test.ts

test(core): add a MemoryStore contract suite and type-check the store contracts (#572)

2026-09-04 01:24:39 +08:00

mimo-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

minimax-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

mode-governance-resolution.test.ts

feat(core): trace execution routing decisions (#430)

2026-07-23 15:25:32 +08:00

multi-perspective-code-review.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

observability-doc-examples.test.ts

chore(observability): complete OBS-5 release readiness (#385)

2026-07-16 18:00:16 +08:00

observability-v2-identity.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

observability-v2-privacy.test.ts

fix(core): record tool input/output on v2 spans under an explicit capture policy (#561)

2026-08-31 16:43:13 +08:00

observability-v2-sink.test.ts

chore(observability): complete OBS-5 release readiness (#385)

2026-07-16 18:00:16 +08:00

observability-v2-spans.test.ts

feat(core): add span lifecycle and hierarchy

2026-07-15 22:08:49 +08:00

onAgentStream.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

openai-adapter.test.ts

feat(core): add framework-owned LLM egress policy (#478)

2026-08-10 15:54:43 +08:00

openai-common.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

openai-fallback-repair.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

openai-fallback.test.ts

chore(deps)!: upgrade OpenAI SDK to v6 and require Node 20 (#453)

2026-07-29 11:21:27 +08:00

openai-sdk-boundary.test.ts

chore(deps)!: upgrade OpenAI SDK to v6 and require Node 20 (#453)

2026-07-29 11:21:27 +08:00

orchestrator.test.ts

docs(core): correct JSDoc that misstates runtime behavior (#585)

2026-09-05 19:47:21 +08:00

otel-package-boundary.test.ts

ci(otel): guard the npm tarball and harden the core boundary test

2026-07-16 12:31:22 +08:00

personalized-interview-simulator.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

pipeline-scheduling.test.ts

feat(core): pipeline DAG tasks as dependencies complete (#434)

2026-07-23 19:52:48 +08:00

process-backend-lifecycle.test.ts

fix(process): clean descendants after parent exit (#390)

2026-07-17 20:22:56 +08:00

process-backend.test.ts

feat(agent): add generic process backend (#378)

2026-07-16 14:40:26 +08:00

public-type-export-coverage.test.ts

feat(core): auditable context rewrites and journal-watermarked checkpoint v5 (#527) (#555)

2026-08-28 23:42:58 +08:00

qiniu-adapter.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

reasoning-fallback.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

redacting-store.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

redaction.test.ts

feat(memory): redact secrets on the persistence path via RedactingStore (#339) (#359)

2026-07-09 22:25:53 +08:00

research-aggregation.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

rich-tool-results.test.ts

feat(core): support rich tool results (#479)

2026-08-08 11:49:55 +08:00

routing-stability-eval.test.ts

fix(core): make hybrid routing opt-in

2026-07-27 01:03:22 +08:00

run-lease.test.ts

feat(core): add authoritative run store and execution leases (#587)

2026-09-07 02:35:32 +08:00

run-store.test.ts

feat(core): add authoritative run store and execution leases (#587)

2026-09-07 02:35:32 +08:00

run-viewer-model.test.ts

feat(dashboard): surface per-task model/provider/cost in Run Viewer (#411)

2026-07-19 15:34:09 +08:00

run-viewer-waterfall.test.ts

feat(core): add offline run viewer for result and trace data (#394)

2026-07-18 11:06:27 +08:00

runner-error-propagation.test.ts

feat(agent): orchestrate external coding agents over ACP (#204) (#360)

2026-07-10 23:26:40 +08:00

runner-reasoning-flags.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

runteam-verify.test.ts

fix(core): fail closed on invalid coordinator plans

2026-07-26 10:55:20 +08:00

sampling-wire-format.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

scheduler.test.ts

fix(core): enforce task requirements globally

2026-07-26 11:21:17 +08:00

semantic-routing-default-adapter.test.ts

fix(core): make hybrid routing opt-in

2026-07-27 01:03:22 +08:00

semantic-routing.test.ts

fix: name the current DeepSeek Flash model instead of a retired alias (#598)

2026-09-12 01:04:31 +08:00

semaphore.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

shared-memory.test.ts

feat(approval): add durable suspend and resume gates (#480)

2026-08-08 03:40:10 +08:00

shell-executor.test.ts

feat(core): add pluggable shell executor (#477)

2026-08-10 15:39:08 +08:00

short-circuit.test.ts

feat(core): add Japanese and Korean routing support (#438)

2026-07-24 00:53:33 +08:00

structured-output.test.ts

feat(release): add OMA-powered release bot (#483)

2026-08-10 17:18:29 +08:00

subpath-exports.test.ts

test(core): cover remaining subpath export barrels in smoke suite (#516)

2026-08-16 14:06:46 +08:00

task-metadata.test.ts

feat(core): add task-scoped results and handoffs (#435)

2026-07-23 23:50:00 +08:00

task-queue.test.ts

feat(core): add adaptive plan recovery

2026-07-26 11:51:57 +08:00

task-results.test.ts

feat(core): add task-scoped results and handoffs (#435)

2026-07-23 23:50:00 +08:00

task-retry.test.ts

feat(release): add OMA-powered release bot (#483)

2026-08-10 17:18:29 +08:00

task-utils.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

team-concurrency.test.ts

feat(core): enforce TeamConfig.maxConcurrency on the team's pool (#593)

2026-09-08 17:55:31 +08:00

team-messaging.test.ts

feat(core): persist MessageBus in checkpoints (#363)

2026-07-10 21:19:16 +08:00

text-tool-extractor.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

token-budget.test.ts

feat(release): add OMA-powered release bot (#483)

2026-08-10 17:18:29 +08:00

tokens.test.ts

fix(core): make unsupported-video terminal and validate video URL sources (#566)

2026-09-03 12:05:03 +08:00

tool-executor.test.ts

fix(core): align JSON Schema required fields with Zod (#549)

2026-08-24 15:35:53 +08:00

tool-filtering.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

tool-result-compression.test.ts

refactor(repo): monorepo groundwork for an easier first run (#303)

2026-06-18 17:31:52 +08:00

trace-store-contract.test.ts

feat(core): add trace store contract and memory reference

2026-07-15 23:59:18 +08:00

trace.test.ts

feat(core): add per-call tool gate hook (#377)

2026-07-16 11:45:47 +08:00

verify-run.test.ts

feat(core): verifyRun journal replay verification (#527) (#556)

2026-08-29 00:43:38 +08:00