mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-22 02:53:26 -05:00
Merge origin/main into feat/hybrid-semantic-routing
This commit is contained in:
Jack Chen committed
commit
4db96dcfd4
29 files changed
+2932
-266
No files matched your search
@@ -60,7 +60,7 @@ OMA combines dynamic orchestration with the control, evidence, and recovery path
|
||||
|
||||
- **Production essentials.** Run reliably, diagnose failures, and prevent quality regressions.
|
||||
|
||||
- **Reliability:** Resume interrupted runs from checkpoints. Retries, timeouts, loop detection, and token and cost budgets keep execution bounded.
|
||||
- **Reliability:** Resume interrupted runs from checkpoints or opt into append-only plan repair at task outcome barriers. Retries, timeouts, loop detection, and token and cost budgets keep execution bounded.
|
||||
- **Observability:** Follow each run through stable identity, execution receipts, and traces; query them in TraceStore, replay the task DAG and span waterfall in the offline Run Viewer, or export through the optional OpenTelemetry adapter.
|
||||
- **Evaluation:** Use the same run records for versioned EvalSets, reference scorers, offline reports, CI gates, regression baselines, and production sampling.
|
||||
- **Safety and privacy:** Keep tools default-deny, gate individual calls, and apply explicit privacy controls to telemetry and persisted state.
|
||||
@@ -153,7 +153,7 @@ Need to embed agent capabilities in an existing product or business system? Emai
|
||||
|---|---|
|
||||
| Install and run | [Core package guide](packages/core/README.md) · [Examples](packages/core/examples/README.md) · [CLI](docs/cli.md) |
|
||||
| Configure models and tools | [Providers](docs/providers.md) · [Tools and sandbox](docs/tool-configuration.md) · [External agents](docs/external-agents.md) |
|
||||
| Operate reliably | [Observability](docs/observability.md) · [Evaluation](docs/evaluation.md) · [Checkpoint and resume](docs/checkpoint.md) · [Context management](docs/context-management.md) |
|
||||
| Operate reliably | [Observability](docs/observability.md) · [Evaluation](docs/evaluation.md) · [Checkpoint and resume](docs/checkpoint.md) · [Adaptive recovery](docs/adaptive-recovery.md) · [Context management](docs/context-management.md) |
|
||||
| Control orchestration | [Consensus](docs/consensus.md) · [Execution routing](docs/execution-routing.md) · [Model routing](docs/model-routing.md) · [Task scheduling](docs/task-scheduling.md) · [Plan replay](docs/plan-replay.md) · [Shared memory](docs/shared-memory.md) |
|
||||
|
||||
## Contributing
|
||||
|
||||
+2
-2
@@ -60,7 +60,7 @@ OMA 将动态编排与生产所需的控制、证据和恢复能力结合起来
|
||||
|
||||
- **生产必备。** 让运行可恢复、故障可诊断，并及时发现质量退化。
|
||||
|
||||
- **可靠性：** 通过 Checkpoint 从断点恢复中断的运行；重试、超时、循环检测与 token、成本双预算让执行始终有明确边界。
|
||||
- **可靠性：** 通过 Checkpoint 从断点恢复中断的运行，或选择在任务结果屏障处启用仅追加式计划修复；重试、超时、循环检测与 token、成本双预算让执行始终有明确边界。
|
||||
- **可观测性：** 通过稳定的运行标识、执行回执与 Trace 跟踪每次运行；使用 TraceStore 查询记录，在离线 Run Viewer 中回放任务 DAG 与 span 瀑布，或通过可选的 OpenTelemetry 适配器导出。
|
||||
- **评测：** 使用同一套运行记录支撑版本化 EvalSet、参考 scorer 打分、离线报告、CI gate、回归基线与线上采样。
|
||||
- **安全与隐私：** 内置工具默认拒绝，支持逐次调用 gate，并对遥测与持久化状态应用显式的隐私控制。
|
||||
@@ -157,7 +157,7 @@ Core 用户可以在本地保存 trace，并用离线 Run Viewer 查看。只有
|
||||
|---|---|
|
||||
| 安装与运行 | [核心包使用指南](packages/core/README_zh.md) · [示例](packages/core/examples/README.md) · [CLI](docs/cli.md) |
|
||||
| 配置模型与工具 | [Provider](docs/providers.md) · [工具与沙箱](docs/tool-configuration.md) · [外部 Agent](docs/external-agents.md) |
|
||||
| 稳定运行 | [可观测性](docs/observability.md) · [评测](docs/evaluation.md) · [Checkpoint 与恢复](docs/checkpoint.md) · [上下文管理](docs/context-management.md) |
|
||||
| 稳定运行 | [可观测性](docs/observability.md) · [评测](docs/evaluation.md) · [Checkpoint 与恢复](docs/checkpoint.md) · [自适应恢复](docs/adaptive-recovery.md) · [上下文管理](docs/context-management.md) |
|
||||
| 控制编排 | [Consensus](docs/consensus.md) · [执行路由](docs/execution-routing.md) · [模型路由](docs/model-routing.md) · [任务调度](docs/task-scheduling.md) · [计划回放](docs/plan-replay.md) · [共享记忆](docs/shared-memory.md) |
|
||||
|
||||
## 参与贡献
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# Adaptive recovery
|
||||
|
||||
OMA keeps task graphs fixed by default. Opt into `recovery.mode: 'repairable'`
|
||||
when an application needs to revise the not-yet-executed part of a run after a
|
||||
task succeeds, fails, or is rejected by consensus verification.
|
||||
|
||||
Adaptive recovery is an outcome barrier, not another retry:
|
||||
|
||||
1. A task produces an outcome.
|
||||
2. The configured `Replanner` (or `onTaskOutcome` callback) may propose a
|
||||
`PlanPatch`.
|
||||
3. OMA validates agent eligibility, limits, task states, references, and the
|
||||
resulting DAG.
|
||||
4. The optional `onPlanPatch` gate approves or rejects the proposal.
|
||||
5. OMA applies the patch atomically, persists a checkpoint when checkpointing
|
||||
is enabled, then publishes newly-ready work.
|
||||
6. Only after that barrier does the triggering task complete or fail and
|
||||
release or cascade its original dependents.
|
||||
|
||||
This ordering prevents an original downstream task from starting while its
|
||||
replacement is still being decided.
|
||||
|
||||
## Configure a replanner
|
||||
|
||||
```ts
|
||||
import {
|
||||
OpenMultiAgent,
|
||||
type Replanner,
|
||||
type TaskOutcome,
|
||||
} from '@open-multi-agent/core'
|
||||
|
||||
const replanner: Replanner = {
|
||||
name: 'fallback-search',
|
||||
replan(outcome: TaskOutcome) {
|
||||
if (outcome.kind !== 'failure' || outcome.task.title !== 'Search') {
|
||||
return undefined
|
||||
}
|
||||
|
||||
const oldAnalysis = outcome.tasks.find((task) => task.title === 'Analysis')
|
||||
if (!oldAnalysis) return undefined
|
||||
|
||||
return {
|
||||
reason: 'Primary search failed; use the fallback source.',
|
||||
supersedePending: [oldAnalysis.id],
|
||||
addTasks: [
|
||||
{
|
||||
key: 'fallback-search',
|
||||
title: 'Fallback Search',
|
||||
description: 'Fetch the source through the fallback path.',
|
||||
assignee: 'researcher-b',
|
||||
},
|
||||
{
|
||||
key: 'replacement-analysis',
|
||||
title: 'Replacement Analysis',
|
||||
description: 'Analyze the fallback result.',
|
||||
assignee: 'analyst',
|
||||
dependsOn: ['fallback-search'],
|
||||
},
|
||||
],
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
const result = await oma.runTasks(team, tasks, {
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
replanner,
|
||||
maxPlanRevisions: 3,
|
||||
maxAddedTasks: 20,
|
||||
onPlanPatch: async (patch, outcome) => {
|
||||
return await applicationPolicyApproves(patch, outcome)
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
`onTaskOutcome` is a shorthand for applications that do not need a named
|
||||
`Replanner` object. Configure one or the other, not both. A custom replanner may
|
||||
call an LLM or another service, but that external I/O and its usage accounting
|
||||
remain owned by the application.
|
||||
|
||||
## Patch operations
|
||||
|
||||
- `addTasks` appends tasks. Every appended task has a patch-local `key`.
|
||||
Dependencies may refer to another key in the same patch or to an existing
|
||||
task ID.
|
||||
- `retargetPending` changes the assignee of a `pending` or `blocked` task after
|
||||
the new agent passes the same eligibility checks used by scheduling.
|
||||
- `supersedePending` marks a `pending` or `blocked` task `skipped`. Replacement
|
||||
tasks are appended rather than rewriting or deleting history.
|
||||
|
||||
Started or terminal tasks cannot be retargeted or superseded. A failure or
|
||||
verification rejection is classified as recovered only when the accepted patch
|
||||
appends at least one replacement task. Successful tasks may append or reshape
|
||||
downstream work without being marked recovered.
|
||||
|
||||
Patch references use task IDs, not titles. This avoids ambiguity when multiple
|
||||
runtime tasks have the same title.
|
||||
|
||||
## Failure, durability, and restore
|
||||
|
||||
Without an accepted patch, existing behavior is unchanged: failed tasks cascade
|
||||
failure to their dependents while independent branches may continue.
|
||||
|
||||
Accepted revisions are stored in queue snapshot version 2. Snapshot version 1
|
||||
remains the fixed-DAG format and is still readable. When checkpointing is
|
||||
enabled, OMA saves the patched graph before it dispatches appended work. If that
|
||||
save fails, the unpublished patch is rolled back and the original failure path
|
||||
continues.
|
||||
|
||||
Restore resets an interrupted task for execution while retaining durable plan
|
||||
revision history. A revision with the same trigger task and outcome is not
|
||||
appended twice after a crash.
|
||||
|
||||
Historical tasks remain truthful in `result.tasks`: a repaired failure is still
|
||||
`failed` with `recoveredByRevision`, and a replaced branch is `skipped` with
|
||||
`supersededByRevision`. Those historical records do not make the overall run
|
||||
fail when the active repaired graph finishes successfully. Accepted revisions
|
||||
are returned in `result.planRevisions`.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Recovery is opt-in. Existing callers remain fixed-DAG.
|
||||
- `runFromPlan()` is exact replay and rejects repairable recovery.
|
||||
- Repairable recovery is incompatible with legacy round-based `onApproval`.
|
||||
Use `onTaskDispatch` and/or `onPlanPatch` gates.
|
||||
- Limits reject further patches; they never silently truncate a patch.
|
||||
- Policy, validation, approval, revision, and checkpoint decisions are exposed
|
||||
through progress events and observability spans.
|
||||
- A repair is forward-only. OMA does not undo external side effects already
|
||||
performed by a task.
|
||||
+17
-16
@@ -97,11 +97,12 @@ enters results, task trace attributes, checkpoint snapshots, or plan artifacts.
|
||||
|
||||
## Assignment strategies
|
||||
|
||||
Unassigned tasks are scheduled when they become ready. `dependency-first` and
|
||||
`composite` order the current ready set by downstream criticality; each selected
|
||||
task is then assigned individually. `round-robin` retains its cursor,
|
||||
`least-busy` reads current `in_progress` load, and `capability-match` retains its
|
||||
hard eligibility filter.
|
||||
Before any task is dispatched, the complete plan is validated against explicit
|
||||
task requirements. Every scheduling strategy filters to eligible agents first;
|
||||
the configured strategy only ranks or rotates within that eligible set.
|
||||
`dependency-first` and `composite` order the current ready set by downstream
|
||||
criticality, `round-robin` retains its cursor, and `least-busy` reads current
|
||||
`in_progress` load.
|
||||
|
||||
Agents may declare `description`, `capabilities`, `costTier`, and
|
||||
`latencyClass`; none is inferred when omitted. Explicit `runTasks()` specs and
|
||||
@@ -109,14 +110,14 @@ coordinator-generated tasks may add `requires` with `requiredTools`,
|
||||
`requiredCapabilities`, `requiredBackend`, and `requiredProvider`. Tool
|
||||
requirements are checked against the final resolved grant set, after presets,
|
||||
allowlists, denylists, and framework rails.
|
||||
Provider requirements are also checked after worker model routing. Incompatible
|
||||
fallback routes are removed rather than crossing the declared provider boundary.
|
||||
|
||||
`capability-match` and `composite` deliberately differ when a task's `requires`
|
||||
cannot be satisfied:
|
||||
|
||||
| Strategy | No eligible agent |
|
||||
|---|---|
|
||||
| `capability-match` | Terminates scheduling with `NO_ELIGIBLE_AGENT`. |
|
||||
| `composite` | Emits a structured `NO_ELIGIBLE_AGENT` warning, then falls back to zero fit plus current load. |
|
||||
When no roster candidate satisfies an unassigned task, validation fails with
|
||||
`INVALID_TASK_REQUIREMENTS` and an issue code of `NO_ELIGIBLE_AGENT`. When an
|
||||
explicit assignee exists but does not satisfy the requirements, the issue code
|
||||
is `ASSIGNEE_REQUIREMENTS_MISMATCH`. Both cases fail before worker execution;
|
||||
hard requirements never fall back to an ineligible agent.
|
||||
|
||||
`composite` maximizes `fitWeight * fit + loadWeight * (1 - normalizedCurrentLoad)`.
|
||||
`schedulingWeights.fit` and `schedulingWeights.load` default to `0.7` and `0.3`,
|
||||
@@ -126,10 +127,10 @@ state. Assignments earlier in one scheduler call are not folded into that same
|
||||
call; in event-driven execution the next ready-task call can observe tasks
|
||||
already marked `in_progress`.
|
||||
|
||||
Coordinator plans that name an agent outside the roster emit an
|
||||
`INVALID_ASSIGNEE` warning, clear that assignment, and use the configured
|
||||
scheduler by default. Set `strictAssignees: true` to stop before task execution
|
||||
with a structured validation error instead.
|
||||
Coordinator plans that name an agent outside the roster fail validation before
|
||||
task execution by default. Set `strictAssignees: false` only to retain the
|
||||
legacy behavior of emitting an `INVALID_ASSIGNEE` warning, clearing that
|
||||
assignment, and using the configured scheduler.
|
||||
|
||||
## Approval modes
|
||||
|
||||
|
||||
@@ -317,7 +317,9 @@ Backend and provider checks use their structured configuration fields.
|
||||
|
||||
Neither permissions nor capabilities are ever inferred from `systemPrompt` or
|
||||
other prose. When no candidate satisfies the hard requirements, the selector
|
||||
returns `NO_ELIGIBLE_AGENT`; a caller must choose any fallback explicitly.
|
||||
returns `NO_ELIGIBLE_AGENT`. Team and explicit-task execution validate the
|
||||
complete plan before dispatch and fail with `INVALID_TASK_REQUIREMENTS`; no
|
||||
scheduling strategy may fall back to an ineligible agent.
|
||||
|
||||
## Per-call gating with `onToolCall`
|
||||
|
||||
|
||||
+14
-12
@@ -161,18 +161,20 @@ const orchestrator = new OpenMultiAgent({
|
||||
|
||||
| Strategy | Assignment behavior | Recommended when |
|
||||
|----------|---------------------|------------------|
|
||||
| `dependency-first` (default) | Assigns tasks that unblock the most downstream work first, rotating agents | The task graph has meaningful dependencies |
|
||||
| `round-robin` | Distributes tasks in queue order across the roster | Agents are interchangeable |
|
||||
| `least-busy` | Chooses the agent with the fewest active or newly assigned tasks | Task duration varies and load balance matters |
|
||||
| `dependency-first` (default) | Assigns tasks that unblock the most downstream work first, rotating eligible agents | The task graph has meaningful dependencies |
|
||||
| `round-robin` | Distributes tasks in queue order across eligible agents | Agents are interchangeable |
|
||||
| `least-busy` | Chooses the eligible agent with the fewest active or newly assigned tasks | Task duration varies and load balance matters |
|
||||
| `capability-match` | Filters explicit task requirements, then prefers declared capability tags before legacy keyword affinity | Tasks or agents declare differentiated requirements/capabilities |
|
||||
| `composite` | Ranks tasks by blocked dependents, hard-filters with `AgentSelector`, then maximizes `fitWeight * fit + loadWeight * (1 - normalizedCurrentLoad)` | Criticality, capability fit, and current load should influence one decision |
|
||||
| `composite` | Ranks tasks by blocked dependents, then maximizes fit and available capacity across eligible agents | Criticality, capability fit, and current load should influence one decision |
|
||||
|
||||
Agents may declare `description`, `capabilities`, `costTier`, and
|
||||
`latencyClass`, and tasks may add hard `requires` constraints; set
|
||||
`strictAssignees: true` to fail fast when a coordinator plan names an agent
|
||||
outside the roster. Weight semantics, load normalization, `NO_ELIGIBLE_AGENT`
|
||||
and `INVALID_ASSIGNEE` behavior, approval compatibility, and progress-event
|
||||
migration are covered in
|
||||
`latencyClass`, and tasks may add hard `requires` constraints. All strategies
|
||||
fail before worker execution when those constraints cannot be satisfied.
|
||||
Coordinator plans also fail fast by default when they name an agent outside
|
||||
the roster; set `strictAssignees: false` only to retain legacy reassignment.
|
||||
Weight semantics, load normalization, `NO_ELIGIBLE_AGENT` and
|
||||
`INVALID_ASSIGNEE` behavior, approval compatibility, and progress-event migration
|
||||
are covered in
|
||||
[Task scheduling and dispatch](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/task-scheduling.md).
|
||||
|
||||
## Capabilities
|
||||
@@ -203,7 +205,7 @@ Coordinator -> Task DAG -> Scheduler -> AgentPool
|
||||
`-- results -> evaluation (offline / sampled, observe-only)
|
||||
```
|
||||
|
||||
The coordinator plans once; the scheduler owns execution order. Agents share results through memory, while checkpoints and traces form separate recovery and observability paths. Evaluation observes completed results and never changes them. Detailed contracts live in the linked subsystem guides below.
|
||||
The coordinator plans once by default; the scheduler owns execution order. Applications can opt into append-only adaptive recovery when task outcomes need to revise the unstarted part of the graph. Agents share results through memory, while checkpoints and traces form separate recovery and observability paths. Evaluation observes completed results and never changes them. Detailed contracts live in the linked subsystem guides below.
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -244,7 +246,7 @@ See [Providers](https://github.com/open-multi-agent/open-multi-agent/blob/main/d
|
||||
| Bound work | `maxTurns`, `timeoutMs`, `callTimeoutMs`, `contextStrategy`, `loopDetection` |
|
||||
| Control spend | `maxTokenBudget`; `maxCostBudget` + application-owned `estimateCost` |
|
||||
| Limit tools | `tools` / `toolPreset`, `cwd` / `defaultCwd`, tool-output caps |
|
||||
| Recover | Task retries, checkpointing, and `restore()` |
|
||||
| Recover | Task retries, checkpointing, `restore()`, and opt-in adaptive plan repair |
|
||||
| Review work | `planOnly`, `onPlanReady`, and approval callbacks |
|
||||
| Observe | Trace sinks, TraceStore, execution receipts, Run Viewer, or the optional OTel adapter |
|
||||
|
||||
@@ -265,7 +267,7 @@ See the [observability guide](https://github.com/open-multi-agent/open-multi-age
|
||||
| Area | Guides |
|
||||
|---|---|
|
||||
| Build agents | [Providers](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md), [tools](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md), [context](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/context-management.md) |
|
||||
| Run reliably | [Evaluation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md), [checkpoint & resume](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/checkpoint.md), [execution routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md), [model routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md), [consensus](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/consensus.md) |
|
||||
| Run reliably | [Evaluation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md), [checkpoint & resume](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/checkpoint.md), [adaptive recovery](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/adaptive-recovery.md), [execution routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md), [model routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md), [consensus](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/consensus.md) |
|
||||
| Control workflows | [Plan preview & replay](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/plan-replay.md), [shared memory](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/shared-memory.md), [external agents](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/external-agents.md) |
|
||||
| Operate | [Observability](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/observability.md), [CLI](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/cli.md), [production examples](examples/production/README.md) |
|
||||
|
||||
|
||||
@@ -145,15 +145,17 @@ const orchestrator = new OpenMultiAgent({
|
||||
|
||||
| 策略 | 分配行为 | 适用场景 |
|
||||
|------|----------|----------|
|
||||
| `dependency-first`（默认） | 优先分配能解锁最多下游工作的任务，并轮转选择 Agent | 任务图存在明确依赖关系 |
|
||||
| `round-robin` | 按队列顺序在 Agent roster 中轮转分配 | Agent 能力可以互换 |
|
||||
| `least-busy` | 选择当前活跃任务或本批新分配任务最少的 Agent | 任务耗时差异较大，需要负载均衡 |
|
||||
| `dependency-first`（默认） | 优先分配能解锁最多下游工作的任务，并在合格 Agent 中轮转选择 | 任务图存在明确依赖关系 |
|
||||
| `round-robin` | 按队列顺序在合格 Agent 中轮转分配 | Agent 能力可以互换 |
|
||||
| `least-busy` | 选择当前活跃任务或本批新分配任务最少的合格 Agent | 任务耗时差异较大，需要负载均衡 |
|
||||
| `capability-match` | 先过滤显式任务要求，再优先匹配声明的能力标签，最后使用兼容的关键词亲和度 | 任务或 Agent 声明了有区分度的要求/能力 |
|
||||
| `composite` | 按阻塞的下游任务数排列任务，经 `AgentSelector` 硬过滤后，最大化 `fitWeight * fit + loadWeight * (1 - normalizedCurrentLoad)` | 需要在一次决策中同时考虑关键度、能力匹配与当前负载 |
|
||||
| `composite` | 按阻塞的下游任务数排列任务，再在合格 Agent 中综合选择匹配度与可用容量最优者 | 需要在一次决策中同时考虑关键度、能力匹配与当前负载 |
|
||||
|
||||
Agent 可声明 `description`、`capabilities`、`costTier` 与 `latencyClass`，
|
||||
任务可通过 `requires` 声明硬性过滤条件；设置 `strictAssignees: true` 可在
|
||||
coordinator 计划引用 roster 之外的 Agent 时提前失败。权重语义、负载归一化、
|
||||
任务可通过 `requires` 声明硬约束。任何调度策略无法满足这些约束时，都会在
|
||||
worker 执行前失败。Coordinator 计划默认也会在引用
|
||||
roster 之外的 Agent 时提前失败；仅在需要保留旧的自动重新分配行为时设置
|
||||
`strictAssignees: false`。权重语义、负载归一化、
|
||||
`NO_ELIGIBLE_AGENT` 与 `INVALID_ASSIGNEE` 行为、审批兼容与 progress 事件迁移
|
||||
见[任务调度与派发](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/task-scheduling.md)。
|
||||
|
||||
@@ -185,7 +187,7 @@ Coordinator -> 任务 DAG -> Scheduler -> AgentPool
|
||||
`-- 结果 -> 评测（离线 / 抽样，仅观察）
|
||||
```
|
||||
|
||||
Coordinator 只负责产生计划，Scheduler 负责执行顺序。Agent 通过记忆共享结果，checkpoint 与 trace 分别形成恢复和可观测路径。评测只观察已完成的结果，不会改变它们。详细契约见下方各子系统指南。
|
||||
默认情况下，Coordinator 只负责产生一次计划，Scheduler 负责执行顺序。当任务结果需要修改任务图中尚未执行的部分时，应用可以选择启用仅追加式自适应恢复。Agent 通过记忆共享结果，checkpoint 与 trace 分别形成恢复和可观测路径。评测只观察已完成的结果，不会改变它们。详细契约见下方各子系统指南。
|
||||
|
||||
## 示例
|
||||
|
||||
@@ -226,7 +228,7 @@ Coordinator 只负责产生计划，Scheduler 负责执行顺序。Agent 通过
|
||||
| 限定工作量 | `maxTurns`、`timeoutMs`、`callTimeoutMs`、`contextStrategy`、`loopDetection` |
|
||||
| 控制成本 | `maxTokenBudget`；`maxCostBudget` + 应用自有 `estimateCost` |
|
||||
| 限制工具 | `tools` / `toolPreset`、`cwd` / `defaultCwd`、工具输出上限 |
|
||||
| 故障恢复 | 任务重试、checkpoint 与 `restore()` |
|
||||
| 故障恢复 | 任务重试、checkpoint、`restore()` 与可选的自适应计划修复 |
|
||||
| 人工把关 | `planOnly`、`onPlanReady` 与审批回调 |
|
||||
| 统一观测 | Trace sink、TraceStore、执行回执、Run Viewer，或可选 OTel adapter |
|
||||
|
||||
@@ -247,7 +249,7 @@ Core 已提供运行标识、trace sink、执行回执、可查询的内存/文
|
||||
| 主题 | 指南 |
|
||||
|---|---|
|
||||
| 构建 agent | [Provider](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md)、[工具](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md)、[上下文](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/context-management.md) |
|
||||
| 稳定运行 | [评测](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md)、[Checkpoint & resume](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/checkpoint.md)、[执行路由](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md)、[模型路由](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md)、[Consensus](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/consensus.md) |
|
||||
| 稳定运行 | [评测](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/evaluation.md)、[Checkpoint & resume](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/checkpoint.md)、[自适应恢复](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/adaptive-recovery.md)、[执行路由](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md)、[模型路由](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md)、[Consensus](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/consensus.md) |
|
||||
| 控制流程 | [计划预览与回放](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/plan-replay.md)、[共享记忆](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/shared-memory.md)、[外部 agent](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/external-agents.md) |
|
||||
| 生产运维 | [可观测性](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/observability.md)、[CLI](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/cli.md)、[生产示例](examples/production/README.md) |
|
||||
|
||||
|
||||
@@ -2,7 +2,24 @@
|
||||
* @fileoverview Framework-specific error classes.
|
||||
*/
|
||||
|
||||
import type { SemanticRoutingAssessment } from './types.js'
|
||||
import type { SemanticRoutingAssessment, TaskRequirementIssue } from './types.js'
|
||||
|
||||
/**
|
||||
* Raised before task execution when a task has no eligible agent or its
|
||||
* explicit assignee does not satisfy the task's hard requirements.
|
||||
*/
|
||||
export class InvalidTaskRequirementsError extends Error {
|
||||
readonly code = 'INVALID_TASK_REQUIREMENTS'
|
||||
|
||||
constructor(readonly issues: readonly TaskRequirementIssue[]) {
|
||||
super(
|
||||
`Task requirements are unsatisfied: ${issues
|
||||
.map((issue) => `${issue.code} for "${issue.taskTitle}"`)
|
||||
.join(', ')}.`,
|
||||
)
|
||||
this.name = 'InvalidTaskRequirementsError'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Raised when an agent or orchestrator run exceeds its configured token budget.
|
||||
@@ -149,6 +166,7 @@ function extractStatus(error: unknown): number | undefined {
|
||||
* limits (429), and all 5xx server errors — is retryable.
|
||||
*/
|
||||
export function isRetryableError(error: unknown): boolean {
|
||||
if (error instanceof InvalidTaskRequirementsError) return false
|
||||
if (error instanceof TokenBudgetExceededError) return false
|
||||
if (error instanceof CostBudgetExceededError) return false
|
||||
if (error instanceof InvalidMessageError) return false
|
||||
|
||||
@@ -96,7 +96,10 @@ export type {
|
||||
SchedulerOptions,
|
||||
SchedulerWarning,
|
||||
} from './orchestrator/scheduler.js'
|
||||
export { AgentSelector } from './orchestrator/agent-selector.js'
|
||||
export {
|
||||
AgentSelector,
|
||||
validateTaskRequirements,
|
||||
} from './orchestrator/agent-selector.js'
|
||||
export type {
|
||||
AgentSelectionFailure,
|
||||
AgentSelectionResult,
|
||||
@@ -186,6 +189,7 @@ export {
|
||||
TokenBudgetExceededError,
|
||||
CostBudgetExceededError,
|
||||
InvalidMessageError,
|
||||
InvalidTaskRequirementsError,
|
||||
LLMCallTimeoutError,
|
||||
RoutingDeclarationRequiredError,
|
||||
RoutingProfilerFailedError,
|
||||
@@ -356,7 +360,16 @@ export type {
|
||||
TaskProfilerResult,
|
||||
RunTasksOptions,
|
||||
RunTaskSpec,
|
||||
PlanPatchTaskSpec,
|
||||
PlanPatchRetarget,
|
||||
PlanPatch,
|
||||
PlanRevision,
|
||||
TaskVerificationOutcome,
|
||||
TaskOutcome,
|
||||
Replanner,
|
||||
RecoveryOptions,
|
||||
TaskMetadata,
|
||||
TaskRequirementIssue,
|
||||
TaskRequirements,
|
||||
RestoreOptions,
|
||||
ModelRouteConfig,
|
||||
@@ -390,6 +403,8 @@ export type {
|
||||
CheckpointRunIdentity,
|
||||
CompletedTaskCheckpoint,
|
||||
TaskQueueSnapshot,
|
||||
TaskQueueSnapshotV1,
|
||||
TaskQueueSnapshotV2,
|
||||
TaskSnapshot,
|
||||
|
||||
// Trace
|
||||
|
||||
@@ -96,7 +96,11 @@ export class Checkpoint {
|
||||
if (value === null || typeof value !== 'object') return false
|
||||
const snapshot = value as Record<string, unknown>
|
||||
if (snapshot['version'] !== 1 && snapshot['version'] !== 2) return false
|
||||
if (snapshot['mode'] !== 'runTeam' && snapshot['mode'] !== 'runTasks') return false
|
||||
if (
|
||||
snapshot['mode'] !== 'runTeam'
|
||||
&& snapshot['mode'] !== 'runTasks'
|
||||
&& snapshot['mode'] !== 'runFromPlan'
|
||||
) return false
|
||||
if (typeof snapshot['createdAt'] !== 'string') return false
|
||||
if (!Array.isArray(snapshot['completedTaskResults'])) return false
|
||||
if (snapshot['metadata'] !== undefined) {
|
||||
@@ -120,6 +124,12 @@ export class Checkpoint {
|
||||
const queue = snapshot['queue']
|
||||
if (queue === null || typeof queue !== 'object') return false
|
||||
const queueRecord = queue as Record<string, unknown>
|
||||
return queueRecord['version'] === 1 && Array.isArray(queueRecord['tasks'])
|
||||
if (queueRecord['version'] !== 1 && queueRecord['version'] !== 2) return false
|
||||
if (!Array.isArray(queueRecord['tasks'])) return false
|
||||
if (queueRecord['version'] === 2) {
|
||||
if (!Number.isInteger(queueRecord['planRevision'])) return false
|
||||
if (!Array.isArray(queueRecord['planRevisions'])) return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,8 @@ export type SpanEventName =
|
||||
| 'loop_detected'
|
||||
| 'stream_chunk'
|
||||
| 'consensus_verdict'
|
||||
| 'recovery_decision'
|
||||
| 'plan_revision_applied'
|
||||
|
||||
export interface TraceRecordBase {
|
||||
readonly schemaVersion: 2
|
||||
|
||||
@@ -10,6 +10,8 @@
|
||||
import type {
|
||||
AgentConfig,
|
||||
OrchestratorConfig,
|
||||
Task,
|
||||
TaskRequirementIssue,
|
||||
TaskRequirements,
|
||||
} from '../types.js'
|
||||
import type { SupportedProvider } from '../llm/adapter.js'
|
||||
@@ -32,6 +34,8 @@ export interface AgentSelectorContext {
|
||||
readonly defaultToolPreset?: OrchestratorConfig['defaultToolPreset']
|
||||
/** Team workers register delegation; standalone selection does not. */
|
||||
readonly includeDelegateTool?: boolean
|
||||
/** Resolve the effective worker config for one task before hard filtering. */
|
||||
readonly resolveCandidate?: (task: Task, candidate: AgentConfig) => AgentConfig
|
||||
}
|
||||
|
||||
export interface EligibleAgentScore {
|
||||
@@ -217,3 +221,79 @@ export class AgentSelector {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function hasExplicitTaskRequirements(
|
||||
requires: TaskRequirements | undefined,
|
||||
): boolean {
|
||||
return requires !== undefined && (
|
||||
(requires.requiredTools?.length ?? 0) > 0
|
||||
|| (requires.requiredCapabilities?.length ?? 0) > 0
|
||||
|| requires.requiredBackend !== undefined
|
||||
|| requires.requiredProvider !== undefined
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate every task's hard requirements before any task is dispatched.
|
||||
*
|
||||
* Unassigned tasks must have at least one eligible roster candidate. Explicitly
|
||||
* assigned tasks are checked only against their named agent so assignment
|
||||
* conflicts fail instead of being silently reassigned.
|
||||
*/
|
||||
export function validateTaskRequirements(
|
||||
tasks: readonly Task[],
|
||||
candidates: readonly AgentConfig[],
|
||||
context: AgentSelectorContext = {},
|
||||
): readonly TaskRequirementIssue[] {
|
||||
const selector = new AgentSelector()
|
||||
const issues: TaskRequirementIssue[] = []
|
||||
|
||||
for (const task of tasks) {
|
||||
if (
|
||||
task.status === 'completed'
|
||||
|| task.status === 'failed'
|
||||
|| task.status === 'skipped'
|
||||
) continue
|
||||
if (!hasExplicitTaskRequirements(task.requires)) continue
|
||||
|
||||
if (task.assignee !== undefined) {
|
||||
const assignee = candidates.find((candidate) => candidate.name === task.assignee)
|
||||
// Unknown assignees retain the existing INVALID_ASSIGNEE/not-found path.
|
||||
if (!assignee) continue
|
||||
const effectiveAssignee = context.resolveCandidate?.(task, assignee) ?? assignee
|
||||
const selection = selector.select({
|
||||
title: task.title,
|
||||
description: task.description,
|
||||
requires: task.requires,
|
||||
}, [effectiveAssignee], context)
|
||||
if (selection.error) {
|
||||
issues.push({
|
||||
code: 'ASSIGNEE_REQUIREMENTS_MISMATCH',
|
||||
taskId: task.id,
|
||||
taskTitle: task.title,
|
||||
assignee: task.assignee,
|
||||
reasons: selection.error.reasons,
|
||||
})
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
const effectiveCandidates = candidates.map((candidate) =>
|
||||
context.resolveCandidate?.(task, candidate) ?? candidate)
|
||||
const selection = selector.select({
|
||||
title: task.title,
|
||||
description: task.description,
|
||||
requires: task.requires,
|
||||
}, effectiveCandidates, context)
|
||||
if (selection.error) {
|
||||
issues.push({
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
taskId: task.id,
|
||||
taskTitle: task.title,
|
||||
reasons: selection.error.reasons,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return issues
|
||||
}
|
||||
@@ -325,7 +325,14 @@ export async function runTaskVerify(
|
||||
result: AgentRunResult,
|
||||
sharedMem: ReturnType<Team['getSharedMemoryInstance']>,
|
||||
ctx: RunContext,
|
||||
): Promise<AgentRunResult> {
|
||||
): Promise<{
|
||||
readonly result: AgentRunResult
|
||||
readonly verification: {
|
||||
readonly verdict: 'accepted' | 'rejected'
|
||||
readonly dissent: readonly string[]
|
||||
readonly rounds: number
|
||||
}
|
||||
}> {
|
||||
const verify = task.verify!
|
||||
const { team, config } = ctx
|
||||
const assigneeConfig = team.getAgents().find((a) => a.name === assignee)
|
||||
@@ -415,8 +422,15 @@ export async function runTaskVerify(
|
||||
const useRevision =
|
||||
consensus.verdict === 'accepted' && consensus.answer && consensus.answer !== result.output
|
||||
return {
|
||||
...result,
|
||||
output: useRevision ? consensus.answer : result.output,
|
||||
tokenUsage: addUsage(result.tokenUsage, consensus.tokenUsage),
|
||||
result: {
|
||||
...result,
|
||||
output: useRevision ? consensus.answer : result.output,
|
||||
tokenUsage: addUsage(result.tokenUsage, consensus.tokenUsage),
|
||||
},
|
||||
verification: {
|
||||
verdict: consensus.verdict,
|
||||
dissent: [...consensus.dissent],
|
||||
rounds: consensus.rounds,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@
|
||||
*/
|
||||
|
||||
import type { RunOptions } from '../agent/runner.js'
|
||||
import { extractJSON } from '../agent/structured-output.js'
|
||||
import type {
|
||||
AgentConfig,
|
||||
AgentRunResult,
|
||||
@@ -35,6 +36,7 @@ import {
|
||||
withModelRoute,
|
||||
routeMatches,
|
||||
} from './agent-config.js'
|
||||
import { z } from 'zod'
|
||||
|
||||
/**
|
||||
* Partial verify config that the coordinator can emit in task JSON.
|
||||
@@ -70,6 +72,119 @@ export interface ParsedTaskSpec {
|
||||
verify?: ConsensusVerifyOptions | CoordinatorVerifySpec | true
|
||||
}
|
||||
|
||||
const coordinatorVerifySchema = z.union([
|
||||
z.literal(true),
|
||||
z.object({
|
||||
mode: z.enum(['refute', 'lens']).optional(),
|
||||
quorum: z.number().int().positive().optional(),
|
||||
maxRounds: z.number().int().positive().optional(),
|
||||
onDissent: z.enum(['revise', 'reject', 'keep']).optional(),
|
||||
}).strict(),
|
||||
])
|
||||
|
||||
const coordinatorTaskSpecSchema = z.object({
|
||||
title: z.string().trim().min(1),
|
||||
description: z.string().trim().min(1),
|
||||
assignee: z.string().trim().min(1).optional(),
|
||||
dependsOn: z.array(z.string().trim().min(1)).optional(),
|
||||
memoryScope: z.enum(['dependencies', 'all']).optional(),
|
||||
maxRetries: z.number().int().nonnegative().optional(),
|
||||
retryDelayMs: z.number().finite().nonnegative().optional(),
|
||||
retryBackoff: z.number().finite().positive().optional(),
|
||||
role: z.string().trim().min(1).optional(),
|
||||
priority: z.enum(['low', 'normal', 'high', 'critical']).optional(),
|
||||
requires: z.object({
|
||||
requiredTools: z.array(z.string().trim().min(1)).optional(),
|
||||
requiredCapabilities: z.array(z.string().trim().min(1)).optional(),
|
||||
requiredBackend: z.enum(['llm', 'process', 'acp']).optional(),
|
||||
requiredProvider: z.string().trim().min(1).optional(),
|
||||
}).strict().optional(),
|
||||
verify: coordinatorVerifySchema.optional(),
|
||||
}).strict()
|
||||
|
||||
const coordinatorTaskSpecsSchema = z.array(coordinatorTaskSpecSchema).min(1)
|
||||
|
||||
/**
|
||||
* Build the coordinator output contract for one team. The roster and task
|
||||
* titles are runtime facts, so they belong in the same schema that triggers
|
||||
* the Agent's single structured-output repair attempt.
|
||||
*/
|
||||
export function buildCoordinatorTaskSpecsSchema(
|
||||
agents: readonly AgentConfig[],
|
||||
strictAssignees: boolean,
|
||||
) {
|
||||
const roster = new Set(agents.map((agent) => agent.name))
|
||||
const normalizeTitle = (title: string) => title.toLowerCase().trim()
|
||||
|
||||
return coordinatorTaskSpecsSchema.superRefine((tasks, ctx) => {
|
||||
const titleIndexes = new Map<string, number[]>()
|
||||
for (let index = 0; index < tasks.length; index++) {
|
||||
const key = normalizeTitle(tasks[index]!.title)
|
||||
const indexes = titleIndexes.get(key) ?? []
|
||||
indexes.push(index)
|
||||
titleIndexes.set(key, indexes)
|
||||
}
|
||||
|
||||
for (let index = 0; index < tasks.length; index++) {
|
||||
const task = tasks[index]!
|
||||
if (strictAssignees && task.assignee !== undefined && !roster.has(task.assignee)) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: [index, 'assignee'],
|
||||
message: `Unknown assignee "${task.assignee}".`,
|
||||
})
|
||||
}
|
||||
for (const [dependencyIndex, dependency] of (task.dependsOn ?? []).entries()) {
|
||||
const matches = titleIndexes.get(normalizeTitle(dependency)) ?? []
|
||||
if (matches.length !== 1) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: [index, 'dependsOn', dependencyIndex],
|
||||
message: matches.length === 0
|
||||
? `Unknown dependency "${dependency}".`
|
||||
: `Ambiguous dependency "${dependency}".`,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const [title, indexes] of titleIndexes) {
|
||||
if (indexes.length > 1) {
|
||||
for (const index of indexes) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: [index, 'title'],
|
||||
message: `Duplicate task title "${title}".`,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const visiting = new Set<number>()
|
||||
const visited = new Set<number>()
|
||||
const visit = (index: number): boolean => {
|
||||
if (visiting.has(index)) return true
|
||||
if (visited.has(index)) return false
|
||||
visiting.add(index)
|
||||
const task = tasks[index]!
|
||||
for (const dependency of task.dependsOn ?? []) {
|
||||
const matches = titleIndexes.get(normalizeTitle(dependency)) ?? []
|
||||
if (matches.length === 1 && visit(matches[0]!)) return true
|
||||
}
|
||||
visiting.delete(index)
|
||||
visited.add(index)
|
||||
return false
|
||||
}
|
||||
if (tasks.some((_, index) => visit(index))) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: [],
|
||||
message: 'Cyclic task dependency detected.',
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
export interface InvalidAssigneeIssue {
|
||||
readonly taskTitle: string
|
||||
readonly assignee: string
|
||||
@@ -139,50 +254,9 @@ export function parseCoordinatorVerify(raw: unknown): CoordinatorVerifySpec | tr
|
||||
* or as a bare array. Returns `null` when no valid array can be extracted.
|
||||
*/
|
||||
export function parseTaskSpecs(raw: string): ParsedTaskSpec[] | null {
|
||||
// Strategy 1: look for a fenced JSON block
|
||||
const fenceMatch = raw.match(/```json\s*([\s\S]*?)```/)
|
||||
const candidate = fenceMatch ? fenceMatch[1]! : raw
|
||||
|
||||
// Strategy 2: find the first '[' and last ']'
|
||||
const arrayStart = candidate.indexOf('[')
|
||||
const arrayEnd = candidate.lastIndexOf(']')
|
||||
if (arrayStart === -1 || arrayEnd === -1 || arrayEnd <= arrayStart) {
|
||||
return null
|
||||
}
|
||||
|
||||
const jsonSlice = candidate.slice(arrayStart, arrayEnd + 1)
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(jsonSlice)
|
||||
if (!Array.isArray(parsed)) return null
|
||||
|
||||
const specs: ParsedTaskSpec[] = []
|
||||
for (const item of parsed) {
|
||||
if (typeof item !== 'object' || item === null) continue
|
||||
const obj = item as Record<string, unknown>
|
||||
if (typeof obj['title'] !== 'string') continue
|
||||
if (typeof obj['description'] !== 'string') continue
|
||||
|
||||
specs.push({
|
||||
title: obj['title'],
|
||||
description: obj['description'],
|
||||
assignee: typeof obj['assignee'] === 'string' ? obj['assignee'] : undefined,
|
||||
dependsOn: Array.isArray(obj['dependsOn'])
|
||||
? (obj['dependsOn'] as unknown[]).filter((x): x is string => typeof x === 'string')
|
||||
: undefined,
|
||||
memoryScope: obj['memoryScope'] === 'all' ? 'all' : undefined,
|
||||
maxRetries: typeof obj['maxRetries'] === 'number' ? obj['maxRetries'] : undefined,
|
||||
retryDelayMs: typeof obj['retryDelayMs'] === 'number' ? obj['retryDelayMs'] : undefined,
|
||||
retryBackoff: typeof obj['retryBackoff'] === 'number' ? obj['retryBackoff'] : undefined,
|
||||
role: typeof obj['role'] === 'string' ? obj['role'] : undefined,
|
||||
priority: obj['priority'] === 'low' || obj['priority'] === 'normal' || obj['priority'] === 'high' || obj['priority'] === 'critical'
|
||||
? obj['priority']
|
||||
: undefined,
|
||||
requires: parseTaskRequirements(obj['requires']),
|
||||
verify: parseCoordinatorVerify(obj['verify']),
|
||||
})
|
||||
}
|
||||
|
||||
return specs.length > 0 ? specs : null
|
||||
const parsed = coordinatorTaskSpecsSchema.safeParse(extractJSON(raw))
|
||||
return parsed.success ? parsed.data as ParsedTaskSpec[] : null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
@@ -384,8 +458,7 @@ export function buildCoordinatorOutputFormatSection(hasVerifyJudges?: boolean):
|
||||
' 3. Avoid adding a dependency just because the information "would be useful" or matches general best practice; if the manifest gives no indication X consumes that input, prefer to leave it out.',
|
||||
' 4. When uncertain, prefer fewer dependencies over more — extra parents cost parallelism and tokens.',
|
||||
'',
|
||||
'Wrap the JSON in a ```json code fence.',
|
||||
'Do not include any text outside the code fence.',
|
||||
'Return only the JSON array. Do not use Markdown code fences or extra text.',
|
||||
)
|
||||
return lines.join('\n')
|
||||
}
|
||||
@@ -408,7 +481,7 @@ export function buildDecompositionPrompt(goal: string, agents: AgentConfig[]): s
|
||||
`## Goal`,
|
||||
goal,
|
||||
'',
|
||||
'Return ONLY the JSON task array in a ```json code fence.',
|
||||
'Return ONLY the JSON task array. Do not use a Markdown code fence.',
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
|
||||
@@ -50,7 +50,9 @@ import type {
|
||||
ConsensusResult,
|
||||
CoordinatorConfig,
|
||||
ExecutionRoutingConfig,
|
||||
ModelRoutingPolicy,
|
||||
PlanArtifact,
|
||||
PlanRevision,
|
||||
PlanTaskArtifact,
|
||||
OrchestratorConfig,
|
||||
OrchestratorEvent,
|
||||
@@ -84,16 +86,21 @@ import { Team } from '../team/team.js'
|
||||
import { TaskQueue } from '../task/queue.js'
|
||||
import { Checkpoint } from '../memory/checkpoint.js'
|
||||
import { InMemoryStore } from '../memory/store.js'
|
||||
import { createTask, validateTaskDependencies } from '../task/task.js'
|
||||
import { validateTaskDependencies } from '../task/task.js'
|
||||
import { validateTaskMetadata } from '../task/metadata.js'
|
||||
import { Scheduler } from './scheduler.js'
|
||||
import {
|
||||
CostBudgetExceededError,
|
||||
InvalidTaskRequirementsError,
|
||||
RoutingDeclarationRequiredError,
|
||||
RoutingProfilerFailedError,
|
||||
RoutingTimeoutError,
|
||||
TokenBudgetExceededError,
|
||||
} from '../errors.js'
|
||||
import {
|
||||
validateTaskRequirements,
|
||||
type AgentSelectorContext,
|
||||
} from './agent-selector.js'
|
||||
import {
|
||||
createRestoreIdentity,
|
||||
resolveRestoreMetadata,
|
||||
@@ -172,6 +179,7 @@ import {
|
||||
type ConsequentialConfirmationState,
|
||||
} from './consequential.js'
|
||||
import { runConsensusCore, applyConsensusDefaults, type ConsensusAgentDefaults } from './consensus.js'
|
||||
import { resolveRecoveryOptions } from './recovery.js'
|
||||
import {
|
||||
createOnlineEvaluator,
|
||||
NOOP_ONLINE_EVALUATION,
|
||||
@@ -181,12 +189,13 @@ import {
|
||||
} from '../eval/online.js'
|
||||
|
||||
import {
|
||||
parseTaskSpecs,
|
||||
buildCoordinatorBaseConfig,
|
||||
buildCoordinatorTaskSpecsSchema,
|
||||
buildDecompositionPrompt,
|
||||
runCoordinatorSynthesis,
|
||||
loadSpecsIntoQueue,
|
||||
findInvalidAssignees,
|
||||
type ParsedTaskSpec,
|
||||
} from './coordinator.js'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -253,8 +262,8 @@ function resolveRunBudgets(
|
||||
*/
|
||||
export class OpenMultiAgent {
|
||||
private readonly config: Required<
|
||||
Omit<OrchestratorConfig, 'onApproval' | 'onTaskDispatch' | 'onAgentStream' | 'onPlanReady' | 'onProgress' | 'onTrace' | 'onToolCall' | 'observability' | 'evaluation' | 'defaultBaseURL' | 'defaultApiKey' | 'maxTokenBudget' | 'maxCostBudget' | 'estimateCost' | 'defaultToolPreset' | 'checkpoint'>
|
||||
> & Pick<OrchestratorConfig, 'onApproval' | 'onTaskDispatch' | 'onAgentStream' | 'onPlanReady' | 'onProgress' | 'onTrace' | 'onToolCall' | 'observability' | 'evaluation' | 'defaultBaseURL' | 'defaultApiKey' | 'maxTokenBudget' | 'maxCostBudget' | 'estimateCost' | 'defaultToolPreset' | 'checkpoint'>
|
||||
Omit<OrchestratorConfig, 'onApproval' | 'onTaskDispatch' | 'onAgentStream' | 'onPlanReady' | 'onProgress' | 'onTrace' | 'onToolCall' | 'observability' | 'evaluation' | 'defaultBaseURL' | 'defaultApiKey' | 'maxTokenBudget' | 'maxCostBudget' | 'estimateCost' | 'defaultToolPreset' | 'checkpoint' | 'recovery'>
|
||||
> & Pick<OrchestratorConfig, 'onApproval' | 'onTaskDispatch' | 'onAgentStream' | 'onPlanReady' | 'onProgress' | 'onTrace' | 'onToolCall' | 'observability' | 'evaluation' | 'defaultBaseURL' | 'defaultApiKey' | 'maxTokenBudget' | 'maxCostBudget' | 'estimateCost' | 'defaultToolPreset' | 'checkpoint' | 'recovery'>
|
||||
|
||||
private readonly teams: Map<string, Team> = new Map()
|
||||
private readonly hasConfiguredCustomExecutionRouter: boolean
|
||||
@@ -274,7 +283,7 @@ export class OpenMultiAgent {
|
||||
* - `maxDelegationDepth`: 3
|
||||
* - `schedulingStrategy`: `'dependency-first'`
|
||||
* - `schedulingWeights`: `{ fit: 0.7, load: 0.3 }`
|
||||
* - `strictAssignees`: `false`
|
||||
* - `strictAssignees`: `true`
|
||||
* - `defaultModel`: `'claude-opus-4-6'`
|
||||
* - `defaultProvider`: `'anthropic'`
|
||||
*/
|
||||
@@ -307,7 +316,7 @@ export class OpenMultiAgent {
|
||||
maxDelegationDepth: config.maxDelegationDepth ?? DEFAULT_MAX_DELEGATION_DEPTH,
|
||||
schedulingStrategy: config.schedulingStrategy ?? 'dependency-first',
|
||||
schedulingWeights: config.schedulingWeights ?? {},
|
||||
strictAssignees: config.strictAssignees ?? false,
|
||||
strictAssignees: config.strictAssignees ?? true,
|
||||
executionRouter: config.executionRouter ?? new DeterministicRouter(),
|
||||
executionRouting: {
|
||||
strategy: config.executionRouting?.strategy ?? 'hybrid',
|
||||
@@ -339,6 +348,7 @@ export class OpenMultiAgent {
|
||||
estimateCost: config.estimateCost,
|
||||
defaultToolPreset: config.defaultToolPreset,
|
||||
checkpoint: config.checkpoint,
|
||||
recovery: config.recovery,
|
||||
onApproval: config.onApproval,
|
||||
onTaskDispatch: config.onTaskDispatch,
|
||||
onPlanReady: config.onPlanReady,
|
||||
@@ -352,14 +362,13 @@ export class OpenMultiAgent {
|
||||
}
|
||||
}
|
||||
|
||||
private createScheduler(): Scheduler {
|
||||
private createScheduler(
|
||||
modelRouting?: ModelRoutingPolicy,
|
||||
tasks: () => readonly Task[] = () => [],
|
||||
): Scheduler {
|
||||
return new Scheduler(
|
||||
this.config.schedulingStrategy,
|
||||
{
|
||||
defaultProvider: this.config.defaultProvider,
|
||||
defaultToolPreset: this.config.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
},
|
||||
this.agentSelectorContext(modelRouting, tasks),
|
||||
{
|
||||
weights: this.config.schedulingWeights,
|
||||
onWarning: (warning) => {
|
||||
@@ -373,6 +382,31 @@ export class OpenMultiAgent {
|
||||
)
|
||||
}
|
||||
|
||||
private agentSelectorContext(
|
||||
modelRouting?: ModelRoutingPolicy,
|
||||
tasks: () => readonly Task[] = () => [],
|
||||
): AgentSelectorContext {
|
||||
return {
|
||||
defaultProvider: this.config.defaultProvider,
|
||||
defaultToolPreset: this.config.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
...(modelRouting ? {
|
||||
resolveCandidate: (task: Task, candidate: AgentConfig): AgentConfig => {
|
||||
const base = applyDefaultToolPreset(
|
||||
applyAgentDefaults(candidate, this.config),
|
||||
this.config.defaultToolPreset,
|
||||
)
|
||||
return withModelRoute(base, routeMatches(modelRouting, {
|
||||
phase: 'worker',
|
||||
agent: candidate.name,
|
||||
task,
|
||||
leaf: isLeafTask(task, tasks()),
|
||||
}))
|
||||
},
|
||||
} : {}),
|
||||
}
|
||||
}
|
||||
|
||||
private resolveExecutionRoutingConfig(
|
||||
override?: ExecutionRoutingConfig,
|
||||
coordinator?: CoordinatorConfig,
|
||||
@@ -1303,7 +1337,10 @@ export class OpenMultiAgent {
|
||||
)
|
||||
|
||||
const decompositionPrompt = buildDecompositionPrompt(goal, agentConfigs)
|
||||
const coordinatorAgent = buildAgent(coordinatorConfig)
|
||||
const coordinatorAgent = buildAgent({
|
||||
...coordinatorConfig,
|
||||
outputSchema: buildCoordinatorTaskSpecsSchema(agentConfigs, this.config.strictAssignees),
|
||||
})
|
||||
const runId = identity.runId
|
||||
const coordinatorDecomposeSpanId = this.config.onTrace ? generateSpanId() : undefined
|
||||
|
||||
@@ -1363,10 +1400,35 @@ export class OpenMultiAgent {
|
||||
// ------------------------------------------------------------------
|
||||
// Step 2: Parse tasks from coordinator output
|
||||
// ------------------------------------------------------------------
|
||||
const taskSpecs = parseTaskSpecs(decompositionResult.output)
|
||||
if (!decompositionResult.success && decompositionResult.errorInfo?.kind !== 'validation') {
|
||||
this.config.onProgress?.({
|
||||
type: 'agent_complete',
|
||||
agent: 'coordinator',
|
||||
data: decompositionResult,
|
||||
})
|
||||
this.config.onProgress?.({
|
||||
type: 'error',
|
||||
data: {
|
||||
code: 'COORDINATOR_DECOMPOSITION_FAILED',
|
||||
error: decompositionResult.errorInfo,
|
||||
},
|
||||
})
|
||||
return finish(this.buildTeamRunResult(
|
||||
agentResults,
|
||||
identity,
|
||||
goal,
|
||||
[],
|
||||
decompositionResult.status ?? statusOnly('error'),
|
||||
decompositionResult.errorInfo,
|
||||
))
|
||||
}
|
||||
|
||||
const taskSpecs = decompositionResult.success && Array.isArray(decompositionResult.structured)
|
||||
? decompositionResult.structured as ParsedTaskSpec[]
|
||||
: null
|
||||
|
||||
const queue = new TaskQueue()
|
||||
const scheduler = this.createScheduler()
|
||||
const scheduler = this.createScheduler(options?.modelRouting, () => queue.list())
|
||||
const taskMetrics = new Map<string, TaskExecutionMetrics>()
|
||||
|
||||
if (taskSpecs && taskSpecs.length > 0) {
|
||||
@@ -1444,16 +1506,59 @@ export class OpenMultiAgent {
|
||||
))
|
||||
}
|
||||
} else {
|
||||
// Coordinator failed to produce structured output — fall back to
|
||||
// one task per agent using the goal as the description.
|
||||
for (const agentConfig of agentConfigs) {
|
||||
const task = createTask({
|
||||
title: `${agentConfig.name}: ${goal.slice(0, 80)}`,
|
||||
description: goal,
|
||||
assignee: agentConfig.name,
|
||||
})
|
||||
queue.add(task)
|
||||
}
|
||||
// A coordinator plan is an execution boundary. Do not turn an invalid
|
||||
// plan into a different topology: that could duplicate side effects.
|
||||
const error = Object.assign(
|
||||
new Error('Coordinator plan failed structured validation after repair.'),
|
||||
{ code: 'COORDINATOR_PLAN_INVALID' },
|
||||
)
|
||||
const classified = classifyRunFailure(error, { kind: 'validation' })
|
||||
this.config.onProgress?.({
|
||||
type: 'agent_complete',
|
||||
agent: 'coordinator',
|
||||
data: decompositionResult,
|
||||
})
|
||||
this.config.onProgress?.({
|
||||
type: 'error',
|
||||
data: {
|
||||
code: 'COORDINATOR_PLAN_INVALID',
|
||||
error: classified.errorInfo,
|
||||
},
|
||||
})
|
||||
return finish(this.buildTeamRunResult(
|
||||
agentResults,
|
||||
identity,
|
||||
goal,
|
||||
[],
|
||||
classified.status,
|
||||
classified.errorInfo,
|
||||
))
|
||||
}
|
||||
|
||||
const requirementIssues = validateTaskRequirements(
|
||||
queue.list(),
|
||||
agentConfigs,
|
||||
this.agentSelectorContext(options?.modelRouting, () => queue.list()),
|
||||
)
|
||||
if (requirementIssues.length > 0) {
|
||||
const error = new InvalidTaskRequirementsError(requirementIssues)
|
||||
const classified = classifyRunFailure(error, { kind: 'validation' })
|
||||
this.config.onProgress?.({
|
||||
type: 'error',
|
||||
data: {
|
||||
code: error.code,
|
||||
issues: requirementIssues,
|
||||
error: classified.errorInfo,
|
||||
},
|
||||
})
|
||||
return finish(this.buildTeamRunResult(
|
||||
agentResults,
|
||||
identity,
|
||||
goal,
|
||||
[],
|
||||
classified.status,
|
||||
classified.errorInfo,
|
||||
))
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
@@ -1510,6 +1615,8 @@ export class OpenMultiAgent {
|
||||
modelRouting: options?.modelRouting,
|
||||
taskById: new Map(queue.list().map((task) => [task.id, task])),
|
||||
taskLeafById: new Map(queue.list().map((task) => [task.id, isLeafTask(task, queue.list())])),
|
||||
recovery: resolveRecoveryOptions(this.config.recovery, options?.recovery),
|
||||
recoveryPatchSignatures: new Set(),
|
||||
}
|
||||
|
||||
const planTasks = queue.list()
|
||||
@@ -1612,6 +1719,8 @@ export class OpenMultiAgent {
|
||||
retryDelayMs: task.retryDelayMs,
|
||||
retryBackoff: task.retryBackoff,
|
||||
verify: task.verify,
|
||||
supersededByRevision: task.supersededByRevision,
|
||||
recoveredByRevision: task.recoveredByRevision,
|
||||
metrics: taskMetrics.get(task.id),
|
||||
}))
|
||||
|
||||
@@ -1640,7 +1749,14 @@ export class OpenMultiAgent {
|
||||
ctx.outcomeErrorInfo = classified.errorInfo
|
||||
}
|
||||
return finish(this.buildTeamRunResult(
|
||||
agentResults, identity, goal, taskRecords, ctx.outcomeStatus, ctx.outcomeErrorInfo,
|
||||
agentResults,
|
||||
identity,
|
||||
goal,
|
||||
taskRecords,
|
||||
ctx.outcomeStatus,
|
||||
ctx.outcomeErrorInfo,
|
||||
false,
|
||||
queue.getPlanRevisions(),
|
||||
))
|
||||
}
|
||||
agentResults.set('coordinator', synthesis.result)
|
||||
@@ -1652,7 +1768,14 @@ export class OpenMultiAgent {
|
||||
// buildTeamRunResult, so we do not increment completedTaskCount here.
|
||||
|
||||
return finish(this.buildTeamRunResult(
|
||||
agentResults, identity, goal, taskRecords, ctx.outcomeStatus, ctx.outcomeErrorInfo,
|
||||
agentResults,
|
||||
identity,
|
||||
goal,
|
||||
taskRecords,
|
||||
ctx.outcomeStatus,
|
||||
ctx.outcomeErrorInfo,
|
||||
false,
|
||||
queue.getPlanRevisions(),
|
||||
))
|
||||
}
|
||||
|
||||
@@ -1715,6 +1838,9 @@ export class OpenMultiAgent {
|
||||
options?: RunTasksOptions,
|
||||
): Promise<TeamRunResult> {
|
||||
const pendingEvaluation = this.beginOnlineEvaluation(plan)
|
||||
if (resolveRecoveryOptions(this.config.recovery, options?.recovery).mode !== 'fixed') {
|
||||
throw new Error('runFromPlan requires fixed recovery so the frozen plan remains exact.')
|
||||
}
|
||||
if (plan.version !== 1) {
|
||||
throw new Error(`Unsupported plan artifact version: ${String(plan.version)}`)
|
||||
}
|
||||
@@ -1726,6 +1852,12 @@ export class OpenMultiAgent {
|
||||
throw new Error(`Invalid plan artifact: ${validation.errors.join(' ')}`)
|
||||
}
|
||||
queue.addBatch(tasks)
|
||||
const activeCheckpoint = this.createActiveCheckpoint(
|
||||
team,
|
||||
options?.checkpoint ?? this.config.checkpoint,
|
||||
'runFromPlan',
|
||||
plan.goal,
|
||||
)
|
||||
|
||||
return this.executeExplicitTaskQueue(
|
||||
team,
|
||||
@@ -1733,7 +1865,7 @@ export class OpenMultiAgent {
|
||||
options,
|
||||
plan.goal,
|
||||
undefined,
|
||||
undefined,
|
||||
activeCheckpoint,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
@@ -1806,6 +1938,7 @@ export class OpenMultiAgent {
|
||||
role: t.role,
|
||||
priority: t.priority,
|
||||
metadata: t.metadata,
|
||||
requires: t.requires,
|
||||
verify: t.verify,
|
||||
})),
|
||||
team.getAgents(),
|
||||
@@ -1828,6 +1961,9 @@ export class OpenMultiAgent {
|
||||
)
|
||||
}
|
||||
if (this.isPlanArtifact(tasksOrOptions)) {
|
||||
if (resolveRecoveryOptions(this.config.recovery, options?.recovery).mode !== 'fixed') {
|
||||
throw new Error('restore from a plan artifact requires fixed recovery so the plan remains exact.')
|
||||
}
|
||||
const queue = new TaskQueue()
|
||||
const tasks = this.tasksFromPlan(tasksOrOptions)
|
||||
const validation = validateTaskDependencies(tasks)
|
||||
@@ -1870,6 +2006,13 @@ export class OpenMultiAgent {
|
||||
)
|
||||
}
|
||||
|
||||
if (
|
||||
snapshot.mode === 'runFromPlan'
|
||||
&& resolveRecoveryOptions(this.config.recovery, options?.recovery).mode !== 'fixed'
|
||||
) {
|
||||
throw new Error('restore of a runFromPlan checkpoint requires fixed recovery so the plan remains exact.')
|
||||
}
|
||||
|
||||
const sharedMem = team.getSharedMemoryInstance()
|
||||
if (sharedMem && snapshot.sharedMemory) {
|
||||
await sharedMem.restore(snapshot.sharedMemory)
|
||||
@@ -2250,6 +2393,25 @@ export class OpenMultiAgent {
|
||||
governanceDeclaration?: GovernanceDeclaration,
|
||||
routingDecisionInput?: RoutingDecisionRecordInput,
|
||||
): Promise<TeamRunResult> {
|
||||
const agentConfigs = team.getAgents()
|
||||
const requirementIssues = validateTaskRequirements(
|
||||
queue.list(),
|
||||
agentConfigs,
|
||||
this.agentSelectorContext(options?.modelRouting, () => queue.list()),
|
||||
)
|
||||
if (requirementIssues.length > 0) {
|
||||
const error = new InvalidTaskRequirementsError(requirementIssues)
|
||||
this.config.onProgress?.({
|
||||
type: 'error',
|
||||
data: {
|
||||
code: error.code,
|
||||
issues: requirementIssues,
|
||||
error: classifyRunFailure(error, { kind: 'validation' }).errorInfo,
|
||||
},
|
||||
})
|
||||
throw error
|
||||
}
|
||||
|
||||
const newRunFacts = identity === undefined
|
||||
? createRunFacts(identityOptionsForRun(options))
|
||||
: undefined
|
||||
@@ -2259,8 +2421,7 @@ export class OpenMultiAgent {
|
||||
const routingDecision = routingDecisionInput
|
||||
? recordRoutingDecision(runIdentity, traceRuntime, routingDecisionInput)
|
||||
: undefined
|
||||
const agentConfigs = team.getAgents()
|
||||
const scheduler = this.createScheduler()
|
||||
const scheduler = this.createScheduler(options?.modelRouting, () => queue.list())
|
||||
if (this.config.onApproval) {
|
||||
scheduler.autoAssign(queue, agentConfigs)
|
||||
}
|
||||
@@ -2298,6 +2459,8 @@ export class OpenMultiAgent {
|
||||
modelRouting: options?.modelRouting,
|
||||
taskById: new Map(queue.list().map((task) => [task.id, task])),
|
||||
taskLeafById: new Map(queue.list().map((task) => [task.id, isLeafTask(task, queue.list())])),
|
||||
recovery: resolveRecoveryOptions(this.config.recovery, options?.recovery),
|
||||
recoveryPatchSignatures: new Set(),
|
||||
}
|
||||
|
||||
await executeQueue(queue, ctx)
|
||||
@@ -2377,6 +2540,8 @@ export class OpenMultiAgent {
|
||||
retryDelayMs: task.retryDelayMs,
|
||||
retryBackoff: task.retryBackoff,
|
||||
verify: task.verify,
|
||||
supersededByRevision: task.supersededByRevision,
|
||||
recoveredByRevision: task.recoveredByRevision,
|
||||
metrics: ctx.taskMetrics.get(task.id),
|
||||
}))
|
||||
|
||||
@@ -2387,6 +2552,8 @@ export class OpenMultiAgent {
|
||||
taskRecords,
|
||||
ctx.outcomeStatus,
|
||||
ctx.outcomeErrorInfo,
|
||||
false,
|
||||
queue.getPlanRevisions(),
|
||||
)
|
||||
const resultWithRouting = {
|
||||
...result,
|
||||
@@ -2508,11 +2675,13 @@ export class OpenMultiAgent {
|
||||
forcedStatus?: RunStatus,
|
||||
forcedErrorInfo?: StructuredTraceError,
|
||||
allowIncompleteTasks = false,
|
||||
planRevisions?: readonly PlanRevision[],
|
||||
): TeamRunResult {
|
||||
let totalUsage: TokenUsage = ZERO_USAGE
|
||||
let overallSuccess = true
|
||||
const collapsed = new Map<string, AgentRunResult>()
|
||||
const taskResults = new Map<string, AgentRunResult>()
|
||||
const taskRecordsById = new Map((tasks ?? []).map((task) => [task.id, task]))
|
||||
|
||||
for (const task of tasks ?? []) {
|
||||
if (!task.assignee) continue
|
||||
@@ -2526,7 +2695,10 @@ export class OpenMultiAgent {
|
||||
const agentName = key.includes(':') ? key.split(':')[0]! : key
|
||||
|
||||
totalUsage = addUsage(totalUsage, result.tokenUsage)
|
||||
if (!result.success) overallSuccess = false
|
||||
const taskId = key.includes(':') ? key.slice(key.indexOf(':') + 1) : undefined
|
||||
const recovered = taskId !== undefined
|
||||
&& taskRecordsById.get(taskId)?.recoveredByRevision !== undefined
|
||||
if (!result.success && !recovered) overallSuccess = false
|
||||
|
||||
const existing = collapsed.get(agentName)
|
||||
if (!existing) {
|
||||
@@ -2560,11 +2732,17 @@ export class OpenMultiAgent {
|
||||
|
||||
const metrics = computeRunMetrics(tasks)
|
||||
|
||||
const statuses = [...agentResults.values()]
|
||||
.map((result) => result.status)
|
||||
const statuses = [...agentResults.entries()]
|
||||
.filter(([key]) => {
|
||||
const taskId = key.includes(':') ? key.slice(key.indexOf(':') + 1) : undefined
|
||||
return taskId === undefined
|
||||
|| taskRecordsById.get(taskId)?.recoveredByRevision === undefined
|
||||
})
|
||||
.map(([, result]) => result.status)
|
||||
.filter((status): status is RunStatus => status !== undefined)
|
||||
const firstStatus = (code: RunStatus['code']) => statuses.find((status) => status.code === code)
|
||||
const taskFailed = tasks?.some((task) => task.status === 'failed') ?? false
|
||||
const taskFailed = tasks?.some((task) =>
|
||||
task.status === 'failed' && task.recoveredByRevision === undefined) ?? false
|
||||
const taskIncomplete = tasks?.some((task) =>
|
||||
task.status === 'pending' || task.status === 'in_progress' || task.status === 'blocked'
|
||||
) ?? false
|
||||
@@ -2587,6 +2765,7 @@ export class OpenMultiAgent {
|
||||
...(errorInfo !== undefined ? { errorInfo } : {}),
|
||||
goal,
|
||||
tasks,
|
||||
...(planRevisions && planRevisions.length > 0 ? { planRevisions } : {}),
|
||||
agentResults: collapsed,
|
||||
taskResults,
|
||||
totalTokenUsage: totalUsage,
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
/**
|
||||
* @fileoverview Opt-in runtime plan repair contracts and validation helpers.
|
||||
*
|
||||
* The executor owns the outcome barrier and durable commit ordering. This
|
||||
* module keeps policy resolution and eligibility checks deterministic.
|
||||
*/
|
||||
|
||||
import type {
|
||||
AgentConfig,
|
||||
PlanPatch,
|
||||
RecoveryOptions,
|
||||
Task,
|
||||
TaskOutcome,
|
||||
TaskVerificationOutcome,
|
||||
AgentRunResult,
|
||||
} from '../types.js'
|
||||
import type { TaskQueue } from '../task/queue.js'
|
||||
import { AgentSelector } from './agent-selector.js'
|
||||
|
||||
export interface ResolvedRecoveryOptions {
|
||||
readonly mode: 'fixed' | 'repairable'
|
||||
readonly onTaskOutcome?: RecoveryOptions['onTaskOutcome']
|
||||
readonly onPlanPatch?: RecoveryOptions['onPlanPatch']
|
||||
readonly maxPlanRevisions: number
|
||||
readonly maxAddedTasks: number
|
||||
}
|
||||
|
||||
const DEFAULT_MAX_PLAN_REVISIONS = 3
|
||||
const DEFAULT_MAX_ADDED_TASKS = 20
|
||||
|
||||
export function resolveRecoveryOptions(
|
||||
configured: RecoveryOptions | undefined,
|
||||
perRun: RecoveryOptions | undefined,
|
||||
): ResolvedRecoveryOptions {
|
||||
const selected = perRun ?? configured
|
||||
if (!selected || (selected.mode ?? 'fixed') === 'fixed') {
|
||||
return {
|
||||
mode: 'fixed',
|
||||
maxPlanRevisions: DEFAULT_MAX_PLAN_REVISIONS,
|
||||
maxAddedTasks: DEFAULT_MAX_ADDED_TASKS,
|
||||
}
|
||||
}
|
||||
if (selected.onTaskOutcome && selected.replanner) {
|
||||
throw new Error("Recovery mode 'repairable' accepts either onTaskOutcome or replanner, not both.")
|
||||
}
|
||||
const onTaskOutcome = selected.onTaskOutcome
|
||||
?? (selected.replanner
|
||||
? (outcome: TaskOutcome) => selected.replanner!.replan(outcome)
|
||||
: undefined)
|
||||
if (!onTaskOutcome) {
|
||||
throw new Error("Recovery mode 'repairable' requires onTaskOutcome or replanner.")
|
||||
}
|
||||
return {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome,
|
||||
onPlanPatch: selected.onPlanPatch,
|
||||
maxPlanRevisions: positiveInteger(
|
||||
selected.maxPlanRevisions,
|
||||
DEFAULT_MAX_PLAN_REVISIONS,
|
||||
'maxPlanRevisions',
|
||||
),
|
||||
maxAddedTasks: positiveInteger(
|
||||
selected.maxAddedTasks,
|
||||
DEFAULT_MAX_ADDED_TASKS,
|
||||
'maxAddedTasks',
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
export function buildTaskOutcome(input: {
|
||||
readonly queue: TaskQueue
|
||||
readonly task: Task
|
||||
readonly result: AgentRunResult
|
||||
readonly verification?: TaskVerificationOutcome
|
||||
readonly tokenBudgetRemaining?: number
|
||||
readonly costBudgetRemaining?: number
|
||||
}): TaskOutcome {
|
||||
const kind = input.verification?.verdict === 'rejected'
|
||||
? 'verification_rejected'
|
||||
: input.result.success
|
||||
? 'success'
|
||||
: 'failure'
|
||||
return {
|
||||
kind,
|
||||
task: cloneTask(input.task),
|
||||
result: cloneAgentResult(input.result),
|
||||
...(input.verification ? { verification: { ...input.verification } } : {}),
|
||||
planRevision: input.queue.getPlanRevision(),
|
||||
tasks: input.queue.list().map(cloneTask),
|
||||
...(input.tokenBudgetRemaining !== undefined
|
||||
? { tokenBudgetRemaining: input.tokenBudgetRemaining }
|
||||
: {}),
|
||||
...(input.costBudgetRemaining !== undefined
|
||||
? { costBudgetRemaining: input.costBudgetRemaining }
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
export function validatePlanPatchEligibility(
|
||||
patch: PlanPatch,
|
||||
queue: TaskQueue,
|
||||
agents: readonly AgentConfig[],
|
||||
context: {
|
||||
readonly defaultProvider?: AgentConfig['provider']
|
||||
readonly defaultToolPreset?: 'readonly' | 'readwrite' | 'full'
|
||||
},
|
||||
): void {
|
||||
const byName = new Map(agents.map((agent) => [agent.name, agent]))
|
||||
const selector = new AgentSelector()
|
||||
|
||||
for (const retarget of patch.retargetPending ?? []) {
|
||||
const task = queue.get(retarget.taskId)
|
||||
const agent = byName.get(retarget.assignee)
|
||||
if (!task) throw new Error(`Plan patch retarget task "${retarget.taskId}" not found.`)
|
||||
if (!agent) throw new Error(`Plan patch retarget agent "${retarget.assignee}" is not in the team roster.`)
|
||||
const selection = selector.select(task, [agent], {
|
||||
defaultProvider: context.defaultProvider,
|
||||
defaultToolPreset: context.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
})
|
||||
if (selection.error) {
|
||||
throw new Error(
|
||||
`Plan patch retarget agent "${retarget.assignee}" is ineligible for task ` +
|
||||
`"${task.id}": ${selection.error.reasons.join(' ')}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
for (const task of patch.addTasks ?? []) {
|
||||
const candidates = task.assignee === undefined
|
||||
? agents
|
||||
: byName.has(task.assignee)
|
||||
? [byName.get(task.assignee)!]
|
||||
: []
|
||||
if (task.assignee !== undefined && candidates.length === 0) {
|
||||
throw new Error(`Plan patch task "${task.key}" names unknown assignee "${task.assignee}".`)
|
||||
}
|
||||
const selection = selector.select(task, candidates, {
|
||||
defaultProvider: context.defaultProvider,
|
||||
defaultToolPreset: context.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
})
|
||||
if (selection.error) {
|
||||
throw new Error(
|
||||
`No eligible agent for plan patch task "${task.key}": ${selection.error.reasons.join(' ')}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function countAddedTasks(revisions: readonly { readonly addedTasks: Readonly<Record<string, string>> }[]): number {
|
||||
return revisions.reduce((sum, revision) => sum + Object.keys(revision.addedTasks).length, 0)
|
||||
}
|
||||
|
||||
export function planPatchSignature(patch: PlanPatch): string {
|
||||
return JSON.stringify({
|
||||
reason: patch.reason,
|
||||
addTasks: patch.addTasks ?? [],
|
||||
retargetPending: patch.retargetPending ?? [],
|
||||
supersedePending: patch.supersedePending ?? [],
|
||||
})
|
||||
}
|
||||
|
||||
function positiveInteger(value: number | undefined, fallback: number, name: string): number {
|
||||
if (value === undefined) return fallback
|
||||
if (!Number.isInteger(value) || value < 1) {
|
||||
throw new Error(`Recovery ${name} must be a positive integer.`)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
function cloneTask(task: Task): Task {
|
||||
return {
|
||||
...task,
|
||||
...(task.dependsOn !== undefined ? { dependsOn: [...task.dependsOn] } : {}),
|
||||
...(task.metadata !== undefined ? { metadata: { ...task.metadata } } : {}),
|
||||
...(task.requires !== undefined
|
||||
? {
|
||||
requires: {
|
||||
...task.requires,
|
||||
...(task.requires.requiredTools !== undefined
|
||||
? { requiredTools: [...task.requires.requiredTools] }
|
||||
: {}),
|
||||
...(task.requires.requiredCapabilities !== undefined
|
||||
? { requiredCapabilities: [...task.requires.requiredCapabilities] }
|
||||
: {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
...(task.verify !== undefined
|
||||
? { verify: { ...task.verify, judges: [...task.verify.judges] } }
|
||||
: {}),
|
||||
createdAt: new Date(task.createdAt),
|
||||
updatedAt: new Date(task.updatedAt),
|
||||
}
|
||||
}
|
||||
|
||||
function cloneAgentResult(result: AgentRunResult): AgentRunResult {
|
||||
return {
|
||||
...result,
|
||||
messages: [...result.messages],
|
||||
tokenUsage: { ...result.tokenUsage },
|
||||
toolCalls: result.toolCalls.map((call) => ({ ...call })),
|
||||
...(result.errorInfo !== undefined ? { errorInfo: { ...result.errorInfo } } : {}),
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ import type { Team } from '../team/team.js'
|
||||
import type { Scheduler } from './scheduler.js'
|
||||
import type { Checkpoint } from '../memory/checkpoint.js'
|
||||
import type { TraceRuntime, TraceSpan } from '../observability/runtime.js'
|
||||
import type { ResolvedRecoveryOptions } from './recovery.js'
|
||||
import { createRunIdentity, validateRunMetadata } from '../observability/identity.js'
|
||||
|
||||
export const ZERO_USAGE: TokenUsage = { input_tokens: 0, output_tokens: 0 }
|
||||
@@ -166,8 +167,10 @@ export interface RunContext {
|
||||
*/
|
||||
readonly revealCoordinatorContext?: RevealCoordinatorContext
|
||||
readonly modelRouting?: ModelRoutingPolicy
|
||||
readonly taskById: ReadonlyMap<string, Task>
|
||||
readonly taskLeafById: ReadonlyMap<string, boolean>
|
||||
readonly taskById: Map<string, Task>
|
||||
readonly taskLeafById: Map<string, boolean>
|
||||
readonly recovery: ResolvedRecoveryOptions
|
||||
readonly recoveryPatchSignatures: Set<string>
|
||||
}
|
||||
|
||||
export interface ActiveCheckpoint {
|
||||
|
||||
@@ -19,6 +19,7 @@ import type { AgentConfig, Task } from '../types.js'
|
||||
import type { TaskQueue } from '../task/queue.js'
|
||||
import {
|
||||
AgentSelector,
|
||||
hasExplicitTaskRequirements,
|
||||
type AgentSelectorContext,
|
||||
} from './agent-selector.js'
|
||||
|
||||
@@ -61,6 +62,7 @@ export const DEFAULT_SCHEDULING_WEIGHTS: SchedulingWeights = {
|
||||
load: 0.3,
|
||||
}
|
||||
|
||||
/** @deprecated Hard requirement failures are now fail-closed errors. */
|
||||
export interface SchedulerWarning {
|
||||
/** Stable machine-readable warning code inherited from AgentSelector. */
|
||||
readonly code: 'NO_ELIGIBLE_AGENT'
|
||||
@@ -75,7 +77,7 @@ export interface SchedulerWarning {
|
||||
export interface SchedulerOptions {
|
||||
/** Per-field composite overrides; omitted fields use the documented defaults. */
|
||||
readonly weights?: Partial<SchedulingWeights>
|
||||
/** Receives structured scheduler degradations without changing assignments. */
|
||||
/** @deprecated Hard requirement failures are no longer downgraded to warnings. */
|
||||
readonly onWarning?: (warning: SchedulerWarning) => void
|
||||
}
|
||||
|
||||
@@ -187,11 +189,15 @@ export class Scheduler {
|
||||
* @returns A `Map<taskId, agentName>` for every unassigned pending task.
|
||||
*/
|
||||
schedule(tasks: Task[], agents: AgentConfig[]): Map<string, string> {
|
||||
if (agents.length === 0) return new Map()
|
||||
|
||||
const unassigned = tasks.filter(
|
||||
(t) => t.status === 'pending' && !t.assignee,
|
||||
)
|
||||
if (agents.length === 0) {
|
||||
const constrained = unassigned.find((task) =>
|
||||
hasExplicitTaskRequirements(task.requires))
|
||||
if (constrained) this.eligibleAgents(constrained, agents)
|
||||
return new Map()
|
||||
}
|
||||
|
||||
switch (this.strategy) {
|
||||
case 'round-robin':
|
||||
@@ -219,7 +225,13 @@ export class Scheduler {
|
||||
agents: AgentConfig[],
|
||||
allTasks: Task[],
|
||||
): string | undefined {
|
||||
if (agents.length === 0 || task.status !== 'pending' || task.assignee) {
|
||||
if (task.status !== 'pending' || task.assignee) {
|
||||
return undefined
|
||||
}
|
||||
if (agents.length === 0) {
|
||||
if (hasExplicitTaskRequirements(task.requires)) {
|
||||
this.eligibleAgents(task, agents)
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
@@ -296,9 +308,10 @@ export class Scheduler {
|
||||
): Map<string, string> {
|
||||
const result = new Map<string, string>()
|
||||
for (const task of unassigned) {
|
||||
const agent = agents[this.roundRobinCursor % agents.length]!
|
||||
const eligible = this.eligibleAgents(task, agents)
|
||||
const agent = eligible[this.roundRobinCursor % eligible.length]!
|
||||
result.set(task.id, agent.name)
|
||||
this.roundRobinCursor = (this.roundRobinCursor + 1) % agents.length
|
||||
this.roundRobinCursor = (this.roundRobinCursor + 1) % eligible.length
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -326,12 +339,13 @@ export class Scheduler {
|
||||
|
||||
const result = new Map<string, string>()
|
||||
for (const task of unassigned) {
|
||||
const eligible = this.eligibleAgents(task, agents)
|
||||
// Pick the agent with the lowest current load.
|
||||
let bestAgent = agents[0]!
|
||||
let bestAgent = eligible[0]!
|
||||
let bestLoad = load.get(bestAgent.name) ?? 0
|
||||
|
||||
for (let i = 1; i < agents.length; i++) {
|
||||
const agent = agents[i]!
|
||||
for (let i = 1; i < eligible.length; i++) {
|
||||
const agent = eligible[i]!
|
||||
const agentLoad = load.get(agent.name) ?? 0
|
||||
if (agentLoad < bestLoad) {
|
||||
bestLoad = agentLoad
|
||||
@@ -365,16 +379,14 @@ export class Scheduler {
|
||||
const selector = new AgentSelector()
|
||||
|
||||
for (const task of unassigned) {
|
||||
const selection = selector.select({
|
||||
title: task.title,
|
||||
description: task.description,
|
||||
requires: task.requires,
|
||||
}, agents, this.selectorContext)
|
||||
const selection = this.selectTaskAgents(task, agents, selector)
|
||||
if (selection.error) {
|
||||
throw new Error(
|
||||
`Scheduler capability-match: ${selection.error.code}: ${selection.error.reasons.join(' ')}`,
|
||||
)
|
||||
}
|
||||
const eligibleAgents = agents.filter((agent) =>
|
||||
selection.eligible.some((entry) => entry.agent === agent))
|
||||
|
||||
// Preserve the legacy scheduler's roster-order tie-break while reusing
|
||||
// the selector's eligibility and scoring kernel. The public selector and
|
||||
@@ -390,8 +402,8 @@ export class Scheduler {
|
||||
}
|
||||
|
||||
if (bestScore === 0) {
|
||||
bestAgent = agents[this.roundRobinCursor % agents.length]!
|
||||
this.roundRobinCursor = (this.roundRobinCursor + 1) % agents.length
|
||||
bestAgent = eligibleAgents[this.roundRobinCursor % eligibleAgents.length]!
|
||||
this.roundRobinCursor = (this.roundRobinCursor + 1) % eligibleAgents.length
|
||||
}
|
||||
|
||||
result.set(task.id, bestAgent.name)
|
||||
@@ -425,9 +437,10 @@ export class Scheduler {
|
||||
let cursor = this.roundRobinCursor
|
||||
|
||||
for (const task of ranked) {
|
||||
const agent = agents[cursor % agents.length]!
|
||||
const eligible = this.eligibleAgents(task, agents)
|
||||
const agent = eligible[cursor % eligible.length]!
|
||||
result.set(task.id, agent.name)
|
||||
cursor = (cursor + 1) % agents.length
|
||||
cursor = (cursor + 1) % eligible.length
|
||||
}
|
||||
|
||||
// Advance the shared cursor for consistency with round-robin.
|
||||
@@ -444,10 +457,8 @@ export class Scheduler {
|
||||
* Assignments made earlier in this call are deliberately not folded into
|
||||
* load, keeping the decision compatible with future one-ready-task calls.
|
||||
*
|
||||
* When hard filtering leaves no eligible agent, the selector's structured
|
||||
* failure is emitted as a warning and the task follows an explicit zero-fit
|
||||
* fallback across the full roster, using current load and ascending agent
|
||||
* name as the deterministic tie-break.
|
||||
* Hard filtering is fail-closed. No load or fit fallback may reintroduce an
|
||||
* ineligible agent.
|
||||
*/
|
||||
private scheduleComposite(
|
||||
unassigned: Task[],
|
||||
@@ -469,27 +480,12 @@ export class Scheduler {
|
||||
const result = new Map<string, string>()
|
||||
|
||||
for (const task of ranked) {
|
||||
const selection = selector.select({
|
||||
title: task.title,
|
||||
description: task.description,
|
||||
requires: task.requires,
|
||||
}, agents, this.selectorContext)
|
||||
const candidates = selection.error
|
||||
? agents.map((agent) => ({ agent, score: 0 }))
|
||||
: selection.eligible
|
||||
|
||||
const selection = this.selectTaskAgents(task, agents, selector)
|
||||
if (selection.error) {
|
||||
this.options.onWarning?.({
|
||||
code: selection.error.code,
|
||||
message: selection.error.message,
|
||||
taskId: task.id,
|
||||
taskTitle: task.title,
|
||||
reasons: selection.error.reasons,
|
||||
fallback: 'zero-fit-current-load',
|
||||
})
|
||||
throw this.noEligibleAgentError(task, selection.error.reasons)
|
||||
}
|
||||
|
||||
const rankedCandidates = candidates.map((candidate) => {
|
||||
const rankedCandidates = selection.eligible.map((candidate) => {
|
||||
const normalizedLoad = (loads.get(candidate.agent.name) ?? 0) / maxLoad
|
||||
return {
|
||||
agent: candidate.agent,
|
||||
@@ -511,6 +507,55 @@ export class Scheduler {
|
||||
return result
|
||||
}
|
||||
|
||||
private eligibleAgents(task: Task, agents: AgentConfig[]): AgentConfig[] {
|
||||
const selection = this.selectTaskAgents(task, agents)
|
||||
if (selection.error) {
|
||||
throw this.noEligibleAgentError(task, selection.error.reasons)
|
||||
}
|
||||
const eligible = new Set(selection.eligible.map((entry) => entry.agent))
|
||||
return agents.filter((agent) => eligible.has(agent))
|
||||
}
|
||||
|
||||
private selectTaskAgents(
|
||||
task: Task,
|
||||
agents: AgentConfig[],
|
||||
selector = new AgentSelector(),
|
||||
): ReturnType<AgentSelector['select']> {
|
||||
const effectiveAgents = agents.map((agent) =>
|
||||
this.selectorContext.resolveCandidate?.(task, agent) ?? agent)
|
||||
const originalByEffective = new Map(
|
||||
effectiveAgents.map((effective, index) => [effective, agents[index]!] as const),
|
||||
)
|
||||
const selection = selector.select({
|
||||
title: task.title,
|
||||
description: task.description,
|
||||
requires: task.requires,
|
||||
}, effectiveAgents, this.selectorContext)
|
||||
const remap = (agent: AgentConfig | undefined): AgentConfig | undefined =>
|
||||
agent === undefined ? undefined : originalByEffective.get(agent)
|
||||
return {
|
||||
...selection,
|
||||
agent: remap(selection.agent),
|
||||
eligible: selection.eligible.map((entry) => ({
|
||||
...entry,
|
||||
agent: originalByEffective.get(entry.agent)!,
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
private noEligibleAgentError(task: Task, reasons: readonly string[]): Error {
|
||||
return Object.assign(
|
||||
new Error(
|
||||
`Scheduler ${this.strategy}: NO_ELIGIBLE_AGENT for task "${task.title}": ${reasons.join(' ')}`,
|
||||
),
|
||||
{
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
taskId: task.id,
|
||||
reasons,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
private resolvedWeights(): SchedulingWeights {
|
||||
return {
|
||||
fit: this.options.weights?.fit ?? DEFAULT_SCHEDULING_WEIGHTS.fit,
|
||||
|
||||
@@ -18,6 +18,7 @@ import type {
|
||||
Task,
|
||||
TaskExecutionMetrics,
|
||||
TaskStatus,
|
||||
TaskVerificationOutcome,
|
||||
TeamInfo,
|
||||
TokenUsage,
|
||||
} from '../types.js'
|
||||
@@ -38,6 +39,8 @@ import {
|
||||
type RunContext,
|
||||
type RevealCoordinatorContext,
|
||||
} from './run-context.js'
|
||||
import { InvalidTaskRequirementsError } from '../errors.js'
|
||||
import { validateTaskRequirements } from './agent-selector.js'
|
||||
import { recordRunUsage, buildCostEstimateContext } from './budget.js'
|
||||
import {
|
||||
routeMatches,
|
||||
@@ -46,9 +49,16 @@ import {
|
||||
applyAgentDefaults,
|
||||
applyDefaultToolPreset,
|
||||
buildAgent,
|
||||
isLeafTask,
|
||||
} from './agent-config.js'
|
||||
import { executeWithRetry } from './retry.js'
|
||||
import { runTaskVerify } from './consensus.js'
|
||||
import {
|
||||
buildTaskOutcome,
|
||||
countAddedTasks,
|
||||
planPatchSignature,
|
||||
validatePlanPatchEligibility,
|
||||
} from './recovery.js'
|
||||
|
||||
/**
|
||||
* Build {@link TeamInfo} for tool context, including nested `runDelegatedAgent`
|
||||
@@ -164,9 +174,9 @@ export function buildTaskAgentTeamInfo(
|
||||
}
|
||||
}
|
||||
|
||||
export async function saveRunCheckpoint(queue: TaskQueue, ctx: RunContext): Promise<void> {
|
||||
export async function saveRunCheckpoint(queue: TaskQueue, ctx: RunContext): Promise<boolean> {
|
||||
const active = ctx.checkpoint
|
||||
if (!active) return
|
||||
if (!active) return true
|
||||
const checkpointSpan = ctx.traceRuntime?.startSpan({
|
||||
kind: 'checkpoint',
|
||||
name: 'save_checkpoint',
|
||||
@@ -232,6 +242,7 @@ export async function saveRunCheckpoint(queue: TaskQueue, ctx: RunContext): Prom
|
||||
try {
|
||||
await nextSave
|
||||
checkpointSpan?.end({ status: statusOnly('ok') })
|
||||
return true
|
||||
} catch (error) {
|
||||
const classified = classifyRunFailure(error, { kind: 'store' })
|
||||
checkpointSpan?.event('checkpoint_failed', {})
|
||||
@@ -240,6 +251,7 @@ export async function saveRunCheckpoint(queue: TaskQueue, ctx: RunContext): Prom
|
||||
type: 'error',
|
||||
data: { kind: 'checkpoint_save_failed', error },
|
||||
} satisfies OrchestratorEvent)
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
@@ -273,6 +285,252 @@ function evaluateDispatchGate(
|
||||
return 'allow'
|
||||
}
|
||||
|
||||
async function applyRecoveryAtOutcome(
|
||||
queue: TaskQueue,
|
||||
task: Task,
|
||||
result: AgentRunResult,
|
||||
verification: TaskVerificationOutcome | undefined,
|
||||
ctx: RunContext,
|
||||
): Promise<boolean> {
|
||||
const recovery = ctx.recovery
|
||||
if (recovery.mode === 'fixed' || !recovery.onTaskOutcome) return false
|
||||
|
||||
const trigger = verification?.verdict === 'rejected'
|
||||
? 'verification_rejected' as const
|
||||
: result.success
|
||||
? 'success' as const
|
||||
: 'failure' as const
|
||||
|
||||
// A patch may have been durably committed immediately before a crash while
|
||||
// this trigger task was still in_progress. Restore resets the task, so do not
|
||||
// append the same repair a second time when it settles again.
|
||||
if (queue.getPlanRevisions().some((revision) =>
|
||||
revision.triggerTaskId === task.id && revision.trigger === trigger)) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'recovery_decision',
|
||||
task: task.id,
|
||||
data: { decision: 'already_applied', trigger, planRevision: queue.getPlanRevision() },
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
const tokenUsed = ctx.cumulativeUsage.input_tokens + ctx.cumulativeUsage.output_tokens
|
||||
const outcome = buildTaskOutcome({
|
||||
queue,
|
||||
task: queue.get(task.id) ?? task,
|
||||
result,
|
||||
verification,
|
||||
...(ctx.maxTokenBudget !== undefined
|
||||
? { tokenBudgetRemaining: Math.max(0, ctx.maxTokenBudget - tokenUsed) }
|
||||
: {}),
|
||||
...(ctx.maxCostBudget !== undefined
|
||||
? { costBudgetRemaining: Math.max(0, ctx.maxCostBudget - ctx.cumulativeCost) }
|
||||
: {}),
|
||||
})
|
||||
|
||||
const policySpan = ctx.traceRuntime?.startSpan({
|
||||
kind: 'callback',
|
||||
name: 'recovery_policy_callback',
|
||||
parent: ctx.taskSpans.get(task.id) ?? ctx.traceRuntime?.root,
|
||||
attributes: {
|
||||
'oma.callback.name': 'onTaskOutcome',
|
||||
'oma.task.id': task.id,
|
||||
'oma.plan.trigger': trigger,
|
||||
},
|
||||
})
|
||||
let patch
|
||||
try {
|
||||
patch = await recovery.onTaskOutcome(outcome)
|
||||
} catch (error) {
|
||||
const classified = classifyRunFailure(error, { kind: 'callback' })
|
||||
policySpan?.end({ status: classified.status, error: classified.errorInfo })
|
||||
ctx.config.onProgress?.({
|
||||
type: 'error',
|
||||
task: task.id,
|
||||
data: { kind: 'recovery_policy_failed', error },
|
||||
})
|
||||
return false
|
||||
}
|
||||
policySpan?.event('recovery_decision', { 'oma.recovery.proposed': patch !== undefined })
|
||||
policySpan?.end({ status: statusOnly('ok') })
|
||||
if (!patch) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'recovery_decision',
|
||||
task: task.id,
|
||||
data: { decision: 'no_patch', trigger, planRevision: queue.getPlanRevision() },
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
if (queue.getPlanRevision() >= recovery.maxPlanRevisions) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: {
|
||||
code: 'RECOVERY_REVISION_LIMIT',
|
||||
maxPlanRevisions: recovery.maxPlanRevisions,
|
||||
},
|
||||
})
|
||||
return false
|
||||
}
|
||||
const addedSoFar = countAddedTasks(queue.getPlanRevisions())
|
||||
const requestedAdded = patch.addTasks?.length ?? 0
|
||||
if (addedSoFar + requestedAdded > recovery.maxAddedTasks) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: {
|
||||
code: 'RECOVERY_ADDED_TASK_LIMIT',
|
||||
maxAddedTasks: recovery.maxAddedTasks,
|
||||
requestedAdded,
|
||||
addedSoFar,
|
||||
},
|
||||
})
|
||||
return false
|
||||
}
|
||||
const signature = planPatchSignature(patch)
|
||||
if (ctx.recoveryPatchSignatures.has(signature)) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: { code: 'RECOVERY_PATCH_REPEATED' },
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
try {
|
||||
validatePlanPatchEligibility(patch, queue, ctx.team.getAgents(), {
|
||||
defaultProvider: ctx.config.defaultProvider,
|
||||
defaultToolPreset: ctx.config.defaultToolPreset,
|
||||
})
|
||||
} catch (error) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: { code: 'RECOVERY_PATCH_INELIGIBLE', error },
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
if (recovery.onPlanPatch) {
|
||||
const approvalSpan = ctx.traceRuntime?.startSpan({
|
||||
kind: 'callback',
|
||||
name: 'plan_patch_approval_callback',
|
||||
parent: ctx.taskSpans.get(task.id) ?? ctx.traceRuntime?.root,
|
||||
attributes: {
|
||||
'oma.callback.name': 'onPlanPatch',
|
||||
'oma.task.id': task.id,
|
||||
'oma.plan.trigger': trigger,
|
||||
},
|
||||
})
|
||||
let approved = false
|
||||
try {
|
||||
approved = await recovery.onPlanPatch(patch, outcome)
|
||||
} catch (error) {
|
||||
const classified = classifyRunFailure(error, { kind: 'callback' })
|
||||
approvalSpan?.end({ status: classified.status, error: classified.errorInfo })
|
||||
ctx.config.onProgress?.({
|
||||
type: 'error',
|
||||
task: task.id,
|
||||
data: { kind: 'recovery_approval_failed', error },
|
||||
})
|
||||
return false
|
||||
}
|
||||
approvalSpan?.event('approval_decision', { 'oma.approval.approved': approved })
|
||||
approvalSpan?.end({ status: statusOnly(approved ? 'ok' : 'rejected') })
|
||||
if (!approved) {
|
||||
ctx.config.onProgress?.({
|
||||
type: 'recovery_decision',
|
||||
task: task.id,
|
||||
data: { decision: 'rejected', trigger, planRevision: queue.getPlanRevision() },
|
||||
})
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
const revisionSpan = ctx.traceRuntime?.startSpan({
|
||||
kind: 'plan',
|
||||
name: 'apply_plan_revision',
|
||||
parent: ctx.taskSpans.get(task.id) ?? ctx.traceRuntime.root,
|
||||
attributes: {
|
||||
'oma.plan.revision.previous': queue.getPlanRevision(),
|
||||
'oma.plan.trigger': trigger,
|
||||
'oma.task.id': task.id,
|
||||
},
|
||||
})
|
||||
|
||||
let applied
|
||||
try {
|
||||
applied = queue.applyPlanPatch(patch, task.id, trigger)
|
||||
} catch (error) {
|
||||
const classified = classifyRunFailure(error, { kind: 'validation' })
|
||||
revisionSpan?.end({ status: classified.status, error: classified.errorInfo })
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: { code: 'RECOVERY_PATCH_INVALID', error },
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
const durable = await saveRunCheckpoint(queue, ctx)
|
||||
if (!durable && ctx.checkpoint) {
|
||||
queue.restorePlanSnapshot(applied.before)
|
||||
revisionSpan?.end({
|
||||
status: statusOnly('error', 'Plan revision checkpoint failed.'),
|
||||
error: {
|
||||
kind: 'store',
|
||||
code: 'PLAN_REVISION_NOT_DURABLE',
|
||||
message: 'Plan revision was rolled back because its checkpoint failed.',
|
||||
},
|
||||
})
|
||||
ctx.config.onProgress?.({
|
||||
type: 'warning',
|
||||
task: task.id,
|
||||
data: {
|
||||
code: 'PLAN_REVISION_NOT_DURABLE',
|
||||
revision: applied.revision.version,
|
||||
},
|
||||
})
|
||||
return false
|
||||
}
|
||||
|
||||
ctx.recoveryPatchSignatures.add(signature)
|
||||
queue.publishPlanRevision(applied.revision)
|
||||
const tasks = queue.list()
|
||||
ctx.taskById.clear()
|
||||
ctx.taskLeafById.clear()
|
||||
for (const candidate of tasks) {
|
||||
ctx.taskById.set(candidate.id, candidate)
|
||||
ctx.taskLeafById.set(candidate.id, isLeafTask(candidate, tasks))
|
||||
}
|
||||
revisionSpan?.event('plan_revision_applied', {
|
||||
'oma.plan.revision': applied.revision.version,
|
||||
'oma.plan.added_tasks': Object.keys(applied.revision.addedTasks).length,
|
||||
'oma.plan.superseded_tasks': applied.revision.supersededTaskIds.length,
|
||||
'oma.plan.retargeted_tasks': applied.revision.retargetedTasks.length,
|
||||
})
|
||||
revisionSpan?.end({
|
||||
status: statusOnly('ok'),
|
||||
attributes: { 'oma.plan.revision': applied.revision.version },
|
||||
})
|
||||
ctx.config.onProgress?.({
|
||||
type: 'plan_revision',
|
||||
task: task.id,
|
||||
data: applied.revision,
|
||||
})
|
||||
ctx.config.onProgress?.({
|
||||
type: 'recovery_decision',
|
||||
task: task.id,
|
||||
data: {
|
||||
decision: 'applied',
|
||||
trigger,
|
||||
planRevision: applied.revision.version,
|
||||
},
|
||||
})
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute all tasks in `queue` using agents in `pool`, respecting dependencies
|
||||
* and running independent tasks in parallel.
|
||||
@@ -289,6 +547,9 @@ export async function executeQueue(
|
||||
): Promise<void> {
|
||||
const { team, pool, scheduler, config } = ctx
|
||||
const legacyRoundMode = config.onApproval !== undefined
|
||||
if (legacyRoundMode && ctx.recovery.mode !== 'fixed') {
|
||||
throw new Error('Runtime recovery is incompatible with legacy onApproval round scheduling.')
|
||||
}
|
||||
const readyTaskIds = new Set<string>()
|
||||
const inFlight = new Map<string, Promise<void>>()
|
||||
const dispatchErrors: unknown[] = []
|
||||
@@ -541,6 +802,31 @@ export async function executeQueue(
|
||||
const completedThisRound: Task[] = []
|
||||
|
||||
const dispatchPromises = pending.map(async (task): Promise<void> => {
|
||||
const requirementIssues = validateTaskRequirements(
|
||||
[task],
|
||||
team.getAgents(),
|
||||
{
|
||||
defaultProvider: config.defaultProvider,
|
||||
defaultToolPreset: config.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
resolveCandidate: (candidateTask, candidate) => {
|
||||
const base = applyDefaultToolPreset(
|
||||
applyAgentDefaults(candidate, config),
|
||||
config.defaultToolPreset,
|
||||
)
|
||||
return withModelRoute(base, routeMatches(ctx.modelRouting, {
|
||||
phase: 'worker',
|
||||
agent: candidate.name,
|
||||
task: candidateTask,
|
||||
leaf: ctx.taskLeafById.get(candidateTask.id),
|
||||
}))
|
||||
},
|
||||
},
|
||||
)
|
||||
if (requirementIssues.length > 0) {
|
||||
throw new InvalidTaskRequirementsError(requirementIssues)
|
||||
}
|
||||
|
||||
// Mark in-progress
|
||||
queue.update(task.id, { status: 'in_progress' as TaskStatus })
|
||||
|
||||
@@ -733,9 +1019,14 @@ export async function executeQueue(
|
||||
applyAgentDefaults(agentConfig, config),
|
||||
config.defaultToolPreset,
|
||||
)
|
||||
const routedConfigs = routeChain(workerRoute).map(route =>
|
||||
withModelRoute(workerBaseConfig, route),
|
||||
)
|
||||
const routedConfigs = routeChain(workerRoute)
|
||||
.map(route => withModelRoute(workerBaseConfig, route))
|
||||
.filter(candidate =>
|
||||
validateTaskRequirements([task], [candidate], {
|
||||
defaultProvider: config.defaultProvider,
|
||||
defaultToolPreset: config.defaultToolPreset,
|
||||
includeDelegateTool: true,
|
||||
}).length === 0)
|
||||
const workerEffectiveConfig = routedConfigs[0] ?? workerBaseConfig
|
||||
const routedAgents = routedConfigs.map(route =>
|
||||
buildAgent(route, { includeDelegateTool: true }),
|
||||
@@ -901,13 +1192,18 @@ export async function executeQueue(
|
||||
// queue, shared memory, progress events, and agentResults as one consistent
|
||||
// result. Judge usage is charged to the same parent budget as the rest of the run.
|
||||
let effective = result
|
||||
let verification: TaskVerificationOutcome | undefined
|
||||
if (task.verify && !ctx.budgetExceededTriggered) {
|
||||
effective = await runTaskVerify(task, assignee, result, sharedMem, ctx)
|
||||
const verified = await runTaskVerify(task, assignee, result, sharedMem, ctx)
|
||||
effective = verified.result
|
||||
verification = verified.verification
|
||||
}
|
||||
|
||||
// Reflect the verified result in the per-task record the caller receives.
|
||||
ctx.agentResults.set(`${assignee}:${task.id}`, effective)
|
||||
|
||||
await applyRecoveryAtOutcome(queue, task, effective, verification, ctx)
|
||||
|
||||
// Persist result into shared memory so other agents can read it
|
||||
if (sharedMem) {
|
||||
await sharedMem.write(assignee, `task:${task.id}:result`, effective.output)
|
||||
@@ -940,6 +1236,7 @@ export async function executeQueue(
|
||||
...(taskLegacyEvent ? { legacyEvent: { ...taskLegacyEvent, success: effective.success } } : {}),
|
||||
})
|
||||
} else {
|
||||
await applyRecoveryAtOutcome(queue, task, result, undefined, ctx)
|
||||
queue.fail(task.id, result.output)
|
||||
taskSpan?.end({
|
||||
status: result.status ?? statusOnly('error', result.output),
|
||||
|
||||
@@ -6,8 +6,16 @@
|
||||
* orchestrators can react without polling.
|
||||
*/
|
||||
|
||||
import type { Task, TaskQueueSnapshot, TaskSnapshot, TaskStatus } from '../types.js'
|
||||
import { isTaskReady } from './task.js'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import type {
|
||||
PlanPatch,
|
||||
PlanRevision,
|
||||
Task,
|
||||
TaskQueueSnapshot,
|
||||
TaskSnapshot,
|
||||
TaskStatus,
|
||||
} from '../types.js'
|
||||
import { createTask, isTaskReady, validateTaskDependencies } from './task.js'
|
||||
import { validateTaskMetadata } from './metadata.js'
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -55,6 +63,8 @@ type HandlerFor<E extends TaskQueueEvent> = E extends 'all:complete'
|
||||
*/
|
||||
export class TaskQueue {
|
||||
private readonly tasks = new Map<string, Task>()
|
||||
private planRevision = 0
|
||||
private planRevisions: PlanRevision[] = []
|
||||
|
||||
/** Listeners keyed by event type, stored as symbol → handler pairs. */
|
||||
private readonly listeners = new Map<
|
||||
@@ -101,8 +111,7 @@ export class TaskQueue {
|
||||
/** Returns a serializable snapshot of all tasks and their status partitions. */
|
||||
snapshot(): TaskQueueSnapshot {
|
||||
const tasks = this.list()
|
||||
return {
|
||||
version: 1,
|
||||
const base = {
|
||||
tasks: tasks.map(TaskQueue.taskToSnapshot),
|
||||
pending: tasks.filter((task) => task.status === 'pending').map((task) => task.id),
|
||||
inProgress: tasks.filter((task) => task.status === 'in_progress').map((task) => task.id),
|
||||
@@ -111,6 +120,14 @@ export class TaskQueue {
|
||||
blocked: tasks.filter((task) => task.status === 'blocked').map((task) => task.id),
|
||||
skipped: tasks.filter((task) => task.status === 'skipped').map((task) => task.id),
|
||||
}
|
||||
return this.planRevision === 0
|
||||
? { version: 1, ...base }
|
||||
: {
|
||||
version: 2,
|
||||
...base,
|
||||
planRevision: this.planRevision,
|
||||
planRevisions: this.planRevisions.map(TaskQueue.clonePlanRevision),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -125,10 +142,6 @@ export class TaskQueue {
|
||||
snapshot: TaskQueueSnapshot,
|
||||
options: { readonly resetInProgress?: boolean } = {},
|
||||
): TaskQueue {
|
||||
if (snapshot.version !== 1) {
|
||||
throw new Error(`TaskQueue.fromSnapshot: unsupported snapshot version ${String(snapshot.version)}.`)
|
||||
}
|
||||
|
||||
const queue = new TaskQueue()
|
||||
const tasks = snapshot.tasks.map((task) => TaskQueue.taskFromSnapshot(task))
|
||||
const restored = options.resetInProgress
|
||||
@@ -138,9 +151,250 @@ export class TaskQueue {
|
||||
for (const task of restored) {
|
||||
queue.tasks.set(task.id, task)
|
||||
}
|
||||
if (snapshot.version === 2) {
|
||||
TaskQueue.validatePlanSnapshot(snapshot)
|
||||
queue.planRevision = snapshot.planRevision
|
||||
queue.planRevisions = snapshot.planRevisions.map(TaskQueue.clonePlanRevision)
|
||||
}
|
||||
return queue
|
||||
}
|
||||
|
||||
/** Current append-only runtime plan revision. */
|
||||
getPlanRevision(): number {
|
||||
return this.planRevision
|
||||
}
|
||||
|
||||
/** Immutable copies of the accepted runtime plan-repair history. */
|
||||
getPlanRevisions(): readonly PlanRevision[] {
|
||||
return this.planRevisions.map(TaskQueue.clonePlanRevision)
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomically apply an append-only plan patch without publishing queue events.
|
||||
*
|
||||
* Call {@link publishPlanRevision} after durable persistence succeeds. Until
|
||||
* then, newly-ready tasks are present in the queue but invisible to the
|
||||
* event-driven executor.
|
||||
*/
|
||||
applyPlanPatch(
|
||||
patch: PlanPatch,
|
||||
triggerTaskId: string,
|
||||
trigger: PlanRevision['trigger'],
|
||||
): { readonly revision: PlanRevision; readonly before: TaskQueueSnapshot } {
|
||||
const reason = patch.reason.trim()
|
||||
if (!reason) throw new Error('TaskQueue.applyPlanPatch: patch reason must not be empty.')
|
||||
|
||||
const addSpecs = [...(patch.addTasks ?? [])]
|
||||
const retargets = [...(patch.retargetPending ?? [])]
|
||||
const supersedeIds = [...(patch.supersedePending ?? [])]
|
||||
if (addSpecs.length === 0 && retargets.length === 0 && supersedeIds.length === 0) {
|
||||
throw new Error('TaskQueue.applyPlanPatch: patch must contain at least one operation.')
|
||||
}
|
||||
if (trigger !== 'success' && addSpecs.length === 0) {
|
||||
throw new Error(
|
||||
'TaskQueue.applyPlanPatch: failure recovery must append at least one replacement task.',
|
||||
)
|
||||
}
|
||||
|
||||
const before = this.snapshot()
|
||||
const draft = new Map<string, Task>(
|
||||
Array.from(this.tasks, ([id, task]) => [id, TaskQueue.cloneTask(task)]),
|
||||
)
|
||||
const nextVersion = this.planRevision + 1
|
||||
const now = new Date()
|
||||
const triggerTask = draft.get(triggerTaskId)
|
||||
if (!triggerTask) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: trigger task "${triggerTaskId}" not found.`)
|
||||
}
|
||||
if (triggerTask.status !== 'in_progress') {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: trigger task "${triggerTaskId}" must be in_progress; ` +
|
||||
`it is ${triggerTask.status}.`,
|
||||
)
|
||||
}
|
||||
|
||||
const retargetIds = new Set<string>()
|
||||
for (const retarget of retargets) {
|
||||
if (retargetIds.has(retarget.taskId)) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: task "${retarget.taskId}" is retargeted more than once.`)
|
||||
}
|
||||
retargetIds.add(retarget.taskId)
|
||||
const task = draft.get(retarget.taskId)
|
||||
if (!task) throw new Error(`TaskQueue.applyPlanPatch: task "${retarget.taskId}" not found.`)
|
||||
if (task.status !== 'pending' && task.status !== 'blocked') {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: only pending or blocked tasks can be retargeted; ` +
|
||||
`"${task.id}" is ${task.status}.`,
|
||||
)
|
||||
}
|
||||
if (!retarget.assignee.trim()) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: retarget assignee for "${task.id}" must not be empty.`)
|
||||
}
|
||||
draft.set(task.id, { ...task, assignee: retarget.assignee, updatedAt: now })
|
||||
}
|
||||
|
||||
const superseded = new Set<string>()
|
||||
for (const taskId of supersedeIds) {
|
||||
if (superseded.has(taskId)) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: task "${taskId}" is superseded more than once.`)
|
||||
}
|
||||
if (retargetIds.has(taskId)) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: task "${taskId}" cannot be retargeted and superseded.`)
|
||||
}
|
||||
superseded.add(taskId)
|
||||
const task = draft.get(taskId)
|
||||
if (!task) throw new Error(`TaskQueue.applyPlanPatch: task "${taskId}" not found.`)
|
||||
if (task.status !== 'pending' && task.status !== 'blocked') {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: only pending or blocked tasks can be superseded; ` +
|
||||
`"${task.id}" is ${task.status}.`,
|
||||
)
|
||||
}
|
||||
draft.set(task.id, {
|
||||
...task,
|
||||
status: 'skipped',
|
||||
result: `Superseded by plan revision ${nextVersion}: ${reason}`,
|
||||
supersededByRevision: nextVersion,
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
const keys = new Set<string>()
|
||||
const created = new Map<string, Task>()
|
||||
for (const spec of addSpecs) {
|
||||
const key = spec.key.trim()
|
||||
if (!key) throw new Error('TaskQueue.applyPlanPatch: appended task key must not be empty.')
|
||||
if (keys.has(key)) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: appended task key "${key}" is duplicated.`)
|
||||
}
|
||||
if (draft.has(key)) {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: appended task key "${key}" conflicts with an existing task id.`,
|
||||
)
|
||||
}
|
||||
keys.add(key)
|
||||
const task = createTask({
|
||||
title: spec.title,
|
||||
description: spec.description,
|
||||
assignee: spec.assignee,
|
||||
memoryScope: spec.memoryScope,
|
||||
dependencyPayload: spec.dependencyPayload,
|
||||
metadata: spec.metadata,
|
||||
maxRetries: spec.maxRetries,
|
||||
retryDelayMs: spec.retryDelayMs,
|
||||
retryBackoff: spec.retryBackoff,
|
||||
role: spec.role,
|
||||
priority: spec.priority,
|
||||
requires: spec.requires,
|
||||
verify: spec.verify,
|
||||
})
|
||||
created.set(key, task)
|
||||
}
|
||||
|
||||
const addedTasks: Record<string, string> = {}
|
||||
for (const spec of addSpecs) {
|
||||
const task = created.get(spec.key.trim())!
|
||||
const dependencies = (spec.dependsOn ?? []).map((reference) => {
|
||||
const newTask = created.get(reference)
|
||||
const dependencyId = newTask?.id ?? reference
|
||||
const dependency = draft.get(dependencyId) ?? newTask
|
||||
if (!dependency) {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: appended task "${spec.key}" has unknown dependency "${reference}".`,
|
||||
)
|
||||
}
|
||||
if (dependency.status === 'failed' || dependency.status === 'skipped') {
|
||||
throw new Error(
|
||||
`TaskQueue.applyPlanPatch: appended task "${spec.key}" cannot depend on ` +
|
||||
`${dependency.status} task "${dependency.id}".`,
|
||||
)
|
||||
}
|
||||
return dependencyId
|
||||
})
|
||||
if (new Set(dependencies).size !== dependencies.length) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: appended task "${spec.key}" has duplicate dependencies.`)
|
||||
}
|
||||
const resolved: Task = {
|
||||
...task,
|
||||
id: task.id || randomUUID(),
|
||||
...(dependencies.length > 0 ? { dependsOn: dependencies } : {}),
|
||||
}
|
||||
created.set(spec.key.trim(), resolved)
|
||||
addedTasks[spec.key.trim()] = resolved.id
|
||||
draft.set(resolved.id, resolved)
|
||||
}
|
||||
|
||||
// Resolve references between appended tasks after every patch-local id exists.
|
||||
for (const spec of addSpecs) {
|
||||
const task = created.get(spec.key.trim())!
|
||||
const dependencies = (spec.dependsOn ?? []).map((reference) =>
|
||||
created.get(reference)?.id ?? reference)
|
||||
const allTasks = Array.from(draft.values())
|
||||
const taskById = new Map(allTasks.map((candidate) => [candidate.id, candidate]))
|
||||
const pendingTask: Task = {
|
||||
...task,
|
||||
...(dependencies.length > 0 ? { dependsOn: dependencies } : { dependsOn: undefined }),
|
||||
status: 'pending',
|
||||
}
|
||||
const status: TaskStatus = isTaskReady(pendingTask, allTasks, taskById)
|
||||
? 'pending'
|
||||
: 'blocked'
|
||||
draft.set(task.id, { ...pendingTask, status })
|
||||
}
|
||||
|
||||
const validation = validateTaskDependencies(Array.from(draft.values()))
|
||||
if (!validation.valid) {
|
||||
throw new Error(`TaskQueue.applyPlanPatch: invalid patched graph: ${validation.errors.join(' ')}`)
|
||||
}
|
||||
|
||||
const revision: PlanRevision = {
|
||||
id: randomUUID(),
|
||||
version: nextVersion,
|
||||
triggerTaskId,
|
||||
trigger,
|
||||
reason,
|
||||
addedTasks,
|
||||
retargetedTasks: retargets.map((item) => ({ ...item })),
|
||||
supersededTaskIds: [...supersedeIds],
|
||||
createdAt: now.toISOString(),
|
||||
}
|
||||
|
||||
if (trigger !== 'success') {
|
||||
draft.set(triggerTaskId, {
|
||||
...draft.get(triggerTaskId)!,
|
||||
recoveredByRevision: nextVersion,
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
this.tasks.clear()
|
||||
for (const [id, task] of draft) this.tasks.set(id, task)
|
||||
this.planRevision = nextVersion
|
||||
this.planRevisions = [...this.planRevisions, revision]
|
||||
return { revision: TaskQueue.clonePlanRevision(revision), before }
|
||||
}
|
||||
|
||||
/** Roll back an unpublished plan patch while preserving queue listeners. */
|
||||
restorePlanSnapshot(snapshot: TaskQueueSnapshot): void {
|
||||
const restored = TaskQueue.fromSnapshot(snapshot)
|
||||
this.tasks.clear()
|
||||
for (const task of restored.list()) this.tasks.set(task.id, task)
|
||||
this.planRevision = restored.planRevision
|
||||
this.planRevisions = [...restored.planRevisions]
|
||||
}
|
||||
|
||||
/** Publish buffered skipped/ready events after a plan revision is durable. */
|
||||
publishPlanRevision(revision: PlanRevision): void {
|
||||
for (const taskId of revision.supersededTaskIds) {
|
||||
const task = this.tasks.get(taskId)
|
||||
if (task) this.emit('task:skipped', task)
|
||||
}
|
||||
for (const taskId of Object.values(revision.addedTasks)) {
|
||||
const task = this.tasks.get(taskId)
|
||||
if (task?.status === 'pending') this.emit('task:ready', task)
|
||||
}
|
||||
if (this.isComplete()) this.emitAllComplete()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Mutation: update / complete / fail
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -535,6 +789,12 @@ export class TaskQueue {
|
||||
...(task.maxRetries !== undefined ? { maxRetries: task.maxRetries } : {}),
|
||||
...(task.retryDelayMs !== undefined ? { retryDelayMs: task.retryDelayMs } : {}),
|
||||
...(task.retryBackoff !== undefined ? { retryBackoff: task.retryBackoff } : {}),
|
||||
...(task.supersededByRevision !== undefined
|
||||
? { supersededByRevision: task.supersededByRevision }
|
||||
: {}),
|
||||
...(task.recoveredByRevision !== undefined
|
||||
? { recoveredByRevision: task.recoveredByRevision }
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -562,6 +822,56 @@ export class TaskQueue {
|
||||
...(snapshot.maxRetries !== undefined ? { maxRetries: snapshot.maxRetries } : {}),
|
||||
...(snapshot.retryDelayMs !== undefined ? { retryDelayMs: snapshot.retryDelayMs } : {}),
|
||||
...(snapshot.retryBackoff !== undefined ? { retryBackoff: snapshot.retryBackoff } : {}),
|
||||
...(snapshot.supersededByRevision !== undefined
|
||||
? { supersededByRevision: snapshot.supersededByRevision }
|
||||
: {}),
|
||||
...(snapshot.recoveredByRevision !== undefined
|
||||
? { recoveredByRevision: snapshot.recoveredByRevision }
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
private static cloneTask(task: Task): Task {
|
||||
return {
|
||||
...task,
|
||||
...(task.dependsOn !== undefined ? { dependsOn: [...task.dependsOn] } : {}),
|
||||
createdAt: new Date(task.createdAt),
|
||||
updatedAt: new Date(task.updatedAt),
|
||||
}
|
||||
}
|
||||
|
||||
private static clonePlanRevision(revision: PlanRevision): PlanRevision {
|
||||
return {
|
||||
...revision,
|
||||
addedTasks: { ...revision.addedTasks },
|
||||
retargetedTasks: revision.retargetedTasks.map((item) => ({ ...item })),
|
||||
supersededTaskIds: [...revision.supersededTaskIds],
|
||||
}
|
||||
}
|
||||
|
||||
private static validatePlanSnapshot(
|
||||
snapshot: Extract<TaskQueueSnapshot, { readonly version: 2 }>,
|
||||
): void {
|
||||
if (
|
||||
!Number.isInteger(snapshot.planRevision)
|
||||
|| snapshot.planRevision < 1
|
||||
|| snapshot.planRevisions.length !== snapshot.planRevision
|
||||
) {
|
||||
throw new Error('TaskQueue.fromSnapshot: invalid plan revision sequence.')
|
||||
}
|
||||
const taskIds = new Set(snapshot.tasks.map((task) => task.id))
|
||||
for (const [index, revision] of snapshot.planRevisions.entries()) {
|
||||
if (revision.version !== index + 1 || !taskIds.has(revision.triggerTaskId)) {
|
||||
throw new Error('TaskQueue.fromSnapshot: invalid plan revision history.')
|
||||
}
|
||||
const referenced = [
|
||||
...Object.values(revision.addedTasks),
|
||||
...revision.retargetedTasks.map((item) => item.taskId),
|
||||
...revision.supersededTaskIds,
|
||||
]
|
||||
if (referenced.some((taskId) => !taskIds.has(taskId))) {
|
||||
throw new Error('TaskQueue.fromSnapshot: plan revision references an unknown task.')
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+155
-11
@@ -1138,6 +1138,15 @@ export interface TaskRequirements {
|
||||
readonly requiredProvider?: SupportedProvider
|
||||
}
|
||||
|
||||
/** Structured reason why a task cannot satisfy its hard requirements. */
|
||||
export interface TaskRequirementIssue {
|
||||
readonly code: 'NO_ELIGIBLE_AGENT' | 'ASSIGNEE_REQUIREMENTS_MISMATCH'
|
||||
readonly taskId: string
|
||||
readonly taskTitle: string
|
||||
readonly assignee?: string
|
||||
readonly reasons: readonly string[]
|
||||
}
|
||||
|
||||
/** Bounded, trace-safe business references attached to one task. */
|
||||
export type TaskMetadata = Readonly<Record<string, TraceAttributeValue>>
|
||||
|
||||
@@ -1172,6 +1181,103 @@ export interface RunTaskSpec {
|
||||
readonly verify?: ConsensusVerifyOptions
|
||||
}
|
||||
|
||||
/** One task appended by an opt-in runtime plan repair. */
|
||||
export interface PlanPatchTaskSpec extends RunTaskSpec {
|
||||
/**
|
||||
* Patch-local stable key. Dependencies of another appended task may refer to
|
||||
* this key; dependencies on the existing graph must use a task id.
|
||||
*/
|
||||
readonly key: string
|
||||
}
|
||||
|
||||
/** Reassign a task that has not started. */
|
||||
export interface PlanPatchRetarget {
|
||||
readonly taskId: string
|
||||
readonly assignee: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Append-only repair of the not-yet-executed portion of a task graph.
|
||||
*
|
||||
* Existing topology is immutable. Replacing a branch means superseding its
|
||||
* pending/blocked nodes and appending replacement tasks with new dependencies.
|
||||
*/
|
||||
export interface PlanPatch {
|
||||
readonly reason: string
|
||||
readonly addTasks?: readonly PlanPatchTaskSpec[]
|
||||
readonly retargetPending?: readonly PlanPatchRetarget[]
|
||||
readonly supersedePending?: readonly string[]
|
||||
}
|
||||
|
||||
/** Accepted description of one runtime plan repair. */
|
||||
export interface PlanRevision {
|
||||
readonly id: string
|
||||
readonly version: number
|
||||
readonly triggerTaskId: string
|
||||
readonly trigger: 'success' | 'failure' | 'verification_rejected'
|
||||
readonly reason: string
|
||||
readonly addedTasks: Readonly<Record<string, string>>
|
||||
readonly retargetedTasks: readonly PlanPatchRetarget[]
|
||||
readonly supersededTaskIds: readonly string[]
|
||||
readonly createdAt: string
|
||||
}
|
||||
|
||||
/** Verification facts exposed to a recovery policy without judge credentials. */
|
||||
export interface TaskVerificationOutcome {
|
||||
readonly verdict: 'accepted' | 'rejected'
|
||||
readonly dissent: readonly string[]
|
||||
readonly rounds: number
|
||||
}
|
||||
|
||||
/** Stable facts available at the task-outcome recovery barrier. */
|
||||
export interface TaskOutcome {
|
||||
readonly kind: 'success' | 'failure' | 'verification_rejected'
|
||||
readonly task: Readonly<Task>
|
||||
readonly result: Readonly<AgentRunResult>
|
||||
readonly verification?: TaskVerificationOutcome
|
||||
readonly planRevision: number
|
||||
readonly tasks: readonly Readonly<Task>[]
|
||||
readonly tokenBudgetRemaining?: number
|
||||
readonly costBudgetRemaining?: number
|
||||
}
|
||||
|
||||
/** Policy object that can propose an append-only repair at a task outcome barrier. */
|
||||
export interface Replanner {
|
||||
readonly name?: string
|
||||
replan(
|
||||
outcome: TaskOutcome,
|
||||
): PlanPatch | undefined | Promise<PlanPatch | undefined>
|
||||
}
|
||||
|
||||
/**
|
||||
* Opt-in runtime plan repair. Omitted or `mode: 'fixed'` preserves the exact
|
||||
* existing DAG lifecycle.
|
||||
*/
|
||||
export interface RecoveryOptions {
|
||||
readonly mode?: 'fixed' | 'repairable'
|
||||
/**
|
||||
* Called before the triggering task is completed or failed, so no dependent
|
||||
* can be dispatched ahead of an accepted patch.
|
||||
*/
|
||||
readonly onTaskOutcome?: (
|
||||
outcome: TaskOutcome,
|
||||
) => PlanPatch | undefined | Promise<PlanPatch | undefined>
|
||||
/**
|
||||
* First-class replanner policy. Mutually exclusive with `onTaskOutcome`.
|
||||
* The application owns any external I/O performed by a custom replanner.
|
||||
*/
|
||||
readonly replanner?: Replanner
|
||||
/** Approval gate for a validated patch. Omitted means the policy owns approval. */
|
||||
readonly onPlanPatch?: (
|
||||
patch: Readonly<PlanPatch>,
|
||||
outcome: TaskOutcome,
|
||||
) => boolean | Promise<boolean>
|
||||
/** Maximum accepted revisions in one run. Default `3`. */
|
||||
readonly maxPlanRevisions?: number
|
||||
/** Maximum cumulative tasks appended by revisions. Default `20`. */
|
||||
readonly maxAddedTasks?: number
|
||||
}
|
||||
|
||||
/** Per-call options for {@link OpenMultiAgent.runTasks}. */
|
||||
export interface RunTasksOptions extends RunIdentityOptions {
|
||||
readonly abortSignal?: AbortSignal
|
||||
@@ -1202,6 +1308,11 @@ export interface RunTasksOptions extends RunIdentityOptions {
|
||||
* coordinator model selection is unchanged.
|
||||
*/
|
||||
readonly modelRouting?: ModelRoutingPolicy
|
||||
/**
|
||||
* Opt-in runtime repair policy. `runFromPlan()` rejects non-fixed recovery so
|
||||
* a frozen plan remains an exact replay contract.
|
||||
*/
|
||||
readonly recovery?: RecoveryOptions
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1549,6 +1660,8 @@ export interface TeamRunResult extends RunOutcomeFields {
|
||||
readonly governanceReason?: GovernanceUnsatisfiedReason
|
||||
readonly goal?: string
|
||||
readonly tasks?: readonly TaskExecutionRecord[]
|
||||
/** Accepted append-only runtime plan revisions, in application order. */
|
||||
readonly planRevisions?: readonly PlanRevision[]
|
||||
/**
|
||||
* True when the run was a plan-only invocation (`runTeam(team, goal, { planOnly: true })`).
|
||||
* The coordinator decomposed the goal but no task agents executed.
|
||||
@@ -1690,6 +1803,10 @@ export interface TaskExecutionRecord {
|
||||
readonly requires?: TaskRequirements
|
||||
/** Verify config attached to this task, if any. Populated in `planOnly` results for inspection. */
|
||||
readonly verify?: ConsensusVerifyOptions
|
||||
/** Plan revision that logically superseded this unstarted task. */
|
||||
readonly supersededByRevision?: number
|
||||
/** Plan revision that repaired this task's failed/rejected outcome. */
|
||||
readonly recoveredByRevision?: number
|
||||
readonly metrics?: TaskExecutionMetrics
|
||||
}
|
||||
|
||||
@@ -1720,7 +1837,7 @@ export interface Task {
|
||||
readonly priority?: 'low' | 'normal' | 'high' | 'critical'
|
||||
/** Validated, bounded business references carried through result/trace/checkpoint. */
|
||||
readonly metadata?: TaskMetadata
|
||||
/** Explicit hard requirements used by capability-aware scheduling. */
|
||||
/** Explicit hard requirements enforced before assignment and execution. */
|
||||
readonly requires?: TaskRequirements
|
||||
result?: string
|
||||
readonly createdAt: Date
|
||||
@@ -1737,6 +1854,10 @@ export interface Task {
|
||||
* parent `maxTokenBudget`. Tasks without `verify` run unchanged.
|
||||
*/
|
||||
readonly verify?: ConsensusVerifyOptions
|
||||
/** Plan revision that logically superseded this unstarted task. */
|
||||
readonly supersededByRevision?: number
|
||||
/** Plan revision that repaired this task's failed/rejected outcome. */
|
||||
readonly recoveredByRevision?: number
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1757,6 +1878,8 @@ export interface OrchestratorEvent {
|
||||
| 'task_complete'
|
||||
| 'task_skipped'
|
||||
| 'task_retry'
|
||||
| 'plan_revision'
|
||||
| 'recovery_decision'
|
||||
| 'budget_exceeded'
|
||||
| 'message'
|
||||
| 'warning'
|
||||
@@ -1776,15 +1899,16 @@ export interface OrchestratorConfig {
|
||||
* agents are interchangeable.
|
||||
* - `'least-busy'` prefers the agent with the fewest active tasks; use it to
|
||||
* balance work when task duration varies.
|
||||
* - `'capability-match'` compares task text with agent names and system
|
||||
* prompts; use it when agents have distinct, clearly described roles.
|
||||
* - `'capability-match'` ranks eligible agents using declared capabilities
|
||||
* and task affinity; use it when agents have distinct roles.
|
||||
* - `'dependency-first'` assigns tasks that unblock the most dependents
|
||||
* first; use it for dependency-heavy DAGs.
|
||||
* - `'composite'` ranks by dependency criticality, hard-filters with the
|
||||
* AgentSelector, then combines fit and current load.
|
||||
*
|
||||
* All strategies hard-filter explicit task requirements before ranking.
|
||||
* Defaults to `'dependency-first'`. Explicit task assignees are preserved
|
||||
* and are never replaced by this strategy.
|
||||
* and must satisfy any declared requirements.
|
||||
*/
|
||||
readonly schedulingStrategy?: SchedulingStrategy
|
||||
/**
|
||||
@@ -1799,10 +1923,9 @@ export interface OrchestratorConfig {
|
||||
/**
|
||||
* Reject coordinator plans that name an assignee outside the team roster.
|
||||
*
|
||||
* Defaults to `false`: invalid names are cleared, a structured `warning`
|
||||
* progress event is emitted, and the configured scheduler assigns the task.
|
||||
* When `true`, the run terminates with a structured `INVALID_ASSIGNEE`
|
||||
* validation error before any planned task executes.
|
||||
* Defaults to `true`: coordinator output that names an unknown agent is
|
||||
* rejected before task execution. Set to `false` only to retain legacy
|
||||
* behavior that clears the assignment and lets the scheduler choose.
|
||||
*/
|
||||
readonly strictAssignees?: boolean
|
||||
/**
|
||||
@@ -1865,6 +1988,8 @@ export interface OrchestratorConfig {
|
||||
* and `restore`. Per-call options override this value. Defaults to off.
|
||||
*/
|
||||
readonly checkpoint?: boolean | CheckpointOptions
|
||||
/** Default opt-in runtime repair policy. Per-run recovery replaces it. */
|
||||
readonly recovery?: RecoveryOptions
|
||||
/**
|
||||
* Fallback tool grant for agents that declare neither {@link AgentConfig.tools}
|
||||
* nor {@link AgentConfig.toolPreset}. Built-in tools are opt-in (default-deny):
|
||||
@@ -2061,10 +2186,12 @@ export interface TaskSnapshot {
|
||||
readonly maxRetries?: number
|
||||
readonly retryDelayMs?: number
|
||||
readonly retryBackoff?: number
|
||||
readonly supersededByRevision?: number
|
||||
readonly recoveredByRevision?: number
|
||||
}
|
||||
|
||||
/** Serializable state of a {@link TaskQueue}. */
|
||||
export interface TaskQueueSnapshot {
|
||||
/** Legacy serializable state of a {@link TaskQueue}. */
|
||||
export interface TaskQueueSnapshotV1 {
|
||||
readonly version: 1
|
||||
readonly tasks: readonly TaskSnapshot[]
|
||||
readonly pending: readonly string[]
|
||||
@@ -2075,6 +2202,23 @@ export interface TaskQueueSnapshot {
|
||||
readonly skipped: readonly string[]
|
||||
}
|
||||
|
||||
/** Serializable adaptive-plan state of a {@link TaskQueue}. */
|
||||
export interface TaskQueueSnapshotV2 {
|
||||
readonly version: 2
|
||||
readonly tasks: readonly TaskSnapshot[]
|
||||
readonly pending: readonly string[]
|
||||
readonly inProgress: readonly string[]
|
||||
readonly completed: readonly string[]
|
||||
readonly failed: readonly string[]
|
||||
readonly blocked: readonly string[]
|
||||
readonly skipped: readonly string[]
|
||||
readonly planRevision: number
|
||||
readonly planRevisions: readonly PlanRevision[]
|
||||
}
|
||||
|
||||
/** Serializable state of a {@link TaskQueue}. */
|
||||
export type TaskQueueSnapshot = TaskQueueSnapshotV1 | TaskQueueSnapshotV2
|
||||
|
||||
/** Result recorded for a completed task inside a checkpoint. */
|
||||
export interface CompletedTaskCheckpoint {
|
||||
readonly taskId: string
|
||||
@@ -2088,7 +2232,7 @@ export interface CompletedTaskCheckpoint {
|
||||
}
|
||||
|
||||
interface CheckpointSnapshotBase {
|
||||
readonly mode: 'runTeam' | 'runTasks'
|
||||
readonly mode: 'runTeam' | 'runTasks' | 'runFromPlan'
|
||||
readonly createdAt: string
|
||||
/** Validated per-run metadata inherited by future restore attempts. */
|
||||
readonly metadata?: Readonly<Record<string, TraceAttributeValue>>
|
||||
|
||||
@@ -0,0 +1,568 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { OpenMultiAgent } from '../src/orchestrator/orchestrator.js'
|
||||
import { Checkpoint } from '../src/memory/checkpoint.js'
|
||||
import { InMemoryStore } from '../src/memory/store.js'
|
||||
import { TaskQueue } from '../src/task/queue.js'
|
||||
import { createTask } from '../src/task/task.js'
|
||||
import type {
|
||||
AgentConfig,
|
||||
LLMAdapter,
|
||||
LLMChatOptions,
|
||||
LLMMessage,
|
||||
LLMResponse,
|
||||
OrchestratorEvent,
|
||||
TaskOutcome,
|
||||
MemoryEntry,
|
||||
MemoryStore,
|
||||
CheckpointSnapshot,
|
||||
PlanArtifact,
|
||||
} from '../src/types.js'
|
||||
|
||||
function userPrompt(messages: LLMMessage[]): string {
|
||||
return [...messages].reverse().find((message) => message.role === 'user')
|
||||
?.content
|
||||
.filter((block): block is { type: 'text'; text: string } => block.type === 'text')
|
||||
.map((block) => block.text)
|
||||
.join('\n') ?? ''
|
||||
}
|
||||
|
||||
function taskTitle(messages: LLMMessage[]): string {
|
||||
return userPrompt(messages).match(/^# Task: (.+)$/m)?.[1] ?? ''
|
||||
}
|
||||
|
||||
function response(text: string, options: LLMChatOptions): LLMResponse {
|
||||
return {
|
||||
id: `response-${text}`,
|
||||
content: [{ type: 'text', text }],
|
||||
model: options.model,
|
||||
stop_reason: 'end_turn',
|
||||
usage: { input_tokens: 1, output_tokens: 1 },
|
||||
}
|
||||
}
|
||||
|
||||
function adapter(
|
||||
handler: (title: string) => string | Error,
|
||||
calls: string[],
|
||||
): LLMAdapter {
|
||||
return {
|
||||
name: 'adaptive-recovery-test',
|
||||
async chat(messages, options) {
|
||||
const title = taskTitle(messages)
|
||||
calls.push(title)
|
||||
const value = handler(title)
|
||||
if (value instanceof Error) throw value
|
||||
return response(value, options)
|
||||
},
|
||||
async *stream() {
|
||||
yield { type: 'done' as const, data: {} }
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function agent(name: string, llm: LLMAdapter): AgentConfig {
|
||||
return {
|
||||
name,
|
||||
model: 'test-model',
|
||||
systemPrompt: `You are ${name}.`,
|
||||
adapter: llm,
|
||||
}
|
||||
}
|
||||
|
||||
describe('adaptive runtime recovery', () => {
|
||||
it('repairs a failed branch by superseding old downstream work and appending replacements', async () => {
|
||||
const calls: string[] = []
|
||||
const events: OrchestratorEvent[] = []
|
||||
const llm = adapter(
|
||||
(title) => title === 'Search' ? new Error('search unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent({
|
||||
maxConcurrency: 2,
|
||||
onProgress: (event) => events.push(event),
|
||||
})
|
||||
const team = oma.createTeam('repair', {
|
||||
name: 'repair',
|
||||
agents: [agent('worker-a', llm), agent('worker-b', llm)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Search', description: 'Search the primary source.', assignee: 'worker-a' },
|
||||
{
|
||||
title: 'Analysis',
|
||||
description: 'Analyze the primary source.',
|
||||
assignee: 'worker-a',
|
||||
dependsOn: ['Search'],
|
||||
},
|
||||
], {
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome: (outcome: TaskOutcome) => {
|
||||
if (outcome.kind !== 'failure' || outcome.task.title !== 'Search') return undefined
|
||||
const oldAnalysis = outcome.tasks.find((task) => task.title === 'Analysis')!
|
||||
return {
|
||||
reason: 'Primary search failed; use the fallback source.',
|
||||
supersedePending: [oldAnalysis.id],
|
||||
addTasks: [
|
||||
{
|
||||
key: 'fallback-search',
|
||||
title: 'Fallback Search',
|
||||
description: 'Search the fallback source.',
|
||||
assignee: 'worker-b',
|
||||
},
|
||||
{
|
||||
key: 'replacement-analysis',
|
||||
title: 'Replacement Analysis',
|
||||
description: 'Analyze the fallback source.',
|
||||
assignee: 'worker-b',
|
||||
dependsOn: ['fallback-search'],
|
||||
},
|
||||
],
|
||||
}
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
expect(calls).toEqual(['Search', 'Fallback Search', 'Replacement Analysis'])
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks?.find((task) => task.title === 'Search')).toMatchObject({
|
||||
status: 'failed',
|
||||
recoveredByRevision: 1,
|
||||
})
|
||||
expect(result.tasks?.find((task) => task.title === 'Analysis')).toMatchObject({
|
||||
status: 'skipped',
|
||||
supersededByRevision: 1,
|
||||
})
|
||||
expect(result.tasks?.find((task) => task.title === 'Fallback Search')?.status).toBe('completed')
|
||||
expect(result.tasks?.find((task) => task.title === 'Replacement Analysis')?.status).toBe('completed')
|
||||
expect(result.planRevisions).toHaveLength(1)
|
||||
expect(result.planRevisions?.[0]).toMatchObject({
|
||||
version: 1,
|
||||
trigger: 'failure',
|
||||
reason: 'Primary search failed; use the fallback source.',
|
||||
})
|
||||
expect(events.some((event) => event.type === 'plan_revision')).toBe(true)
|
||||
})
|
||||
|
||||
it('holds original downstream dispatch until a success expansion is applied', async () => {
|
||||
const calls: string[] = []
|
||||
const llm = adapter((title) => `completed ${title}`, calls)
|
||||
const oma = new OpenMultiAgent({ maxConcurrency: 3 })
|
||||
const team = oma.createTeam('expand', {
|
||||
name: 'expand',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
|
||||
const onTaskOutcome = vi.fn((outcome: TaskOutcome) => {
|
||||
if (outcome.kind !== 'success' || outcome.task.title !== 'Upstream') return undefined
|
||||
const oldDownstream = outcome.tasks.find((task) => task.title === 'Old Downstream')!
|
||||
return {
|
||||
reason: 'Expand work from the actual upstream result.',
|
||||
supersedePending: [oldDownstream.id],
|
||||
addTasks: [
|
||||
{
|
||||
key: 'branch-a',
|
||||
title: 'Branch A',
|
||||
description: 'Process branch A.',
|
||||
assignee: 'worker',
|
||||
dependsOn: [outcome.task.id],
|
||||
},
|
||||
{
|
||||
key: 'branch-b',
|
||||
title: 'Branch B',
|
||||
description: 'Process branch B.',
|
||||
assignee: 'worker',
|
||||
dependsOn: [outcome.task.id],
|
||||
},
|
||||
],
|
||||
}
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Upstream', description: 'Produce structured scope.', assignee: 'worker' },
|
||||
{
|
||||
title: 'Old Downstream',
|
||||
description: 'Original downstream.',
|
||||
assignee: 'worker',
|
||||
dependsOn: ['Upstream'],
|
||||
},
|
||||
], {
|
||||
recovery: { mode: 'repairable', onTaskOutcome },
|
||||
})
|
||||
|
||||
expect(calls).toEqual(['Upstream', 'Branch A', 'Branch B'])
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks?.find((task) => task.title === 'Old Downstream')).toMatchObject({
|
||||
status: 'skipped',
|
||||
supersededByRevision: 1,
|
||||
})
|
||||
expect(result.tasks?.filter((task) => task.status === 'completed').map((task) => task.title))
|
||||
.toEqual(['Upstream', 'Branch A', 'Branch B'])
|
||||
})
|
||||
|
||||
it('preserves fixed DAG cascade behavior when recovery is omitted', async () => {
|
||||
const calls: string[] = []
|
||||
const llm = adapter(
|
||||
(title) => title === 'Search' ? new Error('search unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent()
|
||||
const team = oma.createTeam('fixed', {
|
||||
name: 'fixed',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Search', description: 'Search.', assignee: 'worker' },
|
||||
{
|
||||
title: 'Analysis',
|
||||
description: 'Analyze.',
|
||||
assignee: 'worker',
|
||||
dependsOn: ['Search'],
|
||||
},
|
||||
])
|
||||
|
||||
expect(calls).toEqual(['Search'])
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.tasks?.map((task) => task.status)).toEqual(['failed', 'failed'])
|
||||
})
|
||||
|
||||
it('accepts a first-class replanner policy and applies its repair', async () => {
|
||||
const calls: string[] = []
|
||||
const replan = vi.fn((outcome: TaskOutcome) => outcome.kind === 'failure'
|
||||
? {
|
||||
reason: 'Replanner selected a fallback agent.',
|
||||
addTasks: [{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Recover through the fallback agent.',
|
||||
assignee: 'worker-b',
|
||||
}],
|
||||
}
|
||||
: undefined)
|
||||
const llm = adapter(
|
||||
(title) => title === 'Primary' ? new Error('primary unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent()
|
||||
const team = oma.createTeam('replanner', {
|
||||
name: 'replanner',
|
||||
agents: [agent('worker-a', llm), agent('worker-b', llm)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Primary', description: 'Use the primary path.', assignee: 'worker-a' },
|
||||
], {
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
replanner: { name: 'fallback-policy', replan },
|
||||
},
|
||||
})
|
||||
|
||||
expect(replan).toHaveBeenCalledTimes(2)
|
||||
expect(calls).toEqual(['Primary', 'Fallback'])
|
||||
expect(result.success).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps the original failure path when plan-patch approval rejects a proposal', async () => {
|
||||
const calls: string[] = []
|
||||
const events: OrchestratorEvent[] = []
|
||||
const approval = vi.fn(() => false)
|
||||
const llm = adapter(
|
||||
(title) => title === 'Primary' ? new Error('primary unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent({ onProgress: (event) => events.push(event) })
|
||||
const team = oma.createTeam('approval', {
|
||||
name: 'approval',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Primary', description: 'Use the primary path.', assignee: 'worker' },
|
||||
], {
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome: (outcome) => outcome.kind === 'failure'
|
||||
? {
|
||||
reason: 'Try fallback.',
|
||||
addTasks: [{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Use fallback.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
: undefined,
|
||||
onPlanPatch: approval,
|
||||
},
|
||||
})
|
||||
|
||||
expect(approval).toHaveBeenCalledTimes(1)
|
||||
expect(calls).toEqual(['Primary'])
|
||||
expect(result.success).toBe(false)
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'recovery_decision',
|
||||
data: expect.objectContaining({ decision: 'rejected' }),
|
||||
}))
|
||||
})
|
||||
|
||||
it('appends investigation work when consensus verification rejects evidence', async () => {
|
||||
const workerCalls: string[] = []
|
||||
const judgeCalls: string[] = []
|
||||
const worker = adapter((title) => `completed ${title}`, workerCalls)
|
||||
const judge = adapter(
|
||||
() => '{"accept":false,"critique":"sources conflict"}',
|
||||
judgeCalls,
|
||||
)
|
||||
const oma = new OpenMultiAgent()
|
||||
const team = oma.createTeam('verification-repair', {
|
||||
name: 'verification-repair',
|
||||
agents: [agent('worker', worker)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [{
|
||||
title: 'Verify Evidence',
|
||||
description: 'Validate the available evidence.',
|
||||
assignee: 'worker',
|
||||
verify: {
|
||||
judges: [agent('judge', judge)],
|
||||
quorum: 1,
|
||||
maxRounds: 1,
|
||||
onDissent: 'reject',
|
||||
},
|
||||
}], {
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome: (outcome) => outcome.kind === 'verification_rejected'
|
||||
? {
|
||||
reason: 'The evidence conflicted; gather a second source.',
|
||||
addTasks: [{
|
||||
key: 'supplemental-investigation',
|
||||
title: 'Supplemental Investigation',
|
||||
description: 'Gather and reconcile an independent source.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
})
|
||||
|
||||
expect(judgeCalls).toHaveLength(1)
|
||||
expect(workerCalls).toEqual(['Verify Evidence', 'Supplemental Investigation'])
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks?.find((task) => task.title === 'Verify Evidence')).toMatchObject({
|
||||
status: 'completed',
|
||||
recoveredByRevision: 1,
|
||||
})
|
||||
expect(result.tasks?.find((task) => task.title === 'Supplemental Investigation')?.status)
|
||||
.toBe('completed')
|
||||
})
|
||||
|
||||
it('persists plan revision history in checkpoint queue snapshot v2', async () => {
|
||||
const calls: string[] = []
|
||||
const store = new InMemoryStore()
|
||||
const llm = adapter(
|
||||
(title) => title === 'Search' ? new Error('search unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent()
|
||||
const team = oma.createTeam('checkpoint-repair', {
|
||||
name: 'checkpoint-repair',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
|
||||
await oma.runTasks(team, [
|
||||
{ title: 'Search', description: 'Search.', assignee: 'worker' },
|
||||
], {
|
||||
checkpoint: { store },
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome: (outcome) => outcome.kind === 'failure'
|
||||
? {
|
||||
reason: 'Use fallback.',
|
||||
addTasks: [{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Fetch fallback.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
})
|
||||
|
||||
const snapshot = await new Checkpoint(store).loadLatest()
|
||||
expect(snapshot?.queue.version).toBe(2)
|
||||
if (snapshot?.queue.version !== 2) throw new Error('expected adaptive queue snapshot')
|
||||
expect(snapshot.queue.planRevision).toBe(1)
|
||||
expect(snapshot.queue.planRevisions).toHaveLength(1)
|
||||
expect(snapshot.queue.planRevisions[0]).toMatchObject({
|
||||
trigger: 'failure',
|
||||
reason: 'Use fallback.',
|
||||
})
|
||||
})
|
||||
|
||||
it('rolls back a repair and preserves failure cascade when patch checkpointing fails', async () => {
|
||||
const calls: string[] = []
|
||||
const events: OrchestratorEvent[] = []
|
||||
const store = new FailingStore()
|
||||
const llm = adapter(
|
||||
(title) => title === 'Search' ? new Error('search unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent({ onProgress: (event) => events.push(event) })
|
||||
const team = oma.createTeam('rollback-repair', {
|
||||
name: 'rollback-repair',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
|
||||
const result = await oma.runTasks(team, [
|
||||
{ title: 'Search', description: 'Search.', assignee: 'worker' },
|
||||
{
|
||||
title: 'Analysis',
|
||||
description: 'Analyze.',
|
||||
assignee: 'worker',
|
||||
dependsOn: ['Search'],
|
||||
},
|
||||
], {
|
||||
checkpoint: { store },
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
onTaskOutcome: (outcome) => outcome.kind === 'failure'
|
||||
? {
|
||||
reason: 'Use fallback.',
|
||||
supersedePending: [
|
||||
outcome.tasks.find((task) => task.title === 'Analysis')!.id,
|
||||
],
|
||||
addTasks: [{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Fetch fallback.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
})
|
||||
|
||||
expect(calls).toEqual(['Search'])
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.tasks?.map((task) => task.status)).toEqual(['failed', 'failed'])
|
||||
expect(result.tasks?.some((task) => task.title === 'Fallback')).toBe(false)
|
||||
expect(events.some((event) =>
|
||||
event.type === 'warning'
|
||||
&& (event.data as { code?: string }).code === 'PLAN_REVISION_NOT_DURABLE')).toBe(true)
|
||||
})
|
||||
|
||||
it('restores a crash-boundary revision without appending the repair twice', async () => {
|
||||
const calls: string[] = []
|
||||
const store = new InMemoryStore()
|
||||
const queue = new TaskQueue()
|
||||
const primary = createTask({
|
||||
title: 'Primary',
|
||||
description: 'Use the primary source.',
|
||||
assignee: 'worker',
|
||||
})
|
||||
queue.add(primary)
|
||||
queue.update(primary.id, { status: 'in_progress' })
|
||||
const { revision } = queue.applyPlanPatch({
|
||||
reason: 'Use fallback after the primary failure.',
|
||||
addTasks: [{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Use the fallback source.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}, primary.id, 'failure')
|
||||
queue.publishPlanRevision(revision)
|
||||
|
||||
const checkpoint: CheckpointSnapshot = {
|
||||
version: 2,
|
||||
mode: 'runTasks',
|
||||
createdAt: new Date().toISOString(),
|
||||
identity: {
|
||||
runId: 'adaptive-restore',
|
||||
attempt: 1,
|
||||
lastTraceId: '00000000000000000000000000000001',
|
||||
lastRootSpanId: '0000000000000001',
|
||||
},
|
||||
queue: queue.snapshot(),
|
||||
completedTaskResults: [],
|
||||
}
|
||||
await new Checkpoint(store).save(checkpoint)
|
||||
|
||||
const llm = adapter(
|
||||
(title) => title === 'Primary' ? new Error('primary unavailable') : `completed ${title}`,
|
||||
calls,
|
||||
)
|
||||
const oma = new OpenMultiAgent({ maxConcurrency: 1 })
|
||||
const team = oma.createTeam('adaptive-restore', {
|
||||
name: 'adaptive-restore',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
const replan = vi.fn((outcome: TaskOutcome) => outcome.kind === 'failure'
|
||||
? {
|
||||
reason: 'This duplicate proposal must not be called.',
|
||||
addTasks: [{
|
||||
key: 'duplicate',
|
||||
title: 'Duplicate',
|
||||
description: 'Must not be appended.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
: undefined)
|
||||
|
||||
const result = await oma.restore(team, {
|
||||
checkpoint: { store },
|
||||
recovery: {
|
||||
mode: 'repairable',
|
||||
replanner: { replan },
|
||||
},
|
||||
})
|
||||
|
||||
expect(replan).toHaveBeenCalledTimes(1)
|
||||
expect(calls).toEqual(['Primary', 'Fallback'])
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks?.map((task) => task.title)).toEqual(['Primary', 'Fallback'])
|
||||
expect(result.tasks?.find((task) => task.title === 'Primary')?.recoveredByRevision).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps runFromPlan and its checkpoints exact', async () => {
|
||||
const calls: string[] = []
|
||||
const store = new InMemoryStore()
|
||||
const llm = adapter((title) => `completed ${title}`, calls)
|
||||
const oma = new OpenMultiAgent()
|
||||
const team = oma.createTeam('exact-plan', {
|
||||
name: 'exact-plan',
|
||||
agents: [agent('worker', llm)],
|
||||
})
|
||||
const plan: PlanArtifact = {
|
||||
version: 1,
|
||||
tasks: [{
|
||||
id: 'planned-task',
|
||||
title: 'Planned Task',
|
||||
description: 'Execute exactly.',
|
||||
assignee: 'worker',
|
||||
}],
|
||||
}
|
||||
const recovery = {
|
||||
mode: 'repairable' as const,
|
||||
replanner: { replan: () => undefined },
|
||||
}
|
||||
|
||||
await expect(oma.runFromPlan(team, plan, { recovery }))
|
||||
.rejects.toThrow('runFromPlan requires fixed recovery')
|
||||
await oma.runFromPlan(team, plan, { checkpoint: { store } })
|
||||
expect((await new Checkpoint(store).loadLatest())?.mode).toBe('runFromPlan')
|
||||
await expect(oma.restore(team, { checkpoint: { store }, recovery }))
|
||||
.rejects.toThrow('runFromPlan checkpoint requires fixed recovery')
|
||||
})
|
||||
})
|
||||
|
||||
class FailingStore implements MemoryStore {
|
||||
async get(): Promise<MemoryEntry | null> { return null }
|
||||
async set(): Promise<void> { throw new Error('checkpoint unavailable') }
|
||||
async list(): Promise<MemoryEntry[]> { return [] }
|
||||
async delete(): Promise<void> {}
|
||||
async clear(): Promise<void> {}
|
||||
}
|
||||
@@ -1,9 +1,12 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { AgentSelector } from '../src/orchestrator/agent-selector.js'
|
||||
import {
|
||||
AgentSelector,
|
||||
validateTaskRequirements,
|
||||
} from '../src/orchestrator/agent-selector.js'
|
||||
import { Scheduler } from '../src/orchestrator/scheduler.js'
|
||||
import { createTask } from '../src/task/task.js'
|
||||
import type { AgentConfig } from '../src/types.js'
|
||||
import type { AgentConfig, TaskRequirements } from '../src/types.js'
|
||||
|
||||
describe('AgentSelector', () => {
|
||||
it('hard-filters a high-keyword candidate that lacks a required tool', () => {
|
||||
@@ -78,6 +81,46 @@ describe('AgentSelector', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it.each<{
|
||||
requirements: TaskRequirements
|
||||
candidate: AgentConfig
|
||||
reason: string
|
||||
}>([
|
||||
{
|
||||
requirements: { requiredTools: ['file_edit'] },
|
||||
candidate: { name: 'reader', model: 'test', tools: ['file_read'] },
|
||||
reason: 'missing required tools',
|
||||
},
|
||||
{
|
||||
requirements: { requiredCapabilities: ['typescript'] },
|
||||
candidate: { name: 'generalist', model: 'test' },
|
||||
reason: 'missing required capabilities',
|
||||
},
|
||||
{
|
||||
requirements: { requiredBackend: 'process' },
|
||||
candidate: { name: 'llm-worker', model: 'test' },
|
||||
reason: 'does not satisfy required backend process',
|
||||
},
|
||||
{
|
||||
requirements: { requiredProvider: 'anthropic' },
|
||||
candidate: { name: 'openai-worker', model: 'test', provider: 'openai' },
|
||||
reason: 'does not satisfy required provider anthropic',
|
||||
},
|
||||
])('hard-filters each requirement field independently', ({
|
||||
requirements,
|
||||
candidate,
|
||||
reason,
|
||||
}) => {
|
||||
const result = new AgentSelector().select({
|
||||
title: 'Restricted',
|
||||
description: 'Restricted task',
|
||||
requires: requirements,
|
||||
}, [candidate])
|
||||
|
||||
expect(result.error?.code).toBe('NO_ELIGIBLE_AGENT')
|
||||
expect(result.error?.reasons.join(' ')).toContain(reason)
|
||||
})
|
||||
|
||||
it('handles one candidate and empty requirements as pure soft scoring', () => {
|
||||
const solo: AgentConfig = {
|
||||
name: 'solo',
|
||||
@@ -147,3 +190,42 @@ describe('Scheduler capability-match with AgentSelector', () => {
|
||||
])).toThrow('NO_ELIGIBLE_AGENT')
|
||||
})
|
||||
})
|
||||
|
||||
describe('task requirement preflight', () => {
|
||||
it('reports an explicit assignee that does not satisfy requirements', () => {
|
||||
const task = createTask({
|
||||
title: 'Edit',
|
||||
description: 'Edit a file.',
|
||||
assignee: 'reader',
|
||||
requires: { requiredTools: ['file_edit'] },
|
||||
})
|
||||
|
||||
expect(validateTaskRequirements([task], [
|
||||
{ name: 'reader', model: 'test', tools: ['file_read'] },
|
||||
{ name: 'editor', model: 'test', tools: ['file_edit'] },
|
||||
])).toEqual([
|
||||
expect.objectContaining({
|
||||
code: 'ASSIGNEE_REQUIREMENTS_MISMATCH',
|
||||
taskId: task.id,
|
||||
assignee: 'reader',
|
||||
}),
|
||||
])
|
||||
})
|
||||
|
||||
it('reports an unassigned task when the full roster is ineligible', () => {
|
||||
const task = createTask({
|
||||
title: 'Anthropic task',
|
||||
description: 'Use the required provider.',
|
||||
requires: { requiredProvider: 'anthropic' },
|
||||
})
|
||||
|
||||
expect(validateTaskRequirements([task], [
|
||||
{ name: 'openai-worker', model: 'test', provider: 'openai' },
|
||||
])).toEqual([
|
||||
expect.objectContaining({
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
taskId: task.id,
|
||||
}),
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -336,6 +336,62 @@ describe('OpenMultiAgent checkpoint/restore', () => {
|
||||
expect((await resumedTeam.getSharedMemoryInstance()!.read('seed/note'))?.value).toEqual({ keep: true })
|
||||
})
|
||||
|
||||
it('restores requirements and validates pending work against the resumed roster', async () => {
|
||||
const checkpointStore = new InMemoryStore()
|
||||
const scripted = scriptedAdapter(['first output', 'must not run'])
|
||||
const abort = new AbortController()
|
||||
const orchestrator = new OpenMultiAgent({
|
||||
onProgress(event) {
|
||||
if (event.type === 'task_complete') abort.abort()
|
||||
},
|
||||
})
|
||||
const initialTeam = new Team({
|
||||
name: 'team',
|
||||
agents: [{
|
||||
...worker('worker', scripted.adapter),
|
||||
capabilities: ['typescript'],
|
||||
}],
|
||||
})
|
||||
const constrainedTasks: RunTaskSpec[] = [
|
||||
{
|
||||
title: 'first',
|
||||
description: 'do first',
|
||||
assignee: 'worker',
|
||||
requires: { requiredCapabilities: ['typescript'] },
|
||||
},
|
||||
{
|
||||
title: 'second',
|
||||
description: 'do second',
|
||||
assignee: 'worker',
|
||||
dependsOn: ['first'],
|
||||
requires: { requiredCapabilities: ['typescript'] },
|
||||
},
|
||||
]
|
||||
|
||||
await orchestrator.runTasks(initialTeam, constrainedTasks, {
|
||||
abortSignal: abort.signal,
|
||||
checkpoint: { store: checkpointStore },
|
||||
})
|
||||
expect(scripted.calls()).toBe(1)
|
||||
|
||||
const resumedTeam = new Team({
|
||||
name: 'team',
|
||||
agents: [worker('worker', scripted.adapter)],
|
||||
})
|
||||
await expect(orchestrator.restore(resumedTeam, {
|
||||
checkpoint: { store: checkpointStore },
|
||||
})).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
issues: [
|
||||
expect.objectContaining({
|
||||
code: 'ASSIGNEE_REQUIREMENTS_MISMATCH',
|
||||
taskTitle: 'second',
|
||||
}),
|
||||
],
|
||||
})
|
||||
expect(scripted.calls()).toBe(1)
|
||||
})
|
||||
|
||||
it('restore against an empty store starts a fresh task run', async () => {
|
||||
const store = new InMemoryStore()
|
||||
const scripted = scriptedAdapter(['fresh output'])
|
||||
@@ -355,6 +411,31 @@ describe('OpenMultiAgent checkpoint/restore', () => {
|
||||
expect((await store.list()).some((entry) => isCheckpointKey(entry.key))).toBe(true)
|
||||
})
|
||||
|
||||
it('restore against an empty store preserves and validates task requirements', async () => {
|
||||
const store = new InMemoryStore()
|
||||
const scripted = scriptedAdapter(['must not run'])
|
||||
const team = new Team({
|
||||
name: 'team',
|
||||
agents: [worker('worker', scripted.adapter)],
|
||||
sharedMemoryStore: store,
|
||||
})
|
||||
const orchestrator = new OpenMultiAgent()
|
||||
|
||||
await expect(orchestrator.restore(team, [{
|
||||
title: 'restricted',
|
||||
description: 'requires a missing capability',
|
||||
assignee: 'worker',
|
||||
requires: { requiredCapabilities: ['missing'] },
|
||||
}], { checkpoint: { store } })).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
issues: [
|
||||
expect.objectContaining({ code: 'ASSIGNEE_REQUIREMENTS_MISMATCH' }),
|
||||
],
|
||||
})
|
||||
|
||||
expect(scripted.calls()).toBe(0)
|
||||
})
|
||||
|
||||
it('checkpoint/restore works with a custom async MemoryStore', async () => {
|
||||
const store = new AsyncMapStore()
|
||||
const scripted = scriptedAdapter(['first output', 'second output'])
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import {
|
||||
isRetryableError,
|
||||
InvalidTaskRequirementsError,
|
||||
TokenBudgetExceededError,
|
||||
InvalidMessageError,
|
||||
LLMCallTimeoutError,
|
||||
@@ -37,9 +38,15 @@ describe('isRetryableError', () => {
|
||||
expect(isRetryableError({ statusCode: 503 })).toBe(true)
|
||||
})
|
||||
|
||||
it('classifies budget and invalid-message framework errors as terminal', () => {
|
||||
it('classifies budget, invalid-message, and task-requirement errors as terminal', () => {
|
||||
expect(isRetryableError(new TokenBudgetExceededError('a', 100, 50))).toBe(false)
|
||||
expect(isRetryableError(new InvalidMessageError('bad'))).toBe(false)
|
||||
expect(isRetryableError(new InvalidTaskRequirementsError([{
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
taskId: 'task-1',
|
||||
taskTitle: 'Restricted',
|
||||
reasons: ['worker excluded'],
|
||||
}]))).toBe(false)
|
||||
})
|
||||
|
||||
it('classifies a per-call timeout as retryable', () => {
|
||||
|
||||
@@ -339,6 +339,64 @@ describe('OpenMultiAgent', () => {
|
||||
expect(capturedPrompts).toEqual([])
|
||||
})
|
||||
|
||||
it('rejects an explicit assignee that does not satisfy hard requirements before dispatch', async () => {
|
||||
const events: OrchestratorEvent[] = []
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
onProgress: (event) => events.push(event),
|
||||
})
|
||||
const team = oma.createTeam('t', teamCfg([
|
||||
{ ...agentConfig('reader'), tools: ['file_read'] },
|
||||
{ ...agentConfig('editor'), tools: ['file_edit'] },
|
||||
]))
|
||||
|
||||
await expect(oma.runTasks(team, [{
|
||||
title: 'Edit',
|
||||
description: 'Edit a file.',
|
||||
assignee: 'reader',
|
||||
requires: { requiredTools: ['file_edit'] },
|
||||
}])).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
issues: [
|
||||
expect.objectContaining({
|
||||
code: 'ASSIGNEE_REQUIREMENTS_MISMATCH',
|
||||
assignee: 'reader',
|
||||
}),
|
||||
],
|
||||
})
|
||||
|
||||
expect(capturedPrompts).toEqual([])
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'error',
|
||||
data: expect.objectContaining({ code: 'INVALID_TASK_REQUIREMENTS' }),
|
||||
}))
|
||||
})
|
||||
|
||||
it('rejects a plan with no eligible agent before dispatch', async () => {
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = oma.createTeam('t', teamCfg([agentConfig('worker')]))
|
||||
|
||||
await expect(oma.runTasks(team, [
|
||||
{
|
||||
title: 'Unrestricted predecessor',
|
||||
description: 'Must not start before the full plan is validated.',
|
||||
},
|
||||
{
|
||||
title: 'Restricted',
|
||||
description: 'Requires a missing capability.',
|
||||
dependsOn: ['Unrestricted predecessor'],
|
||||
requires: { requiredCapabilities: ['missing'] },
|
||||
},
|
||||
])).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
issues: [
|
||||
expect.objectContaining({ code: 'NO_ELIGIBLE_AGENT' }),
|
||||
],
|
||||
})
|
||||
|
||||
expect(capturedPrompts).toEqual([])
|
||||
})
|
||||
|
||||
it.each<{
|
||||
strategy: SchedulingStrategy
|
||||
expected: Record<string, string>
|
||||
@@ -524,6 +582,31 @@ describe('OpenMultiAgent', () => {
|
||||
expect(team.getAgents()[0]?.model).toBe('default-worker-model')
|
||||
})
|
||||
|
||||
it('rejects a model route that overrides a required provider before dispatch', async () => {
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'default-model' })
|
||||
const team = oma.createTeam('t', teamCfg([
|
||||
{ ...agentConfig('worker-a'), provider: 'anthropic' },
|
||||
]))
|
||||
|
||||
await expect(oma.runTasks(team, [{
|
||||
title: 'Anthropic only',
|
||||
description: 'Run on the required provider.',
|
||||
assignee: 'worker-a',
|
||||
requires: { requiredProvider: 'anthropic' },
|
||||
}], {
|
||||
modelRouting: {
|
||||
rules: [{
|
||||
match: { phase: 'worker' },
|
||||
route: { model: 'routed-model', provider: 'openai' },
|
||||
}],
|
||||
},
|
||||
})).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
})
|
||||
|
||||
expect(capturedPrompts).toEqual([])
|
||||
})
|
||||
|
||||
it('routes critical dependent tasks to an explicit premium model', async () => {
|
||||
mockAdapterResponses = ['research output', 'review output']
|
||||
|
||||
@@ -597,6 +680,43 @@ describe('OpenMultiAgent', () => {
|
||||
expect(capturedAdapterProviders).toEqual(['anthropic', 'openai'])
|
||||
})
|
||||
|
||||
it('does not use a fallback route that violates a required provider', async () => {
|
||||
mockAdapterHandler = () =>
|
||||
Object.assign(new Error('provider unavailable'), { status: 503 })
|
||||
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'default-model' })
|
||||
const team = oma.createTeam('t', teamCfg([
|
||||
{ ...agentConfig('worker-a'), provider: 'anthropic' },
|
||||
]))
|
||||
|
||||
const result = await oma.runTasks(team, [{
|
||||
title: 'Anthropic fallback boundary',
|
||||
description: 'Do not cross the provider boundary.',
|
||||
assignee: 'worker-a',
|
||||
requires: { requiredProvider: 'anthropic' },
|
||||
maxRetries: 1,
|
||||
retryDelayMs: 0,
|
||||
}], {
|
||||
modelRouting: {
|
||||
rules: [{
|
||||
match: { phase: 'worker' },
|
||||
route: {
|
||||
model: 'primary-model',
|
||||
provider: 'anthropic',
|
||||
fallback: [{ model: 'backup-model', provider: 'openai' }],
|
||||
},
|
||||
}],
|
||||
},
|
||||
})
|
||||
|
||||
expect(result.success).toBe(false)
|
||||
expect(capturedAdapterProviders).toEqual(['anthropic'])
|
||||
expect(capturedChatOptions.map(options => options.model)).toEqual([
|
||||
'primary-model',
|
||||
'primary-model',
|
||||
])
|
||||
})
|
||||
|
||||
it('fails over after a retryable provider error while onAgentStream is enabled', async () => {
|
||||
const streamedErrorKinds: Array<{ kind: string | undefined; retryable: boolean | undefined }> = []
|
||||
let defaultModelCalls = 0
|
||||
@@ -971,9 +1091,12 @@ describe('OpenMultiAgent', () => {
|
||||
expect(result.agentResults.has('coordinator')).toBe(true)
|
||||
})
|
||||
|
||||
it('returns a validation failure for a coordinator-generated cyclic DAG', async () => {
|
||||
it('repairs a coordinator-generated cyclic DAG before dispatch', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[{"title":"Task A","description":"Do A","assignee":"worker-a","dependsOn":["Task B"]},{"title":"Task B","description":"Do B","assignee":"worker-b","dependsOn":["Task A"]}]\n```',
|
||||
'```json\n[{"title":"Research","description":"Research the topic","assignee":"worker-a"}]\n```',
|
||||
'research output',
|
||||
'final answer',
|
||||
]
|
||||
const events: OrchestratorEvent[] = []
|
||||
const oma = new OpenMultiAgent({
|
||||
@@ -987,31 +1110,56 @@ describe('OpenMultiAgent', () => {
|
||||
'First research the topic, then produce a detailed report with recommendations.',
|
||||
)
|
||||
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.status?.code).toBe('error')
|
||||
expect(result.errorInfo).toMatchObject({ kind: 'validation' })
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(capturedPrompts).toHaveLength(1)
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'error',
|
||||
data: expect.objectContaining({ code: 'INVALID_TASK_DEPENDENCIES' }),
|
||||
}))
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks).toHaveLength(1)
|
||||
expect(capturedPrompts).toHaveLength(4)
|
||||
expect(events.some((event) => event.type === 'error')).toBe(false)
|
||||
})
|
||||
|
||||
it('falls back to one-task-per-agent when coordinator output is unparseable', async () => {
|
||||
it('fails closed when coordinator output remains unparseable after repair', async () => {
|
||||
mockAdapterResponses = [
|
||||
'I cannot produce JSON output', // invalid coordinator output
|
||||
'worker-a result',
|
||||
'worker-b result',
|
||||
'synthesis',
|
||||
'Still not JSON', // repair also fails
|
||||
]
|
||||
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const events: OrchestratorEvent[] = []
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
onProgress: (event) => events.push(event),
|
||||
})
|
||||
const team = oma.createTeam('t', teamCfg())
|
||||
|
||||
const result = await oma.runTeam(team, 'First design the database schema, then implement the REST API endpoints')
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.status?.code).toBe('error')
|
||||
expect(result.errorInfo).toMatchObject({ kind: 'validation', code: 'COORDINATOR_PLAN_INVALID' })
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(capturedPrompts).toHaveLength(2)
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'error',
|
||||
data: expect.objectContaining({ code: 'COORDINATOR_PLAN_INVALID' }),
|
||||
}))
|
||||
})
|
||||
|
||||
it('rejects a partially malformed coordinator plan instead of executing its valid subset', async () => {
|
||||
const malformedPlan = '```json\n[' +
|
||||
'{"title":"Research","description":"Research the topic","assignee":"worker-a"},' +
|
||||
'{"title":"Broken","assignee":"worker-b"}' +
|
||||
']\n```'
|
||||
mockAdapterResponses = [malformedPlan, malformedPlan]
|
||||
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = oma.createTeam('t', teamCfg())
|
||||
|
||||
const result = await oma.runTeam(
|
||||
team,
|
||||
'First research the topic, then produce a detailed report with recommendations.',
|
||||
)
|
||||
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(capturedPrompts).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('uses schedulingStrategy for unassigned coordinator tasks', async () => {
|
||||
@@ -1083,7 +1231,7 @@ describe('OpenMultiAgent', () => {
|
||||
expect(result.tasks?.[0]?.assignee).toBe('aaa-idle')
|
||||
})
|
||||
|
||||
it('surfaces composite no-eligible fallback through onProgress warning', async () => {
|
||||
it('rejects a coordinator plan with unsatisfied requirements before planOnly succeeds', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[{"title":"Restricted","description":"Restricted task","requires":{"requiredCapabilities":["missing"]}}]\n```',
|
||||
]
|
||||
@@ -1101,24 +1249,56 @@ describe('OpenMultiAgent', () => {
|
||||
{ planOnly: true },
|
||||
)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.tasks?.[0]?.assignee).toBe('worker-a')
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.errorInfo).toMatchObject({
|
||||
kind: 'validation',
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
})
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'warning',
|
||||
type: 'error',
|
||||
data: expect.objectContaining({
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
fallback: 'zero-fit-current-load',
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
issues: [
|
||||
expect.objectContaining({ code: 'NO_ELIGIBLE_AGENT' }),
|
||||
],
|
||||
}),
|
||||
}))
|
||||
expect(capturedPrompts).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('warns and schedules a coordinator task with an invalid assignee by default', async () => {
|
||||
it('rejects a coordinator assignee that conflicts with hard requirements', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[{"title":"Edit","description":"Edit a file","assignee":"reader","requires":{"requiredTools":["file_edit"]}}]\n```',
|
||||
]
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = oma.createTeam('t', teamCfg([
|
||||
{ ...agentConfig('reader'), tools: ['file_read'] },
|
||||
{ ...agentConfig('editor'), tools: ['file_edit'] },
|
||||
]))
|
||||
|
||||
const result = await oma.runTeam(
|
||||
team,
|
||||
'First edit the file, then review the result.',
|
||||
{ planOnly: true },
|
||||
)
|
||||
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.errorInfo).toMatchObject({
|
||||
kind: 'validation',
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
})
|
||||
expect(capturedPrompts).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('warns and schedules a coordinator task with an invalid assignee only when legacy reassignment is explicit', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[{"title":"Research","description":"Research the topic","assignee":"ghost"}]\n```',
|
||||
]
|
||||
const events: OrchestratorEvent[] = []
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
strictAssignees: false,
|
||||
onProgress: (event) => events.push(event),
|
||||
})
|
||||
const team = oma.createTeam('t', teamCfg([agentConfig('worker-a')]))
|
||||
@@ -1143,14 +1323,14 @@ describe('OpenMultiAgent', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('returns a structured validation error for an invalid assignee in strict mode', async () => {
|
||||
it('returns a structured validation error for an invalid assignee by default', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[{"title":"Research","description":"Research the topic","assignee":"ghost"}]\n```',
|
||||
'```json\n[{"title":"Research","description":"Research the topic","assignee":"ghost"}]\n```',
|
||||
]
|
||||
const events: OrchestratorEvent[] = []
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
strictAssignees: true,
|
||||
onProgress: (event) => events.push(event),
|
||||
})
|
||||
const team = oma.createTeam('t', teamCfg([agentConfig('worker-a')]))
|
||||
@@ -1165,12 +1345,12 @@ describe('OpenMultiAgent', () => {
|
||||
expect(result.status?.code).toBe('error')
|
||||
expect(result.errorInfo).toMatchObject({
|
||||
kind: 'validation',
|
||||
code: 'INVALID_ASSIGNEE',
|
||||
code: 'COORDINATOR_PLAN_INVALID',
|
||||
})
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(events).toContainEqual(expect.objectContaining({
|
||||
type: 'error',
|
||||
data: expect.objectContaining({ code: 'INVALID_ASSIGNEE' }),
|
||||
data: expect.objectContaining({ code: 'COORDINATOR_PLAN_INVALID' }),
|
||||
}))
|
||||
})
|
||||
|
||||
@@ -1470,17 +1650,13 @@ describe('OpenMultiAgent', () => {
|
||||
expect(result.agentResults.get('worker')?.output).toBe('worker output')
|
||||
})
|
||||
|
||||
it('marks tasks with unknown coordinator dependencies as failed instead of dropping them', async () => {
|
||||
it('fails closed when coordinator dependencies remain unknown after repair', async () => {
|
||||
let coordinatorCalls = 0
|
||||
let workerCalls = 0
|
||||
let synthesisPrompt = ''
|
||||
const coordinatorAdapter: LLMAdapter = {
|
||||
name: 'coordinator-mock',
|
||||
async chat(messages: LLMMessage[]): Promise<LLMResponse> {
|
||||
const prompt = extractUserPrompt(messages)
|
||||
if (prompt.includes('Task Results')) {
|
||||
synthesisPrompt = prompt
|
||||
return textResponse('final with gap')
|
||||
}
|
||||
async chat(): Promise<LLMResponse> {
|
||||
coordinatorCalls++
|
||||
return textResponse('```json\n[{"title": "Use missing", "description": "Use a missing result", "assignee": "worker", "dependsOn": ["Missing research"]}]\n```')
|
||||
},
|
||||
async *stream() { yield { type: 'done' as const, data: {} } },
|
||||
@@ -1503,22 +1679,18 @@ describe('OpenMultiAgent', () => {
|
||||
coordinator: { adapter: coordinatorAdapter },
|
||||
})
|
||||
|
||||
const task = result.tasks?.find((t) => t.title === 'Use missing')
|
||||
expect(task?.status).toBe('failed')
|
||||
expect(synthesisPrompt).toContain('Unresolved dependency reference(s): Missing research')
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(coordinatorCalls).toBe(2)
|
||||
expect(workerCalls).toBe(0)
|
||||
})
|
||||
|
||||
it('fails ambiguous title dependencies when coordinator emits duplicate task titles', async () => {
|
||||
let synthesisPrompt = ''
|
||||
it('fails closed when duplicate coordinator task titles remain ambiguous after repair', async () => {
|
||||
let coordinatorCalls = 0
|
||||
const coordinatorAdapter: LLMAdapter = {
|
||||
name: 'coordinator-mock',
|
||||
async chat(messages: LLMMessage[]): Promise<LLMResponse> {
|
||||
const prompt = extractUserPrompt(messages)
|
||||
if (prompt.includes('Task Results')) {
|
||||
synthesisPrompt = prompt
|
||||
return textResponse('final with ambiguity')
|
||||
}
|
||||
async chat(): Promise<LLMResponse> {
|
||||
coordinatorCalls++
|
||||
return textResponse([
|
||||
'```json',
|
||||
'[',
|
||||
@@ -1539,9 +1711,9 @@ describe('OpenMultiAgent', () => {
|
||||
coordinator: { adapter: coordinatorAdapter },
|
||||
})
|
||||
|
||||
const synth = result.tasks?.find((t) => t.title === 'Synthesize')
|
||||
expect(synth?.status).toBe('failed')
|
||||
expect(synthesisPrompt).toContain('Research (ambiguous duplicate title)')
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.tasks).toEqual([])
|
||||
expect(coordinatorCalls).toBe(2)
|
||||
})
|
||||
|
||||
it('includes failed and skipped task sections in final synthesis prompt', async () => {
|
||||
@@ -2077,6 +2249,33 @@ describe('OpenMultiAgent', () => {
|
||||
retryBackoff: 3,
|
||||
})
|
||||
})
|
||||
|
||||
it('round-trips requirements and revalidates them before plan replay', async () => {
|
||||
mockAdapterResponses = [
|
||||
'```json\n[' +
|
||||
'{"title":"Typed work","description":"Implement TypeScript","assignee":"worker","requires":{"requiredCapabilities":["typescript"]}}' +
|
||||
']\n```',
|
||||
]
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const planningTeam = oma.createTeam('planning', teamCfg([
|
||||
{ ...agentConfig('worker'), capabilities: ['typescript'] },
|
||||
]))
|
||||
|
||||
const planOnlyResult = await oma.runTeam(planningTeam, complexGoal, { planOnly: true })
|
||||
const plan = oma.createPlanArtifact(planOnlyResult)
|
||||
expect(plan.tasks[0]?.requires).toEqual({
|
||||
requiredCapabilities: ['typescript'],
|
||||
})
|
||||
|
||||
capturedPrompts = []
|
||||
const replayTeam = oma.createTeam('replay', teamCfg([
|
||||
agentConfig('worker'),
|
||||
]))
|
||||
await expect(oma.runFromPlan(replayTeam, plan)).rejects.toMatchObject({
|
||||
code: 'INVALID_TASK_REQUIREMENTS',
|
||||
})
|
||||
expect(capturedPrompts).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('stream trace events', () => {
|
||||
|
||||
@@ -169,7 +169,7 @@ describe('runTeam verify hook integration', () => {
|
||||
expect(taskD!.verify).toBeUndefined()
|
||||
})
|
||||
|
||||
it('verify field in coordinator JSON is ignored when invalid (null, number, string)', async () => {
|
||||
it('rejects an invalid coordinator verify field instead of silently dropping it', async () => {
|
||||
const worker = agent('worker')
|
||||
const judge = agent('judge')
|
||||
|
||||
@@ -179,6 +179,11 @@ describe('runTeam verify hook integration', () => {
|
||||
{ title: 'Task E2', description: 'Do E2', assignee: 'worker', dependsOn: [], verify: 42 },
|
||||
{ title: 'Task E3', description: 'Do E3', assignee: 'worker', dependsOn: [], verify: 'yes' },
|
||||
]),
|
||||
coordinatorPlan([
|
||||
{ title: 'Task E1', description: 'Do E1', assignee: 'worker', dependsOn: [], verify: null },
|
||||
{ title: 'Task E2', description: 'Do E2', assignee: 'worker', dependsOn: [], verify: 42 },
|
||||
{ title: 'Task E3', description: 'Do E3', assignee: 'worker', dependsOn: [], verify: 'yes' },
|
||||
]),
|
||||
]
|
||||
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
@@ -189,9 +194,8 @@ describe('runTeam verify hook integration', () => {
|
||||
verifyJudges: [judge],
|
||||
})
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
for (const task of result.tasks ?? []) {
|
||||
expect(task.verify).toBeUndefined()
|
||||
}
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.errorInfo).toMatchObject({ kind: 'validation', code: 'COORDINATOR_PLAN_INVALID' })
|
||||
expect(result.tasks).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -1,5 +1,6 @@
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { Scheduler } from '../src/orchestrator/scheduler.js'
|
||||
import type { SchedulingStrategy } from '../src/orchestrator/scheduler.js'
|
||||
import { TaskQueue } from '../src/task/queue.js'
|
||||
import { createTask } from '../src/task/task.js'
|
||||
import type { AgentConfig, Task, TaskRequirements } from '../src/types.js'
|
||||
@@ -67,7 +68,10 @@ describe('Scheduler: round-robin', () => {
|
||||
|
||||
it('returns empty map when no agents', () => {
|
||||
const s = new Scheduler('round-robin')
|
||||
const tasks = [pendingTask('t1')]
|
||||
const tasks = [
|
||||
pendingTask('t1'),
|
||||
pendingTask('t2', { requires: {} }),
|
||||
]
|
||||
expect(s.schedule(tasks, []).size).toBe(0)
|
||||
})
|
||||
|
||||
@@ -258,6 +262,61 @@ describe('Scheduler: compatibility strategies', () => {
|
||||
.keys(),
|
||||
][0]).toBe(critical.id)
|
||||
})
|
||||
|
||||
it.each([
|
||||
'round-robin',
|
||||
'least-busy',
|
||||
'capability-match',
|
||||
'dependency-first',
|
||||
'composite',
|
||||
] satisfies SchedulingStrategy[])(
|
||||
'%s applies hard requirements before strategy-specific ranking',
|
||||
(strategy) => {
|
||||
const scheduler = new Scheduler(strategy)
|
||||
const agents: AgentConfig[] = [
|
||||
{
|
||||
...agent('ineligible'),
|
||||
capabilities: ['typescript'],
|
||||
tools: ['file_read'],
|
||||
provider: 'openai',
|
||||
},
|
||||
{
|
||||
...agent('eligible'),
|
||||
capabilities: ['typescript'],
|
||||
tools: ['file_edit'],
|
||||
provider: 'anthropic',
|
||||
},
|
||||
]
|
||||
const task = pendingTask('Restricted', {
|
||||
requires: {
|
||||
requiredTools: ['file_edit'],
|
||||
requiredCapabilities: ['typescript'],
|
||||
requiredBackend: 'llm',
|
||||
requiredProvider: 'anthropic',
|
||||
},
|
||||
})
|
||||
|
||||
expect(scheduler.schedule([task], agents).get(task.id)).toBe('eligible')
|
||||
},
|
||||
)
|
||||
|
||||
it.each([
|
||||
'round-robin',
|
||||
'least-busy',
|
||||
'capability-match',
|
||||
'dependency-first',
|
||||
'composite',
|
||||
] satisfies SchedulingStrategy[])(
|
||||
'%s fails closed when no agent satisfies hard requirements',
|
||||
(strategy) => {
|
||||
const task = pendingTask('Restricted', {
|
||||
requires: { requiredCapabilities: ['missing'] },
|
||||
})
|
||||
|
||||
expect(() => new Scheduler(strategy).schedule([task], [agent('worker')]))
|
||||
.toThrow('NO_ELIGIBLE_AGENT')
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -325,7 +384,7 @@ describe('Scheduler: composite', () => {
|
||||
expect(loadOnly.get(task.id)).toBe('idle')
|
||||
})
|
||||
|
||||
it('warns and uses the explicit zero-fit load fallback when nobody is eligible', () => {
|
||||
it('fails closed when nobody is eligible', () => {
|
||||
const warnings: unknown[] = []
|
||||
const s = new Scheduler('composite', {}, {
|
||||
onWarning: (warning) => warnings.push(warning),
|
||||
@@ -340,16 +399,8 @@ describe('Scheduler: composite', () => {
|
||||
requires: { requiredCapabilities: ['missing'] },
|
||||
})
|
||||
|
||||
const assignments = s.schedule([busy, task], agents)
|
||||
|
||||
expect(assignments.get(task.id)).toBe('idle')
|
||||
expect(warnings).toEqual([
|
||||
expect.objectContaining({
|
||||
code: 'NO_ELIGIBLE_AGENT',
|
||||
taskId: task.id,
|
||||
fallback: 'zero-fit-current-load',
|
||||
}),
|
||||
])
|
||||
expect(() => s.schedule([busy, task], agents)).toThrow('NO_ELIGIBLE_AGENT')
|
||||
expect(warnings).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -131,6 +131,146 @@ describe('TaskQueue', () => {
|
||||
expect(q.list().find((t) => t.id === 'c')!.status).toBe('failed')
|
||||
})
|
||||
|
||||
it('applies an append-only plan patch atomically and publishes events later', () => {
|
||||
const q = new TaskQueue()
|
||||
const readyHandler = vi.fn()
|
||||
const skippedHandler = vi.fn()
|
||||
q.on('task:ready', readyHandler)
|
||||
q.on('task:skipped', skippedHandler)
|
||||
q.add(task('source'))
|
||||
q.add(task('old-downstream', { dependsOn: ['source'] }))
|
||||
q.update('source', { status: 'in_progress' })
|
||||
readyHandler.mockClear()
|
||||
|
||||
const applied = q.applyPlanPatch({
|
||||
reason: 'Use a fallback source.',
|
||||
supersedePending: ['old-downstream'],
|
||||
addTasks: [
|
||||
{
|
||||
key: 'fallback',
|
||||
title: 'Fallback',
|
||||
description: 'Fetch from the fallback source.',
|
||||
assignee: 'worker',
|
||||
},
|
||||
{
|
||||
key: 'replacement',
|
||||
title: 'Replacement',
|
||||
description: 'Consume the fallback.',
|
||||
assignee: 'worker',
|
||||
dependsOn: ['fallback'],
|
||||
},
|
||||
],
|
||||
}, 'source', 'failure')
|
||||
|
||||
expect(q.getPlanRevision()).toBe(1)
|
||||
expect(q.get('source')?.recoveredByRevision).toBe(1)
|
||||
expect(q.get('old-downstream')?.status).toBe('skipped')
|
||||
expect(q.get('old-downstream')?.supersededByRevision).toBe(1)
|
||||
expect(q.get(applied.revision.addedTasks['fallback']!)?.status).toBe('pending')
|
||||
expect(q.get(applied.revision.addedTasks['replacement']!)?.status).toBe('blocked')
|
||||
expect(readyHandler).not.toHaveBeenCalled()
|
||||
expect(skippedHandler).not.toHaveBeenCalled()
|
||||
|
||||
q.publishPlanRevision(applied.revision)
|
||||
expect(readyHandler).toHaveBeenCalledTimes(1)
|
||||
expect(skippedHandler).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('rejects an invalid plan patch without changing the queue', () => {
|
||||
const q = new TaskQueue()
|
||||
q.add(task('source'))
|
||||
q.add(task('downstream', { dependsOn: ['source'] }))
|
||||
q.update('source', { status: 'in_progress' })
|
||||
const before = q.snapshot()
|
||||
|
||||
expect(() => q.applyPlanPatch({
|
||||
reason: 'Invalid repair.',
|
||||
addTasks: [{
|
||||
key: 'replacement',
|
||||
title: 'Replacement',
|
||||
description: 'Cannot depend on a missing task.',
|
||||
dependsOn: ['missing'],
|
||||
}],
|
||||
}, 'source', 'failure')).toThrow('unknown dependency')
|
||||
|
||||
expect(q.snapshot()).toEqual(before)
|
||||
})
|
||||
|
||||
it('rejects a patch outside the trigger outcome barrier', () => {
|
||||
const q = new TaskQueue()
|
||||
q.add(task('source'))
|
||||
const before = q.snapshot()
|
||||
|
||||
expect(() => q.applyPlanPatch({
|
||||
reason: 'Too early.',
|
||||
addTasks: [{
|
||||
key: 'replacement',
|
||||
title: 'Replacement',
|
||||
description: 'Must not be appended.',
|
||||
}],
|
||||
}, 'source', 'failure')).toThrow('must be in_progress')
|
||||
|
||||
expect(q.snapshot()).toEqual(before)
|
||||
})
|
||||
|
||||
it('does not classify a failure as recovered without an appended replacement', () => {
|
||||
const q = new TaskQueue()
|
||||
q.add(task('source'))
|
||||
q.add(task('downstream', { dependsOn: ['source'] }))
|
||||
const before = q.snapshot()
|
||||
|
||||
expect(() => q.applyPlanPatch({
|
||||
reason: 'Only retarget downstream.',
|
||||
retargetPending: [{ taskId: 'downstream', assignee: 'worker-b' }],
|
||||
}, 'source', 'failure')).toThrow('must append at least one replacement task')
|
||||
|
||||
expect(q.snapshot()).toEqual(before)
|
||||
})
|
||||
|
||||
it('round-trips adaptive plan revisions through snapshot v2', () => {
|
||||
const q = new TaskQueue()
|
||||
q.add(task('source'))
|
||||
q.update('source', { status: 'in_progress' })
|
||||
const { revision } = q.applyPlanPatch({
|
||||
reason: 'Add follow-up.',
|
||||
addTasks: [{
|
||||
key: 'follow-up',
|
||||
title: 'Follow-up',
|
||||
description: 'Continue after source.',
|
||||
dependsOn: ['source'],
|
||||
}],
|
||||
}, 'source', 'success')
|
||||
q.publishPlanRevision(revision)
|
||||
|
||||
const snapshot = q.snapshot()
|
||||
expect(snapshot.version).toBe(2)
|
||||
const restored = TaskQueue.fromSnapshot(snapshot)
|
||||
expect(restored.getPlanRevision()).toBe(1)
|
||||
expect(restored.getPlanRevisions()).toEqual([revision])
|
||||
expect(restored.list()).toEqual(q.list())
|
||||
})
|
||||
|
||||
it('rejects a corrupt adaptive revision sequence during restore', () => {
|
||||
const q = new TaskQueue()
|
||||
q.add(task('source'))
|
||||
q.update('source', { status: 'in_progress' })
|
||||
q.applyPlanPatch({
|
||||
reason: 'Add follow-up.',
|
||||
addTasks: [{
|
||||
key: 'follow-up',
|
||||
title: 'Follow-up',
|
||||
description: 'Continue.',
|
||||
}],
|
||||
}, 'source', 'success')
|
||||
const snapshot = q.snapshot()
|
||||
if (snapshot.version !== 2) throw new Error('expected adaptive snapshot')
|
||||
|
||||
expect(() => TaskQueue.fromSnapshot({
|
||||
...snapshot,
|
||||
planRevision: 2,
|
||||
})).toThrow('invalid plan revision sequence')
|
||||
})
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Completion
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
Reference in new issue
Block a user