mirror of
https://github.com/open-multi-agent/open-multi-agent.git
synced 2026-10-19 01:31:07 -05:00
feat(orchestrator): add declared governance and post-execution floor (#420)
This commit is contained in:
1 parent
7dcde7d8fc
commit
26be338ff7
9 files changed
+816
-2
No files matched your search
@@ -114,6 +114,28 @@ and run duration because those facts are not fields on that result type. When
|
||||
the available result or trace cannot establish a fact, the builder returns an
|
||||
empty array or `null` for that field and sets `partial: true`.
|
||||
|
||||
### Post-execution governance conclusion
|
||||
|
||||
Every OMA-produced `TeamRunResult` carries `governanceConclusion`:
|
||||
|
||||
- `satisfied` — a `governanceIntent: 'required'` run executed every declared
|
||||
role, every adjacent `requiredOrder` pair appears in the observed execution
|
||||
order and is connected by a dependency path, and declarations with two or
|
||||
more roles produced `independentReviewOccurred: true`;
|
||||
- `unsatisfied` — at least one of those required execution facts is absent;
|
||||
- `not-applicable` — governance was omitted, `none`, or `preferred`.
|
||||
|
||||
The conclusion is computed by passing the result through
|
||||
`buildExecutionReceipt()` and then `evaluateGovernance()`. It is therefore a
|
||||
post-execution topology verdict, not a judgment about answer wording or quality.
|
||||
Review labels or approval markers in model output cannot satisfy the gate.
|
||||
|
||||
`success` is intentionally independent: it continues to mean that the run
|
||||
finished without runtime errors. A run can be `success: true` and
|
||||
`governanceConclusion: 'unsatisfied'`; applications that require governance
|
||||
must gate on the latter field. In particular, multiple required roles without
|
||||
a dependency-backed review chain do not count as independent review.
|
||||
|
||||
## TraceRecord schema v2
|
||||
|
||||
The core package exports the `TraceRecord` schema used by the internal
|
||||
|
||||
@@ -2,6 +2,42 @@
|
||||
|
||||
Agents can be configured with fine-grained tool access control using presets, allowlists, and denylists.
|
||||
|
||||
## Declared governance roles in `runTeam()`
|
||||
|
||||
Tool and credential boundaries are often tied to named agents. When a goal must actually pass through specific roster roles, declare the topology structurally on the `runTeam()` call:
|
||||
|
||||
```typescript
|
||||
const result = await orchestrator.runTeam(team, 'Review this change before release.', {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
requiredOrder: ['reviewer', 'security'],
|
||||
})
|
||||
|
||||
if (result.governanceConclusion !== 'satisfied') {
|
||||
throw new Error('Required governance was not satisfied by the executed topology.')
|
||||
}
|
||||
```
|
||||
|
||||
For both `required` and `preferred`, OMA skips coordinator decomposition and the simple-goal single-agent short circuit. It creates one task for every `requiredRoles` entry, keeps each task assigned to that roster agent, and uses `requiredOrder` to create dependency edges. Each task receives the original goal unchanged; the agent's `systemPrompt`, tools, and credentials define the role. Downstream tasks receive prerequisite outputs through dependency-scoped memory.
|
||||
|
||||
The goal text is not inspected to choose this topology. The same declaration therefore produces the same roles and dependency order for English, Chinese, or any other language. Every `requiredRoles` name must exist in the team roster, and `requiredOrder`, when present, must be a permutation of those roles. Invalid declarations throw before any agent runs.
|
||||
|
||||
Use `governanceIntent: 'none'` to opt into the existing automatic `runTeam()` route explicitly. Omitting `governanceIntent` also preserves the existing behavior, including simple-goal short circuit and coordinator-generated task planning.
|
||||
|
||||
After execution, required declarations are checked against the execution
|
||||
receipt. The `governanceConclusion` value is `satisfied`, `unsatisfied`, or
|
||||
`not-applicable`. `required` is the only enforced intent;
|
||||
`preferred`, `none`, and an omitted intent return `not-applicable`. An
|
||||
`unsatisfied` conclusion means that a required role, dependency path/order, or
|
||||
independent review fact was not observed. It does not rewrite `result.success`,
|
||||
which keeps its existing runtime-error meaning, so governance-sensitive callers
|
||||
must check `governanceConclusion` explicitly.
|
||||
|
||||
The gate reads only the structured execution topology produced by
|
||||
`buildExecutionReceipt()`. Agent answer text cannot prove that another role ran
|
||||
or that an independent review occurred, even if it contains reviewer names,
|
||||
approval labels, or audit markers.
|
||||
|
||||
## Built-in tools are opt-in (default-deny)
|
||||
|
||||
Built-in tools — `bash` and the filesystem tools (`file_read`, `file_write`, `file_edit`, `grep`, `glob`) — are **default-deny**. An agent receives a built-in tool only when it is granted explicitly via `tools` (an allowlist of names) or `toolPreset`. An agent that sets **neither** resolves to **zero** built-in tools:
|
||||
|
||||
@@ -109,6 +109,18 @@ Set `OPENAI_API_KEY` for this example. For other hosted or local models, see [Pr
|
||||
|
||||
Use `planOnly` to inspect a generated task graph before execution, then `createPlanArtifact()` and `runFromPlan()` to replay it. `runConsensus()` adds a proposer→judge verification loop when one answer needs extra scrutiny.
|
||||
|
||||
When an application must enforce named independent roles, declare that governance intent instead of relying on wording in the goal:
|
||||
|
||||
```typescript
|
||||
const governed = await orchestrator.runTeam(team, 'Review the evidence and assess the risk.', {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['researcher', 'analyst'],
|
||||
requiredOrder: ['researcher', 'analyst'],
|
||||
})
|
||||
```
|
||||
|
||||
`required` and `preferred` both bypass automatic decomposition and the simple-goal short circuit. OMA creates one task per declared roster name, assigns it to that agent, and chains tasks in `requiredOrder`; dependency outputs are passed to downstream roles. The topology comes only from these structured fields, so equivalent goals in different languages use the same roles and order. `none` or an omitted `governanceIntent` preserves the existing automatic `runTeam()` behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
| Capability | What you get |
|
||||
|
||||
@@ -55,6 +55,8 @@
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export { OpenMultiAgent, executeWithRetry, computeRetryDelay } from './orchestrator/orchestrator.js'
|
||||
export { evaluateGovernance } from './orchestrator/governance.js'
|
||||
export type { GovernanceDeclaration } from './orchestrator/governance.js'
|
||||
export { Scheduler } from './orchestrator/scheduler.js'
|
||||
export type { SchedulingStrategy } from './orchestrator/scheduler.js'
|
||||
|
||||
@@ -282,6 +284,7 @@ export type {
|
||||
// Team
|
||||
TeamConfig,
|
||||
TeamRunResult,
|
||||
GovernanceConclusion,
|
||||
RunMetrics,
|
||||
RunTeamOptions,
|
||||
RunTasksOptions,
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
import type {
|
||||
AgentConfig,
|
||||
GovernanceConclusion,
|
||||
RunTaskSpec,
|
||||
RunTeamOptions,
|
||||
} from '../types.js'
|
||||
import type { ExecutionReceipt } from '../observability/execution-receipt.js'
|
||||
|
||||
/** The structured governance fields accepted by {@link RunTeamOptions}. */
|
||||
export type GovernanceDeclaration = Pick<
|
||||
RunTeamOptions,
|
||||
'governanceIntent' | 'requiredRoles' | 'requiredOrder'
|
||||
>
|
||||
|
||||
/**
|
||||
* Compare required governance with an execution receipt.
|
||||
*
|
||||
* The evaluator deliberately has no access to agent output text. Required
|
||||
* order is established by both observed start order and a dependency path
|
||||
* between every adjacent declared role.
|
||||
*/
|
||||
export function evaluateGovernance(
|
||||
declaration: GovernanceDeclaration | undefined,
|
||||
receipt: ExecutionReceipt,
|
||||
): GovernanceConclusion {
|
||||
if (declaration?.governanceIntent !== 'required') return 'not-applicable'
|
||||
|
||||
const requiredRoles = declaration.requiredRoles ?? []
|
||||
if (requiredRoles.length === 0 || new Set(requiredRoles).size !== requiredRoles.length) {
|
||||
return 'unsatisfied'
|
||||
}
|
||||
|
||||
const executedRoles = new Set(receipt.rolesExecuted)
|
||||
if (requiredRoles.some((role) => !executedRoles.has(role))) return 'unsatisfied'
|
||||
|
||||
const requiredOrder = declaration.requiredOrder
|
||||
if (requiredOrder !== undefined) {
|
||||
if (!isPermutation(requiredOrder, requiredRoles)) return 'unsatisfied'
|
||||
if (!matchesRequiredOrder(requiredOrder, receipt, executedRoles)) return 'unsatisfied'
|
||||
}
|
||||
|
||||
if (requiredRoles.length >= 2 && !receipt.independentReviewOccurred) {
|
||||
return 'unsatisfied'
|
||||
}
|
||||
|
||||
return 'satisfied'
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the explicit role topology requested by a structured governance
|
||||
* declaration. `undefined` means the caller selected the legacy automatic
|
||||
* `runTeam()` route.
|
||||
*/
|
||||
export function buildGovernanceTaskSpecs(
|
||||
goal: string,
|
||||
agentConfigs: readonly AgentConfig[],
|
||||
options?: RunTeamOptions,
|
||||
): readonly RunTaskSpec[] | undefined {
|
||||
const intent = options?.governanceIntent
|
||||
if (intent !== 'required' && intent !== 'preferred') return undefined
|
||||
|
||||
const requiredRoles = options?.requiredRoles ?? []
|
||||
if (requiredRoles.length === 0) {
|
||||
throw new Error(
|
||||
`runTeam governanceIntent '${intent}' requires at least one role in requiredRoles.`,
|
||||
)
|
||||
}
|
||||
|
||||
const duplicateRoles = findDuplicates(requiredRoles)
|
||||
if (duplicateRoles.length > 0) {
|
||||
throw new Error(
|
||||
`runTeam requiredRoles must contain unique team agent names; duplicate role(s): ${duplicateRoles.join(', ')}.`,
|
||||
)
|
||||
}
|
||||
|
||||
const rosterNames = new Set(agentConfigs.map((agent) => agent.name))
|
||||
const unknownRoles = requiredRoles.filter((role) => !rosterNames.has(role))
|
||||
if (unknownRoles.length > 0) {
|
||||
throw new Error(
|
||||
`runTeam requiredRoles must exist in the team roster; unknown role(s): ${unknownRoles.join(', ')}.`,
|
||||
)
|
||||
}
|
||||
|
||||
const requiredOrder = options?.requiredOrder
|
||||
if (requiredOrder !== undefined) {
|
||||
const declaredRoles = new Set(requiredRoles)
|
||||
const unknownOrderRoles = requiredOrder.filter((role) => !declaredRoles.has(role))
|
||||
if (unknownOrderRoles.length > 0) {
|
||||
throw new Error(
|
||||
`runTeam requiredOrder may reference only requiredRoles; invalid role(s): ${unknownOrderRoles.join(', ')}.`,
|
||||
)
|
||||
}
|
||||
|
||||
const duplicateOrderRoles = findDuplicates(requiredOrder)
|
||||
if (duplicateOrderRoles.length > 0) {
|
||||
throw new Error(
|
||||
`runTeam requiredOrder must contain each required role once; duplicate role(s): ${duplicateOrderRoles.join(', ')}.`,
|
||||
)
|
||||
}
|
||||
|
||||
if (requiredOrder.length !== requiredRoles.length) {
|
||||
const orderedRoles = new Set(requiredOrder)
|
||||
const missingRoles = requiredRoles.filter((role) => !orderedRoles.has(role))
|
||||
throw new Error(
|
||||
`runTeam requiredOrder must be a permutation of requiredRoles; missing role(s): ${missingRoles.join(', ')}.`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
const executionOrder = requiredOrder ?? requiredRoles
|
||||
// Keep task text role-neutral: assignment selects the roster agent, whose
|
||||
// systemPrompt owns the role semantics. Titles exist only for dependency IDs.
|
||||
const titleByRole = new Map(
|
||||
executionOrder.map((role, index) => [role, `Governance task ${index + 1}`]),
|
||||
)
|
||||
|
||||
return executionOrder.map((role, index) => ({
|
||||
title: titleByRole.get(role)!,
|
||||
description: goal,
|
||||
assignee: role,
|
||||
...(index > 0 && requiredOrder !== undefined
|
||||
? { dependsOn: [titleByRole.get(executionOrder[index - 1]!)!] }
|
||||
: {}),
|
||||
memoryScope: 'dependencies',
|
||||
}))
|
||||
}
|
||||
|
||||
function findDuplicates(values: readonly string[]): string[] {
|
||||
const seen = new Set<string>()
|
||||
const duplicates = new Set<string>()
|
||||
for (const value of values) {
|
||||
if (seen.has(value)) duplicates.add(value)
|
||||
seen.add(value)
|
||||
}
|
||||
return [...duplicates]
|
||||
}
|
||||
|
||||
function isPermutation(values: readonly string[], expected: readonly string[]): boolean {
|
||||
if (values.length !== expected.length || new Set(values).size !== values.length) return false
|
||||
const expectedValues = new Set(expected)
|
||||
return values.every((value) => expectedValues.has(value))
|
||||
}
|
||||
|
||||
function matchesRequiredOrder(
|
||||
requiredOrder: readonly string[],
|
||||
receipt: ExecutionReceipt,
|
||||
executedRoles: ReadonlySet<string>,
|
||||
): boolean {
|
||||
const positionByRole = new Map(receipt.executionOrder.map((role, index) => [role, index]))
|
||||
const adjacency = new Map<string, Set<string>>()
|
||||
for (const edge of receipt.dependencyEdges) {
|
||||
if (!executedRoles.has(edge.from) || !executedRoles.has(edge.to)) continue
|
||||
const targets = adjacency.get(edge.from) ?? new Set<string>()
|
||||
targets.add(edge.to)
|
||||
adjacency.set(edge.from, targets)
|
||||
}
|
||||
|
||||
for (let index = 1; index < requiredOrder.length; index++) {
|
||||
const predecessor = requiredOrder[index - 1]!
|
||||
const successor = requiredOrder[index]!
|
||||
const predecessorPosition = positionByRole.get(predecessor)
|
||||
const successorPosition = positionByRole.get(successor)
|
||||
if (
|
||||
predecessorPosition === undefined
|
||||
|| successorPosition === undefined
|
||||
|| predecessorPosition >= successorPosition
|
||||
|| !hasDependencyPath(predecessor, successor, adjacency)
|
||||
) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
function hasDependencyPath(
|
||||
from: string,
|
||||
to: string,
|
||||
adjacency: ReadonlyMap<string, ReadonlySet<string>>,
|
||||
): boolean {
|
||||
const pending = [from]
|
||||
const visited = new Set([from])
|
||||
|
||||
while (pending.length > 0) {
|
||||
const current = pending.shift()!
|
||||
for (const next of adjacency.get(current) ?? []) {
|
||||
if (next === to) return true
|
||||
if (visited.has(next)) continue
|
||||
visited.add(next)
|
||||
pending.push(next)
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
@@ -88,6 +88,7 @@ import {
|
||||
type RestoreMetadataResolution,
|
||||
} from '../observability/identity.js'
|
||||
import { classifyRunFailure, statusOnly } from '../observability/status.js'
|
||||
import { buildExecutionReceipt } from '../observability/execution-receipt.js'
|
||||
import {
|
||||
createTraceRuntime,
|
||||
LEGACY_TRACE_METADATA_ONLY,
|
||||
@@ -129,6 +130,11 @@ import {
|
||||
} from './agent-config.js'
|
||||
import { isSimpleGoal, selectBestAgent } from './short-circuit.js'
|
||||
import { executeQueue, saveRunCheckpoint } from './task-execution.js'
|
||||
import {
|
||||
buildGovernanceTaskSpecs,
|
||||
evaluateGovernance,
|
||||
type GovernanceDeclaration,
|
||||
} from './governance.js'
|
||||
import { runConsensusCore, applyConsensusDefaults, type ConsensusAgentDefaults } from './consensus.js'
|
||||
import {
|
||||
createOnlineEvaluator,
|
||||
@@ -453,6 +459,29 @@ export class OpenMultiAgent {
|
||||
options?: RunTeamOptions,
|
||||
): Promise<TeamRunResult> {
|
||||
const pendingEvaluation = this.beginOnlineEvaluation(goal)
|
||||
const agentConfigs = team.getAgents()
|
||||
const governanceTaskSpecs = buildGovernanceTaskSpecs(goal, agentConfigs, options)
|
||||
if (options?.governanceIntent === 'required' && governanceTaskSpecs === undefined) {
|
||||
throw new Error('Invariant violation: required runTeam governance must use an explicit task topology.')
|
||||
}
|
||||
if (governanceTaskSpecs !== undefined) {
|
||||
const queue = new TaskQueue()
|
||||
loadSpecsIntoQueue(governanceTaskSpecs, agentConfigs, queue)
|
||||
return this.executeExplicitTaskQueue(
|
||||
team,
|
||||
queue,
|
||||
options,
|
||||
goal,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
pendingEvaluation,
|
||||
options,
|
||||
)
|
||||
}
|
||||
|
||||
const { identity, metadata } = createRunFacts(identityOptionsForRun(options))
|
||||
const traceRuntime = this.startTrace(identity, metadata)
|
||||
const finish = (result: TeamRunResult): TeamRunResult => {
|
||||
@@ -467,7 +496,6 @@ export class OpenMultiAgent {
|
||||
this.completeOnlineEvaluation(pendingEvaluation, completedResult)
|
||||
return completedResult
|
||||
}
|
||||
const agentConfigs = team.getAgents()
|
||||
const coordinatorOverrides = options?.coordinator
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
@@ -480,7 +508,12 @@ export class OpenMultiAgent {
|
||||
// The best-matching agent is selected via keyword affinity scoring
|
||||
// (same algorithm as the `capability-match` scheduler strategy).
|
||||
// ------------------------------------------------------------------
|
||||
if (!options?.planOnly && agentConfigs.length > 0 && isSimpleGoal(goal)) {
|
||||
if (
|
||||
options?.governanceIntent !== 'required'
|
||||
&& !options?.planOnly
|
||||
&& agentConfigs.length > 0
|
||||
&& isSimpleGoal(goal)
|
||||
) {
|
||||
const bestAgent = selectBestAgent(goal, agentConfigs)
|
||||
|
||||
// Use buildAgent() + agent.run() directly instead of this.runAgent()
|
||||
@@ -1456,6 +1489,7 @@ export class OpenMultiAgent {
|
||||
identity?: RunIdentity,
|
||||
restoreMetadata?: RestoreMetadataResolution,
|
||||
pendingEvaluation?: PendingOnlineEvaluation,
|
||||
governanceDeclaration?: GovernanceDeclaration,
|
||||
): Promise<TeamRunResult> {
|
||||
const newRunFacts = identity === undefined
|
||||
? createRunFacts(identityOptionsForRun(options))
|
||||
@@ -1586,6 +1620,10 @@ export class OpenMultiAgent {
|
||||
)
|
||||
const completedResult: TeamRunResult = {
|
||||
...result,
|
||||
governanceConclusion: evaluateGovernance(
|
||||
governanceDeclaration,
|
||||
buildExecutionReceipt(result),
|
||||
),
|
||||
...(metadata !== undefined ? { metadata } : {}),
|
||||
}
|
||||
traceRuntime?.close({
|
||||
@@ -1747,6 +1785,7 @@ export class OpenMultiAgent {
|
||||
|
||||
return {
|
||||
success: status.code === 'ok',
|
||||
governanceConclusion: 'not-applicable',
|
||||
identity,
|
||||
status,
|
||||
...(errorInfo !== undefined ? { errorInfo } : {}),
|
||||
|
||||
@@ -1101,6 +1101,28 @@ export interface RunTasksOptions extends RunIdentityOptions {
|
||||
*/
|
||||
export interface RunTeamOptions extends RunTasksOptions {
|
||||
readonly coordinator?: CoordinatorConfig
|
||||
/**
|
||||
* Optional structured governance signal for this goal.
|
||||
*
|
||||
* `'required'` and `'preferred'` both bypass coordinator decomposition and
|
||||
* the simple-goal short circuit. Instead, `runTeam()` executes one task per
|
||||
* {@link requiredRoles} entry using the declared roster assignments.
|
||||
* `'none'` and an omitted value preserve the existing automatic routing
|
||||
* behavior.
|
||||
*/
|
||||
readonly governanceIntent?: 'required' | 'preferred' | 'none'
|
||||
/**
|
||||
* Agent names from the team roster that must execute independent tasks when
|
||||
* {@link governanceIntent} is `'required'` or `'preferred'`.
|
||||
*/
|
||||
readonly requiredRoles?: readonly string[]
|
||||
/**
|
||||
* Optional execution order for {@link requiredRoles}. When provided, it must
|
||||
* be a permutation of `requiredRoles`; each role depends on the previous one.
|
||||
* When omitted, the declared role tasks have no dependencies and may run in
|
||||
* parallel.
|
||||
*/
|
||||
readonly requiredOrder?: readonly string[]
|
||||
/**
|
||||
* When true, the coordinator decomposes the goal but no task agents run.
|
||||
* The returned {@link TeamRunResult} has `planOnly: true`, `success: true`,
|
||||
@@ -1148,6 +1170,9 @@ export interface RunTeamOptions extends RunTasksOptions {
|
||||
readonly verifyJudges?: readonly AgentConfig[]
|
||||
}
|
||||
|
||||
/** Post-execution result of comparing required governance with execution facts. */
|
||||
export type GovernanceConclusion = 'satisfied' | 'unsatisfied' | 'not-applicable'
|
||||
|
||||
/** Run-level aggregated metrics summary. */
|
||||
export interface RunMetrics {
|
||||
readonly totalTokens: TokenUsage
|
||||
@@ -1164,6 +1189,14 @@ export interface RunMetrics {
|
||||
/** Aggregated result for a full team run. */
|
||||
export interface TeamRunResult extends RunOutcomeFields {
|
||||
readonly success: boolean
|
||||
/**
|
||||
* Post-execution governance verdict for this run.
|
||||
*
|
||||
* OMA-produced results always set this field. It remains optional on the
|
||||
* interface so older serialized results and caller-created fixtures continue
|
||||
* to type-check. `success` retains its existing runtime-error semantics.
|
||||
*/
|
||||
readonly governanceConclusion?: GovernanceConclusion
|
||||
readonly goal?: string
|
||||
readonly tasks?: readonly TaskExecutionRecord[]
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
buildExecutionReceipt,
|
||||
evaluateGovernance,
|
||||
OpenMultiAgent,
|
||||
} from '../src/index.js'
|
||||
import type {
|
||||
AgentConfig,
|
||||
AgentRunResult,
|
||||
AgentTrace,
|
||||
ExecutionReceipt,
|
||||
GovernanceDeclaration,
|
||||
LLMAdapter,
|
||||
LLMResponse,
|
||||
RunTeamOptions,
|
||||
TaskExecutionRecord,
|
||||
} from '../src/index.js'
|
||||
|
||||
const requiredChain = {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
requiredOrder: ['reviewer', 'security'],
|
||||
} as const satisfies GovernanceDeclaration
|
||||
|
||||
function receipt(overrides: Partial<ExecutionReceipt> = {}): ExecutionReceipt {
|
||||
return {
|
||||
mode: 'single',
|
||||
rolesExecuted: [],
|
||||
executionOrder: [],
|
||||
dependencyEdges: [],
|
||||
independentRolesCount: 0,
|
||||
independentReviewOccurred: false,
|
||||
totalTokens: { input: 0, output: 0 },
|
||||
durationMs: 0,
|
||||
partial: false,
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
function textAdapter(output: string): LLMAdapter {
|
||||
return {
|
||||
name: 'governance-floor-test',
|
||||
async chat(): Promise<LLMResponse> {
|
||||
return {
|
||||
id: `response-${output}`,
|
||||
content: [{ type: 'text', text: output }],
|
||||
model: 'mock-model',
|
||||
stop_reason: 'end_turn',
|
||||
usage: { input_tokens: 1, output_tokens: 1 },
|
||||
}
|
||||
},
|
||||
async *stream() {
|
||||
yield { type: 'done' as const, data: {} }
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function errorAdapter(message: string): LLMAdapter {
|
||||
return {
|
||||
name: 'governance-floor-error-test',
|
||||
async chat(): Promise<LLMResponse> {
|
||||
throw new Error(message)
|
||||
},
|
||||
async *stream() {
|
||||
yield { type: 'done' as const, data: {} }
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function agent(name: string, adapter: LLMAdapter): AgentConfig {
|
||||
return {
|
||||
name,
|
||||
model: 'mock-model',
|
||||
systemPrompt: `You are the ${name}.`,
|
||||
adapter,
|
||||
}
|
||||
}
|
||||
|
||||
async function runTeam(
|
||||
options?: RunTeamOptions,
|
||||
reviewerAdapter: LLMAdapter = textAdapter('reviewer output'),
|
||||
securityAdapter: LLMAdapter = textAdapter('security output'),
|
||||
) {
|
||||
const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = orchestrator.createTeam('governance-floor-team', {
|
||||
name: 'governance-floor-team',
|
||||
agents: [
|
||||
agent('reviewer', reviewerAdapter),
|
||||
agent('security', securityAdapter),
|
||||
],
|
||||
sharedMemory: true,
|
||||
})
|
||||
return orchestrator.runTeam(team, 'Say hello.', options)
|
||||
}
|
||||
|
||||
function topology(tasks: readonly TaskExecutionRecord[] | undefined) {
|
||||
const records = tasks ?? []
|
||||
const assigneeById = new Map(records.map((task) => [task.id, task.assignee]))
|
||||
return records.map((task) => ({
|
||||
assignee: task.assignee,
|
||||
status: task.status,
|
||||
dependsOn: task.dependsOn.map((dependency) => assigneeById.get(dependency)),
|
||||
executed: task.metrics !== undefined,
|
||||
}))
|
||||
}
|
||||
|
||||
describe('evaluateGovernance', () => {
|
||||
it('is satisfied when all required roles execute in the declared review chain', () => {
|
||||
expect(evaluateGovernance(requiredChain, receipt({
|
||||
mode: 'multi-agent',
|
||||
rolesExecuted: ['reviewer', 'security'],
|
||||
executionOrder: ['reviewer', 'security'],
|
||||
dependencyEdges: [{ from: 'reviewer', to: 'security' }],
|
||||
independentRolesCount: 2,
|
||||
independentReviewOccurred: true,
|
||||
}))).toBe('satisfied')
|
||||
})
|
||||
|
||||
it('is unsatisfied when one required role did not execute', () => {
|
||||
const declaration = {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security', 'operator'],
|
||||
requiredOrder: ['reviewer', 'security', 'operator'],
|
||||
} as const satisfies GovernanceDeclaration
|
||||
|
||||
expect(evaluateGovernance(declaration, receipt({
|
||||
mode: 'multi-agent',
|
||||
rolesExecuted: ['reviewer', 'security'],
|
||||
executionOrder: ['reviewer', 'security'],
|
||||
dependencyEdges: [{ from: 'reviewer', to: 'security' }],
|
||||
independentRolesCount: 2,
|
||||
independentReviewOccurred: true,
|
||||
}))).toBe('unsatisfied')
|
||||
})
|
||||
|
||||
it.each([
|
||||
{
|
||||
name: 'parallel roles with no dependency edge',
|
||||
executionOrder: ['reviewer', 'security'],
|
||||
dependencyEdges: [],
|
||||
},
|
||||
{
|
||||
name: 'roles executing in the opposite dependency order',
|
||||
executionOrder: ['security', 'reviewer'],
|
||||
dependencyEdges: [{ from: 'security', to: 'reviewer' }],
|
||||
},
|
||||
{
|
||||
name: 'the declared start order with a reversed dependency edge',
|
||||
executionOrder: ['reviewer', 'security'],
|
||||
dependencyEdges: [{ from: 'security', to: 'reviewer' }],
|
||||
},
|
||||
])('is unsatisfied for $name', ({ executionOrder, dependencyEdges }) => {
|
||||
expect(evaluateGovernance(requiredChain, receipt({
|
||||
mode: 'multi-agent',
|
||||
rolesExecuted: ['reviewer', 'security'],
|
||||
executionOrder,
|
||||
dependencyEdges,
|
||||
independentRolesCount: 2,
|
||||
independentReviewOccurred: dependencyEdges.length > 0,
|
||||
}))).toBe('unsatisfied')
|
||||
})
|
||||
|
||||
it('accepts a dependency path between adjacent required roles', () => {
|
||||
expect(evaluateGovernance(requiredChain, receipt({
|
||||
mode: 'multi-agent',
|
||||
rolesExecuted: ['reviewer', 'analyst', 'security'],
|
||||
executionOrder: ['reviewer', 'analyst', 'security'],
|
||||
dependencyEdges: [
|
||||
{ from: 'reviewer', to: 'analyst' },
|
||||
{ from: 'analyst', to: 'security' },
|
||||
],
|
||||
independentRolesCount: 3,
|
||||
independentReviewOccurred: true,
|
||||
}))).toBe('satisfied')
|
||||
})
|
||||
|
||||
it('is unsatisfied when two roles are required but only one agent executed', () => {
|
||||
expect(evaluateGovernance(requiredChain, receipt({
|
||||
rolesExecuted: ['reviewer'],
|
||||
executionOrder: ['reviewer'],
|
||||
independentRolesCount: 1,
|
||||
independentReviewOccurred: false,
|
||||
}))).toBe('unsatisfied')
|
||||
})
|
||||
|
||||
it('ignores review markers in answer text and judges only the I1 receipt', () => {
|
||||
const answer: AgentRunResult = {
|
||||
success: true,
|
||||
output: 'reviewer approved; security approved; INDEPENDENT_REVIEW_COMPLETE',
|
||||
messages: [],
|
||||
tokenUsage: { input_tokens: 1, output_tokens: 1 },
|
||||
toolCalls: [],
|
||||
}
|
||||
const trace: AgentTrace = {
|
||||
type: 'agent',
|
||||
runId: 'run-1',
|
||||
spanId: 'span-1',
|
||||
agent: 'reviewer',
|
||||
startMs: 0,
|
||||
endMs: 1,
|
||||
durationMs: 1,
|
||||
turns: 1,
|
||||
tokens: { input_tokens: 1, output_tokens: 1 },
|
||||
toolCalls: 0,
|
||||
}
|
||||
|
||||
expect(evaluateGovernance(requiredChain, buildExecutionReceipt(answer, [trace])))
|
||||
.toBe('unsatisfied')
|
||||
})
|
||||
|
||||
it.each([
|
||||
{ name: 'omitted intent', declaration: {} },
|
||||
{
|
||||
name: 'none intent',
|
||||
declaration: { governanceIntent: 'none', requiredRoles: ['reviewer', 'security'] },
|
||||
},
|
||||
{
|
||||
name: 'preferred intent',
|
||||
declaration: { governanceIntent: 'preferred', requiredRoles: ['reviewer', 'security'] },
|
||||
},
|
||||
] as const)('is not applicable for $name', ({ declaration }) => {
|
||||
expect(evaluateGovernance(declaration, receipt())).toBe('not-applicable')
|
||||
})
|
||||
})
|
||||
|
||||
describe('runTeam governance floor', () => {
|
||||
it('returns satisfied for a required run with the declared executed topology', async () => {
|
||||
const result = await runTeam(requiredChain)
|
||||
const execution = buildExecutionReceipt(result)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.governanceConclusion).toBe('satisfied')
|
||||
expect(result.tasks?.some((task) => task.id === 'short-circuit')).toBe(false)
|
||||
expect(topology(result.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', status: 'completed', dependsOn: [], executed: true },
|
||||
{ assignee: 'security', status: 'completed', dependsOn: ['reviewer'], executed: true },
|
||||
])
|
||||
expect(execution).toMatchObject({
|
||||
rolesExecuted: ['reviewer', 'security'],
|
||||
executionOrder: ['reviewer', 'security'],
|
||||
dependencyEdges: [{ from: 'reviewer', to: 'security' }],
|
||||
independentReviewOccurred: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('returns unsatisfied when an upstream required task fails and a downstream role never runs', async () => {
|
||||
const result = await runTeam(
|
||||
requiredChain,
|
||||
errorAdapter('reviewer failed'),
|
||||
textAdapter('security output'),
|
||||
)
|
||||
const execution = buildExecutionReceipt(result)
|
||||
|
||||
expect(result.success).toBe(false)
|
||||
expect(result.governanceConclusion).toBe('unsatisfied')
|
||||
expect(topology(result.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', status: 'failed', dependsOn: [], executed: true },
|
||||
{ assignee: 'security', status: 'failed', dependsOn: ['reviewer'], executed: false },
|
||||
])
|
||||
expect(execution.executionOrder).not.toEqual(['reviewer', 'security'])
|
||||
})
|
||||
|
||||
it('keeps runtime success separate from an unsatisfied governance conclusion', async () => {
|
||||
const result = await runTeam({
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
})
|
||||
const execution = buildExecutionReceipt(result)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.governanceConclusion).toBe('unsatisfied')
|
||||
expect(execution.dependencyEdges).toEqual([])
|
||||
expect(execution.independentReviewOccurred).toBe(false)
|
||||
})
|
||||
|
||||
it('returns not-applicable and preserves the existing simple-goal route when undeclared', async () => {
|
||||
const result = await runTeam()
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.governanceConclusion).toBe('not-applicable')
|
||||
expect(result.tasks?.map((task) => task.id)).toEqual(['short-circuit'])
|
||||
})
|
||||
|
||||
it('does not enforce the floor for preferred governance', async () => {
|
||||
const result = await runTeam({
|
||||
...requiredChain,
|
||||
governanceIntent: 'preferred',
|
||||
})
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.governanceConclusion).toBe('not-applicable')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,181 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { OpenMultiAgent } from '../src/orchestrator/orchestrator.js'
|
||||
import type {
|
||||
AgentConfig,
|
||||
LLMAdapter,
|
||||
LLMMessage,
|
||||
LLMResponse,
|
||||
RunTeamOptions,
|
||||
TaskExecutionRecord,
|
||||
} from '../src/types.js'
|
||||
|
||||
function userPrompt(messages: LLMMessage[]): string {
|
||||
const message = [...messages].reverse().find((candidate) => candidate.role === 'user')
|
||||
return (message?.content ?? [])
|
||||
.filter((block): block is { type: 'text'; text: string } => block.type === 'text')
|
||||
.map((block) => block.text)
|
||||
.join('\n')
|
||||
}
|
||||
|
||||
function responseAdapter(reply: string, captures: string[] = []): LLMAdapter {
|
||||
return {
|
||||
name: 'governance-test',
|
||||
async chat(messages): Promise<LLMResponse> {
|
||||
captures.push(userPrompt(messages))
|
||||
return {
|
||||
id: `response-${captures.length}`,
|
||||
content: [{ type: 'text', text: reply }],
|
||||
model: 'mock-model',
|
||||
stop_reason: 'end_turn',
|
||||
usage: { input_tokens: 1, output_tokens: 1 },
|
||||
}
|
||||
},
|
||||
async *stream() {
|
||||
yield { type: 'done' as const, data: {} }
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function agent(name: string, reply: string, captures?: string[]): AgentConfig {
|
||||
return {
|
||||
name,
|
||||
model: 'mock-model',
|
||||
systemPrompt: `You are the ${name}.`,
|
||||
adapter: responseAdapter(reply, captures),
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeTopology(tasks: readonly TaskExecutionRecord[] | undefined) {
|
||||
const records = tasks ?? []
|
||||
const assigneeById = new Map(records.map((task) => [task.id, task.assignee]))
|
||||
return records.map((task) => ({
|
||||
assignee: task.assignee,
|
||||
dependsOn: task.dependsOn.map((dependencyId) => assigneeById.get(dependencyId)),
|
||||
}))
|
||||
}
|
||||
|
||||
async function runGovernedGoal(goal: string, options: RunTeamOptions) {
|
||||
const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = orchestrator.createTeam('governance-team', {
|
||||
name: 'governance-team',
|
||||
agents: [
|
||||
agent('reviewer', 'reviewer output'),
|
||||
agent('security', 'security output'),
|
||||
],
|
||||
sharedMemory: true,
|
||||
})
|
||||
return orchestrator.runTeam(team, goal, options)
|
||||
}
|
||||
|
||||
describe('runTeam declared governance intent', () => {
|
||||
const requiredGovernance = {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
requiredOrder: ['reviewer', 'security'],
|
||||
} as const satisfies RunTeamOptions
|
||||
|
||||
it('executes every required role in the declared dependency order', async () => {
|
||||
const reviewerPrompts: string[] = []
|
||||
const securityPrompts: string[] = []
|
||||
const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = orchestrator.createTeam('governance-team', {
|
||||
name: 'governance-team',
|
||||
agents: [
|
||||
agent('reviewer', 'reviewer output', reviewerPrompts),
|
||||
agent('security', 'security output', securityPrompts),
|
||||
],
|
||||
sharedMemory: true,
|
||||
})
|
||||
const goal = 'Review this transfer request.'
|
||||
|
||||
const result = await orchestrator.runTeam(team, goal, requiredGovernance)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(result.agentResults.has('coordinator')).toBe(false)
|
||||
expect(result.agentResults.has('reviewer')).toBe(true)
|
||||
expect(result.agentResults.has('security')).toBe(true)
|
||||
expect(normalizeTopology(result.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', dependsOn: [] },
|
||||
{ assignee: 'security', dependsOn: ['reviewer'] },
|
||||
])
|
||||
expect(result.tasks?.every((task) => task.description === goal)).toBe(true)
|
||||
expect(result.tasks?.every((task) => task.memoryScope === 'dependencies')).toBe(true)
|
||||
expect(reviewerPrompts).toHaveLength(1)
|
||||
expect(securityPrompts[0]).toContain('## Context from prerequisite tasks')
|
||||
expect(securityPrompts[0]).toContain('reviewer output')
|
||||
})
|
||||
|
||||
it('builds the same role topology for equivalent English and Chinese goals', async () => {
|
||||
const english = await runGovernedGoal(
|
||||
'Review the key rotation and verify that it is safe.',
|
||||
requiredGovernance,
|
||||
)
|
||||
const chinese = await runGovernedGoal(
|
||||
'审核密钥轮换并确认其安全性。',
|
||||
requiredGovernance,
|
||||
)
|
||||
|
||||
expect(normalizeTopology(english.tasks)).toEqual(normalizeTopology(chinese.tasks))
|
||||
expect(normalizeTopology(chinese.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', dependsOn: [] },
|
||||
{ assignee: 'security', dependsOn: ['reviewer'] },
|
||||
])
|
||||
})
|
||||
|
||||
it('treats preferred as the same satisfying topology in this release', async () => {
|
||||
const result = await runGovernedGoal('Check this request.', {
|
||||
...requiredGovernance,
|
||||
governanceIntent: 'preferred',
|
||||
})
|
||||
|
||||
expect(normalizeTopology(result.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', dependsOn: [] },
|
||||
{ assignee: 'security', dependsOn: ['reviewer'] },
|
||||
])
|
||||
})
|
||||
|
||||
it('leaves declared roles unordered when requiredOrder is omitted', async () => {
|
||||
const result = await runGovernedGoal('Check this request.', {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
})
|
||||
|
||||
expect(normalizeTopology(result.tasks)).toEqual([
|
||||
{ assignee: 'reviewer', dependsOn: [] },
|
||||
{ assignee: 'security', dependsOn: [] },
|
||||
])
|
||||
})
|
||||
|
||||
it('preserves the existing simple-goal route for none and an omitted intent', async () => {
|
||||
const withoutIntent = await runGovernedGoal('Say hello.', {
|
||||
requiredRoles: ['reviewer', 'security'],
|
||||
requiredOrder: ['reviewer', 'security'],
|
||||
})
|
||||
const withNone = await runGovernedGoal('Say hello.', { governanceIntent: 'none' })
|
||||
|
||||
expect(normalizeTopology(withNone.tasks)).toEqual(normalizeTopology(withoutIntent.tasks))
|
||||
expect(withNone.tasks?.map((task) => task.id)).toEqual(['short-circuit'])
|
||||
expect(withNone.agentResults.has('coordinator')).toBe(false)
|
||||
expect(withoutIntent.tasks?.map((task) => task.id)).toEqual(['short-circuit'])
|
||||
expect(withoutIntent.agentResults.has('coordinator')).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects required roles that are absent from the team roster', async () => {
|
||||
await expect(runGovernedGoal('Review this request.', {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer', 'auditor'],
|
||||
})).rejects.toThrow(
|
||||
'runTeam requiredRoles must exist in the team roster; unknown role(s): auditor.',
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects requiredOrder entries that are not declared in requiredRoles', async () => {
|
||||
await expect(runGovernedGoal('Review this request.', {
|
||||
governanceIntent: 'required',
|
||||
requiredRoles: ['reviewer'],
|
||||
requiredOrder: ['security'],
|
||||
})).rejects.toThrow(
|
||||
'runTeam requiredOrder may reference only requiredRoles; invalid role(s): security.',
|
||||
)
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user