diff --git a/docs/observability.md b/docs/observability.md index 4e16a24..dd94d54 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -114,6 +114,28 @@ and run duration because those facts are not fields on that result type. When the available result or trace cannot establish a fact, the builder returns an empty array or `null` for that field and sets `partial: true`. +### Post-execution governance conclusion + +Every OMA-produced `TeamRunResult` carries `governanceConclusion`: + +- `satisfied` — a `governanceIntent: 'required'` run executed every declared + role, every adjacent `requiredOrder` pair appears in the observed execution + order and is connected by a dependency path, and declarations with two or + more roles produced `independentReviewOccurred: true`; +- `unsatisfied` — at least one of those required execution facts is absent; +- `not-applicable` — governance was omitted, `none`, or `preferred`. + +The conclusion is computed by passing the result through +`buildExecutionReceipt()` and then `evaluateGovernance()`. It is therefore a +post-execution topology verdict, not a judgment about answer wording or quality. +Review labels or approval markers in model output cannot satisfy the gate. + +`success` is intentionally independent: it continues to mean that the run +finished without runtime errors. A run can be `success: true` and +`governanceConclusion: 'unsatisfied'`; applications that require governance +must gate on the latter field. In particular, multiple required roles without +a dependency-backed review chain do not count as independent review. + ## TraceRecord schema v2 The core package exports the `TraceRecord` schema used by the internal diff --git a/docs/tool-configuration.md b/docs/tool-configuration.md index 4371794..fe4b4f3 100644 --- a/docs/tool-configuration.md +++ b/docs/tool-configuration.md @@ -2,6 +2,42 @@ Agents can be configured with fine-grained tool access control using presets, allowlists, and denylists. +## Declared governance roles in `runTeam()` + +Tool and credential boundaries are often tied to named agents. When a goal must actually pass through specific roster roles, declare the topology structurally on the `runTeam()` call: + +```typescript +const result = await orchestrator.runTeam(team, 'Review this change before release.', { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security'], + requiredOrder: ['reviewer', 'security'], +}) + +if (result.governanceConclusion !== 'satisfied') { + throw new Error('Required governance was not satisfied by the executed topology.') +} +``` + +For both `required` and `preferred`, OMA skips coordinator decomposition and the simple-goal single-agent short circuit. It creates one task for every `requiredRoles` entry, keeps each task assigned to that roster agent, and uses `requiredOrder` to create dependency edges. Each task receives the original goal unchanged; the agent's `systemPrompt`, tools, and credentials define the role. Downstream tasks receive prerequisite outputs through dependency-scoped memory. + +The goal text is not inspected to choose this topology. The same declaration therefore produces the same roles and dependency order for English, Chinese, or any other language. Every `requiredRoles` name must exist in the team roster, and `requiredOrder`, when present, must be a permutation of those roles. Invalid declarations throw before any agent runs. + +Use `governanceIntent: 'none'` to opt into the existing automatic `runTeam()` route explicitly. Omitting `governanceIntent` also preserves the existing behavior, including simple-goal short circuit and coordinator-generated task planning. + +After execution, required declarations are checked against the execution +receipt. The `governanceConclusion` value is `satisfied`, `unsatisfied`, or +`not-applicable`. `required` is the only enforced intent; +`preferred`, `none`, and an omitted intent return `not-applicable`. An +`unsatisfied` conclusion means that a required role, dependency path/order, or +independent review fact was not observed. It does not rewrite `result.success`, +which keeps its existing runtime-error meaning, so governance-sensitive callers +must check `governanceConclusion` explicitly. + +The gate reads only the structured execution topology produced by +`buildExecutionReceipt()`. Agent answer text cannot prove that another role ran +or that an independent review occurred, even if it contains reviewer names, +approval labels, or audit markers. + ## Built-in tools are opt-in (default-deny) Built-in tools — `bash` and the filesystem tools (`file_read`, `file_write`, `file_edit`, `grep`, `glob`) — are **default-deny**. An agent receives a built-in tool only when it is granted explicitly via `tools` (an allowlist of names) or `toolPreset`. An agent that sets **neither** resolves to **zero** built-in tools: diff --git a/packages/core/README.md b/packages/core/README.md index 1c4dda1..0a3cca8 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -109,6 +109,18 @@ Set `OPENAI_API_KEY` for this example. For other hosted or local models, see [Pr Use `planOnly` to inspect a generated task graph before execution, then `createPlanArtifact()` and `runFromPlan()` to replay it. `runConsensus()` adds a proposer→judge verification loop when one answer needs extra scrutiny. +When an application must enforce named independent roles, declare that governance intent instead of relying on wording in the goal: + +```typescript +const governed = await orchestrator.runTeam(team, 'Review the evidence and assess the risk.', { + governanceIntent: 'required', + requiredRoles: ['researcher', 'analyst'], + requiredOrder: ['researcher', 'analyst'], +}) +``` + +`required` and `preferred` both bypass automatic decomposition and the simple-goal short circuit. OMA creates one task per declared roster name, assigns it to that agent, and chains tasks in `requiredOrder`; dependency outputs are passed to downstream roles. The topology comes only from these structured fields, so equivalent goals in different languages use the same roles and order. `none` or an omitted `governanceIntent` preserves the existing automatic `runTeam()` behavior. + ## Capabilities | Capability | What you get | diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 27b3098..9ba19f1 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -55,6 +55,8 @@ // --------------------------------------------------------------------------- export { OpenMultiAgent, executeWithRetry, computeRetryDelay } from './orchestrator/orchestrator.js' +export { evaluateGovernance } from './orchestrator/governance.js' +export type { GovernanceDeclaration } from './orchestrator/governance.js' export { Scheduler } from './orchestrator/scheduler.js' export type { SchedulingStrategy } from './orchestrator/scheduler.js' @@ -282,6 +284,7 @@ export type { // Team TeamConfig, TeamRunResult, + GovernanceConclusion, RunMetrics, RunTeamOptions, RunTasksOptions, diff --git a/packages/core/src/orchestrator/governance.ts b/packages/core/src/orchestrator/governance.ts new file mode 100644 index 0000000..b20a814 --- /dev/null +++ b/packages/core/src/orchestrator/governance.ts @@ -0,0 +1,195 @@ +import type { + AgentConfig, + GovernanceConclusion, + RunTaskSpec, + RunTeamOptions, +} from '../types.js' +import type { ExecutionReceipt } from '../observability/execution-receipt.js' + +/** The structured governance fields accepted by {@link RunTeamOptions}. */ +export type GovernanceDeclaration = Pick< + RunTeamOptions, + 'governanceIntent' | 'requiredRoles' | 'requiredOrder' +> + +/** + * Compare required governance with an execution receipt. + * + * The evaluator deliberately has no access to agent output text. Required + * order is established by both observed start order and a dependency path + * between every adjacent declared role. + */ +export function evaluateGovernance( + declaration: GovernanceDeclaration | undefined, + receipt: ExecutionReceipt, +): GovernanceConclusion { + if (declaration?.governanceIntent !== 'required') return 'not-applicable' + + const requiredRoles = declaration.requiredRoles ?? [] + if (requiredRoles.length === 0 || new Set(requiredRoles).size !== requiredRoles.length) { + return 'unsatisfied' + } + + const executedRoles = new Set(receipt.rolesExecuted) + if (requiredRoles.some((role) => !executedRoles.has(role))) return 'unsatisfied' + + const requiredOrder = declaration.requiredOrder + if (requiredOrder !== undefined) { + if (!isPermutation(requiredOrder, requiredRoles)) return 'unsatisfied' + if (!matchesRequiredOrder(requiredOrder, receipt, executedRoles)) return 'unsatisfied' + } + + if (requiredRoles.length >= 2 && !receipt.independentReviewOccurred) { + return 'unsatisfied' + } + + return 'satisfied' +} + +/** + * Build the explicit role topology requested by a structured governance + * declaration. `undefined` means the caller selected the legacy automatic + * `runTeam()` route. + */ +export function buildGovernanceTaskSpecs( + goal: string, + agentConfigs: readonly AgentConfig[], + options?: RunTeamOptions, +): readonly RunTaskSpec[] | undefined { + const intent = options?.governanceIntent + if (intent !== 'required' && intent !== 'preferred') return undefined + + const requiredRoles = options?.requiredRoles ?? [] + if (requiredRoles.length === 0) { + throw new Error( + `runTeam governanceIntent '${intent}' requires at least one role in requiredRoles.`, + ) + } + + const duplicateRoles = findDuplicates(requiredRoles) + if (duplicateRoles.length > 0) { + throw new Error( + `runTeam requiredRoles must contain unique team agent names; duplicate role(s): ${duplicateRoles.join(', ')}.`, + ) + } + + const rosterNames = new Set(agentConfigs.map((agent) => agent.name)) + const unknownRoles = requiredRoles.filter((role) => !rosterNames.has(role)) + if (unknownRoles.length > 0) { + throw new Error( + `runTeam requiredRoles must exist in the team roster; unknown role(s): ${unknownRoles.join(', ')}.`, + ) + } + + const requiredOrder = options?.requiredOrder + if (requiredOrder !== undefined) { + const declaredRoles = new Set(requiredRoles) + const unknownOrderRoles = requiredOrder.filter((role) => !declaredRoles.has(role)) + if (unknownOrderRoles.length > 0) { + throw new Error( + `runTeam requiredOrder may reference only requiredRoles; invalid role(s): ${unknownOrderRoles.join(', ')}.`, + ) + } + + const duplicateOrderRoles = findDuplicates(requiredOrder) + if (duplicateOrderRoles.length > 0) { + throw new Error( + `runTeam requiredOrder must contain each required role once; duplicate role(s): ${duplicateOrderRoles.join(', ')}.`, + ) + } + + if (requiredOrder.length !== requiredRoles.length) { + const orderedRoles = new Set(requiredOrder) + const missingRoles = requiredRoles.filter((role) => !orderedRoles.has(role)) + throw new Error( + `runTeam requiredOrder must be a permutation of requiredRoles; missing role(s): ${missingRoles.join(', ')}.`, + ) + } + } + + const executionOrder = requiredOrder ?? requiredRoles + // Keep task text role-neutral: assignment selects the roster agent, whose + // systemPrompt owns the role semantics. Titles exist only for dependency IDs. + const titleByRole = new Map( + executionOrder.map((role, index) => [role, `Governance task ${index + 1}`]), + ) + + return executionOrder.map((role, index) => ({ + title: titleByRole.get(role)!, + description: goal, + assignee: role, + ...(index > 0 && requiredOrder !== undefined + ? { dependsOn: [titleByRole.get(executionOrder[index - 1]!)!] } + : {}), + memoryScope: 'dependencies', + })) +} + +function findDuplicates(values: readonly string[]): string[] { + const seen = new Set() + const duplicates = new Set() + for (const value of values) { + if (seen.has(value)) duplicates.add(value) + seen.add(value) + } + return [...duplicates] +} + +function isPermutation(values: readonly string[], expected: readonly string[]): boolean { + if (values.length !== expected.length || new Set(values).size !== values.length) return false + const expectedValues = new Set(expected) + return values.every((value) => expectedValues.has(value)) +} + +function matchesRequiredOrder( + requiredOrder: readonly string[], + receipt: ExecutionReceipt, + executedRoles: ReadonlySet, +): boolean { + const positionByRole = new Map(receipt.executionOrder.map((role, index) => [role, index])) + const adjacency = new Map>() + for (const edge of receipt.dependencyEdges) { + if (!executedRoles.has(edge.from) || !executedRoles.has(edge.to)) continue + const targets = adjacency.get(edge.from) ?? new Set() + targets.add(edge.to) + adjacency.set(edge.from, targets) + } + + for (let index = 1; index < requiredOrder.length; index++) { + const predecessor = requiredOrder[index - 1]! + const successor = requiredOrder[index]! + const predecessorPosition = positionByRole.get(predecessor) + const successorPosition = positionByRole.get(successor) + if ( + predecessorPosition === undefined + || successorPosition === undefined + || predecessorPosition >= successorPosition + || !hasDependencyPath(predecessor, successor, adjacency) + ) { + return false + } + } + + return true +} + +function hasDependencyPath( + from: string, + to: string, + adjacency: ReadonlyMap>, +): boolean { + const pending = [from] + const visited = new Set([from]) + + while (pending.length > 0) { + const current = pending.shift()! + for (const next of adjacency.get(current) ?? []) { + if (next === to) return true + if (visited.has(next)) continue + visited.add(next) + pending.push(next) + } + } + + return false +} diff --git a/packages/core/src/orchestrator/orchestrator.ts b/packages/core/src/orchestrator/orchestrator.ts index 2efcc65..af46e17 100644 --- a/packages/core/src/orchestrator/orchestrator.ts +++ b/packages/core/src/orchestrator/orchestrator.ts @@ -88,6 +88,7 @@ import { type RestoreMetadataResolution, } from '../observability/identity.js' import { classifyRunFailure, statusOnly } from '../observability/status.js' +import { buildExecutionReceipt } from '../observability/execution-receipt.js' import { createTraceRuntime, LEGACY_TRACE_METADATA_ONLY, @@ -129,6 +130,11 @@ import { } from './agent-config.js' import { isSimpleGoal, selectBestAgent } from './short-circuit.js' import { executeQueue, saveRunCheckpoint } from './task-execution.js' +import { + buildGovernanceTaskSpecs, + evaluateGovernance, + type GovernanceDeclaration, +} from './governance.js' import { runConsensusCore, applyConsensusDefaults, type ConsensusAgentDefaults } from './consensus.js' import { createOnlineEvaluator, @@ -453,6 +459,29 @@ export class OpenMultiAgent { options?: RunTeamOptions, ): Promise { const pendingEvaluation = this.beginOnlineEvaluation(goal) + const agentConfigs = team.getAgents() + const governanceTaskSpecs = buildGovernanceTaskSpecs(goal, agentConfigs, options) + if (options?.governanceIntent === 'required' && governanceTaskSpecs === undefined) { + throw new Error('Invariant violation: required runTeam governance must use an explicit task topology.') + } + if (governanceTaskSpecs !== undefined) { + const queue = new TaskQueue() + loadSpecsIntoQueue(governanceTaskSpecs, agentConfigs, queue) + return this.executeExplicitTaskQueue( + team, + queue, + options, + goal, + undefined, + undefined, + undefined, + undefined, + undefined, + pendingEvaluation, + options, + ) + } + const { identity, metadata } = createRunFacts(identityOptionsForRun(options)) const traceRuntime = this.startTrace(identity, metadata) const finish = (result: TeamRunResult): TeamRunResult => { @@ -467,7 +496,6 @@ export class OpenMultiAgent { this.completeOnlineEvaluation(pendingEvaluation, completedResult) return completedResult } - const agentConfigs = team.getAgents() const coordinatorOverrides = options?.coordinator // ------------------------------------------------------------------ @@ -480,7 +508,12 @@ export class OpenMultiAgent { // The best-matching agent is selected via keyword affinity scoring // (same algorithm as the `capability-match` scheduler strategy). // ------------------------------------------------------------------ - if (!options?.planOnly && agentConfigs.length > 0 && isSimpleGoal(goal)) { + if ( + options?.governanceIntent !== 'required' + && !options?.planOnly + && agentConfigs.length > 0 + && isSimpleGoal(goal) + ) { const bestAgent = selectBestAgent(goal, agentConfigs) // Use buildAgent() + agent.run() directly instead of this.runAgent() @@ -1456,6 +1489,7 @@ export class OpenMultiAgent { identity?: RunIdentity, restoreMetadata?: RestoreMetadataResolution, pendingEvaluation?: PendingOnlineEvaluation, + governanceDeclaration?: GovernanceDeclaration, ): Promise { const newRunFacts = identity === undefined ? createRunFacts(identityOptionsForRun(options)) @@ -1586,6 +1620,10 @@ export class OpenMultiAgent { ) const completedResult: TeamRunResult = { ...result, + governanceConclusion: evaluateGovernance( + governanceDeclaration, + buildExecutionReceipt(result), + ), ...(metadata !== undefined ? { metadata } : {}), } traceRuntime?.close({ @@ -1747,6 +1785,7 @@ export class OpenMultiAgent { return { success: status.code === 'ok', + governanceConclusion: 'not-applicable', identity, status, ...(errorInfo !== undefined ? { errorInfo } : {}), diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 0c89ede..4beb84e 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -1101,6 +1101,28 @@ export interface RunTasksOptions extends RunIdentityOptions { */ export interface RunTeamOptions extends RunTasksOptions { readonly coordinator?: CoordinatorConfig + /** + * Optional structured governance signal for this goal. + * + * `'required'` and `'preferred'` both bypass coordinator decomposition and + * the simple-goal short circuit. Instead, `runTeam()` executes one task per + * {@link requiredRoles} entry using the declared roster assignments. + * `'none'` and an omitted value preserve the existing automatic routing + * behavior. + */ + readonly governanceIntent?: 'required' | 'preferred' | 'none' + /** + * Agent names from the team roster that must execute independent tasks when + * {@link governanceIntent} is `'required'` or `'preferred'`. + */ + readonly requiredRoles?: readonly string[] + /** + * Optional execution order for {@link requiredRoles}. When provided, it must + * be a permutation of `requiredRoles`; each role depends on the previous one. + * When omitted, the declared role tasks have no dependencies and may run in + * parallel. + */ + readonly requiredOrder?: readonly string[] /** * When true, the coordinator decomposes the goal but no task agents run. * The returned {@link TeamRunResult} has `planOnly: true`, `success: true`, @@ -1148,6 +1170,9 @@ export interface RunTeamOptions extends RunTasksOptions { readonly verifyJudges?: readonly AgentConfig[] } +/** Post-execution result of comparing required governance with execution facts. */ +export type GovernanceConclusion = 'satisfied' | 'unsatisfied' | 'not-applicable' + /** Run-level aggregated metrics summary. */ export interface RunMetrics { readonly totalTokens: TokenUsage @@ -1164,6 +1189,14 @@ export interface RunMetrics { /** Aggregated result for a full team run. */ export interface TeamRunResult extends RunOutcomeFields { readonly success: boolean + /** + * Post-execution governance verdict for this run. + * + * OMA-produced results always set this field. It remains optional on the + * interface so older serialized results and caller-created fixtures continue + * to type-check. `success` retains its existing runtime-error semantics. + */ + readonly governanceConclusion?: GovernanceConclusion readonly goal?: string readonly tasks?: readonly TaskExecutionRecord[] /** diff --git a/packages/core/tests/governance-floor.test.ts b/packages/core/tests/governance-floor.test.ts new file mode 100644 index 0000000..de6d8c9 --- /dev/null +++ b/packages/core/tests/governance-floor.test.ts @@ -0,0 +1,293 @@ +import { describe, expect, it } from 'vitest' +import { + buildExecutionReceipt, + evaluateGovernance, + OpenMultiAgent, +} from '../src/index.js' +import type { + AgentConfig, + AgentRunResult, + AgentTrace, + ExecutionReceipt, + GovernanceDeclaration, + LLMAdapter, + LLMResponse, + RunTeamOptions, + TaskExecutionRecord, +} from '../src/index.js' + +const requiredChain = { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security'], + requiredOrder: ['reviewer', 'security'], +} as const satisfies GovernanceDeclaration + +function receipt(overrides: Partial = {}): ExecutionReceipt { + return { + mode: 'single', + rolesExecuted: [], + executionOrder: [], + dependencyEdges: [], + independentRolesCount: 0, + independentReviewOccurred: false, + totalTokens: { input: 0, output: 0 }, + durationMs: 0, + partial: false, + ...overrides, + } +} + +function textAdapter(output: string): LLMAdapter { + return { + name: 'governance-floor-test', + async chat(): Promise { + return { + id: `response-${output}`, + content: [{ type: 'text', text: output }], + model: 'mock-model', + stop_reason: 'end_turn', + usage: { input_tokens: 1, output_tokens: 1 }, + } + }, + async *stream() { + yield { type: 'done' as const, data: {} } + }, + } +} + +function errorAdapter(message: string): LLMAdapter { + return { + name: 'governance-floor-error-test', + async chat(): Promise { + throw new Error(message) + }, + async *stream() { + yield { type: 'done' as const, data: {} } + }, + } +} + +function agent(name: string, adapter: LLMAdapter): AgentConfig { + return { + name, + model: 'mock-model', + systemPrompt: `You are the ${name}.`, + adapter, + } +} + +async function runTeam( + options?: RunTeamOptions, + reviewerAdapter: LLMAdapter = textAdapter('reviewer output'), + securityAdapter: LLMAdapter = textAdapter('security output'), +) { + const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' }) + const team = orchestrator.createTeam('governance-floor-team', { + name: 'governance-floor-team', + agents: [ + agent('reviewer', reviewerAdapter), + agent('security', securityAdapter), + ], + sharedMemory: true, + }) + return orchestrator.runTeam(team, 'Say hello.', options) +} + +function topology(tasks: readonly TaskExecutionRecord[] | undefined) { + const records = tasks ?? [] + const assigneeById = new Map(records.map((task) => [task.id, task.assignee])) + return records.map((task) => ({ + assignee: task.assignee, + status: task.status, + dependsOn: task.dependsOn.map((dependency) => assigneeById.get(dependency)), + executed: task.metrics !== undefined, + })) +} + +describe('evaluateGovernance', () => { + it('is satisfied when all required roles execute in the declared review chain', () => { + expect(evaluateGovernance(requiredChain, receipt({ + mode: 'multi-agent', + rolesExecuted: ['reviewer', 'security'], + executionOrder: ['reviewer', 'security'], + dependencyEdges: [{ from: 'reviewer', to: 'security' }], + independentRolesCount: 2, + independentReviewOccurred: true, + }))).toBe('satisfied') + }) + + it('is unsatisfied when one required role did not execute', () => { + const declaration = { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security', 'operator'], + requiredOrder: ['reviewer', 'security', 'operator'], + } as const satisfies GovernanceDeclaration + + expect(evaluateGovernance(declaration, receipt({ + mode: 'multi-agent', + rolesExecuted: ['reviewer', 'security'], + executionOrder: ['reviewer', 'security'], + dependencyEdges: [{ from: 'reviewer', to: 'security' }], + independentRolesCount: 2, + independentReviewOccurred: true, + }))).toBe('unsatisfied') + }) + + it.each([ + { + name: 'parallel roles with no dependency edge', + executionOrder: ['reviewer', 'security'], + dependencyEdges: [], + }, + { + name: 'roles executing in the opposite dependency order', + executionOrder: ['security', 'reviewer'], + dependencyEdges: [{ from: 'security', to: 'reviewer' }], + }, + { + name: 'the declared start order with a reversed dependency edge', + executionOrder: ['reviewer', 'security'], + dependencyEdges: [{ from: 'security', to: 'reviewer' }], + }, + ])('is unsatisfied for $name', ({ executionOrder, dependencyEdges }) => { + expect(evaluateGovernance(requiredChain, receipt({ + mode: 'multi-agent', + rolesExecuted: ['reviewer', 'security'], + executionOrder, + dependencyEdges, + independentRolesCount: 2, + independentReviewOccurred: dependencyEdges.length > 0, + }))).toBe('unsatisfied') + }) + + it('accepts a dependency path between adjacent required roles', () => { + expect(evaluateGovernance(requiredChain, receipt({ + mode: 'multi-agent', + rolesExecuted: ['reviewer', 'analyst', 'security'], + executionOrder: ['reviewer', 'analyst', 'security'], + dependencyEdges: [ + { from: 'reviewer', to: 'analyst' }, + { from: 'analyst', to: 'security' }, + ], + independentRolesCount: 3, + independentReviewOccurred: true, + }))).toBe('satisfied') + }) + + it('is unsatisfied when two roles are required but only one agent executed', () => { + expect(evaluateGovernance(requiredChain, receipt({ + rolesExecuted: ['reviewer'], + executionOrder: ['reviewer'], + independentRolesCount: 1, + independentReviewOccurred: false, + }))).toBe('unsatisfied') + }) + + it('ignores review markers in answer text and judges only the I1 receipt', () => { + const answer: AgentRunResult = { + success: true, + output: 'reviewer approved; security approved; INDEPENDENT_REVIEW_COMPLETE', + messages: [], + tokenUsage: { input_tokens: 1, output_tokens: 1 }, + toolCalls: [], + } + const trace: AgentTrace = { + type: 'agent', + runId: 'run-1', + spanId: 'span-1', + agent: 'reviewer', + startMs: 0, + endMs: 1, + durationMs: 1, + turns: 1, + tokens: { input_tokens: 1, output_tokens: 1 }, + toolCalls: 0, + } + + expect(evaluateGovernance(requiredChain, buildExecutionReceipt(answer, [trace]))) + .toBe('unsatisfied') + }) + + it.each([ + { name: 'omitted intent', declaration: {} }, + { + name: 'none intent', + declaration: { governanceIntent: 'none', requiredRoles: ['reviewer', 'security'] }, + }, + { + name: 'preferred intent', + declaration: { governanceIntent: 'preferred', requiredRoles: ['reviewer', 'security'] }, + }, + ] as const)('is not applicable for $name', ({ declaration }) => { + expect(evaluateGovernance(declaration, receipt())).toBe('not-applicable') + }) +}) + +describe('runTeam governance floor', () => { + it('returns satisfied for a required run with the declared executed topology', async () => { + const result = await runTeam(requiredChain) + const execution = buildExecutionReceipt(result) + + expect(result.success).toBe(true) + expect(result.governanceConclusion).toBe('satisfied') + expect(result.tasks?.some((task) => task.id === 'short-circuit')).toBe(false) + expect(topology(result.tasks)).toEqual([ + { assignee: 'reviewer', status: 'completed', dependsOn: [], executed: true }, + { assignee: 'security', status: 'completed', dependsOn: ['reviewer'], executed: true }, + ]) + expect(execution).toMatchObject({ + rolesExecuted: ['reviewer', 'security'], + executionOrder: ['reviewer', 'security'], + dependencyEdges: [{ from: 'reviewer', to: 'security' }], + independentReviewOccurred: true, + }) + }) + + it('returns unsatisfied when an upstream required task fails and a downstream role never runs', async () => { + const result = await runTeam( + requiredChain, + errorAdapter('reviewer failed'), + textAdapter('security output'), + ) + const execution = buildExecutionReceipt(result) + + expect(result.success).toBe(false) + expect(result.governanceConclusion).toBe('unsatisfied') + expect(topology(result.tasks)).toEqual([ + { assignee: 'reviewer', status: 'failed', dependsOn: [], executed: true }, + { assignee: 'security', status: 'failed', dependsOn: ['reviewer'], executed: false }, + ]) + expect(execution.executionOrder).not.toEqual(['reviewer', 'security']) + }) + + it('keeps runtime success separate from an unsatisfied governance conclusion', async () => { + const result = await runTeam({ + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security'], + }) + const execution = buildExecutionReceipt(result) + + expect(result.success).toBe(true) + expect(result.governanceConclusion).toBe('unsatisfied') + expect(execution.dependencyEdges).toEqual([]) + expect(execution.independentReviewOccurred).toBe(false) + }) + + it('returns not-applicable and preserves the existing simple-goal route when undeclared', async () => { + const result = await runTeam() + + expect(result.success).toBe(true) + expect(result.governanceConclusion).toBe('not-applicable') + expect(result.tasks?.map((task) => task.id)).toEqual(['short-circuit']) + }) + + it('does not enforce the floor for preferred governance', async () => { + const result = await runTeam({ + ...requiredChain, + governanceIntent: 'preferred', + }) + + expect(result.success).toBe(true) + expect(result.governanceConclusion).toBe('not-applicable') + }) +}) diff --git a/packages/core/tests/governance-intent.test.ts b/packages/core/tests/governance-intent.test.ts new file mode 100644 index 0000000..01ee338 --- /dev/null +++ b/packages/core/tests/governance-intent.test.ts @@ -0,0 +1,181 @@ +import { describe, expect, it } from 'vitest' +import { OpenMultiAgent } from '../src/orchestrator/orchestrator.js' +import type { + AgentConfig, + LLMAdapter, + LLMMessage, + LLMResponse, + RunTeamOptions, + TaskExecutionRecord, +} from '../src/types.js' + +function userPrompt(messages: LLMMessage[]): string { + const message = [...messages].reverse().find((candidate) => candidate.role === 'user') + return (message?.content ?? []) + .filter((block): block is { type: 'text'; text: string } => block.type === 'text') + .map((block) => block.text) + .join('\n') +} + +function responseAdapter(reply: string, captures: string[] = []): LLMAdapter { + return { + name: 'governance-test', + async chat(messages): Promise { + captures.push(userPrompt(messages)) + return { + id: `response-${captures.length}`, + content: [{ type: 'text', text: reply }], + model: 'mock-model', + stop_reason: 'end_turn', + usage: { input_tokens: 1, output_tokens: 1 }, + } + }, + async *stream() { + yield { type: 'done' as const, data: {} } + }, + } +} + +function agent(name: string, reply: string, captures?: string[]): AgentConfig { + return { + name, + model: 'mock-model', + systemPrompt: `You are the ${name}.`, + adapter: responseAdapter(reply, captures), + } +} + +function normalizeTopology(tasks: readonly TaskExecutionRecord[] | undefined) { + const records = tasks ?? [] + const assigneeById = new Map(records.map((task) => [task.id, task.assignee])) + return records.map((task) => ({ + assignee: task.assignee, + dependsOn: task.dependsOn.map((dependencyId) => assigneeById.get(dependencyId)), + })) +} + +async function runGovernedGoal(goal: string, options: RunTeamOptions) { + const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' }) + const team = orchestrator.createTeam('governance-team', { + name: 'governance-team', + agents: [ + agent('reviewer', 'reviewer output'), + agent('security', 'security output'), + ], + sharedMemory: true, + }) + return orchestrator.runTeam(team, goal, options) +} + +describe('runTeam declared governance intent', () => { + const requiredGovernance = { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security'], + requiredOrder: ['reviewer', 'security'], + } as const satisfies RunTeamOptions + + it('executes every required role in the declared dependency order', async () => { + const reviewerPrompts: string[] = [] + const securityPrompts: string[] = [] + const orchestrator = new OpenMultiAgent({ defaultModel: 'mock-model' }) + const team = orchestrator.createTeam('governance-team', { + name: 'governance-team', + agents: [ + agent('reviewer', 'reviewer output', reviewerPrompts), + agent('security', 'security output', securityPrompts), + ], + sharedMemory: true, + }) + const goal = 'Review this transfer request.' + + const result = await orchestrator.runTeam(team, goal, requiredGovernance) + + expect(result.success).toBe(true) + expect(result.agentResults.has('coordinator')).toBe(false) + expect(result.agentResults.has('reviewer')).toBe(true) + expect(result.agentResults.has('security')).toBe(true) + expect(normalizeTopology(result.tasks)).toEqual([ + { assignee: 'reviewer', dependsOn: [] }, + { assignee: 'security', dependsOn: ['reviewer'] }, + ]) + expect(result.tasks?.every((task) => task.description === goal)).toBe(true) + expect(result.tasks?.every((task) => task.memoryScope === 'dependencies')).toBe(true) + expect(reviewerPrompts).toHaveLength(1) + expect(securityPrompts[0]).toContain('## Context from prerequisite tasks') + expect(securityPrompts[0]).toContain('reviewer output') + }) + + it('builds the same role topology for equivalent English and Chinese goals', async () => { + const english = await runGovernedGoal( + 'Review the key rotation and verify that it is safe.', + requiredGovernance, + ) + const chinese = await runGovernedGoal( + '审核密钥轮换并确认其安全性。', + requiredGovernance, + ) + + expect(normalizeTopology(english.tasks)).toEqual(normalizeTopology(chinese.tasks)) + expect(normalizeTopology(chinese.tasks)).toEqual([ + { assignee: 'reviewer', dependsOn: [] }, + { assignee: 'security', dependsOn: ['reviewer'] }, + ]) + }) + + it('treats preferred as the same satisfying topology in this release', async () => { + const result = await runGovernedGoal('Check this request.', { + ...requiredGovernance, + governanceIntent: 'preferred', + }) + + expect(normalizeTopology(result.tasks)).toEqual([ + { assignee: 'reviewer', dependsOn: [] }, + { assignee: 'security', dependsOn: ['reviewer'] }, + ]) + }) + + it('leaves declared roles unordered when requiredOrder is omitted', async () => { + const result = await runGovernedGoal('Check this request.', { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'security'], + }) + + expect(normalizeTopology(result.tasks)).toEqual([ + { assignee: 'reviewer', dependsOn: [] }, + { assignee: 'security', dependsOn: [] }, + ]) + }) + + it('preserves the existing simple-goal route for none and an omitted intent', async () => { + const withoutIntent = await runGovernedGoal('Say hello.', { + requiredRoles: ['reviewer', 'security'], + requiredOrder: ['reviewer', 'security'], + }) + const withNone = await runGovernedGoal('Say hello.', { governanceIntent: 'none' }) + + expect(normalizeTopology(withNone.tasks)).toEqual(normalizeTopology(withoutIntent.tasks)) + expect(withNone.tasks?.map((task) => task.id)).toEqual(['short-circuit']) + expect(withNone.agentResults.has('coordinator')).toBe(false) + expect(withoutIntent.tasks?.map((task) => task.id)).toEqual(['short-circuit']) + expect(withoutIntent.agentResults.has('coordinator')).toBe(false) + }) + + it('rejects required roles that are absent from the team roster', async () => { + await expect(runGovernedGoal('Review this request.', { + governanceIntent: 'required', + requiredRoles: ['reviewer', 'auditor'], + })).rejects.toThrow( + 'runTeam requiredRoles must exist in the team roster; unknown role(s): auditor.', + ) + }) + + it('rejects requiredOrder entries that are not declared in requiredRoles', async () => { + await expect(runGovernedGoal('Review this request.', { + governanceIntent: 'required', + requiredRoles: ['reviewer'], + requiredOrder: ['security'], + })).rejects.toThrow( + 'runTeam requiredOrder may reference only requiredRoles; invalid role(s): security.', + ) + }) +})