fix(core): make hybrid routing opt-in

This commit is contained in:
Jack Chen committed 2026-07-27 01:03:22 +08:00
1 parent 2880d88a21
commit e0e4cf6479
11 files changed
+354 -45

No files matched your search

+37
View File
@@ -673,6 +673,43 @@ corpus against these gates:
usage; and
- P95 end-to-end latency regression no greater than 10%.
### Running the real-provider Shadow gate
`packages/core/tests/e2e/semantic-routing-shadow.test.ts` runs the reviewed
synthetic corpus through the actual `LLMTaskProfiler`, then applies the same
deterministic policy that Hybrid routing uses. It is skipped unless
`SEMANTIC_ROUTING_SHADOW=1` is set, executes no workers or tools, and does not
send user data. The profiler uses the shipping `LLMTaskProfiler` default of 800
output tokens per fixture, so the canary validates the production contract. The
optional `SEMANTIC_ROUTING_SHADOW_BASE_URL` and
`SEMANTIC_ROUTING_SHADOW_REGION` values map to the selected adapter; Bedrock
uses its normal AWS credential chain instead of the API-key variable.
For DeepSeek V4, the built-in profiler explicitly uses non-thinking mode: this
is a bounded JSON classification call, and DeepSeek otherwise enables thinking
by default. The production route and this gate use the same setting.
Set credentials only in your local shell or CI secret store; never paste a key
into a command, repository file, or test log. For example, map an existing
local provider secret without printing it:
```bash
export SEMANTIC_ROUTING_SHADOW=1
export SEMANTIC_ROUTING_SHADOW_PROVIDER=openai
export SEMANTIC_ROUTING_SHADOW_MODEL=gpt-4o-mini
export SEMANTIC_ROUTING_SHADOW_API_KEY="$OPENAI_API_KEY"
npm run test:semantic-routing-shadow -w @open-multi-agent/core
```
The report contains only provider/model identifiers, aggregate accuracy,
failure and mismatch case IDs, P95 profiler latency, and aggregate token use.
It deliberately omits goals, inferred reasons, raw model output, and all
configuration secrets. The executable gate enforces zero invalid profiles,
zero critical false-Single outcomes, and at least 95% reviewed routing
accuracy. Record the output as release evidence. Measure the remaining
end-to-end token and latency regression gates in the production-shadow canary
before declaring a provider/model supported for Hybrid routing.
Historical success rates, online adaptive learning, and Team-to-Single
optimization require separately versioned evaluation, monitoring, and rollback
and are not part of V1.
+17 -23
View File
@@ -10,8 +10,9 @@ Execution Routing decides the execution topology for an automatic `runTeam()` ca
2. Declared governance topology or `preferredUnderBudget` degradation policy.
3. The per-run or orchestrator-level custom `executionRouter`.
4. The built-in `DeterministicRouter`.
5. When the default/fallback deterministic result is Single, the semantic
`TaskProfiler` and deterministic semantic policy.
5. When `executionRouting.strategy` is `hybrid` and the default/fallback
deterministic result is Single, the semantic `TaskProfiler` and deterministic
semantic policy.
Routers run only for automatic, non-`planOnly` topology selection. They never override an explicit mode, declared role topology, or governance budget policy. `governanceIntent: 'none'` retains automatic topology selection, while still counting as an explicit governance declaration for consequential-tool confirmation.
@@ -21,9 +22,9 @@ Single, Hybrid routing may profile that fallback candidate.
## Hybrid semantic routing
Hybrid is the default strategy. It preserves the cheap deterministic Team
decision and adds at most one no-tool model call only when that router would
otherwise choose Single:
Hybrid is opt-in. It preserves the cheap deterministic Team decision and adds
at most one no-tool model call only when that router would otherwise choose
Single:
```ts
const orchestrator = new OpenMultiAgent({
@@ -158,9 +159,10 @@ deterministic legacy decision, semantic recommendation, actual topology, usage,
and structured fallback state. The executed task topology, final tool grants,
and `ExecutionReceipt` remain governance truth.
## Deterministic compatibility mode
## Deterministic default
Use one line to restore the previous no-profiler behavior:
Omitting `executionRouting` uses the no-profiler deterministic router. Set the
strategy explicitly when you want configuration-level clarity:
```ts
const orchestrator = new OpenMultiAgent({
@@ -168,10 +170,9 @@ const orchestrator = new OpenMultiAgent({
})
```
This makes no extra model call and retains the existing deterministic Router
result and custom-Router fallback behavior. The compatibility path remains
available for offline use, incident degradation, and the entire major release
cycle.
This makes no extra model call and retains deterministic Router results and
custom-Router fallback behavior. Hybrid should be enabled only for providers
and models that have passed the documented Shadow gate.
Explicitly installing `new DeterministicRouter()` as `executionRouter` also
makes that valid Router decision final under the precedence rules, so the
@@ -210,18 +211,11 @@ which router selected Single or Team. Router decisions and TaskProfiles cannot
satisfy named-role or independent-review requirements; those facts continue to
come from structured governance declarations and the executed topology.
## Major-version migration
This implementation is intended for the next major release because it changes
automatic `runTeam()` from deterministic-only to Hybrid by default. In this
code, Hybrid is already the runtime default. A short goal may therefore make
one additional model call and upgrade from Single to Team, increasing latency,
token use, and cost.
Applications that require exact old call counts or fully offline routing should
set `executionRouting: { strategy: 'deterministic' }` before upgrading.
## Promotion criteria
Shadow evaluation is a release-engineering technique, not a user-facing runtime
mode: run fixed fixtures in CI and canaries, compare its recommendation without
changing production topology, and promote only after the documented accuracy,
invalid-output, cost, and latency gates pass. Online historical-success learning
and Team-to-Single optimization are outside V1.
changing production topology, and promote a provider/model to supported Hybrid
use only after the documented accuracy, invalid-output, cost, and latency gates
pass. Online historical-success learning and Team-to-Single optimization are
outside V1.
+10 -9
View File
@@ -109,20 +109,21 @@ Set `OPENAI_API_KEY` for this example. For other hosted or local models, see [Pr
Use `planOnly` to inspect a generated task graph before execution, then `createPlanArtifact()` and `runFromPlan()` to replay it. `runConsensus()` adds a proposer→judge verification loop when one answer needs extra scrutiny.
Automatic `runTeam()` uses Hybrid Semantic Routing by default. The cheap
`DeterministicRouter` keeps Team decisions and sends only Single candidates to
a one-call, no-tool `TaskProfiler`; deterministic policy may keep Single,
upgrade to Team, or require an explicit governance declaration. Valid custom
`executionRouter` decisions, explicit `mode`, and declared governance retain
higher priority. Set `executionRouting: { strategy: 'deterministic' }` for the
legacy no-profiler path. Auto results expose `routingDecision` and, when
profiling ran, `semanticRoutingAssessment`. See [Execution
Automatic `runTeam()` uses the deterministic router by default: no extra model
call is made. Opt into Hybrid Semantic Routing with
`executionRouting: { strategy: 'hybrid' }`; it keeps deterministic Team
decisions and sends only Single candidates to a one-call, no-tool
`TaskProfiler`. Deterministic policy may keep Single, upgrade to Team, or
require an explicit governance declaration. Valid custom `executionRouter`
decisions, explicit `mode`, and declared governance retain higher priority.
Auto results expose `routingDecision` and, when profiling ran,
`semanticRoutingAssessment`. See [Execution
Routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md).
Execution Routing selects Single versus Team; [Model
Routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md)
selects models inside that topology.
The Profiler sends the goal to the configured routing adapter, then the
When Hybrid is enabled, the Profiler sends the goal to the configured routing adapter, then the
Coordinator adapter, or finally the orchestrator's default provider. That last
fallback can make a provider call even when every worker has its own adapter.
Configure `executionRouting.adapter` or use deterministic strategy when the goal
+1
View File
@@ -67,6 +67,7 @@
"test:coverage": "vitest run --coverage",
"lint": "tsc --noEmit",
"test:e2e": "RUN_E2E=1 vitest run tests/e2e/",
"test:semantic-routing-shadow": "RUN_E2E=1 vitest run --reporter=dot tests/e2e/semantic-routing-shadow.test.ts",
"prepublishOnly": "npm run build"
},
"keywords": [
@@ -364,7 +364,7 @@ export class OpenMultiAgent {
strictAssignees: config.strictAssignees ?? true,
executionRouter: config.executionRouter ?? new DeterministicRouter(),
executionRouting: {
strategy: config.executionRouting?.strategy ?? 'hybrid',
strategy: config.executionRouting?.strategy ?? 'deterministic',
confidenceThreshold: config.executionRouting?.confidenceThreshold ?? 0.7,
failurePolicy: config.executionRouting?.failurePolicy ?? 'fallback',
...(config.executionRouting?.profiler !== undefined
@@ -469,7 +469,7 @@ export class OpenMultiAgent {
?? this.config.executionRouting.timeoutMs
?? coordinator?.callTimeoutMs
const resolved = {
strategy: override?.strategy ?? this.config.executionRouting.strategy ?? 'hybrid',
strategy: override?.strategy ?? this.config.executionRouting.strategy ?? 'deterministic',
confidenceThreshold:
override?.confidenceThreshold
?? this.config.executionRouting.confidenceThreshold
@@ -131,6 +131,13 @@ export class LLMTaskProfiler implements TaskProfiler {
model: this.model,
maxTokens: this.maxTokens,
temperature: 0,
// DeepSeek V4 enables thinking by default. Profiling is a bounded
// classification call whose only useful output is the JSON profile;
// leaving thinking enabled can spend its whole output budget on
// reasoning_content before producing that profile.
...(this.adapter.name === 'deepseek'
? { extraBody: { thinking: { type: 'disabled' } } }
: {}),
systemPrompt:
TASK_PROFILER_SYSTEM_PROMPT
+ buildStructuredOutputInstruction(rawTaskProfileSchema),
+4 -4
View File
@@ -1420,7 +1420,7 @@ export type RoutingFailurePolicy = 'fallback' | 'fail'
/** Hybrid execution-routing controls shared by orchestrator and per-run config. */
export interface ExecutionRoutingConfig {
/** Defaults to `hybrid`. Use `deterministic` for the legacy no-profiler path. */
/** Defaults to `deterministic`. Set `hybrid` to opt into semantic profiling. */
readonly strategy?: ExecutionRoutingStrategy
/** Custom profiler. When omitted, OMA builds an {@link LLMTaskProfiler}. */
readonly profiler?: TaskProfiler
@@ -1944,10 +1944,10 @@ export interface OrchestratorConfig {
*/
readonly executionRouter?: ExecutionRouter
/**
* Default hybrid execution-routing configuration.
* Default execution-routing configuration.
*
* Omitted means `strategy: 'hybrid'`. Use
* `{ strategy: 'deterministic' }` to restore the legacy no-profiler path.
* Omitted means `strategy: 'deterministic'`. Set
* `{ strategy: 'hybrid' }` to opt into semantic profiling.
*/
readonly executionRouting?: ExecutionRoutingConfig
/**
@@ -0,0 +1,203 @@
/**
* Release-canary evaluation for the real LLMTaskProfiler.
*
* This suite is deliberately skipped unless the caller explicitly enables it.
* It sends only the reviewed synthetic goals in semantic-routing-set.json to
* the selected provider; it never executes workers, tools, or user data.
*
* Run with the `test:semantic-routing-shadow` workspace script after setting:
* SEMANTIC_ROUTING_SHADOW=1
* SEMANTIC_ROUTING_SHADOW_PROVIDER=<supported provider>
* SEMANTIC_ROUTING_SHADOW_MODEL=<model id>
* SEMANTIC_ROUTING_SHADOW_API_KEY=<local secret> (not needed for Bedrock)
*/
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import { loadEvalSet } from '../../src/eval/file.js'
import { createAdapter, type SupportedProvider } from '../../src/llm/adapter.js'
import {
evaluateSemanticRoutingPolicy,
LLMTaskProfiler,
taskProfileSchema,
} from '../../src/orchestrator/task-profiler.js'
import type { TaskProfile } from '../../src/types.js'
const SEMANTIC_EVAL_SET_PATH = fileURLToPath(
new URL('../fixtures/eval/semantic-routing-set.json', import.meta.url),
)
const SUPPORTED_PROVIDERS = new Set<SupportedProvider>([
'anthropic',
'azure-openai',
'bedrock',
'copilot',
'deepseek',
'doubao',
'gemini',
'grok',
'hunyuan',
'minimax',
'mimo',
'openai',
'qiniu',
])
const SHADOW_ENABLED = process.env['SEMANTIC_ROUTING_SHADOW'] === '1'
const describeShadow = SHADOW_ENABLED ? describe : describe.skip
interface ShadowFixture {
readonly goal: string
readonly facts: {
readonly confidenceThreshold: number
readonly hasConsequentialTools: boolean
readonly permissionBoundaryCount: number
}
readonly expected: 'single' | 'team' | 'needs-declaration'
}
interface ShadowObservation {
readonly caseId: string
readonly tags: readonly string[]
readonly expected: ShadowFixture['expected']
readonly actual?: ShadowFixture['expected']
readonly durationMs: number
readonly inputTokens?: number
readonly outputTokens?: number
readonly failure?: string
}
function percentile95(values: readonly number[]): number {
if (values.length === 0) return 0
const ordered = [...values].sort((left, right) => left - right)
return ordered[Math.ceil(ordered.length * 0.95) - 1]!
}
function requireShadowConfig(): {
readonly provider: SupportedProvider
readonly model: string
readonly apiKey?: string
readonly baseURL?: string
readonly region?: string
} {
const providerValue = process.env['SEMANTIC_ROUTING_SHADOW_PROVIDER']
const model = process.env['SEMANTIC_ROUTING_SHADOW_MODEL']
const apiKey = process.env['SEMANTIC_ROUTING_SHADOW_API_KEY']
const baseURL = process.env['SEMANTIC_ROUTING_SHADOW_BASE_URL']
const region = process.env['SEMANTIC_ROUTING_SHADOW_REGION']
if (providerValue === undefined || !SUPPORTED_PROVIDERS.has(providerValue as SupportedProvider)) {
throw new Error('SEMANTIC_ROUTING_SHADOW_PROVIDER must name a supported provider.')
}
if (model === undefined || model.length === 0) {
throw new Error('SEMANTIC_ROUTING_SHADOW_MODEL must be a non-empty model id.')
}
if ((apiKey === undefined || apiKey.length === 0) && providerValue !== 'bedrock') {
throw new Error('SEMANTIC_ROUTING_SHADOW_API_KEY must be set locally when Shadow testing is enabled.')
}
return {
provider: providerValue as SupportedProvider,
model,
...(apiKey !== undefined && apiKey.length > 0 ? { apiKey } : {}),
...(baseURL !== undefined && baseURL.length > 0 ? { baseURL } : {}),
...(region !== undefined && region.length > 0 ? { region } : {}),
}
}
describeShadow('semantic routing Shadow evaluation', () => {
it('meets the reviewed semantic-routing release gates with a real profiler', async () => {
const config = requireShadowConfig()
const evalSet = await loadEvalSet(SEMANTIC_EVAL_SET_PATH)
const adapter = await createAdapter(
config.provider,
config.apiKey,
config.baseURL,
config.region,
)
const profiler = new LLMTaskProfiler({
adapter,
model: config.model,
// Match the shipping LLMTaskProfiler default. A lower canary-only cap
// would validate truncation behavior rather than the release contract.
maxTokens: 800,
})
const observations: ShadowObservation[] = []
for (const evalCase of evalSet.cases) {
const fixture = evalCase.input as ShadowFixture
const startedAt = Date.now()
try {
const profiled = await profiler.profile({
goal: fixture.goal,
roster: [
{ name: 'researcher', model: config.model, capabilities: ['research'] },
{ name: 'reviewer', model: config.model, capabilities: ['review'] },
],
})
const profile: TaskProfile = taskProfileSchema.parse(profiled.profile)
const actual = evaluateSemanticRoutingPolicy(profile, fixture.facts).recommendation
observations.push({
caseId: evalCase.id,
tags: evalCase.tags ?? [],
expected: fixture.expected,
actual,
durationMs: Date.now() - startedAt,
inputTokens: profiled.usage?.input_tokens,
outputTokens: profiled.usage?.output_tokens,
})
} catch (error) {
observations.push({
caseId: evalCase.id,
tags: evalCase.tags ?? [],
expected: fixture.expected,
durationMs: Date.now() - startedAt,
failure: error instanceof Error ? error.name : 'unknown',
})
}
}
const failures = observations.filter((observation) => observation.failure !== undefined)
const mismatches = observations.filter((observation) =>
observation.failure === undefined && observation.actual !== observation.expected)
const criticalFalseSingles = observations.filter((observation) =>
observation.tags.includes('critical')
&& observation.expected !== 'single'
&& observation.actual === 'single')
const successful = observations.length - failures.length
const accuracy = observations.length === 0
? 0
: (successful - mismatches.length) / observations.length
const inputTokens = observations.reduce(
(total, observation) => total + (observation.inputTokens ?? 0),
0,
)
const outputTokens = observations.reduce(
(total, observation) => total + (observation.outputTokens ?? 0),
0,
)
// Deliberately excludes goal text, profile reasons, raw responses, and API
// configuration so the canary report stays safe to retain in CI output.
console.info('[semantic-routing-shadow] %s', JSON.stringify({
provider: config.provider,
model: config.model,
caseCount: observations.length,
successful,
accuracy,
invalidOrFailedProfiles: failures.length,
criticalFalseSingles: criticalFalseSingles.map((observation) => observation.caseId),
mismatches: mismatches.map((observation) => ({
caseId: observation.caseId,
expected: observation.expected,
actual: observation.actual,
})),
failureCases: failures.map((observation) => ({
caseId: observation.caseId,
failure: observation.failure,
})),
p95ProfilerLatencyMs: percentile95(observations.map((observation) => observation.durationMs)),
totalTokens: { input_tokens: inputTokens, output_tokens: outputTokens },
}))
expect(failures, 'Profiler must return a valid profile for every reviewed fixture.').toHaveLength(0)
expect(criticalFalseSingles, 'Critical tasks must never remain on the Single path.').toHaveLength(0)
expect(accuracy, 'Reviewed end-to-end routing accuracy must be at least 95%.').toBeGreaterThanOrEqual(0.95)
}, 120_000)
})
@@ -303,7 +303,7 @@ async function runActualRoute(
const adapter = fixedAdapter()
const orchestrator = new OpenMultiAgent({
defaultModel: 'mock-model',
executionRouting: { profiler: fixedProfiler() },
executionRouting: { strategy: 'hybrid', profiler: fixedProfiler() },
})
const team = orchestrator.createTeam(`routing-stability-${family.id}-${variant.id}`, {
name: `routing-stability-${family.id}`,
@@ -58,6 +58,7 @@ describe('semantic routing default adapter resolution', () => {
const oma = new OpenMultiAgent({
defaultProvider: 'openai',
defaultModel: 'default-routing-model',
executionRouting: { strategy: 'hybrid' },
})
const team = oma.createTeam('default-adapter', {
name: 'default-adapter',
+71 -6
View File
@@ -97,10 +97,18 @@ function createRun(
) {
const agentAdapter = adapter()
const coordinatorAdapter = adapter(PLAN)
const {
executionRouting: extraExecutionRouting,
...remainingConfig
} = extraConfig
const oma = new OpenMultiAgent({
defaultModel: 'mock-model',
executionRouting: { profiler: taskProfiler },
...extraConfig,
...remainingConfig,
executionRouting: {
strategy: 'hybrid',
profiler: taskProfiler,
...extraExecutionRouting,
},
})
const team = oma.createTeam('semantic-routing', {
name: 'semantic-routing',
@@ -229,6 +237,37 @@ describe('LLMTaskProfiler', () => {
source: 'inferred',
})
})
it('disables DeepSeek V4 thinking so profiling returns within its output budget', async () => {
const mockAdapter = {
...adapter(JSON.stringify({
evidenceSources: 'single',
independentReview: 'none',
conflictingObjectives: false,
sideEffectIntent: 'none',
permissionIsolation: 'none',
decomposable: false,
parallelizable: false,
complexity: 'low',
confidence: 0.95,
reasons: ['The task is a bounded classification request.'],
})),
name: 'deepseek',
}
const taskProfiler = new LLMTaskProfiler({
adapter: mockAdapter,
model: 'deepseek-v4-flash',
})
await taskProfiler.profile({
goal: 'Summarize this note.',
roster: [{ name: 'alpha', model: 'deepseek-v4-flash' }],
})
expect(mockAdapter.chat.mock.calls[0]?.[1]).toMatchObject({
extraBody: { thinking: { type: 'disabled' } },
})
})
})
describe('hybrid runTeam routing', () => {
@@ -266,7 +305,7 @@ describe('hybrid runTeam routing', () => {
await sink.shutdown({ timeoutMs: 500 })
})
it('is the default and upgrades only a deterministic Single to Team', async () => {
it('upgrades only a deterministic Single to Team when Hybrid is enabled', async () => {
const taskProfiler = profiler(profile({ evidenceSources: 'multiple' }))
const { run } = createRun(taskProfiler)
@@ -285,6 +324,24 @@ describe('hybrid runTeam routing', () => {
})
})
it('defaults to deterministic routing without creating a profiler call', async () => {
const agentAdapter = adapter('hello')
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
const team = oma.createTeam('default-deterministic', {
name: 'default-deterministic',
agents: [{ name: 'alpha', model: 'mock-model', adapter: agentAdapter }],
})
const result = await oma.runTeam(team, 'Say hello')
expect(agentAdapter.chat).toHaveBeenCalledOnce()
expect(result.routingDecision).toMatchObject({
mode: 'single',
routerVersion: 'deterministic-v1',
})
expect(result.semanticRoutingAssessment).toBeUndefined()
})
it('keeps deterministic Team without making a profiler call', async () => {
const taskProfiler = profiler(profile({ evidenceSources: 'multiple' }))
const { run } = createRun(taskProfiler)
@@ -506,7 +563,7 @@ describe('hybrid runTeam routing', () => {
const coordinatorAdapter = adapter(PLAN)
const oma = new OpenMultiAgent({
defaultModel: 'mock-model',
executionRouting: { profiler: taskProfiler },
executionRouting: { strategy: 'hybrid', profiler: taskProfiler },
})
const team = oma.createTeam('permission-boundaries', {
name: 'permission-boundaries',
@@ -625,6 +682,7 @@ describe('hybrid runTeam routing', () => {
const oma = new OpenMultiAgent({
defaultModel: 'mock-model',
executionRouting: {
strategy: 'hybrid',
adapter: configuredAdapter,
model: 'configured-routing-model',
},
@@ -636,7 +694,11 @@ describe('hybrid runTeam routing', () => {
await oma.runTeam(team, 'Say hello', {
coordinator: { model: 'mock-model', adapter: coordinatorAdapter },
executionRouting: { adapter: perRunAdapter, model: 'per-run-routing-model' },
executionRouting: {
strategy: 'hybrid',
adapter: perRunAdapter,
model: 'per-run-routing-model',
},
})
expect(perRunAdapter.chat).toHaveBeenCalledOnce()
@@ -656,7 +718,10 @@ describe('hybrid runTeam routing', () => {
: PLAN,
))
const agentAdapter = adapter()
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
const oma = new OpenMultiAgent({
defaultModel: 'mock-model',
executionRouting: { strategy: 'hybrid' },
})
const team = oma.createTeam('coordinator-adapter-fallback', {
name: 'coordinator-adapter-fallback',
agents: [{ name: 'alpha', model: 'mock-model', adapter: agentAdapter }],