mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-21 18:42:50 -05:00
fix(core): make hybrid routing opt-in
This commit is contained in:
1 parent
2880d88a21
commit
e0e4cf6479
11 files changed
+354
-45
No files matched your search
@@ -673,6 +673,43 @@ corpus against these gates:
|
||||
usage; and
|
||||
- P95 end-to-end latency regression no greater than 10%.
|
||||
|
||||
### Running the real-provider Shadow gate
|
||||
|
||||
`packages/core/tests/e2e/semantic-routing-shadow.test.ts` runs the reviewed
|
||||
synthetic corpus through the actual `LLMTaskProfiler`, then applies the same
|
||||
deterministic policy that Hybrid routing uses. It is skipped unless
|
||||
`SEMANTIC_ROUTING_SHADOW=1` is set, executes no workers or tools, and does not
|
||||
send user data. The profiler uses the shipping `LLMTaskProfiler` default of 800
|
||||
output tokens per fixture, so the canary validates the production contract. The
|
||||
optional `SEMANTIC_ROUTING_SHADOW_BASE_URL` and
|
||||
`SEMANTIC_ROUTING_SHADOW_REGION` values map to the selected adapter; Bedrock
|
||||
uses its normal AWS credential chain instead of the API-key variable.
|
||||
|
||||
For DeepSeek V4, the built-in profiler explicitly uses non-thinking mode: this
|
||||
is a bounded JSON classification call, and DeepSeek otherwise enables thinking
|
||||
by default. The production route and this gate use the same setting.
|
||||
|
||||
Set credentials only in your local shell or CI secret store; never paste a key
|
||||
into a command, repository file, or test log. For example, map an existing
|
||||
local provider secret without printing it:
|
||||
|
||||
```bash
|
||||
export SEMANTIC_ROUTING_SHADOW=1
|
||||
export SEMANTIC_ROUTING_SHADOW_PROVIDER=openai
|
||||
export SEMANTIC_ROUTING_SHADOW_MODEL=gpt-4o-mini
|
||||
export SEMANTIC_ROUTING_SHADOW_API_KEY="$OPENAI_API_KEY"
|
||||
npm run test:semantic-routing-shadow -w @open-multi-agent/core
|
||||
```
|
||||
|
||||
The report contains only provider/model identifiers, aggregate accuracy,
|
||||
failure and mismatch case IDs, P95 profiler latency, and aggregate token use.
|
||||
It deliberately omits goals, inferred reasons, raw model output, and all
|
||||
configuration secrets. The executable gate enforces zero invalid profiles,
|
||||
zero critical false-Single outcomes, and at least 95% reviewed routing
|
||||
accuracy. Record the output as release evidence. Measure the remaining
|
||||
end-to-end token and latency regression gates in the production-shadow canary
|
||||
before declaring a provider/model supported for Hybrid routing.
|
||||
|
||||
Historical success rates, online adaptive learning, and Team-to-Single
|
||||
optimization require separately versioned evaluation, monitoring, and rollback
|
||||
and are not part of V1.
|
||||
|
||||
+17
-23
@@ -10,8 +10,9 @@ Execution Routing decides the execution topology for an automatic `runTeam()` ca
|
||||
2. Declared governance topology or `preferredUnderBudget` degradation policy.
|
||||
3. The per-run or orchestrator-level custom `executionRouter`.
|
||||
4. The built-in `DeterministicRouter`.
|
||||
5. When the default/fallback deterministic result is Single, the semantic
|
||||
`TaskProfiler` and deterministic semantic policy.
|
||||
5. When `executionRouting.strategy` is `hybrid` and the default/fallback
|
||||
deterministic result is Single, the semantic `TaskProfiler` and deterministic
|
||||
semantic policy.
|
||||
|
||||
Routers run only for automatic, non-`planOnly` topology selection. They never override an explicit mode, declared role topology, or governance budget policy. `governanceIntent: 'none'` retains automatic topology selection, while still counting as an explicit governance declaration for consequential-tool confirmation.
|
||||
|
||||
@@ -21,9 +22,9 @@ Single, Hybrid routing may profile that fallback candidate.
|
||||
|
||||
## Hybrid semantic routing
|
||||
|
||||
Hybrid is the default strategy. It preserves the cheap deterministic Team
|
||||
decision and adds at most one no-tool model call only when that router would
|
||||
otherwise choose Single:
|
||||
Hybrid is opt-in. It preserves the cheap deterministic Team decision and adds
|
||||
at most one no-tool model call only when that router would otherwise choose
|
||||
Single:
|
||||
|
||||
```ts
|
||||
const orchestrator = new OpenMultiAgent({
|
||||
@@ -158,9 +159,10 @@ deterministic legacy decision, semantic recommendation, actual topology, usage,
|
||||
and structured fallback state. The executed task topology, final tool grants,
|
||||
and `ExecutionReceipt` remain governance truth.
|
||||
|
||||
## Deterministic compatibility mode
|
||||
## Deterministic default
|
||||
|
||||
Use one line to restore the previous no-profiler behavior:
|
||||
Omitting `executionRouting` uses the no-profiler deterministic router. Set the
|
||||
strategy explicitly when you want configuration-level clarity:
|
||||
|
||||
```ts
|
||||
const orchestrator = new OpenMultiAgent({
|
||||
@@ -168,10 +170,9 @@ const orchestrator = new OpenMultiAgent({
|
||||
})
|
||||
```
|
||||
|
||||
This makes no extra model call and retains the existing deterministic Router
|
||||
result and custom-Router fallback behavior. The compatibility path remains
|
||||
available for offline use, incident degradation, and the entire major release
|
||||
cycle.
|
||||
This makes no extra model call and retains deterministic Router results and
|
||||
custom-Router fallback behavior. Hybrid should be enabled only for providers
|
||||
and models that have passed the documented Shadow gate.
|
||||
|
||||
Explicitly installing `new DeterministicRouter()` as `executionRouter` also
|
||||
makes that valid Router decision final under the precedence rules, so the
|
||||
@@ -210,18 +211,11 @@ which router selected Single or Team. Router decisions and TaskProfiles cannot
|
||||
satisfy named-role or independent-review requirements; those facts continue to
|
||||
come from structured governance declarations and the executed topology.
|
||||
|
||||
## Major-version migration
|
||||
|
||||
This implementation is intended for the next major release because it changes
|
||||
automatic `runTeam()` from deterministic-only to Hybrid by default. In this
|
||||
code, Hybrid is already the runtime default. A short goal may therefore make
|
||||
one additional model call and upgrade from Single to Team, increasing latency,
|
||||
token use, and cost.
|
||||
Applications that require exact old call counts or fully offline routing should
|
||||
set `executionRouting: { strategy: 'deterministic' }` before upgrading.
|
||||
## Promotion criteria
|
||||
|
||||
Shadow evaluation is a release-engineering technique, not a user-facing runtime
|
||||
mode: run fixed fixtures in CI and canaries, compare its recommendation without
|
||||
changing production topology, and promote only after the documented accuracy,
|
||||
invalid-output, cost, and latency gates pass. Online historical-success learning
|
||||
and Team-to-Single optimization are outside V1.
|
||||
changing production topology, and promote a provider/model to supported Hybrid
|
||||
use only after the documented accuracy, invalid-output, cost, and latency gates
|
||||
pass. Online historical-success learning and Team-to-Single optimization are
|
||||
outside V1.
|
||||
+10
-9
@@ -109,20 +109,21 @@ Set `OPENAI_API_KEY` for this example. For other hosted or local models, see [Pr
|
||||
|
||||
Use `planOnly` to inspect a generated task graph before execution, then `createPlanArtifact()` and `runFromPlan()` to replay it. `runConsensus()` adds a proposer→judge verification loop when one answer needs extra scrutiny.
|
||||
|
||||
Automatic `runTeam()` uses Hybrid Semantic Routing by default. The cheap
|
||||
`DeterministicRouter` keeps Team decisions and sends only Single candidates to
|
||||
a one-call, no-tool `TaskProfiler`; deterministic policy may keep Single,
|
||||
upgrade to Team, or require an explicit governance declaration. Valid custom
|
||||
`executionRouter` decisions, explicit `mode`, and declared governance retain
|
||||
higher priority. Set `executionRouting: { strategy: 'deterministic' }` for the
|
||||
legacy no-profiler path. Auto results expose `routingDecision` and, when
|
||||
profiling ran, `semanticRoutingAssessment`. See [Execution
|
||||
Automatic `runTeam()` uses the deterministic router by default: no extra model
|
||||
call is made. Opt into Hybrid Semantic Routing with
|
||||
`executionRouting: { strategy: 'hybrid' }`; it keeps deterministic Team
|
||||
decisions and sends only Single candidates to a one-call, no-tool
|
||||
`TaskProfiler`. Deterministic policy may keep Single, upgrade to Team, or
|
||||
require an explicit governance declaration. Valid custom `executionRouter`
|
||||
decisions, explicit `mode`, and declared governance retain higher priority.
|
||||
Auto results expose `routingDecision` and, when profiling ran,
|
||||
`semanticRoutingAssessment`. See [Execution
|
||||
Routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/execution-routing.md).
|
||||
Execution Routing selects Single versus Team; [Model
|
||||
Routing](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/model-routing.md)
|
||||
selects models inside that topology.
|
||||
|
||||
The Profiler sends the goal to the configured routing adapter, then the
|
||||
When Hybrid is enabled, the Profiler sends the goal to the configured routing adapter, then the
|
||||
Coordinator adapter, or finally the orchestrator's default provider. That last
|
||||
fallback can make a provider call even when every worker has its own adapter.
|
||||
Configure `executionRouting.adapter` or use deterministic strategy when the goal
|
||||
|
||||
@@ -67,6 +67,7 @@
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"lint": "tsc --noEmit",
|
||||
"test:e2e": "RUN_E2E=1 vitest run tests/e2e/",
|
||||
"test:semantic-routing-shadow": "RUN_E2E=1 vitest run --reporter=dot tests/e2e/semantic-routing-shadow.test.ts",
|
||||
"prepublishOnly": "npm run build"
|
||||
},
|
||||
"keywords": [
|
||||
|
||||
@@ -364,7 +364,7 @@ export class OpenMultiAgent {
|
||||
strictAssignees: config.strictAssignees ?? true,
|
||||
executionRouter: config.executionRouter ?? new DeterministicRouter(),
|
||||
executionRouting: {
|
||||
strategy: config.executionRouting?.strategy ?? 'hybrid',
|
||||
strategy: config.executionRouting?.strategy ?? 'deterministic',
|
||||
confidenceThreshold: config.executionRouting?.confidenceThreshold ?? 0.7,
|
||||
failurePolicy: config.executionRouting?.failurePolicy ?? 'fallback',
|
||||
...(config.executionRouting?.profiler !== undefined
|
||||
@@ -469,7 +469,7 @@ export class OpenMultiAgent {
|
||||
?? this.config.executionRouting.timeoutMs
|
||||
?? coordinator?.callTimeoutMs
|
||||
const resolved = {
|
||||
strategy: override?.strategy ?? this.config.executionRouting.strategy ?? 'hybrid',
|
||||
strategy: override?.strategy ?? this.config.executionRouting.strategy ?? 'deterministic',
|
||||
confidenceThreshold:
|
||||
override?.confidenceThreshold
|
||||
?? this.config.executionRouting.confidenceThreshold
|
||||
|
||||
@@ -131,6 +131,13 @@ export class LLMTaskProfiler implements TaskProfiler {
|
||||
model: this.model,
|
||||
maxTokens: this.maxTokens,
|
||||
temperature: 0,
|
||||
// DeepSeek V4 enables thinking by default. Profiling is a bounded
|
||||
// classification call whose only useful output is the JSON profile;
|
||||
// leaving thinking enabled can spend its whole output budget on
|
||||
// reasoning_content before producing that profile.
|
||||
...(this.adapter.name === 'deepseek'
|
||||
? { extraBody: { thinking: { type: 'disabled' } } }
|
||||
: {}),
|
||||
systemPrompt:
|
||||
TASK_PROFILER_SYSTEM_PROMPT
|
||||
+ buildStructuredOutputInstruction(rawTaskProfileSchema),
|
||||
|
||||
@@ -1420,7 +1420,7 @@ export type RoutingFailurePolicy = 'fallback' | 'fail'
|
||||
|
||||
/** Hybrid execution-routing controls shared by orchestrator and per-run config. */
|
||||
export interface ExecutionRoutingConfig {
|
||||
/** Defaults to `hybrid`. Use `deterministic` for the legacy no-profiler path. */
|
||||
/** Defaults to `deterministic`. Set `hybrid` to opt into semantic profiling. */
|
||||
readonly strategy?: ExecutionRoutingStrategy
|
||||
/** Custom profiler. When omitted, OMA builds an {@link LLMTaskProfiler}. */
|
||||
readonly profiler?: TaskProfiler
|
||||
@@ -1944,10 +1944,10 @@ export interface OrchestratorConfig {
|
||||
*/
|
||||
readonly executionRouter?: ExecutionRouter
|
||||
/**
|
||||
* Default hybrid execution-routing configuration.
|
||||
* Default execution-routing configuration.
|
||||
*
|
||||
* Omitted means `strategy: 'hybrid'`. Use
|
||||
* `{ strategy: 'deterministic' }` to restore the legacy no-profiler path.
|
||||
* Omitted means `strategy: 'deterministic'`. Set
|
||||
* `{ strategy: 'hybrid' }` to opt into semantic profiling.
|
||||
*/
|
||||
readonly executionRouting?: ExecutionRoutingConfig
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* Release-canary evaluation for the real LLMTaskProfiler.
|
||||
*
|
||||
* This suite is deliberately skipped unless the caller explicitly enables it.
|
||||
* It sends only the reviewed synthetic goals in semantic-routing-set.json to
|
||||
* the selected provider; it never executes workers, tools, or user data.
|
||||
*
|
||||
* Run with the `test:semantic-routing-shadow` workspace script after setting:
|
||||
* SEMANTIC_ROUTING_SHADOW=1
|
||||
* SEMANTIC_ROUTING_SHADOW_PROVIDER=<supported provider>
|
||||
* SEMANTIC_ROUTING_SHADOW_MODEL=<model id>
|
||||
* SEMANTIC_ROUTING_SHADOW_API_KEY=<local secret> (not needed for Bedrock)
|
||||
*/
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { loadEvalSet } from '../../src/eval/file.js'
|
||||
import { createAdapter, type SupportedProvider } from '../../src/llm/adapter.js'
|
||||
import {
|
||||
evaluateSemanticRoutingPolicy,
|
||||
LLMTaskProfiler,
|
||||
taskProfileSchema,
|
||||
} from '../../src/orchestrator/task-profiler.js'
|
||||
import type { TaskProfile } from '../../src/types.js'
|
||||
|
||||
const SEMANTIC_EVAL_SET_PATH = fileURLToPath(
|
||||
new URL('../fixtures/eval/semantic-routing-set.json', import.meta.url),
|
||||
)
|
||||
const SUPPORTED_PROVIDERS = new Set<SupportedProvider>([
|
||||
'anthropic',
|
||||
'azure-openai',
|
||||
'bedrock',
|
||||
'copilot',
|
||||
'deepseek',
|
||||
'doubao',
|
||||
'gemini',
|
||||
'grok',
|
||||
'hunyuan',
|
||||
'minimax',
|
||||
'mimo',
|
||||
'openai',
|
||||
'qiniu',
|
||||
])
|
||||
const SHADOW_ENABLED = process.env['SEMANTIC_ROUTING_SHADOW'] === '1'
|
||||
const describeShadow = SHADOW_ENABLED ? describe : describe.skip
|
||||
|
||||
interface ShadowFixture {
|
||||
readonly goal: string
|
||||
readonly facts: {
|
||||
readonly confidenceThreshold: number
|
||||
readonly hasConsequentialTools: boolean
|
||||
readonly permissionBoundaryCount: number
|
||||
}
|
||||
readonly expected: 'single' | 'team' | 'needs-declaration'
|
||||
}
|
||||
|
||||
interface ShadowObservation {
|
||||
readonly caseId: string
|
||||
readonly tags: readonly string[]
|
||||
readonly expected: ShadowFixture['expected']
|
||||
readonly actual?: ShadowFixture['expected']
|
||||
readonly durationMs: number
|
||||
readonly inputTokens?: number
|
||||
readonly outputTokens?: number
|
||||
readonly failure?: string
|
||||
}
|
||||
|
||||
function percentile95(values: readonly number[]): number {
|
||||
if (values.length === 0) return 0
|
||||
const ordered = [...values].sort((left, right) => left - right)
|
||||
return ordered[Math.ceil(ordered.length * 0.95) - 1]!
|
||||
}
|
||||
|
||||
function requireShadowConfig(): {
|
||||
readonly provider: SupportedProvider
|
||||
readonly model: string
|
||||
readonly apiKey?: string
|
||||
readonly baseURL?: string
|
||||
readonly region?: string
|
||||
} {
|
||||
const providerValue = process.env['SEMANTIC_ROUTING_SHADOW_PROVIDER']
|
||||
const model = process.env['SEMANTIC_ROUTING_SHADOW_MODEL']
|
||||
const apiKey = process.env['SEMANTIC_ROUTING_SHADOW_API_KEY']
|
||||
const baseURL = process.env['SEMANTIC_ROUTING_SHADOW_BASE_URL']
|
||||
const region = process.env['SEMANTIC_ROUTING_SHADOW_REGION']
|
||||
if (providerValue === undefined || !SUPPORTED_PROVIDERS.has(providerValue as SupportedProvider)) {
|
||||
throw new Error('SEMANTIC_ROUTING_SHADOW_PROVIDER must name a supported provider.')
|
||||
}
|
||||
if (model === undefined || model.length === 0) {
|
||||
throw new Error('SEMANTIC_ROUTING_SHADOW_MODEL must be a non-empty model id.')
|
||||
}
|
||||
if ((apiKey === undefined || apiKey.length === 0) && providerValue !== 'bedrock') {
|
||||
throw new Error('SEMANTIC_ROUTING_SHADOW_API_KEY must be set locally when Shadow testing is enabled.')
|
||||
}
|
||||
return {
|
||||
provider: providerValue as SupportedProvider,
|
||||
model,
|
||||
...(apiKey !== undefined && apiKey.length > 0 ? { apiKey } : {}),
|
||||
...(baseURL !== undefined && baseURL.length > 0 ? { baseURL } : {}),
|
||||
...(region !== undefined && region.length > 0 ? { region } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
describeShadow('semantic routing Shadow evaluation', () => {
|
||||
it('meets the reviewed semantic-routing release gates with a real profiler', async () => {
|
||||
const config = requireShadowConfig()
|
||||
const evalSet = await loadEvalSet(SEMANTIC_EVAL_SET_PATH)
|
||||
const adapter = await createAdapter(
|
||||
config.provider,
|
||||
config.apiKey,
|
||||
config.baseURL,
|
||||
config.region,
|
||||
)
|
||||
const profiler = new LLMTaskProfiler({
|
||||
adapter,
|
||||
model: config.model,
|
||||
// Match the shipping LLMTaskProfiler default. A lower canary-only cap
|
||||
// would validate truncation behavior rather than the release contract.
|
||||
maxTokens: 800,
|
||||
})
|
||||
const observations: ShadowObservation[] = []
|
||||
|
||||
for (const evalCase of evalSet.cases) {
|
||||
const fixture = evalCase.input as ShadowFixture
|
||||
const startedAt = Date.now()
|
||||
try {
|
||||
const profiled = await profiler.profile({
|
||||
goal: fixture.goal,
|
||||
roster: [
|
||||
{ name: 'researcher', model: config.model, capabilities: ['research'] },
|
||||
{ name: 'reviewer', model: config.model, capabilities: ['review'] },
|
||||
],
|
||||
})
|
||||
const profile: TaskProfile = taskProfileSchema.parse(profiled.profile)
|
||||
const actual = evaluateSemanticRoutingPolicy(profile, fixture.facts).recommendation
|
||||
observations.push({
|
||||
caseId: evalCase.id,
|
||||
tags: evalCase.tags ?? [],
|
||||
expected: fixture.expected,
|
||||
actual,
|
||||
durationMs: Date.now() - startedAt,
|
||||
inputTokens: profiled.usage?.input_tokens,
|
||||
outputTokens: profiled.usage?.output_tokens,
|
||||
})
|
||||
} catch (error) {
|
||||
observations.push({
|
||||
caseId: evalCase.id,
|
||||
tags: evalCase.tags ?? [],
|
||||
expected: fixture.expected,
|
||||
durationMs: Date.now() - startedAt,
|
||||
failure: error instanceof Error ? error.name : 'unknown',
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const failures = observations.filter((observation) => observation.failure !== undefined)
|
||||
const mismatches = observations.filter((observation) =>
|
||||
observation.failure === undefined && observation.actual !== observation.expected)
|
||||
const criticalFalseSingles = observations.filter((observation) =>
|
||||
observation.tags.includes('critical')
|
||||
&& observation.expected !== 'single'
|
||||
&& observation.actual === 'single')
|
||||
const successful = observations.length - failures.length
|
||||
const accuracy = observations.length === 0
|
||||
? 0
|
||||
: (successful - mismatches.length) / observations.length
|
||||
const inputTokens = observations.reduce(
|
||||
(total, observation) => total + (observation.inputTokens ?? 0),
|
||||
0,
|
||||
)
|
||||
const outputTokens = observations.reduce(
|
||||
(total, observation) => total + (observation.outputTokens ?? 0),
|
||||
0,
|
||||
)
|
||||
|
||||
// Deliberately excludes goal text, profile reasons, raw responses, and API
|
||||
// configuration so the canary report stays safe to retain in CI output.
|
||||
console.info('[semantic-routing-shadow] %s', JSON.stringify({
|
||||
provider: config.provider,
|
||||
model: config.model,
|
||||
caseCount: observations.length,
|
||||
successful,
|
||||
accuracy,
|
||||
invalidOrFailedProfiles: failures.length,
|
||||
criticalFalseSingles: criticalFalseSingles.map((observation) => observation.caseId),
|
||||
mismatches: mismatches.map((observation) => ({
|
||||
caseId: observation.caseId,
|
||||
expected: observation.expected,
|
||||
actual: observation.actual,
|
||||
})),
|
||||
failureCases: failures.map((observation) => ({
|
||||
caseId: observation.caseId,
|
||||
failure: observation.failure,
|
||||
})),
|
||||
p95ProfilerLatencyMs: percentile95(observations.map((observation) => observation.durationMs)),
|
||||
totalTokens: { input_tokens: inputTokens, output_tokens: outputTokens },
|
||||
}))
|
||||
|
||||
expect(failures, 'Profiler must return a valid profile for every reviewed fixture.').toHaveLength(0)
|
||||
expect(criticalFalseSingles, 'Critical tasks must never remain on the Single path.').toHaveLength(0)
|
||||
expect(accuracy, 'Reviewed end-to-end routing accuracy must be at least 95%.').toBeGreaterThanOrEqual(0.95)
|
||||
}, 120_000)
|
||||
})
|
||||
@@ -303,7 +303,7 @@ async function runActualRoute(
|
||||
const adapter = fixedAdapter()
|
||||
const orchestrator = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
executionRouting: { profiler: fixedProfiler() },
|
||||
executionRouting: { strategy: 'hybrid', profiler: fixedProfiler() },
|
||||
})
|
||||
const team = orchestrator.createTeam(`routing-stability-${family.id}-${variant.id}`, {
|
||||
name: `routing-stability-${family.id}`,
|
||||
|
||||
@@ -58,6 +58,7 @@ describe('semantic routing default adapter resolution', () => {
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultProvider: 'openai',
|
||||
defaultModel: 'default-routing-model',
|
||||
executionRouting: { strategy: 'hybrid' },
|
||||
})
|
||||
const team = oma.createTeam('default-adapter', {
|
||||
name: 'default-adapter',
|
||||
|
||||
@@ -97,10 +97,18 @@ function createRun(
|
||||
) {
|
||||
const agentAdapter = adapter()
|
||||
const coordinatorAdapter = adapter(PLAN)
|
||||
const {
|
||||
executionRouting: extraExecutionRouting,
|
||||
...remainingConfig
|
||||
} = extraConfig
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
executionRouting: { profiler: taskProfiler },
|
||||
...extraConfig,
|
||||
...remainingConfig,
|
||||
executionRouting: {
|
||||
strategy: 'hybrid',
|
||||
profiler: taskProfiler,
|
||||
...extraExecutionRouting,
|
||||
},
|
||||
})
|
||||
const team = oma.createTeam('semantic-routing', {
|
||||
name: 'semantic-routing',
|
||||
@@ -229,6 +237,37 @@ describe('LLMTaskProfiler', () => {
|
||||
source: 'inferred',
|
||||
})
|
||||
})
|
||||
|
||||
it('disables DeepSeek V4 thinking so profiling returns within its output budget', async () => {
|
||||
const mockAdapter = {
|
||||
...adapter(JSON.stringify({
|
||||
evidenceSources: 'single',
|
||||
independentReview: 'none',
|
||||
conflictingObjectives: false,
|
||||
sideEffectIntent: 'none',
|
||||
permissionIsolation: 'none',
|
||||
decomposable: false,
|
||||
parallelizable: false,
|
||||
complexity: 'low',
|
||||
confidence: 0.95,
|
||||
reasons: ['The task is a bounded classification request.'],
|
||||
})),
|
||||
name: 'deepseek',
|
||||
}
|
||||
const taskProfiler = new LLMTaskProfiler({
|
||||
adapter: mockAdapter,
|
||||
model: 'deepseek-v4-flash',
|
||||
})
|
||||
|
||||
await taskProfiler.profile({
|
||||
goal: 'Summarize this note.',
|
||||
roster: [{ name: 'alpha', model: 'deepseek-v4-flash' }],
|
||||
})
|
||||
|
||||
expect(mockAdapter.chat.mock.calls[0]?.[1]).toMatchObject({
|
||||
extraBody: { thinking: { type: 'disabled' } },
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('hybrid runTeam routing', () => {
|
||||
@@ -266,7 +305,7 @@ describe('hybrid runTeam routing', () => {
|
||||
await sink.shutdown({ timeoutMs: 500 })
|
||||
})
|
||||
|
||||
it('is the default and upgrades only a deterministic Single to Team', async () => {
|
||||
it('upgrades only a deterministic Single to Team when Hybrid is enabled', async () => {
|
||||
const taskProfiler = profiler(profile({ evidenceSources: 'multiple' }))
|
||||
const { run } = createRun(taskProfiler)
|
||||
|
||||
@@ -285,6 +324,24 @@ describe('hybrid runTeam routing', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('defaults to deterministic routing without creating a profiler call', async () => {
|
||||
const agentAdapter = adapter('hello')
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const team = oma.createTeam('default-deterministic', {
|
||||
name: 'default-deterministic',
|
||||
agents: [{ name: 'alpha', model: 'mock-model', adapter: agentAdapter }],
|
||||
})
|
||||
|
||||
const result = await oma.runTeam(team, 'Say hello')
|
||||
|
||||
expect(agentAdapter.chat).toHaveBeenCalledOnce()
|
||||
expect(result.routingDecision).toMatchObject({
|
||||
mode: 'single',
|
||||
routerVersion: 'deterministic-v1',
|
||||
})
|
||||
expect(result.semanticRoutingAssessment).toBeUndefined()
|
||||
})
|
||||
|
||||
it('keeps deterministic Team without making a profiler call', async () => {
|
||||
const taskProfiler = profiler(profile({ evidenceSources: 'multiple' }))
|
||||
const { run } = createRun(taskProfiler)
|
||||
@@ -506,7 +563,7 @@ describe('hybrid runTeam routing', () => {
|
||||
const coordinatorAdapter = adapter(PLAN)
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
executionRouting: { profiler: taskProfiler },
|
||||
executionRouting: { strategy: 'hybrid', profiler: taskProfiler },
|
||||
})
|
||||
const team = oma.createTeam('permission-boundaries', {
|
||||
name: 'permission-boundaries',
|
||||
@@ -625,6 +682,7 @@ describe('hybrid runTeam routing', () => {
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
executionRouting: {
|
||||
strategy: 'hybrid',
|
||||
adapter: configuredAdapter,
|
||||
model: 'configured-routing-model',
|
||||
},
|
||||
@@ -636,7 +694,11 @@ describe('hybrid runTeam routing', () => {
|
||||
|
||||
await oma.runTeam(team, 'Say hello', {
|
||||
coordinator: { model: 'mock-model', adapter: coordinatorAdapter },
|
||||
executionRouting: { adapter: perRunAdapter, model: 'per-run-routing-model' },
|
||||
executionRouting: {
|
||||
strategy: 'hybrid',
|
||||
adapter: perRunAdapter,
|
||||
model: 'per-run-routing-model',
|
||||
},
|
||||
})
|
||||
|
||||
expect(perRunAdapter.chat).toHaveBeenCalledOnce()
|
||||
@@ -656,7 +718,10 @@ describe('hybrid runTeam routing', () => {
|
||||
: PLAN,
|
||||
))
|
||||
const agentAdapter = adapter()
|
||||
const oma = new OpenMultiAgent({ defaultModel: 'mock-model' })
|
||||
const oma = new OpenMultiAgent({
|
||||
defaultModel: 'mock-model',
|
||||
executionRouting: { strategy: 'hybrid' },
|
||||
})
|
||||
const team = oma.createTeam('coordinator-adapter-fallback', {
|
||||
name: 'coordinator-adapter-fallback',
|
||||
agents: [{ name: 'alpha', model: 'mock-model', adapter: agentAdapter }],
|
||||
|
||||
Reference in new issue
Block a user