mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-21 10:40:12 -05:00
Jack Chen
16562ed894
bench/ measures a multi-agent run against a single-agent baseline pursuing the identical goal, on tokens, wall-clock, cost and judge-scored quality. Three groups per task: A runs runTasks() with deterministic model routing and a token budget, while B and C run one agent on the strong and the cheap tier, bracketing A from both sides so an effect that holds against both is attributable to the orchestration rather than to the model mix. Two tasks ship with it, both from packages/core/examples/cookbook/: a four-role contract review DAG and a single-pass meeting report. Role prompts are read out of the example source at load time rather than restated in the harness, so what gets measured cannot drift from what the examples say. Token counts come off each provider response through a loopback recording proxy, because TokenUsage carries no cache field and no per-model split and so cannot price a group A run that deliberately mixes two tiers. The framework's own counts are recorded alongside, and any disagreement lands in the row's notes instead of being reconciled silently. Quality is judge-scored, blind, and run in both presentation orders so position preference cancels. A is judged once per challenger and each of those scores is kept separately: A's score against B and its score against C are two measurements taken under two different contrasts, and only the first belongs in an A-minus-B comparison. The report states only what it computed. Whether the token budget bound is read off budget_exceeded. What dropping cache-contaminated repetitions would do to the headline is recomputed and printed, including whether any comparison reverses direction. Which rate schedule the invocation was billed on is checked against the manifest's UTC window rather than assumed, because the configured prices are peak rates and a run that happened off-peak was billed half of what the cost column says. Vendor-specific facts are emitted only for the vendor that actually ran, and a missing manifest is an error rather than a report whose models and controlled variables all read unknown. bench/ is not an npm workspace, so it is reached through explicit root scripts: npm run lint ends with lint:bench, and npm test ends with bench:ab:test. Both run in CI, the tests across Node 20, 22 and 24. npm run bench:ab:mock drives the full pipeline against an offline stand-in upstream, so the wiring is checkable without a key or a provider call. Results (bench/REPORT.md, bench/results-*.csv, bench/runs/) are gitignored.
47 lines
1.5 KiB
JSON
47 lines
1.5 KiB
JSON
{
|
|
"$comment": "Controlled variables for the OMA A/B benchmark. Every field here is written into bench/REPORT.md verbatim. Do not change values between the pilot and the full run without re-running the pilot.",
|
|
"provider": "deepseek",
|
|
"models": {
|
|
"strong": "deepseek-v4-pro",
|
|
"cheap": "deepseek-v4-flash"
|
|
},
|
|
"temperature": 0.2,
|
|
"thinking": {
|
|
"enabled": false
|
|
},
|
|
"maxTurns": 1,
|
|
"cacheBusting": true,
|
|
"maxTokenBudget": 400000,
|
|
"repetitions": 5,
|
|
"tasks": [
|
|
"contract-review",
|
|
"meeting-report"
|
|
],
|
|
"groups": [
|
|
"A",
|
|
"B",
|
|
"C"
|
|
],
|
|
"judge": {
|
|
"enabled": true,
|
|
"provider": "deepseek",
|
|
"model": "deepseek-v4-pro",
|
|
"temperature": 0,
|
|
"rubricVersion": "1",
|
|
"$comment": "Same vendor as the models under test. Accepted here because both candidates in every pair come from that same vendor, so vendor self-preference applies symmetrically; position bias is handled by scoring each pair in both orders. REPORT.md must state this."
|
|
},
|
|
"pricing": {
|
|
"$comment": "USD per 1,000,000 tokens, PEAK rates from https://api-docs.deepseek.com/quick_start/pricing (verified 2026-08-18). DeepSeek charges peak rates 01:00-04:00 and 06:00-10:00 UTC and half that off-peak; the whole benchmark invocation must sit inside one window for these numbers to be exact, and REPORT.md records the window actually used.",
|
|
"deepseek-v4-pro": {
|
|
"input": 1.32,
|
|
"cachedInput": 0.044,
|
|
"output": 3.96
|
|
},
|
|
"deepseek-v4-flash": {
|
|
"input": 0.44,
|
|
"cachedInput": 0.014,
|
|
"output": 1.32
|
|
}
|
|
}
|
|
}
|