mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-21 10:40:12 -05:00
isaacgao4396
d6153c27d8
`judgePrompt` ran once per `score()` call and could only return a string, so every judge in a quorum received the same text prompt. A judge scorer could therefore never see a non-text artifact: an agent that produces an image or a rendered document could only be scored on whatever text it emitted alongside the work, which is the agent describing itself rather than the work. It now runs once per judge and may return either a string, wrapped by the existing text template exactly as before, or a complete `readonly LLMMessage[]` passed straight to `Agent.run()`. A caller can branch on `judge.capabilities` to give a vision-tagged judge an image block while a cheaper text-only judge in the same quorum gets a text fallback. The string path is unchanged, and the old `(context) => string` signature still assigns to the widened type. The structured path deliberately hands the caller everything: OMA adds neither the case template nor the verdict-schema instruction, so a caller returning messages must append `buildStructuredOutputInstruction` itself or the judge is never told to emit the JSON `parseVerdict` requires. `docs/evaluation.md` says so in prose, and the tests fail when that instruction is missing rather than passing on a mock that returns JSON regardless. Locating the artifact stays the caller's job. `ScorerContext` exposes the run transcript, not a produced artifact, and a tool-produced image sits nested in a `tool_result` block rather than as a top-level image block, so the documented helper searches both shapes. Still out of scope and unchanged here: no per-judge failure isolation, so one judge failing on content it cannot handle fails the whole `score()` call. No new options, no `ScorerContext` change, no persistence change.