open-multi-agent-open-multi-agent/docs/structured-input.md

# Structured agent input

`Agent.run()`, `Agent.stream()`, and `OpenMultiAgent.runAgent()` accept either a
string or a complete `LLMMessage[]`. The string form is unchanged shorthand for
one user text message. Use the message form when the application owns prior
conversation turns or needs content blocks such as images and videos:

```ts
import {
  OpenMultiAgent,
  type LLMMessage,
} from '@open-multi-agent/core'

const messages: LLMMessage[] = [
  { role: 'user', content: [{ type: 'text', text: 'Keep answers concise.' }] },
  { role: 'assistant', content: [{ type: 'text', text: 'Understood.' }] },
  {
    role: 'user',
    content: [
      { type: 'text', text: 'What is shown here?' },
      {
        type: 'image',
        source: {
          type: 'base64',
          media_type: 'image/png',
          data: imageBytes.toString('base64'),
        },
      },
    ],
  },
]

const result = await new OpenMultiAgent().runAgent(
  { name: 'vision', model: 'claude-sonnet-4-6' },
  messages,
)
```

A `video` block carries either inline base64 bytes or a remote reference:

```ts
{
  type: 'video',
  source: { type: 'url', media_type: 'video/mp4', url: 'https://example.com/clip.mp4' },
}
```

Unlike an image, a `url` source is fetched by the provider rather than by OMA,
so the reference leaves the process. Both source shapes are therefore validated
at every public entry point under the same rules as tool-result media: an
absolute `http:`/`https:` URL, or raw canonical base64 with a parameterless MIME
type. Anything else — a `file:` path, a provider-private scheme, a data URL
pasted into the `url` field — is rejected with `InvalidMessageError` before an
adapter sees it. Providers that expose their own upload handles for large videos
(MiniMax's `mm_file://`, for instance) are not reachable through this block
today.

## Which adapters accept which blocks

The selected model/provider must support every supplied content block. OMA does
not infer a vision-capability flag or silently remove unsupported blocks. Two
things follow from that, and they look different at the call site:

- **OpenAI-shaped adapters forward and let the provider decide.** A `video`
  block becomes a `video_url` content part on the Chat Completions route.
  MiniMax accepts it; the other OpenAI-compatible providers reject it, and that
  provider error follows the normal agent failure path.
- **Adapters with a typed content union reject locally.** Anthropic, Bedrock,
  Gemini, and the AI SDK bridge have no `video` mapping, so they raise
  `UnsupportedContentBlockError` before the request is sent. That error is
  terminal: retrying cannot add a wire-format capability, so orchestrator retry
  skips it rather than spending the backoff ladder on it. Bedrock and Gemini do
  have native video support upstream — the gap is OMA's mapping, not theirs.

## API boundaries

| API | String form | Structured form | Conversation behavior |
|---|---|---|---|
| `Agent.run(input)` | One user text message | Complete `readonly LLMMessage[]` | Fresh; does not read or update persistent history |
| `Agent.stream(input)` | One user text message | Complete `readonly LLMMessage[]` | Fresh; same input semantics as `run()` |
| `OpenMultiAgent.runAgent(config, input)` | One user text message | Complete `readonly LLMMessage[]` | Fresh one-off Agent with orchestration, tracing, progress, budgets, and evaluation |
| `Agent.prompt(input)` | One user text message | One user turn as `readonly ContentBlock[]` | Appends that turn and the response to persistent history |

`AgentConfig.history` restores earlier persistent turns for `prompt()`. Passing a
content-block list to `prompt()` does not replace that history and cannot insert
assistant turns; use `run(messages)` for a caller-owned complete conversation.
`runTeam()` goals and `runTasks()` task descriptions remain text-only.

## Copying and validation

Structured inputs are runtime-validated with the same `LLMMessage` shape guard
used at adapter boundaries, then defensively deep-copied before hooks, runners,
progress callbacks, evaluation, or persistent history retain them. Mutating the
caller's arrays, content blocks, image source, or tool input after calling an API
does not change that run. `Agent.getHistory()` also returns a deep copy.

Invalid message/content shapes or data that cannot be cloned throw
`InvalidMessageError` before `beforeRun`, provider/backend execution, progress,
or online evaluation. Invalid `prompt()` input is not appended to history.
For `stream()`, whose events are documented in
[streaming](streaming.md), this validation happens when `stream()` is called,
before the returned iterator starts.

## `beforeRun` semantics

`beforeRun` receives both views of the effective input:

```ts
beforeRun(ctx) {
  return {
    ...ctx,
    messages: ctx.messages, // complete defensive message copy
    prompt: ctx.prompt,     // text blocks from the latest user message
  }
}
```

`ctx.prompt` remains the backwards-compatible concatenation of text blocks from
the latest user message. An image-only turn therefore has an empty `prompt` but
is fully available in `ctx.messages`.

Returning `messages` replaces the complete input. If the hook also changes
`prompt`, OMA applies the message replacement first, then replaces the latest
user message's text blocks with one text block. Non-text blocks keep their
relative order. Hook inputs and execution messages are copies, so a hook rewrite
does not mutate caller-owned data or the original user turn stored by
`Agent.prompt()`.

## Progress and online evaluation

String `runAgent()` calls retain the existing `agent_start` payload
`{ prompt: string }` and string evaluation input. Structured calls emit
`{ messages: LLMMessage[] }` and submit a separate message copy to online
evaluation. Progress callback mutations cannot affect execution or evaluation.
The evaluator's existing `storePayloads` policy still applies: `none` omits the
content, while `redacted` or `full` serializes a bounded payload according to the
documented privacy contract.

## External process and ACP backends

Process and ACP backends expose text prompt transports, not OMA's structured
message protocol. Supplying `LLMMessage[]` to `run()` / `stream()` / `runAgent()`,
or `ContentBlock[]` to `prompt()`, throws `InvalidMessageError` before a process
is spawned or an ACP session is opened. This fail-fast boundary prevents images
or caller-owned history from being silently discarded. Pass a string instead.

For the same reason, an external agent's `beforeRun` may rewrite `prompt` but may
not change `messages`. Existing string execution, including the process backend's
fresh-per-run behavior and ACP's protocol session behavior, is unchanged.
`AgentConfig.history` does not seed either external transport; it restores
messages only for LLM-backed `prompt()` conversations.