feat(core): add OpenRouter image adapter (#613)

* feat(core): add OpenRouter image adapter

OpenRouter serves image generation and editing through its own /images endpoint, with input images in input_references, so OpenAIImageAdapter pointed at an OpenRouter baseURL cannot reach it. OpenRouterImageAdapter calls that endpoint with fetch, reads base64 output only, rejects a mask instead of dropping it, and honors egressPolicy.

Safety rejections arrive in two shapes. OpenRouter's own moderation answers 403 with reasons or flagged_input in error.metadata; a rejection by the upstream provider carries the provider's code in error.metadata.provider_code and its original body in error.metadata.raw. Both are classified as content_policy, reusing the OpenAI and Seedream rules for upstream codes. The shared content-policy rule now also receives the HTTP status and the full error object.

Follow-up to #610.

* fix(core): classify forwarded safety rejections before retryable 5xx

A gateway such as OpenRouter can forward an upstream safety rejection with a 5xx status. The shared image error classifier checked the 5xx branch first, so the adapter's content-policy rule never ran and the rejection was retried on the same model. The content-policy rule now runs before the timeout and 5xx branches; a 429 is still classified as a rate limit.

* fix(core): reject image providerOptions that replace adapter-set fields

providerOptions was spread last into the request body, so a key such as input_references, image, prompt, or model silently replaced what the adapter built from the request, and an edit could run on different images or none. The OpenAI, OpenRouter, and Seedream adapters now reject those keys with a TypeError at construction and spread the remaining options before their own fields.

* fix(core): keep size and watermark as image providerOptions defaults

The reserved-key check rejected size in every image adapter and watermark in Seedream, which removed a reasonable way to set a default output size. Only keys that carry the request itself (model, prompt, n, the input image fields, mask, and response_format) are rejected now. A size in providerOptions is a default that request.size overrides, and Seedream watermark in providerOptions again overrides the adapter default; the OpenAI multipart body sends a single size field.
This commit is contained in:
Jack Chen authored and GitHub committed 2026-09-23 23:49:17 +08:00
1 parent deab8804dd
commit 213bbf531b
12 files changed
+467 -34

No files matched your search

+4
View File
@@ -15,6 +15,10 @@
Volcengine Ark. Both classify safety rejections as `content_policy`, fail
rather than drop input they cannot use, and honor `egressPolicy`.
- Added `ImageModelError`; `isRetryableError()` returns its `retryable` flag.
- Added `OpenRouterImageAdapter` for OpenRouter's `/images` endpoint. It sends
input images as `input_references`, and classifies both OpenRouter's own
moderation rejections and upstream rejections forwarded in `error.metadata`
as `content_policy`.
## 1.20.0 - 2026-09-18
+1 -1
View File
@@ -73,7 +73,7 @@ endpoint or `AZURE_OPENAI_ENDPOINT` when a policy is active.
| Surface | Behavior while `egressPolicy` is configured |
|---|---|
| Anthropic, OpenAI, Azure OpenAI, DeepSeek, Doubao, Grok, Hunyuan, MiniMax, MiMo, and Qiniu built-in adapters | Enforced. OMA resolves and checks the effective provider origin before loading the optional SDK, injects a guarded fetch transport, checks every request again, and rejects redirects. |
| Built-in image adapters (`OpenAIImageAdapter`, `SeedreamImageAdapter`) | Enforced when `egressPolicy` is passed to the adapter. Every request is checked against the policy before it opens, and redirects are rejected. The adapters request base64 output and never fetch a returned image URL, so no second origin is contacted. A denial fails the attempt as a non-retryable `invalid_request`; see [image generation](image-generation.md). |
| Built-in image adapters (`OpenAIImageAdapter`, `OpenRouterImageAdapter`, `SeedreamImageAdapter`) | Enforced when `egressPolicy` is passed to the adapter. Every request is checked against the policy before it opens, and redirects are rejected. The adapters request base64 output and never fetch a returned image URL, so no second origin is contacted. A denial fails the attempt as a non-retryable `invalid_request`; see [image generation](image-generation.md). |
| Gemini built-in adapter | Unsupported and fail-closed before import or connection. The current Google GenAI SDK path uses module-global fetch and exposes no per-client transport hook to this adapter. |
| AWS Bedrock built-in adapter | Unsupported and fail-closed before import or connection. The current adapter cannot bound all request and AWS credential-provider endpoints, including identity/metadata paths. |
| AI SDK bridge or another custom `LLMAdapter` | Unsupported and fail-closed before adapter invocation. The model object does not expose a reliable target and transport contract to OMA. This also applies when a custom adapter is supplied to the built-in semantic profiler or coordinator. |
+16 -5
View File
@@ -93,7 +93,12 @@ Content-policy detection is per provider, because providers signal it
differently. The OpenAI adapter matches the error codes `moderation_blocked`
and `content_policy_violation`. The Seedream adapter matches Ark error codes
ending in `SensitiveContentDetected`, both in an error response and in a
per-image error inside a 200 response. Anything else lands in
per-image error inside a 200 response. The OpenRouter adapter matches a 403
carrying OpenRouter's moderation metadata (`reasons` or `flagged_input`), and
an upstream error whose code, in `error.metadata.provider_code` or in the
original body in `error.metadata.raw`, one of the two rules above recognizes.
These rules run before the timeout and 5xx rules, so a rejection a gateway
forwards with a 5xx status is not retried. Anything else lands in
`invalid_request`, which is also non-retryable, so an unrecognized rejection
still moves to the next model rather than being retried.
@@ -125,13 +130,19 @@ record to durable storage from the callback if you need it to survive a crash;
| Adapter | Endpoint | Credentials | Notes |
|---|---|---|---|
| `OpenAIImageAdapter` | `POST /images/generations` without images, `POST /images/edits` (multipart) with images | `apiKey` or `OPENAI_API_KEY`; `baseURL` or `OPENAI_BASE_URL` | Reads `data[0].b64_json` and never downloads a returned URL, so it targets models that return base64, such as the `gpt-image` family. Works with OpenAI-compatible endpoints through `baseURL`. |
| `OpenRouterImageAdapter` | `POST /images` on OpenRouter | `apiKey` or `OPENROUTER_API_KEY` | Sends input images as `input_references` data URLs and reads `data[0].b64_json`. OpenRouter's request shape differs from the OpenAI Images API, so `OpenAIImageAdapter` with an OpenRouter `baseURL` does not work. Rejects a mask. Its classification follows OpenRouter's documented error format and has not been checked against live responses. |
| `SeedreamImageAdapter` | `POST /images/generations` on Volcengine Ark | `apiKey` or `ARK_API_KEY` | Same endpoint and key as the Doubao text adapter. Sends input images as data URLs, always requests `b64_json`, and sets `watermark: false` unless configured. Rejects a mask. |
Both accept `providerOptions` for extra body fields, such as `quality` or
`moderation` for OpenAI and `seed` for Seedream, and `maxInputImages` to fail
over-long requests before any network call.
All three accept `providerOptions` for extra body fields, such as `quality` or
`moderation` for OpenAI, `aspect_ratio` or a `provider` routing object for
OpenRouter, and `seed` for Seedream, and `maxInputImages` to fail over-long
requests before any network call. A `size` in `providerOptions` is a default
that `request.size` overrides. A key that carries the request itself, such as
`model`, `prompt`, or the field that holds input images, is rejected with a
`TypeError` when the adapter is constructed, so an option can never silently
replace the prompt or the images.
Both honor `egressPolicy` the same way the text adapters do: every request is
All three honor `egressPolicy` the same way the text adapters do: every request is
checked against the policy and redirects are rejected. See
[LLM egress policy](egress-policy.md).
+5 -2
View File
@@ -253,8 +253,11 @@ The coordinator accepts the same hook via `runTeam(team, goal, { coordinator: {
Image generation and editing do not go through `provider` or `LLMAdapter`. They
use `ImageModelAdapter` and `runImage()`, with built-in adapters for the OpenAI
Images API (`OPENAI_API_KEY`, `OPENAI_BASE_URL`) and Seedream on Volcengine Ark
(`ARK_API_KEY`). See [image generation](image-generation.md).
Images API (`OPENAI_API_KEY`, `OPENAI_BASE_URL`), OpenRouter
(`OPENROUTER_API_KEY`), and Seedream on Volcengine Ark (`ARK_API_KEY`). The
OpenRouter image adapter is separate from the text route above because its
image endpoint has its own request shape. See
[image generation](image-generation.md).
## Extended thinking / reasoning
+1 -1
View File
@@ -245,7 +245,7 @@ Change `provider`, `model`, and credentials; the agent shape stays the same.
Optional integrations load only when used: core directly installs only `@anthropic-ai/sdk`, `openai`, and `zod`; other SDKs are lazy-loading opt-in peers, and OpenTelemetry lives entirely in `@open-multi-agent/otel`. Dependency changes are weighed on demonstrated value plus security, size, maintenance, and compatibility cost, not a fixed count.
Image generation and editing use a separate interface: `runImage()` tries an ordered chain of image models with per-model retries, fallback, output validation, and a record for every provider call, and ships native adapters for the OpenAI Images API and Seedream on Volcengine Ark. See [image generation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md).
Image generation and editing use a separate interface: `runImage()` tries an ordered chain of image models with per-model retries, fallback, output validation, and a record for every provider call, and ships native adapters for the OpenAI Images API, OpenRouter, and Seedream on Volcengine Ark. See [image generation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md).
See [Providers](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md), [framework-owned LLM egress policy](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/egress-policy.md), [Self-hosting and data residency](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/self-hosting.md), and [Tool configuration](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md) for credentials, models, the AI SDK bridge, reasoning settings, MCP, local endpoints, self-hosted deployment, and the exact network-enforcement boundary.
+1 -1
View File
@@ -238,7 +238,7 @@ Coordinator -> 任务 DAG -> Scheduler -> AgentPool
可选集成只在使用时加载:core 直接安装的只有 `@anthropic-ai/sdk`、`openai` 和 `zod`,其余 SDK 都是按需懒加载的可选 peer,OpenTelemetry 完全归属 `@open-multi-agent/otel`。依赖变更按实际价值与安全、体积、维护、兼容成本权衡,不设固定数量上限。
图像生成与编辑使用独立的接口:`runImage()` 按顺序尝试一组图像模型,支持单模型重试、降级、出图校验,并为每次服务商调用留存记录;内置 OpenAI Images API 与火山方舟 Seedream 的原生 adapter。详见[图像生成](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md)。
图像生成与编辑使用独立的接口:`runImage()` 按顺序尝试一组图像模型,支持单模型重试、降级、出图校验,并为每次服务商调用留存记录;内置 OpenAI Images API、OpenRouter 与火山方舟 Seedream 的原生 adapter。详见[图像生成](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md)。
凭证、模型、AI SDK 桥接、推理设置、MCP、本地端点配置、自托管部署,以及出网管控的确切生效边界,见 [Provider](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md)、[框架级 LLM 出网策略](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/egress-policy.md)、[自托管与数据驻留](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/self-hosting.md)和[工具配置](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md)。
+30 -5
View File
@@ -21,6 +21,24 @@ export function imageFetch(egressPolicy: EgressPolicy | undefined, provider: str
: createEgressFetch(egressPolicy, provider, lateFetch)
}
/**
* Reject `providerOptions` keys the adapter sets from the request or its own
* options. Letting them through would let a stray option replace the prompt or
* the input images without any error, so the conflict fails at construction.
*/
export function assertNoReservedOptions(
provider: string,
providerOptions: Readonly<Record<string, unknown>>,
isReserved: (key: string) => boolean,
): void {
const clashes = Object.keys(providerOptions).filter(isReserved)
if (clashes.length > 0) {
throw new TypeError(
`${provider} providerOptions cannot set ${clashes.join(', ')}; the adapter sets ${clashes.length === 1 ? 'it' : 'them'} from the request or its own options`,
)
}
}
/** Join a base URL and a path without doubling or dropping the slash. */
export function joinUrl(baseURL: string, path: string): string {
return `${baseURL.replace(/\/+$/, '')}/${path.replace(/^\/+/, '')}`
@@ -51,13 +69,18 @@ function truncate(value: string): string {
return value.length > MAX_DETAIL_CHARS ? `${value.slice(0, MAX_DETAIL_CHARS)}…` : value
}
/** The `{ error: { code, message } }` shape both built-in providers use. */
/** Fields read from the `{ error: { code, type, message } }` shape the built-in providers use. */
export interface ProviderErrorBody {
readonly code?: string
readonly type?: string
readonly message?: string
/** The whole `error` object (or body), for provider rules that need more fields. */
readonly error?: Readonly<Record<string, unknown>>
}
/** An adapter's own rule for recognizing a safety rejection. */
export type ContentPolicyRule = (body: ProviderErrorBody, status: number) => boolean
export function parseProviderErrorBody(text: string): ProviderErrorBody {
let parsed: unknown
try {
@@ -74,7 +97,7 @@ export function parseProviderErrorBody(text: string): ProviderErrorBody {
const value = inner[key]
return typeof value === 'string' || typeof value === 'number' ? String(value) : undefined
}
return { code: pick('code'), type: pick('type'), message: pick('message') }
return { code: pick('code'), type: pick('type'), message: pick('message'), error: inner }
}
/**
@@ -87,7 +110,7 @@ export function classifyHttpError(
status: number,
bodyText: string,
retryAfterHeader: string | null,
isContentPolicy: (body: ProviderErrorBody) => boolean,
isContentPolicy: ContentPolicyRule,
): ImageModelError {
const body = parseProviderErrorBody(bodyText)
const detail = truncate(body.message ?? bodyText)
@@ -99,9 +122,11 @@ export function classifyHttpError(
retryAfterMs: parseRetryAfter(retryAfterHeader),
})
}
// Checked before the retryable branches: a gateway can forward an upstream
// safety rejection with a 5xx status, and retrying it cannot succeed.
if (isContentPolicy(body, status)) return new ImageModelError('content_policy', message, false, options)
if (status === 408) return new ImageModelError('timeout', message, true, options)
if (status >= 500) return new ImageModelError('api_error', message, true, options)
if (isContentPolicy(body)) return new ImageModelError('content_policy', message, false, options)
return new ImageModelError('invalid_request', message, false, options)
}
@@ -116,7 +141,7 @@ export async function sendImageRequest(
url: string,
init: RequestInit,
signal: AbortSignal,
isContentPolicy: (body: ProviderErrorBody) => boolean,
isContentPolicy: ContentPolicyRule,
): Promise<unknown> {
let response: Response
let text: string
+15 -5
View File
@@ -11,6 +11,7 @@
import { ImageModelError } from '../errors.js'
import type { EgressPolicy } from '../types.js'
import {
assertNoReservedOptions,
extensionFor,
firstBase64Image,
imageFetch,
@@ -26,6 +27,7 @@ import type {
} from './types.js'
const DEFAULT_BASE_URL = 'https://api.openai.com/v1'
const RESERVED_OPTIONS = new Set(['model', 'prompt', 'n', 'image', 'image[]', 'mask'])
/**
* Error codes the Images API uses for a safety rejection:
@@ -53,7 +55,10 @@ export interface OpenAIImageAdapterOptions {
readonly maxInputImages?: number
/**
* Extra body fields sent with every call, for example `quality`,
* `background`, or `moderation`. Values are sent as-is.
* `background`, or `moderation`. Values are sent as-is. A `size` here is a
* default that `request.size` overrides. Fields that carry the request itself
* (`model`, `prompt`, `n`, `image`, `image[]`, `mask`) are rejected at
* construction.
*/
readonly providerOptions?: Readonly<Record<string, string | number | boolean>>
/** Restrict outbound requests; see the egress policy docs. */
@@ -75,6 +80,7 @@ export class OpenAIImageAdapter implements ImageModelAdapter {
this.baseURL = options.baseURL ?? process.env['OPENAI_BASE_URL'] ?? DEFAULT_BASE_URL
this.maxInputImages = options.maxInputImages
this.providerOptions = options.providerOptions ?? {}
assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key))
this.fetchImpl = imageFetch(options.egressPolicy, this.provider)
}
@@ -102,11 +108,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter {
const headers = { Authorization: `Bearer ${this.apiKey}` }
const params: Record<string, unknown> = {
...this.providerOptions,
model: this.model,
size: request.size,
...(request.size !== undefined ? { size: request.size } : {}),
inputImages: images.length,
mask: request.mask !== undefined,
...this.providerOptions,
}
let body: unknown
@@ -119,11 +125,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter {
method: 'POST',
headers: { ...headers, 'Content-Type': 'application/json' },
body: JSON.stringify({
...this.providerOptions,
model: this.model,
prompt: request.prompt,
n: 1,
...(request.size !== undefined ? { size: request.size } : {}),
...this.providerOptions,
}),
},
options.signal,
@@ -135,7 +141,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter {
form.append('prompt', request.prompt)
form.append('n', '1')
if (request.size !== undefined) form.append('size', request.size)
for (const [key, value] of Object.entries(this.providerOptions)) form.append(key, String(value))
for (const [key, value] of Object.entries(this.providerOptions)) {
// request.size, appended above, takes precedence over a default size.
if (key === 'size' && request.size !== undefined) continue
form.append(key, String(value))
}
const field = images.length > 1 ? 'image[]' : 'image'
images.forEach((image, index) => {
form.append(
+180
View File
@@ -0,0 +1,180 @@
/**
* @fileoverview Image adapter for OpenRouter's image generation endpoint.
*
* `POST /images` (JSON) serves both text-to-image and edits: input images go
* in `input_references` as data URLs. The response carries base64 images in
* `data[].b64_json`, so the adapter never downloads from a second origin.
* This is a different request shape from the OpenAI Images API, so it cannot
* be reached with `OpenAIImageAdapter` and a changed `baseURL`.
*/
import { ImageModelError } from '../errors.js'
import type { EgressPolicy } from '../types.js'
import {
assertNoReservedOptions,
firstBase64Image,
imageFetch,
joinUrl,
parseProviderErrorBody,
sendImageRequest,
toDataUrl,
type ProviderErrorBody,
} from './http.js'
import { isOpenAIContentPolicyError } from './openai.js'
import { isSeedreamContentPolicyError } from './seedream.js'
import type {
ImageCallOptions,
ImageModelAdapter,
ImageModelOutput,
ImageRequest,
} from './types.js'
const DEFAULT_BASE_URL = 'https://openrouter.ai/api/v1'
const RESERVED_OPTIONS = new Set(['model', 'prompt', 'n', 'input_references'])
function asRecord(value: unknown): Record<string, unknown> | undefined {
return value !== null && typeof value === 'object' && !Array.isArray(value)
? value as Record<string, unknown>
: undefined
}
/**
* OpenRouter reports a safety rejection in two ways. Its own moderation
* answers 403 with `error.metadata.reasons` or `flagged_input`. A rejection by
* the upstream provider carries the provider's error code in
* `error.metadata.provider_code` and its original body in
* `error.metadata.raw`; both are checked against the upstream rules OMA
* already knows.
*/
export function isOpenRouterContentPolicyError(body: ProviderErrorBody, status: number): boolean {
const metadata = asRecord(body.error?.['metadata'])
if (metadata === undefined) return false
if (status === 403 && (metadata['reasons'] !== undefined || metadata['flagged_input'] !== undefined)) {
return true
}
const providerCode = metadata['provider_code']
if (typeof providerCode === 'string') {
const upstreamCode = { code: providerCode }
if (isOpenAIContentPolicyError(upstreamCode) || isSeedreamContentPolicyError(upstreamCode)) return true
}
const raw = metadata['raw']
const upstream = typeof raw === 'string'
? parseProviderErrorBody(raw)
: asRecord(raw) !== undefined
? parseProviderErrorBody(JSON.stringify(raw))
: undefined
if (upstream === undefined) return false
return isOpenAIContentPolicyError(upstream) || isSeedreamContentPolicyError(upstream)
}
export interface OpenRouterImageAdapterOptions {
/** OpenRouter model slug, for example `openai/gpt-image-1`. */
readonly model: string
/** Defaults to `OPENROUTER_API_KEY`. */
readonly apiKey?: string
/** Defaults to the public OpenRouter endpoint. */
readonly baseURL?: string
/**
* Most input images to send. A request with more fails as `invalid_request`
* before any network call. Unset means the endpoint enforces its own limit.
*/
readonly maxInputImages?: number
/**
* Extra body fields sent with every call, for example `aspect_ratio`,
* `seed`, or a `provider` routing object. Values are sent as-is. A `size`
* here is a default that `request.size` overrides. Fields that carry the
* request itself (`model`, `prompt`, `n`, `input_references`) are rejected
* at construction.
*/
readonly providerOptions?: Readonly<Record<string, unknown>>
/** Restrict outbound requests; see the egress policy docs. */
readonly egressPolicy?: EgressPolicy
}
export class OpenRouterImageAdapter implements ImageModelAdapter {
readonly provider = 'openrouter'
readonly model: string
private readonly apiKey: string | undefined
private readonly baseURL: string
private readonly maxInputImages: number | undefined
private readonly providerOptions: Readonly<Record<string, unknown>>
private readonly fetchImpl: typeof globalThis.fetch
constructor(options: OpenRouterImageAdapterOptions) {
this.model = options.model
this.apiKey = options.apiKey ?? process.env['OPENROUTER_API_KEY']
this.baseURL = options.baseURL ?? DEFAULT_BASE_URL
this.maxInputImages = options.maxInputImages
this.providerOptions = options.providerOptions ?? {}
assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key))
this.fetchImpl = imageFetch(options.egressPolicy, this.provider)
}
async generate(request: ImageRequest, options: ImageCallOptions): Promise<ImageModelOutput> {
const images = request.images ?? []
if (request.mask !== undefined) {
throw new ImageModelError(
'invalid_request',
'openrouter does not accept a separate mask; describe the edit in the prompt instead',
false,
{ provider: this.provider },
)
}
if (this.maxInputImages !== undefined && images.length > this.maxInputImages) {
throw new ImageModelError(
'invalid_request',
`openrouter accepts at most ${this.maxInputImages} input images, got ${images.length}`,
false,
{ provider: this.provider },
)
}
if (this.apiKey === undefined || this.apiKey === '') {
throw new ImageModelError('invalid_request', 'openrouter API key is not set', false, { provider: this.provider })
}
const body = await sendImageRequest(
this.fetchImpl,
this.provider,
joinUrl(this.baseURL, 'images'),
{
method: 'POST',
headers: {
Authorization: `Bearer ${this.apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
...this.providerOptions,
model: this.model,
prompt: request.prompt,
n: 1,
...(request.size !== undefined ? { size: request.size } : {}),
...(images.length > 0
? {
input_references: images.map(image => ({
type: 'image_url',
image_url: { url: toDataUrl(image) },
})),
}
: {}),
}),
},
options.signal,
isOpenRouterContentPolicyError,
)
const { data } = firstBase64Image(this.provider, body)
const usage = asRecord(body)?.['usage']
const params: Record<string, unknown> = {
...this.providerOptions,
model: this.model,
...(request.size !== undefined ? { size: request.size } : {}),
inputImages: images.length,
}
return {
data,
// runImage re-derives the real type from the bytes.
mediaType: 'image/png',
params: usage === undefined ? params : { ...params, usage },
}
}
}
+12 -6
View File
@@ -10,6 +10,7 @@
import { ImageModelError } from '../errors.js'
import type { EgressPolicy } from '../types.js'
import {
assertNoReservedOptions,
firstBase64Image,
imageFetch,
joinUrl,
@@ -25,6 +26,7 @@ import type {
} from './types.js'
const DEFAULT_BASE_URL = 'https://ark.cn-beijing.volces.com/api/v3'
const RESERVED_OPTIONS = new Set(['model', 'prompt', 'image', 'response_format'])
/**
* Ark reports safety rejections with codes such as
@@ -50,7 +52,10 @@ export interface SeedreamImageAdapterOptions {
readonly watermark?: boolean
/**
* Extra body fields sent with every call, for example `seed`. Values are
* sent as-is and may override the defaults above.
* sent as-is and override `watermark`; a `size` here is a default that
* `request.size` overrides. Fields that carry the request itself or the
* output format (`model`, `prompt`, `image`, `response_format`) are
* rejected at construction.
*/
readonly providerOptions?: Readonly<Record<string, unknown>>
/** Restrict outbound requests; see the egress policy docs. */
@@ -74,6 +79,7 @@ export class SeedreamImageAdapter implements ImageModelAdapter {
this.maxInputImages = options.maxInputImages
this.watermark = options.watermark ?? false
this.providerOptions = options.providerOptions ?? {}
assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key))
this.fetchImpl = imageFetch(options.egressPolicy, this.provider)
}
@@ -115,12 +121,12 @@ export class SeedreamImageAdapter implements ImageModelAdapter {
'Content-Type': 'application/json',
},
body: JSON.stringify({
watermark: this.watermark,
...this.providerOptions,
model: this.model,
prompt: request.prompt,
...(image !== undefined ? { image } : {}),
...(request.size !== undefined ? { size: request.size } : {}),
watermark: this.watermark,
...this.providerOptions,
response_format: 'b64_json',
}),
},
@@ -153,11 +159,11 @@ export class SeedreamImageAdapter implements ImageModelAdapter {
? (body as Record<string, unknown>)['usage']
: undefined
const params: Record<string, unknown> = {
model: this.model,
size: request.size,
inputImages: images.length,
watermark: this.watermark,
...this.providerOptions,
model: this.model,
...(request.size !== undefined ? { size: request.size } : {}),
inputImages: images.length,
}
return {
data,
+2
View File
@@ -284,6 +284,8 @@ export type {
export { runImage } from './image/run-image.js'
export { OpenAIImageAdapter } from './image/openai.js'
export type { OpenAIImageAdapterOptions } from './image/openai.js'
export { OpenRouterImageAdapter } from './image/openrouter.js'
export type { OpenRouterImageAdapterOptions } from './image/openrouter.js'
export { SeedreamImageAdapter } from './image/seedream.js'
export type { SeedreamImageAdapterOptions } from './image/seedream.js'
export type {
+200 -8
View File
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { ImageModelError } from '../src/errors.js'
import { OpenAIImageAdapter } from '../src/image/openai.js'
import { OpenRouterImageAdapter } from '../src/image/openrouter.js'
import { runImage } from '../src/image/run-image.js'
import { SeedreamImageAdapter } from '../src/image/seedream.js'
import type { ImageRequest } from '../src/image/types.js'
@@ -104,6 +105,26 @@ describe('OpenAIImageAdapter', () => {
expect(form.getAll('image[]')).toHaveLength(0)
})
it('rejects providerOptions that would replace adapter-set fields', () => {
for (const key of ['model', 'prompt', 'n', 'image', 'image[]', 'mask']) {
expect(() => new OpenAIImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { [key]: 'x' } }))
.toThrow(`openai providerOptions cannot set ${key}`)
}
})
it('treats a size in providerOptions as a default that request.size overrides, without a duplicate field', async () => {
const fetchMock = stubFetch(
jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }),
jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }),
)
const custom = new OpenAIImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '1024x1024' } })
await custom.generate({ prompt: 'x', images: [input(pngHeader(4, 4))] }, { signal })
await custom.generate({ prompt: 'x', images: [input(pngHeader(4, 4))], size: '1536x1024' }, { signal })
const forms = fetchMock.mock.calls.map(([, init]) => init?.body as FormData)
expect(forms[0]!.getAll('size')).toEqual(['1024x1024'])
expect(forms[1]!.getAll('size')).toEqual(['1536x1024'])
})
it('refuses more images than maxInputImages without calling the API', async () => {
const fetchMock = stubFetch()
const limited = new OpenAIImageAdapter({ model: 'm', apiKey: 'k', maxInputImages: 1 })
@@ -214,16 +235,28 @@ describe('SeedreamImageAdapter', () => {
expect(typeof body.image).toBe('string')
})
it('does not let providerOptions switch the output to a URL', async () => {
it('sends free providerOptions and rejects ones that would replace adapter-set fields', async () => {
const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] }))
const custom = new SeedreamImageAdapter({
model: 'm',
apiKey: 'k',
providerOptions: { seed: 7, response_format: 'url' },
})
await new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { seed: 7 } }).generate({ prompt: 'x' }, { signal })
expect(JSON.parse(String(fetchMock.mock.calls[0]![1]?.body))).toMatchObject({ seed: 7, response_format: 'b64_json' })
for (const key of ['response_format', 'image', 'prompt', 'model']) {
expect(() => new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { [key]: 'x' } }))
.toThrow(`seedream providerOptions cannot set ${key}`)
}
})
it('treats size and watermark in providerOptions as defaults, with request.size taking precedence', async () => {
const fetchMock = stubFetch(
jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] }),
jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] }),
)
const custom = new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '2k', watermark: true } })
await custom.generate({ prompt: 'x' }, { signal })
const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body))
expect(body).toMatchObject({ seed: 7, response_format: 'b64_json' })
await custom.generate({ prompt: 'x', size: '4k' }, { signal })
const bodies = fetchMock.mock.calls.map(([, init]) => JSON.parse(String(init?.body)))
expect(bodies[0]).toMatchObject({ size: '2k', watermark: true })
expect(bodies[1]).toMatchObject({ size: '4k', watermark: true })
})
it('refuses a mask instead of dropping it', async () => {
@@ -252,6 +285,165 @@ describe('SeedreamImageAdapter', () => {
})
})
describe('OpenRouterImageAdapter', () => {
const adapter = () => new OpenRouterImageAdapter({
model: 'openai/gpt-image-1',
apiKey: 'or-test',
providerOptions: { aspect_ratio: '4:3', provider: { order: ['openai'] } },
})
it('sends input images as input_references data URLs to /images', async () => {
const first = pngHeader(4, 4)
const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(pngHeader(1024, 768)) }], usage: { total_tokens: 9 } }))
const output = await adapter().generate(
{ prompt: 'swap the chair', images: [input(first), input(jpegHeader(4, 4), 'image/jpeg')], size: '1024x768' },
{ signal },
)
const [url, init] = fetchMock.mock.calls[0]!
expect(url).toBe('https://openrouter.ai/api/v1/images')
expect((init?.headers as Record<string, string>)['Authorization']).toBe('Bearer or-test')
const body = JSON.parse(String(init?.body))
expect(body).toMatchObject({
model: 'openai/gpt-image-1',
prompt: 'swap the chair',
n: 1,
size: '1024x768',
aspect_ratio: '4:3',
provider: { order: ['openai'] },
})
expect(body.input_references).toEqual([
{ type: 'image_url', image_url: { url: `data:image/png;base64,${b64(first)}` } },
{ type: 'image_url', image_url: { url: expect.stringMatching(/^data:image\/jpeg;base64,/) } },
])
expect(output.params).toMatchObject({ inputImages: 2, usage: { total_tokens: 9 } })
})
it('rejects providerOptions that would replace the prompt or the input images', () => {
expect(() => new OpenRouterImageAdapter({
model: 'm',
apiKey: 'k',
providerOptions: { input_references: [], prompt: 'other' },
})).toThrow('openrouter providerOptions cannot set input_references, prompt')
})
it('treats a size in providerOptions as a default that request.size overrides', async () => {
const fetchMock = stubFetch(
jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }),
jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }),
)
const custom = new OpenRouterImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '1024x1024' } })
await custom.generate({ prompt: 'x' }, { signal })
await custom.generate({ prompt: 'x', size: '1536x1024' }, { signal })
const bodies = fetchMock.mock.calls.map(([, init]) => JSON.parse(String(init?.body)))
expect(bodies.map(body => body.size)).toEqual(['1024x1024', '1536x1024'])
})
it('omits input_references for text-to-image', async () => {
const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }))
await adapter().generate({ prompt: 'a desk' }, { signal })
const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body))
expect(body).not.toHaveProperty('input_references')
})
it('refuses a mask instead of dropping it', async () => {
const fetchMock = stubFetch()
const error = await failure(adapter().generate(
{ prompt: 'x', images: [input(pngHeader(4, 4))], mask: input(pngHeader(4, 4)) },
{ signal },
))
expect(error).toMatchObject({ type: 'invalid_request', retryable: false })
expect(fetchMock).not.toHaveBeenCalled()
})
it('classifies an OpenRouter moderation 403 as content_policy', async () => {
stubFetch(jsonResponse({
error: {
code: 403,
message: 'Your chosen model requires moderation and your input was flagged',
metadata: { reasons: ['sexual'], flagged_input: 'x', provider_name: 'OpenAI', model_slug: 'openai/gpt-image-1' },
},
}, 403))
const error = await failure(adapter().generate({ prompt: 'x' }, { signal }))
expect(error).toMatchObject({ type: 'content_policy', retryable: false, status: 403, provider: 'openrouter' })
})
it('classifies an upstream rejection forwarded in metadata.raw as content_policy', async () => {
stubFetch(
jsonResponse({
error: {
code: 400,
message: 'Provider returned error',
metadata: { provider_name: 'OpenAI', raw: JSON.stringify({ error: { code: 'moderation_blocked', message: 'rejected' } }) },
},
}, 400),
jsonResponse({
error: {
code: 400,
message: 'Provider returned error',
metadata: { provider_name: 'ByteDance', raw: { error: { code: 'OutputImageSensitiveContentDetected' } } },
},
}, 400),
)
expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false })
expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false })
})
it('classifies an upstream rejection forwarded with a 5xx status as content_policy, not a retryable api_error', async () => {
stubFetch(
jsonResponse({
error: {
code: 502,
message: 'Provider returned error',
metadata: { provider_name: 'OpenAI', raw: JSON.stringify({ error: { code: 'moderation_blocked' } }) },
},
}, 502),
jsonResponse({ error: { code: 502, message: 'Provider returned error', metadata: { provider_name: 'OpenAI' } } }, 502),
)
expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false, status: 502 })
expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'api_error', retryable: true, status: 502 })
})
it('classifies an upstream rejection reported in metadata.provider_code as content_policy', async () => {
stubFetch(jsonResponse({
error: { code: 400, message: 'Provider returned error', metadata: { provider_name: 'OpenAI', provider_code: 'moderation_blocked' } },
}, 400))
const error = await failure(adapter().generate({ prompt: 'x' }, { signal }))
expect(error).toMatchObject({ type: 'content_policy', retryable: false })
})
it('treats other 4xx, including a 403 without moderation metadata, as invalid_request', async () => {
stubFetch(
jsonResponse({ error: { code: 402, message: 'Insufficient credits' } }, 402),
jsonResponse({ error: { code: 403, message: 'Key disabled' } }, 403),
jsonResponse({ error: { code: 400, message: 'bad', metadata: { raw: 'not json' } } }, 400),
jsonResponse({ error: { code: 400, message: 'bad', metadata: { provider_code: 'invalid_size' } } }, 400),
)
for (let i = 0; i < 4; i++) {
expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'invalid_request', retryable: false })
}
})
it('treats an empty data array as a retryable invalid_output', async () => {
stubFetch(jsonResponse({ data: [] }))
const error = await failure(adapter().generate({ prompt: 'x' }, { signal }))
expect(error).toMatchObject({ type: 'invalid_output', retryable: true })
})
it('fails without a network call when the egress policy denies OpenRouter', async () => {
const fetchMock = stubFetch()
const guarded = new OpenRouterImageAdapter({
model: 'm',
apiKey: 'k',
egressPolicy: { mode: 'allowlist', allowedOrigins: ['https://api.openai.com'] },
})
const error = await failure(guarded.generate({ prompt: 'x' }, { signal }))
expect(error).toMatchObject({ type: 'invalid_request', retryable: false })
expect(fetchMock).not.toHaveBeenCalled()
})
})
describe('runImage with the built-in adapters', () => {
it('falls back from a moderated provider to the next one and records both calls', async () => {
stubFetch(