diff --git a/CHANGELOG.md b/CHANGELOG.md index b95218e..bda9d95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ Volcengine Ark. Both classify safety rejections as `content_policy`, fail rather than drop input they cannot use, and honor `egressPolicy`. - Added `ImageModelError`; `isRetryableError()` returns its `retryable` flag. +- Added `OpenRouterImageAdapter` for OpenRouter's `/images` endpoint. It sends + input images as `input_references`, and classifies both OpenRouter's own + moderation rejections and upstream rejections forwarded in `error.metadata` + as `content_policy`. ## 1.20.0 - 2026-09-18 diff --git a/docs/egress-policy.md b/docs/egress-policy.md index 821dd12..3102bf2 100644 --- a/docs/egress-policy.md +++ b/docs/egress-policy.md @@ -73,7 +73,7 @@ endpoint or `AZURE_OPENAI_ENDPOINT` when a policy is active. | Surface | Behavior while `egressPolicy` is configured | |---|---| | Anthropic, OpenAI, Azure OpenAI, DeepSeek, Doubao, Grok, Hunyuan, MiniMax, MiMo, and Qiniu built-in adapters | Enforced. OMA resolves and checks the effective provider origin before loading the optional SDK, injects a guarded fetch transport, checks every request again, and rejects redirects. | -| Built-in image adapters (`OpenAIImageAdapter`, `SeedreamImageAdapter`) | Enforced when `egressPolicy` is passed to the adapter. Every request is checked against the policy before it opens, and redirects are rejected. The adapters request base64 output and never fetch a returned image URL, so no second origin is contacted. A denial fails the attempt as a non-retryable `invalid_request`; see [image generation](image-generation.md). | +| Built-in image adapters (`OpenAIImageAdapter`, `OpenRouterImageAdapter`, `SeedreamImageAdapter`) | Enforced when `egressPolicy` is passed to the adapter. Every request is checked against the policy before it opens, and redirects are rejected. The adapters request base64 output and never fetch a returned image URL, so no second origin is contacted. A denial fails the attempt as a non-retryable `invalid_request`; see [image generation](image-generation.md). | | Gemini built-in adapter | Unsupported and fail-closed before import or connection. The current Google GenAI SDK path uses module-global fetch and exposes no per-client transport hook to this adapter. | | AWS Bedrock built-in adapter | Unsupported and fail-closed before import or connection. The current adapter cannot bound all request and AWS credential-provider endpoints, including identity/metadata paths. | | AI SDK bridge or another custom `LLMAdapter` | Unsupported and fail-closed before adapter invocation. The model object does not expose a reliable target and transport contract to OMA. This also applies when a custom adapter is supplied to the built-in semantic profiler or coordinator. | diff --git a/docs/image-generation.md b/docs/image-generation.md index 5ba3c01..63cb587 100644 --- a/docs/image-generation.md +++ b/docs/image-generation.md @@ -93,7 +93,12 @@ Content-policy detection is per provider, because providers signal it differently. The OpenAI adapter matches the error codes `moderation_blocked` and `content_policy_violation`. The Seedream adapter matches Ark error codes ending in `SensitiveContentDetected`, both in an error response and in a -per-image error inside a 200 response. Anything else lands in +per-image error inside a 200 response. The OpenRouter adapter matches a 403 +carrying OpenRouter's moderation metadata (`reasons` or `flagged_input`), and +an upstream error whose code, in `error.metadata.provider_code` or in the +original body in `error.metadata.raw`, one of the two rules above recognizes. +These rules run before the timeout and 5xx rules, so a rejection a gateway +forwards with a 5xx status is not retried. Anything else lands in `invalid_request`, which is also non-retryable, so an unrecognized rejection still moves to the next model rather than being retried. @@ -125,13 +130,19 @@ record to durable storage from the callback if you need it to survive a crash; | Adapter | Endpoint | Credentials | Notes | |---|---|---|---| | `OpenAIImageAdapter` | `POST /images/generations` without images, `POST /images/edits` (multipart) with images | `apiKey` or `OPENAI_API_KEY`; `baseURL` or `OPENAI_BASE_URL` | Reads `data[0].b64_json` and never downloads a returned URL, so it targets models that return base64, such as the `gpt-image` family. Works with OpenAI-compatible endpoints through `baseURL`. | +| `OpenRouterImageAdapter` | `POST /images` on OpenRouter | `apiKey` or `OPENROUTER_API_KEY` | Sends input images as `input_references` data URLs and reads `data[0].b64_json`. OpenRouter's request shape differs from the OpenAI Images API, so `OpenAIImageAdapter` with an OpenRouter `baseURL` does not work. Rejects a mask. Its classification follows OpenRouter's documented error format and has not been checked against live responses. | | `SeedreamImageAdapter` | `POST /images/generations` on Volcengine Ark | `apiKey` or `ARK_API_KEY` | Same endpoint and key as the Doubao text adapter. Sends input images as data URLs, always requests `b64_json`, and sets `watermark: false` unless configured. Rejects a mask. | -Both accept `providerOptions` for extra body fields, such as `quality` or -`moderation` for OpenAI and `seed` for Seedream, and `maxInputImages` to fail -over-long requests before any network call. +All three accept `providerOptions` for extra body fields, such as `quality` or +`moderation` for OpenAI, `aspect_ratio` or a `provider` routing object for +OpenRouter, and `seed` for Seedream, and `maxInputImages` to fail over-long +requests before any network call. A `size` in `providerOptions` is a default +that `request.size` overrides. A key that carries the request itself, such as +`model`, `prompt`, or the field that holds input images, is rejected with a +`TypeError` when the adapter is constructed, so an option can never silently +replace the prompt or the images. -Both honor `egressPolicy` the same way the text adapters do: every request is +All three honor `egressPolicy` the same way the text adapters do: every request is checked against the policy and redirects are rejected. See [LLM egress policy](egress-policy.md). diff --git a/docs/providers.md b/docs/providers.md index cbd0dfb..9e3f668 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -253,8 +253,11 @@ The coordinator accepts the same hook via `runTeam(team, goal, { coordinator: { Image generation and editing do not go through `provider` or `LLMAdapter`. They use `ImageModelAdapter` and `runImage()`, with built-in adapters for the OpenAI -Images API (`OPENAI_API_KEY`, `OPENAI_BASE_URL`) and Seedream on Volcengine Ark -(`ARK_API_KEY`). See [image generation](image-generation.md). +Images API (`OPENAI_API_KEY`, `OPENAI_BASE_URL`), OpenRouter +(`OPENROUTER_API_KEY`), and Seedream on Volcengine Ark (`ARK_API_KEY`). The +OpenRouter image adapter is separate from the text route above because its +image endpoint has its own request shape. See +[image generation](image-generation.md). ## Extended thinking / reasoning diff --git a/packages/core/README.md b/packages/core/README.md index 516cdac..b5f860f 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -245,7 +245,7 @@ Change `provider`, `model`, and credentials; the agent shape stays the same. Optional integrations load only when used: core directly installs only `@anthropic-ai/sdk`, `openai`, and `zod`; other SDKs are lazy-loading opt-in peers, and OpenTelemetry lives entirely in `@open-multi-agent/otel`. Dependency changes are weighed on demonstrated value plus security, size, maintenance, and compatibility cost, not a fixed count. -Image generation and editing use a separate interface: `runImage()` tries an ordered chain of image models with per-model retries, fallback, output validation, and a record for every provider call, and ships native adapters for the OpenAI Images API and Seedream on Volcengine Ark. See [image generation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md). +Image generation and editing use a separate interface: `runImage()` tries an ordered chain of image models with per-model retries, fallback, output validation, and a record for every provider call, and ships native adapters for the OpenAI Images API, OpenRouter, and Seedream on Volcengine Ark. See [image generation](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md). See [Providers](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md), [framework-owned LLM egress policy](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/egress-policy.md), [Self-hosting and data residency](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/self-hosting.md), and [Tool configuration](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md) for credentials, models, the AI SDK bridge, reasoning settings, MCP, local endpoints, self-hosted deployment, and the exact network-enforcement boundary. diff --git a/packages/core/README_zh.md b/packages/core/README_zh.md index 25d1a34..875b96b 100644 --- a/packages/core/README_zh.md +++ b/packages/core/README_zh.md @@ -238,7 +238,7 @@ Coordinator -> 任务 DAG -> Scheduler -> AgentPool 可选集成只在使用时加载：core 直接安装的只有 `@anthropic-ai/sdk`、`openai` 和 `zod`，其余 SDK 都是按需懒加载的可选 peer，OpenTelemetry 完全归属 `@open-multi-agent/otel`。依赖变更按实际价值与安全、体积、维护、兼容成本权衡，不设固定数量上限。 -图像生成与编辑使用独立的接口：`runImage()` 按顺序尝试一组图像模型，支持单模型重试、降级、出图校验，并为每次服务商调用留存记录；内置 OpenAI Images API 与火山方舟 Seedream 的原生 adapter。详见[图像生成](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md)。 +图像生成与编辑使用独立的接口：`runImage()` 按顺序尝试一组图像模型，支持单模型重试、降级、出图校验，并为每次服务商调用留存记录；内置 OpenAI Images API、OpenRouter 与火山方舟 Seedream 的原生 adapter。详见[图像生成](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/image-generation.md)。 凭证、模型、AI SDK 桥接、推理设置、MCP、本地端点配置、自托管部署，以及出网管控的确切生效边界，见 [Provider](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/providers.md)、[框架级 LLM 出网策略](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/egress-policy.md)、[自托管与数据驻留](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/self-hosting.md)和[工具配置](https://github.com/open-multi-agent/open-multi-agent/blob/main/docs/tool-configuration.md)。 diff --git a/packages/core/src/image/http.ts b/packages/core/src/image/http.ts index 69352d6..cb425b0 100644 --- a/packages/core/src/image/http.ts +++ b/packages/core/src/image/http.ts @@ -21,6 +21,24 @@ export function imageFetch(egressPolicy: EgressPolicy | undefined, provider: str : createEgressFetch(egressPolicy, provider, lateFetch) } +/** + * Reject `providerOptions` keys the adapter sets from the request or its own + * options. Letting them through would let a stray option replace the prompt or + * the input images without any error, so the conflict fails at construction. + */ +export function assertNoReservedOptions( + provider: string, + providerOptions: Readonly>, + isReserved: (key: string) => boolean, +): void { + const clashes = Object.keys(providerOptions).filter(isReserved) + if (clashes.length > 0) { + throw new TypeError( + `${provider} providerOptions cannot set ${clashes.join(', ')}; the adapter sets ${clashes.length === 1 ? 'it' : 'them'} from the request or its own options`, + ) + } +} + /** Join a base URL and a path without doubling or dropping the slash. */ export function joinUrl(baseURL: string, path: string): string { return `${baseURL.replace(/\/+$/, '')}/${path.replace(/^\/+/, '')}` @@ -51,13 +69,18 @@ function truncate(value: string): string { return value.length > MAX_DETAIL_CHARS ? `${value.slice(0, MAX_DETAIL_CHARS)}…` : value } -/** The `{ error: { code, message } }` shape both built-in providers use. */ +/** Fields read from the `{ error: { code, type, message } }` shape the built-in providers use. */ export interface ProviderErrorBody { readonly code?: string readonly type?: string readonly message?: string + /** The whole `error` object (or body), for provider rules that need more fields. */ + readonly error?: Readonly> } +/** An adapter's own rule for recognizing a safety rejection. */ +export type ContentPolicyRule = (body: ProviderErrorBody, status: number) => boolean + export function parseProviderErrorBody(text: string): ProviderErrorBody { let parsed: unknown try { @@ -74,7 +97,7 @@ export function parseProviderErrorBody(text: string): ProviderErrorBody { const value = inner[key] return typeof value === 'string' || typeof value === 'number' ? String(value) : undefined } - return { code: pick('code'), type: pick('type'), message: pick('message') } + return { code: pick('code'), type: pick('type'), message: pick('message'), error: inner } } /** @@ -87,7 +110,7 @@ export function classifyHttpError( status: number, bodyText: string, retryAfterHeader: string | null, - isContentPolicy: (body: ProviderErrorBody) => boolean, + isContentPolicy: ContentPolicyRule, ): ImageModelError { const body = parseProviderErrorBody(bodyText) const detail = truncate(body.message ?? bodyText) @@ -99,9 +122,11 @@ export function classifyHttpError( retryAfterMs: parseRetryAfter(retryAfterHeader), }) } + // Checked before the retryable branches: a gateway can forward an upstream + // safety rejection with a 5xx status, and retrying it cannot succeed. + if (isContentPolicy(body, status)) return new ImageModelError('content_policy', message, false, options) if (status === 408) return new ImageModelError('timeout', message, true, options) if (status >= 500) return new ImageModelError('api_error', message, true, options) - if (isContentPolicy(body)) return new ImageModelError('content_policy', message, false, options) return new ImageModelError('invalid_request', message, false, options) } @@ -116,7 +141,7 @@ export async function sendImageRequest( url: string, init: RequestInit, signal: AbortSignal, - isContentPolicy: (body: ProviderErrorBody) => boolean, + isContentPolicy: ContentPolicyRule, ): Promise { let response: Response let text: string diff --git a/packages/core/src/image/openai.ts b/packages/core/src/image/openai.ts index 5532c6b..2248d0d 100644 --- a/packages/core/src/image/openai.ts +++ b/packages/core/src/image/openai.ts @@ -11,6 +11,7 @@ import { ImageModelError } from '../errors.js' import type { EgressPolicy } from '../types.js' import { + assertNoReservedOptions, extensionFor, firstBase64Image, imageFetch, @@ -26,6 +27,7 @@ import type { } from './types.js' const DEFAULT_BASE_URL = 'https://api.openai.com/v1' +const RESERVED_OPTIONS = new Set(['model', 'prompt', 'n', 'image', 'image[]', 'mask']) /** * Error codes the Images API uses for a safety rejection: @@ -53,7 +55,10 @@ export interface OpenAIImageAdapterOptions { readonly maxInputImages?: number /** * Extra body fields sent with every call, for example `quality`, - * `background`, or `moderation`. Values are sent as-is. + * `background`, or `moderation`. Values are sent as-is. A `size` here is a + * default that `request.size` overrides. Fields that carry the request itself + * (`model`, `prompt`, `n`, `image`, `image[]`, `mask`) are rejected at + * construction. */ readonly providerOptions?: Readonly> /** Restrict outbound requests; see the egress policy docs. */ @@ -75,6 +80,7 @@ export class OpenAIImageAdapter implements ImageModelAdapter { this.baseURL = options.baseURL ?? process.env['OPENAI_BASE_URL'] ?? DEFAULT_BASE_URL this.maxInputImages = options.maxInputImages this.providerOptions = options.providerOptions ?? {} + assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key)) this.fetchImpl = imageFetch(options.egressPolicy, this.provider) } @@ -102,11 +108,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter { const headers = { Authorization: `Bearer ${this.apiKey}` } const params: Record = { + ...this.providerOptions, model: this.model, - size: request.size, + ...(request.size !== undefined ? { size: request.size } : {}), inputImages: images.length, mask: request.mask !== undefined, - ...this.providerOptions, } let body: unknown @@ -119,11 +125,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter { method: 'POST', headers: { ...headers, 'Content-Type': 'application/json' }, body: JSON.stringify({ + ...this.providerOptions, model: this.model, prompt: request.prompt, n: 1, ...(request.size !== undefined ? { size: request.size } : {}), - ...this.providerOptions, }), }, options.signal, @@ -135,7 +141,11 @@ export class OpenAIImageAdapter implements ImageModelAdapter { form.append('prompt', request.prompt) form.append('n', '1') if (request.size !== undefined) form.append('size', request.size) - for (const [key, value] of Object.entries(this.providerOptions)) form.append(key, String(value)) + for (const [key, value] of Object.entries(this.providerOptions)) { + // request.size, appended above, takes precedence over a default size. + if (key === 'size' && request.size !== undefined) continue + form.append(key, String(value)) + } const field = images.length > 1 ? 'image[]' : 'image' images.forEach((image, index) => { form.append( diff --git a/packages/core/src/image/openrouter.ts b/packages/core/src/image/openrouter.ts new file mode 100644 index 0000000..b1bb5c6 --- /dev/null +++ b/packages/core/src/image/openrouter.ts @@ -0,0 +1,180 @@ +/** + * @fileoverview Image adapter for OpenRouter's image generation endpoint. + * + * `POST /images` (JSON) serves both text-to-image and edits: input images go + * in `input_references` as data URLs. The response carries base64 images in + * `data[].b64_json`, so the adapter never downloads from a second origin. + * This is a different request shape from the OpenAI Images API, so it cannot + * be reached with `OpenAIImageAdapter` and a changed `baseURL`. + */ + +import { ImageModelError } from '../errors.js' +import type { EgressPolicy } from '../types.js' +import { + assertNoReservedOptions, + firstBase64Image, + imageFetch, + joinUrl, + parseProviderErrorBody, + sendImageRequest, + toDataUrl, + type ProviderErrorBody, +} from './http.js' +import { isOpenAIContentPolicyError } from './openai.js' +import { isSeedreamContentPolicyError } from './seedream.js' +import type { + ImageCallOptions, + ImageModelAdapter, + ImageModelOutput, + ImageRequest, +} from './types.js' + +const DEFAULT_BASE_URL = 'https://openrouter.ai/api/v1' +const RESERVED_OPTIONS = new Set(['model', 'prompt', 'n', 'input_references']) + +function asRecord(value: unknown): Record | undefined { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : undefined +} + +/** + * OpenRouter reports a safety rejection in two ways. Its own moderation + * answers 403 with `error.metadata.reasons` or `flagged_input`. A rejection by + * the upstream provider carries the provider's error code in + * `error.metadata.provider_code` and its original body in + * `error.metadata.raw`; both are checked against the upstream rules OMA + * already knows. + */ +export function isOpenRouterContentPolicyError(body: ProviderErrorBody, status: number): boolean { + const metadata = asRecord(body.error?.['metadata']) + if (metadata === undefined) return false + if (status === 403 && (metadata['reasons'] !== undefined || metadata['flagged_input'] !== undefined)) { + return true + } + const providerCode = metadata['provider_code'] + if (typeof providerCode === 'string') { + const upstreamCode = { code: providerCode } + if (isOpenAIContentPolicyError(upstreamCode) || isSeedreamContentPolicyError(upstreamCode)) return true + } + const raw = metadata['raw'] + const upstream = typeof raw === 'string' + ? parseProviderErrorBody(raw) + : asRecord(raw) !== undefined + ? parseProviderErrorBody(JSON.stringify(raw)) + : undefined + if (upstream === undefined) return false + return isOpenAIContentPolicyError(upstream) || isSeedreamContentPolicyError(upstream) +} + +export interface OpenRouterImageAdapterOptions { + /** OpenRouter model slug, for example `openai/gpt-image-1`. */ + readonly model: string + /** Defaults to `OPENROUTER_API_KEY`. */ + readonly apiKey?: string + /** Defaults to the public OpenRouter endpoint. */ + readonly baseURL?: string + /** + * Most input images to send. A request with more fails as `invalid_request` + * before any network call. Unset means the endpoint enforces its own limit. + */ + readonly maxInputImages?: number + /** + * Extra body fields sent with every call, for example `aspect_ratio`, + * `seed`, or a `provider` routing object. Values are sent as-is. A `size` + * here is a default that `request.size` overrides. Fields that carry the + * request itself (`model`, `prompt`, `n`, `input_references`) are rejected + * at construction. + */ + readonly providerOptions?: Readonly> + /** Restrict outbound requests; see the egress policy docs. */ + readonly egressPolicy?: EgressPolicy +} + +export class OpenRouterImageAdapter implements ImageModelAdapter { + readonly provider = 'openrouter' + readonly model: string + private readonly apiKey: string | undefined + private readonly baseURL: string + private readonly maxInputImages: number | undefined + private readonly providerOptions: Readonly> + private readonly fetchImpl: typeof globalThis.fetch + + constructor(options: OpenRouterImageAdapterOptions) { + this.model = options.model + this.apiKey = options.apiKey ?? process.env['OPENROUTER_API_KEY'] + this.baseURL = options.baseURL ?? DEFAULT_BASE_URL + this.maxInputImages = options.maxInputImages + this.providerOptions = options.providerOptions ?? {} + assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key)) + this.fetchImpl = imageFetch(options.egressPolicy, this.provider) + } + + async generate(request: ImageRequest, options: ImageCallOptions): Promise { + const images = request.images ?? [] + if (request.mask !== undefined) { + throw new ImageModelError( + 'invalid_request', + 'openrouter does not accept a separate mask; describe the edit in the prompt instead', + false, + { provider: this.provider }, + ) + } + if (this.maxInputImages !== undefined && images.length > this.maxInputImages) { + throw new ImageModelError( + 'invalid_request', + `openrouter accepts at most ${this.maxInputImages} input images, got ${images.length}`, + false, + { provider: this.provider }, + ) + } + if (this.apiKey === undefined || this.apiKey === '') { + throw new ImageModelError('invalid_request', 'openrouter API key is not set', false, { provider: this.provider }) + } + + const body = await sendImageRequest( + this.fetchImpl, + this.provider, + joinUrl(this.baseURL, 'images'), + { + method: 'POST', + headers: { + Authorization: `Bearer ${this.apiKey}`, + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + ...this.providerOptions, + model: this.model, + prompt: request.prompt, + n: 1, + ...(request.size !== undefined ? { size: request.size } : {}), + ...(images.length > 0 + ? { + input_references: images.map(image => ({ + type: 'image_url', + image_url: { url: toDataUrl(image) }, + })), + } + : {}), + }), + }, + options.signal, + isOpenRouterContentPolicyError, + ) + + const { data } = firstBase64Image(this.provider, body) + const usage = asRecord(body)?.['usage'] + const params: Record = { + ...this.providerOptions, + model: this.model, + ...(request.size !== undefined ? { size: request.size } : {}), + inputImages: images.length, + } + return { + data, + // runImage re-derives the real type from the bytes. + mediaType: 'image/png', + params: usage === undefined ? params : { ...params, usage }, + } + } +} diff --git a/packages/core/src/image/seedream.ts b/packages/core/src/image/seedream.ts index 2112aaf..68fff7a 100644 --- a/packages/core/src/image/seedream.ts +++ b/packages/core/src/image/seedream.ts @@ -10,6 +10,7 @@ import { ImageModelError } from '../errors.js' import type { EgressPolicy } from '../types.js' import { + assertNoReservedOptions, firstBase64Image, imageFetch, joinUrl, @@ -25,6 +26,7 @@ import type { } from './types.js' const DEFAULT_BASE_URL = 'https://ark.cn-beijing.volces.com/api/v3' +const RESERVED_OPTIONS = new Set(['model', 'prompt', 'image', 'response_format']) /** * Ark reports safety rejections with codes such as @@ -50,7 +52,10 @@ export interface SeedreamImageAdapterOptions { readonly watermark?: boolean /** * Extra body fields sent with every call, for example `seed`. Values are - * sent as-is and may override the defaults above. + * sent as-is and override `watermark`; a `size` here is a default that + * `request.size` overrides. Fields that carry the request itself or the + * output format (`model`, `prompt`, `image`, `response_format`) are + * rejected at construction. */ readonly providerOptions?: Readonly> /** Restrict outbound requests; see the egress policy docs. */ @@ -74,6 +79,7 @@ export class SeedreamImageAdapter implements ImageModelAdapter { this.maxInputImages = options.maxInputImages this.watermark = options.watermark ?? false this.providerOptions = options.providerOptions ?? {} + assertNoReservedOptions(this.provider, this.providerOptions, key => RESERVED_OPTIONS.has(key)) this.fetchImpl = imageFetch(options.egressPolicy, this.provider) } @@ -115,12 +121,12 @@ export class SeedreamImageAdapter implements ImageModelAdapter { 'Content-Type': 'application/json', }, body: JSON.stringify({ + watermark: this.watermark, + ...this.providerOptions, model: this.model, prompt: request.prompt, ...(image !== undefined ? { image } : {}), ...(request.size !== undefined ? { size: request.size } : {}), - watermark: this.watermark, - ...this.providerOptions, response_format: 'b64_json', }), }, @@ -153,11 +159,11 @@ export class SeedreamImageAdapter implements ImageModelAdapter { ? (body as Record)['usage'] : undefined const params: Record = { - model: this.model, - size: request.size, - inputImages: images.length, watermark: this.watermark, ...this.providerOptions, + model: this.model, + ...(request.size !== undefined ? { size: request.size } : {}), + inputImages: images.length, } return { data, diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 780603c..fcd195c 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -284,6 +284,8 @@ export type { export { runImage } from './image/run-image.js' export { OpenAIImageAdapter } from './image/openai.js' export type { OpenAIImageAdapterOptions } from './image/openai.js' +export { OpenRouterImageAdapter } from './image/openrouter.js' +export type { OpenRouterImageAdapterOptions } from './image/openrouter.js' export { SeedreamImageAdapter } from './image/seedream.js' export type { SeedreamImageAdapterOptions } from './image/seedream.js' export type { diff --git a/packages/core/tests/image-adapters.test.ts b/packages/core/tests/image-adapters.test.ts index b351883..ae34742 100644 --- a/packages/core/tests/image-adapters.test.ts +++ b/packages/core/tests/image-adapters.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { ImageModelError } from '../src/errors.js' import { OpenAIImageAdapter } from '../src/image/openai.js' +import { OpenRouterImageAdapter } from '../src/image/openrouter.js' import { runImage } from '../src/image/run-image.js' import { SeedreamImageAdapter } from '../src/image/seedream.js' import type { ImageRequest } from '../src/image/types.js' @@ -104,6 +105,26 @@ describe('OpenAIImageAdapter', () => { expect(form.getAll('image[]')).toHaveLength(0) }) + it('rejects providerOptions that would replace adapter-set fields', () => { + for (const key of ['model', 'prompt', 'n', 'image', 'image[]', 'mask']) { + expect(() => new OpenAIImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { [key]: 'x' } })) + .toThrow(`openai providerOptions cannot set ${key}`) + } + }) + + it('treats a size in providerOptions as a default that request.size overrides, without a duplicate field', async () => { + const fetchMock = stubFetch( + jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }), + jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }), + ) + const custom = new OpenAIImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '1024x1024' } }) + await custom.generate({ prompt: 'x', images: [input(pngHeader(4, 4))] }, { signal }) + await custom.generate({ prompt: 'x', images: [input(pngHeader(4, 4))], size: '1536x1024' }, { signal }) + const forms = fetchMock.mock.calls.map(([, init]) => init?.body as FormData) + expect(forms[0]!.getAll('size')).toEqual(['1024x1024']) + expect(forms[1]!.getAll('size')).toEqual(['1536x1024']) + }) + it('refuses more images than maxInputImages without calling the API', async () => { const fetchMock = stubFetch() const limited = new OpenAIImageAdapter({ model: 'm', apiKey: 'k', maxInputImages: 1 }) @@ -214,16 +235,28 @@ describe('SeedreamImageAdapter', () => { expect(typeof body.image).toBe('string') }) - it('does not let providerOptions switch the output to a URL', async () => { + it('sends free providerOptions and rejects ones that would replace adapter-set fields', async () => { const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] })) - const custom = new SeedreamImageAdapter({ - model: 'm', - apiKey: 'k', - providerOptions: { seed: 7, response_format: 'url' }, - }) + await new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { seed: 7 } }).generate({ prompt: 'x' }, { signal }) + expect(JSON.parse(String(fetchMock.mock.calls[0]![1]?.body))).toMatchObject({ seed: 7, response_format: 'b64_json' }) + + for (const key of ['response_format', 'image', 'prompt', 'model']) { + expect(() => new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { [key]: 'x' } })) + .toThrow(`seedream providerOptions cannot set ${key}`) + } + }) + + it('treats size and watermark in providerOptions as defaults, with request.size taking precedence', async () => { + const fetchMock = stubFetch( + jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] }), + jsonResponse({ data: [{ b64_json: b64(jpegHeader(8, 8)) }] }), + ) + const custom = new SeedreamImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '2k', watermark: true } }) await custom.generate({ prompt: 'x' }, { signal }) - const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) - expect(body).toMatchObject({ seed: 7, response_format: 'b64_json' }) + await custom.generate({ prompt: 'x', size: '4k' }, { signal }) + const bodies = fetchMock.mock.calls.map(([, init]) => JSON.parse(String(init?.body))) + expect(bodies[0]).toMatchObject({ size: '2k', watermark: true }) + expect(bodies[1]).toMatchObject({ size: '4k', watermark: true }) }) it('refuses a mask instead of dropping it', async () => { @@ -252,6 +285,165 @@ describe('SeedreamImageAdapter', () => { }) }) +describe('OpenRouterImageAdapter', () => { + const adapter = () => new OpenRouterImageAdapter({ + model: 'openai/gpt-image-1', + apiKey: 'or-test', + providerOptions: { aspect_ratio: '4:3', provider: { order: ['openai'] } }, + }) + + it('sends input images as input_references data URLs to /images', async () => { + const first = pngHeader(4, 4) + const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(pngHeader(1024, 768)) }], usage: { total_tokens: 9 } })) + + const output = await adapter().generate( + { prompt: 'swap the chair', images: [input(first), input(jpegHeader(4, 4), 'image/jpeg')], size: '1024x768' }, + { signal }, + ) + + const [url, init] = fetchMock.mock.calls[0]! + expect(url).toBe('https://openrouter.ai/api/v1/images') + expect((init?.headers as Record)['Authorization']).toBe('Bearer or-test') + const body = JSON.parse(String(init?.body)) + expect(body).toMatchObject({ + model: 'openai/gpt-image-1', + prompt: 'swap the chair', + n: 1, + size: '1024x768', + aspect_ratio: '4:3', + provider: { order: ['openai'] }, + }) + expect(body.input_references).toEqual([ + { type: 'image_url', image_url: { url: `data:image/png;base64,${b64(first)}` } }, + { type: 'image_url', image_url: { url: expect.stringMatching(/^data:image\/jpeg;base64,/) } }, + ]) + expect(output.params).toMatchObject({ inputImages: 2, usage: { total_tokens: 9 } }) + }) + + it('rejects providerOptions that would replace the prompt or the input images', () => { + expect(() => new OpenRouterImageAdapter({ + model: 'm', + apiKey: 'k', + providerOptions: { input_references: [], prompt: 'other' }, + })).toThrow('openrouter providerOptions cannot set input_references, prompt') + }) + + it('treats a size in providerOptions as a default that request.size overrides', async () => { + const fetchMock = stubFetch( + jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }), + jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] }), + ) + const custom = new OpenRouterImageAdapter({ model: 'm', apiKey: 'k', providerOptions: { size: '1024x1024' } }) + await custom.generate({ prompt: 'x' }, { signal }) + await custom.generate({ prompt: 'x', size: '1536x1024' }, { signal }) + const bodies = fetchMock.mock.calls.map(([, init]) => JSON.parse(String(init?.body))) + expect(bodies.map(body => body.size)).toEqual(['1024x1024', '1536x1024']) + }) + + it('omits input_references for text-to-image', async () => { + const fetchMock = stubFetch(jsonResponse({ data: [{ b64_json: b64(pngHeader(8, 8)) }] })) + await adapter().generate({ prompt: 'a desk' }, { signal }) + const body = JSON.parse(String(fetchMock.mock.calls[0]![1]?.body)) + expect(body).not.toHaveProperty('input_references') + }) + + it('refuses a mask instead of dropping it', async () => { + const fetchMock = stubFetch() + const error = await failure(adapter().generate( + { prompt: 'x', images: [input(pngHeader(4, 4))], mask: input(pngHeader(4, 4)) }, + { signal }, + )) + expect(error).toMatchObject({ type: 'invalid_request', retryable: false }) + expect(fetchMock).not.toHaveBeenCalled() + }) + + it('classifies an OpenRouter moderation 403 as content_policy', async () => { + stubFetch(jsonResponse({ + error: { + code: 403, + message: 'Your chosen model requires moderation and your input was flagged', + metadata: { reasons: ['sexual'], flagged_input: 'x', provider_name: 'OpenAI', model_slug: 'openai/gpt-image-1' }, + }, + }, 403)) + const error = await failure(adapter().generate({ prompt: 'x' }, { signal })) + expect(error).toMatchObject({ type: 'content_policy', retryable: false, status: 403, provider: 'openrouter' }) + }) + + it('classifies an upstream rejection forwarded in metadata.raw as content_policy', async () => { + stubFetch( + jsonResponse({ + error: { + code: 400, + message: 'Provider returned error', + metadata: { provider_name: 'OpenAI', raw: JSON.stringify({ error: { code: 'moderation_blocked', message: 'rejected' } }) }, + }, + }, 400), + jsonResponse({ + error: { + code: 400, + message: 'Provider returned error', + metadata: { provider_name: 'ByteDance', raw: { error: { code: 'OutputImageSensitiveContentDetected' } } }, + }, + }, 400), + ) + expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false }) + expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false }) + }) + + it('classifies an upstream rejection forwarded with a 5xx status as content_policy, not a retryable api_error', async () => { + stubFetch( + jsonResponse({ + error: { + code: 502, + message: 'Provider returned error', + metadata: { provider_name: 'OpenAI', raw: JSON.stringify({ error: { code: 'moderation_blocked' } }) }, + }, + }, 502), + jsonResponse({ error: { code: 502, message: 'Provider returned error', metadata: { provider_name: 'OpenAI' } } }, 502), + ) + expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'content_policy', retryable: false, status: 502 }) + expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'api_error', retryable: true, status: 502 }) + }) + + it('classifies an upstream rejection reported in metadata.provider_code as content_policy', async () => { + stubFetch(jsonResponse({ + error: { code: 400, message: 'Provider returned error', metadata: { provider_name: 'OpenAI', provider_code: 'moderation_blocked' } }, + }, 400)) + const error = await failure(adapter().generate({ prompt: 'x' }, { signal })) + expect(error).toMatchObject({ type: 'content_policy', retryable: false }) + }) + + it('treats other 4xx, including a 403 without moderation metadata, as invalid_request', async () => { + stubFetch( + jsonResponse({ error: { code: 402, message: 'Insufficient credits' } }, 402), + jsonResponse({ error: { code: 403, message: 'Key disabled' } }, 403), + jsonResponse({ error: { code: 400, message: 'bad', metadata: { raw: 'not json' } } }, 400), + jsonResponse({ error: { code: 400, message: 'bad', metadata: { provider_code: 'invalid_size' } } }, 400), + ) + for (let i = 0; i < 4; i++) { + expect(await failure(adapter().generate({ prompt: 'x' }, { signal }))).toMatchObject({ type: 'invalid_request', retryable: false }) + } + }) + + it('treats an empty data array as a retryable invalid_output', async () => { + stubFetch(jsonResponse({ data: [] })) + const error = await failure(adapter().generate({ prompt: 'x' }, { signal })) + expect(error).toMatchObject({ type: 'invalid_output', retryable: true }) + }) + + it('fails without a network call when the egress policy denies OpenRouter', async () => { + const fetchMock = stubFetch() + const guarded = new OpenRouterImageAdapter({ + model: 'm', + apiKey: 'k', + egressPolicy: { mode: 'allowlist', allowedOrigins: ['https://api.openai.com'] }, + }) + const error = await failure(guarded.generate({ prompt: 'x' }, { signal })) + expect(error).toMatchObject({ type: 'invalid_request', retryable: false }) + expect(fetchMock).not.toHaveBeenCalled() + }) +}) + describe('runImage with the built-in adapters', () => { it('falls back from a moderated provider to the next one and records both calls', async () => { stubFetch(