mirror of
https://github.com/JackChen-me/open-multi-agent.git
synced 2026-10-19 01:29:57 -05:00
Jack Chen
5033842802
Follow-up to #548, which introduced VideoBlock. Two gaps were missed when that PR was reviewed. The four adapters that cannot map video threw a bare Error. isRetryableError returns true for any Error without a status, so executeWithRetry treated a deterministic capability gap as transient: it ran the full backoff ladder and rewrote a checkpoint per attempt before failing the same way, and the failure was recorded as retryable. UnsupportedContentBlockError makes that terminal, names the adapter that rejected the block, and carries a code alongside the existing UnsupportedToolResultContentError. VideoBlock is the first top-level content block with a URL source, so it inherited no validation. That source is dereferenced by the provider rather than by OMA, and a file: path or a data URL pasted into the field was forwarded verbatim; an unvalidated media_type could also alter the data URL built for inline bytes. Public entry points now run video sources through the same copyMediaSource rules already applied to tool-result media, so the documented HTTP(S) constraint is enforced rather than aspirational. Provider-private upload handles such as MiniMax's mm_file:// are rejected: supporting them needs a source variant of its own, not an unchecked string. Also from the same review: - estimateTokens charged an inline video the 64-character fixed cost images get, hiding a multi-megabyte payload from every context strategy keyed off the estimate. It is now sized like rich tool-result media, so the attachment that needs compacting is the one that triggers compaction. - stripImageBlocksForSummary handles video too, so it is now stripMediaBlocksForSummary. - The two double casts in openai-common are narrowed to a single seam, and the video_url extension type now records that it is not an OpenAI part. - docs/structured-input.md states the rule the adapters actually follow: OpenAI-shaped adapters forward and let the provider decide, adapters with a typed content union reject locally. Bedrock and Gemini support video upstream, so the gap there is OMA's mapping rather than theirs.
60 lines
1.7 KiB
TypeScript
60 lines
1.7 KiB
TypeScript
import { describe, expect, it } from 'vitest'
|
|
import { estimateTokens } from '../src/utils/tokens.js'
|
|
import type { LLMMessage } from '../src/types.js'
|
|
|
|
describe('estimateTokens', () => {
|
|
it('counts retained reasoning text', () => {
|
|
const messages: LLMMessage[] = [
|
|
{
|
|
role: 'assistant',
|
|
content: [{ type: 'reasoning', text: 'x'.repeat(400) }],
|
|
},
|
|
]
|
|
|
|
expect(estimateTokens(messages)).toBe(100)
|
|
})
|
|
|
|
it('sizes an inline video by its payload, not by the image fixed cost', () => {
|
|
// The fixed 64-character charge images get would hide a multi-megabyte
|
|
// video from every context strategy that keys off this estimate.
|
|
const data = 'A'.repeat(400_000)
|
|
const messages: LLMMessage[] = [
|
|
{
|
|
role: 'user',
|
|
content: [{
|
|
type: 'video',
|
|
source: { type: 'base64', media_type: 'video/mp4', data },
|
|
}],
|
|
},
|
|
]
|
|
|
|
const image: LLMMessage[] = [
|
|
{
|
|
role: 'user',
|
|
content: [{
|
|
type: 'image',
|
|
source: { type: 'base64', media_type: 'image/png', data },
|
|
}],
|
|
},
|
|
]
|
|
|
|
expect(estimateTokens(messages)).toBe(Math.ceil((400_000 + 'video/mp4'.length + 32) / 4))
|
|
expect(estimateTokens(messages)).toBeGreaterThan(estimateTokens(image) * 1000)
|
|
})
|
|
|
|
it('sizes a video url source by the reference rather than the fetched bytes', () => {
|
|
const url = `https://example.com/${'a'.repeat(80)}.mp4`
|
|
const messages: LLMMessage[] = [
|
|
{
|
|
role: 'user',
|
|
content: [{
|
|
type: 'video',
|
|
source: { type: 'url', media_type: 'video/mp4', url },
|
|
}],
|
|
},
|
|
]
|
|
|
|
expect(estimateTokens(messages)).toBe(Math.ceil((url.length + 'video/mp4'.length + 32) / 4))
|
|
})
|
|
})
|