fix(libSceVideodec2): decode after a flush without a reset

sceVideodec2Decode threw when it followed sceVideodec2Flush without
sceVideodec2Reset. It now resets FFmpeg's drained decoder and decodes the
unit, keeping the pictures the flush has not returned yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

core/libs/prx/libSceVideodec2/Export.cpp

@@ -555,7 +555,10 @@ int APS5_VABI sceVideodec2Decode_nid_postfix(std::uint64_t handle, const InputDa
     if (Trace() && decoder.decoded < 3) std::fprintf(stderr, "[videodec2] Decode input=%llu frame=%llu output=%llu au=%llu pts=%llu size=%ux%u fb=%p/%llu\n", static_cast<unsigned long long>(input->thisSize), static_cast<unsigned long long>(frame ? frame->thisSize : 0), static_cast<unsigned long long>(output->thisSize), static_cast<unsigned long long>(input->auSize), static_cast<unsigned long long>(input->ptsData), decoder.width, decoder.height, frame ? frame->frameBuffer : nullptr, static_cast<unsigned long long>(frame ? frame->frameBufferSize : 0));
     ++decoder.decoded;
     const Picture picture{input->ptsData, input->dtsData, input->attachedData};
-    if (decoder.flushing) throw std::runtime_error("Videodec2: reset required after flushing");
+    if (decoder.flushing) {
+        avcodec_flush_buffers(decoder.context);
+        decoder.flushing = false;
+    }
     if (!input->auData || input->auSize == 0 || input->auSize > static_cast<std::uint64_t>(std::numeric_limits<int>::max())) throw std::runtime_error("Videodec2: invalid access unit");
     if (decoder.codec == CodecVp9) {
         auto references = decoder.vp9References;

core/libs/tests/GuestVideodec2.cpp

@@ -646,6 +646,42 @@ std::uint64_t frameBufferBytes() {
     return memory.maxFrameBufferSize;
 }
 
+void testDecodeAfterFlush() {
+    std::vector<std::uint8_t> buffer(frameBufferBytes());
+    const auto handle = createDecoder(1, 100);
+    {
+        FrameBuffer frame{sizeof(FrameBuffer), buffer.data(), buffer.size(), false};
+        OutputInfo output{sizeof(OutputInfo)};
+        check(sceVideodec2Flush_nid_postfix(handle, &frame, &output) == 0 && !output.isValid && !frame.isAccepted, "flushing a decoder that decoded nothing gave a picture");
+    }
+    const auto units = accessUnits(false);
+    for (const std::uint64_t base : {1000u, 5000u}) {
+        std::size_t pictures = 0;
+        const auto take = [&](const OutputInfo& output) {
+            if (!output.isValid) return false;
+            check(pictures < PictureHashes.size() && owned(buffer, base + DisplayOrderUnits[pictures]), "a picture decoded after a flush is out of display order");
+            check(hashNv12(buffer.data(), output.framePitch, output.frameHeight) == PictureHashes[pictures], "picture " + std::to_string(pictures) + " decoded after a flush differs from the reference");
+            ++pictures;
+            return true;
+        };
+        for (std::size_t unit = 0; unit < units.size(); ++unit) {
+            const InputData input{sizeof(InputData), units[unit].data(), units[unit].size(), base + unit, unit, base + unit};
+            FrameBuffer frame{sizeof(FrameBuffer), buffer.data(), buffer.size(), false};
+            OutputInfo output{sizeof(OutputInfo)};
+            check(sceVideodec2Decode_nid_postfix(handle, &input, &frame, &output) == 0, "decode after a flush failed");
+            take(output);
+        }
+        for (;;) {
+            FrameBuffer frame{sizeof(FrameBuffer), buffer.data(), buffer.size(), false};
+            OutputInfo output{sizeof(OutputInfo)};
+            check(sceVideodec2Flush_nid_postfix(handle, &frame, &output) == 0, "flush failed");
+            if (!take(output)) break;
+        }
+        check(pictures == PictureHashes.size(), "missing pictures of a stream decoded after a flush");
+    }
+    check(sceVideodec2DeleteDecoder_nid_postfix(handle) == 0, "delete failed");
+}
+
 void testPictureOwnership() {
     std::vector<std::uint8_t> shared(frameBufferBytes()), other(frameBufferBytes());
     {
@@ -774,6 +810,7 @@ int main() {
         testClip(CodecVp9, 0, ivfFrames(), Vp9Hashes);
         testVp9PictureMetadata();
         testTenBit();
+        testDecodeAfterFlush();
         testPictureOwnership();
         testConcurrentOwnership();
         std::puts("Videodec2 tests passed");

docs/dev/debt/libSceVideodec2.md

@@ -11,4 +11,5 @@
 - [sceVideodec2CreateDecoder](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) decodes HEVC (codec type 974921) and VP9 (2382845) through FFmpeg, besides H.264. The tags come from SharpProspero's `Videodec2CodecType` and EVO-PLAYER-PS5's `sce_videodec2.h`, which records them from hardware runs. HEVC input is converted from length-prefixed NAL units to Annex B as for H.264, and VP9 frames are passed through. Output is the same 8-bit NV12 for every codec; a decoder for HEVC Main10 or VP9 Profile 2 is created, but a frame deeper than 8 bits throws, since the console's 10-bit output layout is unknown, and `QueryDecoderMemoryInfo` reports the 8-bit frame size for it. `sceVideodec2GetPictureInfo` and `sceVideodec2GetAvcPictureInfo` throw for HEVC and VP9 pictures, which use other layouts (`SceVideodec2HevcPictureInfo`, `SceVideodec2Vp9PictureInfo`), and `sceVideodec2GetHevcPictureInfo` is not exported; `sceVideodec2GetVp9PictureInfo` has its own VP9 layout
 
 - [sceVideodec2GetVp9PictureInfo](../../../core/libs/prx/libSceVideodec2/Export.cpp) uses the 88-byte layout and two-pointer signature from [SharpProspero Videodec2.cs](https://github.com/SvenGDK/SharpProspero/blob/5dfdb4b0b166e83dc9ae275001afb695f12c7174/src/SharpProspero/Interop/Video/Videodec2.cs). Frame metadata follows the [VP9 specification](https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.7-20170222-draft.pdf), sections 5.26 and 6.2: profile, key/intra-only flags, bit depth, color space/range, coded/render dimensions and superframe count are retained per emitted buffer. Level is 0 because the elementary stream does not signal a level; the console value is unverified. Null pointers, undersized structures and querying another codec throw; console error codes and support for other structure sizes are unknown. Existing NV12 output rejects pictures deeper than 8 bits. Superframes are submitted as individual coded frames to keep each output associated with its input metadata.
+- [sceVideodec2Decode after sceVideodec2Flush](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) without `sceVideodec2Reset` restarts FFmpeg's decoder (`avcodec_flush_buffers`) and decodes the unit, where it threw before. TEKKEN DARK RESURRECTION (PPSA31371), which runs in Sony's PSP emulator, decodes after a flush without a reset (#7118), so the console accepts the call. Pictures the flush decoded but has not returned yet are returned before the new ones. What the console returns for them, and for a stream that continues across the flush without a new IDR picture, is not measured.
 - [sceVideodec2Decode](../../../core/libs/prx/libSceVideodec2/Export.cpp) (libSceVideodec2) and `sceVideodec2Flush` output the coded picture, with the chroma plane after its luma rows, as the console does for 1080p H.264 and HEVC streams coded at 1920x1088. An HEVC or VP9 picture whose size is not a multiple of 16, such as HEVC coded at 1920x1080, is output at that size; whether the console pads it is unverified. `sceVideodec2GetPictureInfo` reports no frame cropping