From 0286a767e213cbfb5cef72b367f1e09e39bae80e Mon Sep 17 00:00:00 2001 From: fdfawef324 <287205748+fdfawef324@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:16:17 -0400 Subject: [PATCH] feat(agc): capture submissions and replay their decode and wait releases offline A hang with threads in Driver::waitMemory takes a ten-minute boot to reproduce, and the existing switches show only the first submissions of one queue (APS5_DUMP_QUEUE) or a rejected one (APS5_DUMP_REJECTED). Capture. APS5_DUMP_SUBMITS= writes every submission the driver accepts. /index.txt starts with "aps5-submissions 1" and "ordinal queue type serial time_us dwords kind file held conditional", then one line per submission: acceptance ordinal, queue id, graphics or compute, serial, microseconds on the APS5_TRACE_GPU clock, dword count, kind, file name, the dword offsets of the waits held at submit (Submission::heldAtSubmit) and the COND_EXEC and predicated indirect buffer ranges (Submission::conditionalEnds) as start:end, "-" for none. /submission_.bin holds the copied dwords, indirect buffers inlined. REWIND tails are listed as rewind with the serial of their submission, suspend points as suspend without a file. SubmissionCapture::Prepare copies the commands before the driver lock; Take, under it after reserveOutputs, sets only the ordinal, time and held waits and does not allocate; the file and then its index line are written after the lock. A failed write reports the failure to the driver (Driver::ReportFailure) and throws, so every later driver call throws too. Unset, Submit tests a static pointer. An index written before the held and conditional columns ("ordinal queue type serial time_us dwords kind file") is read with both unknown. Analysis. agc_pm4_replay [--full] decodes each submission and throws on a packet the driver rejects (a corrupt capture). It does not model cross-queue timing: a wait is released only when the capture alone proves that the driver releases it (WaitMemory.cpp, LabelHistory.cpp), and unknown, with the reason, when its outcome depends on when another queue's stores run. A memory wait is released when its compare always passes, when it was held at submit, when the last store to the awaited bytes before it on its queue covers them with an accepted value, is not conditional and no other queue stores to those bytes anywhere in the capture, or when a RELEASE_MEM or WRITE_DATA label earlier in the waiting submission that the label history keeps (KeepsLabelStore, not conditional) stores an accepted value and fewer than LabelStoreHistory other stores to the awaited bytes can run between it and the wait: those between them on the waiting queue and every store of another queue to those bytes, since the history keeps the last stores per dword over all queues and finds only a store that ran after the waiting submission entered the driver. A REWIND is released when its tail is in the capture; one valid at submit without a captured tail is unknown, because the driver reads the valid bit again when the queue reaches it. A wait is unreleased when it was not held at submit and no store that can run before it may write an accepted value (a store of part of the awaited bytes or of a run-time value may; a store later on its queue runs after it): the GPU does not release it, a CPU or shader store could. A REWIND without a tail that was not valid at submit is unreleased. Every other wait is unknown: "depends on cross-queue timing (stores on queue X at #...)" when another queue stores to the awaited bytes, otherwise because the last store before it on its queue is conditional, stores a run-time value or part of the bytes, or overwrites an accepted store the capture does not prove still in the label history. Without the held column a wait that would be unreleased is unknown; without the conditional column every packet after a COND_EXEC or predicated INDIRECT_BUFFER is conditional to the end of its submission. The outcomes depend on the order of each queue's stores and on which stores other queues make, not on the submit order across queues. The report counts the outcomes, routes and cross-queue unknowns, says when the index lacks the columns, lists up to 32 unreleased and then unknown waits newest first with the packets around them and their stores, then the last submissions of each queue; exit code 1 when a wait is unreleased, 2 on errors. Pm4 gains DecodeStores, DecodeWait, DecodeConditionalWrite, DecodeConstantRamDump, the COPY_DATA and DMA_DATA field helpers, the sample dump layout and RewindValid; Execute, ResolveStore, Validate, DecodeMemoryCopy, the driver's sample dump, noteHeldAtSubmit and executeRewindTail use them. LabelStoreHistory and KeepsLabelStore move to LabelStores.hpp, used by noteLabelStore and the analysis. Test. agc_driver_pm4_replay submits command buffers built with the libSceAgc builders through the driver with the switch set and checks the index, the held waits, the COND_EXEC range, the dwords, the decode and the outcome of eight waits, including WRITE_DATA 1, WRITE_DATA 0, WAIT == 1 in one submission, which the driver releases through its label history. Hand-made captures check each release route, a store of another queue to the awaited bytes that entered before or after (unknown), the label history limit with later stores on the waiting queue or on another queue, a fill the history does not keep, a store in an earlier submission overwritten later, later stores, rejected values and a missing rewind tail (unreleased), the six captures of the reviews (none released but the wait held at submit), five groups of waits in eight submission orders with identical outcomes, an index without the held and conditional columns and corrupt captures. A directory in place of the next submission file makes the write fail, and the submit, the next submit and shutdown throw. agc_driver_pm4_replay_tool runs the tool through tests/Pm4ReplayTool.cmake and checks exit codes 1 and 2. Co-Authored-By: Claude Opus 5.5 --- core/libs/prx/libSceAgcDriver/CMakeLists.txt | 32 + .../Driver/Synchronization/LabelStores.hpp | 18 + .../Driver/Synchronization/WaitMemory.hpp | 3 +- .../libSceAgcDriver/Execution/include/Pm4.hpp | 47 ++ .../Execution/include/SubmissionCapture.hpp | 68 ++ .../src/Driver/Packets/PacketExecution.cpp | 2 +- .../src/Driver/Packets/PacketMemory.cpp | 8 +- .../src/Driver/Queues/Submission.cpp | 43 +- .../Driver/Synchronization/LabelHistory.cpp | 2 +- .../prx/libSceAgcDriver/Execution/src/Pm4.cpp | 146 +++- .../Execution/src/SubmissionCapture.cpp | 237 ++++++ .../Execution/src/VulkanDevice.cpp | 2 +- .../prx/libSceAgcDriver/tests/Pm4Replay.cpp | 730 +++++++++++++++++ .../libSceAgcDriver/tests/Pm4ReplayTool.cmake | 29 + .../prx/libSceAgcDriver/tools/Pm4Analysis.cpp | 765 ++++++++++++++++++ .../prx/libSceAgcDriver/tools/Pm4Analysis.hpp | 91 +++ .../prx/libSceAgcDriver/tools/Pm4Replay.cpp | 35 + docs/dev/BUILD.md | 13 + 18 files changed, 2222 insertions(+), 49 deletions(-) create mode 100644 core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp create mode 100644 core/libs/prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp create mode 100644 core/libs/prx/libSceAgcDriver/Execution/src/SubmissionCapture.cpp create mode 100644 core/libs/prx/libSceAgcDriver/tests/Pm4Replay.cpp create mode 100644 core/libs/prx/libSceAgcDriver/tests/Pm4ReplayTool.cmake create mode 100644 core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.cpp create mode 100644 core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.hpp create mode 100644 core/libs/prx/libSceAgcDriver/tools/Pm4Replay.cpp diff --git a/core/libs/prx/libSceAgcDriver/CMakeLists.txt b/core/libs/prx/libSceAgcDriver/CMakeLists.txt index 3fdab7d9a8..b4d09e16fa 100644 --- a/core/libs/prx/libSceAgcDriver/CMakeLists.txt +++ b/core/libs/prx/libSceAgcDriver/CMakeLists.txt @@ -171,6 +171,7 @@ set(agcDriverSources ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/SubgroupClock.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/ShaderDeviceProfile.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/Pm4.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/SubmissionCapture.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/VulkanDevice.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/AspectFit.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Execution/src/PresentationScaler.cpp @@ -1558,6 +1559,37 @@ endif() set_target_properties(agc_shader_replay PROPERTIES CXX_EXTENSIONS OFF) configure_windows_unwind(agc_shader_replay) +add_executable(agc_pm4_replay EXCLUDE_FROM_ALL + tools/Pm4Replay.cpp + tools/Pm4Analysis.cpp +) +target_include_directories(agc_pm4_replay PRIVATE ${LIBS_INCLUDE_DIR}) +target_link_libraries(agc_pm4_replay PRIVATE libSceAgcDriver libc) +set_target_properties(agc_pm4_replay PROPERTIES CXX_EXTENSIONS OFF) +configure_windows_unwind(agc_pm4_replay) + +if(BUILD_TESTING) + add_test_executable(agc_driver_pm4_replay_tests tests/Pm4Replay.cpp tools/Pm4Analysis.cpp) + target_include_directories(agc_driver_pm4_replay_tests PRIVATE ${LIBS_INCLUDE_DIR}) + target_link_libraries(agc_driver_pm4_replay_tests PRIVATE libSceAgc libSceAgcDriver libkernel libc) + set_target_properties(agc_driver_pm4_replay_tests PROPERTIES CXX_EXTENSIONS OFF) + configure_windows_unwind(agc_driver_pm4_replay_tests) + add_dependencies(agc_driver_pm4_replay_tests agc_pm4_replay) + set(agcPm4ReplayCapture "${CMAKE_CURRENT_BINARY_DIR}/pm4-replay-capture") + add_test(NAME agc_driver_pm4_replay COMMAND agc_driver_pm4_replay_tests) + set_tests_properties(agc_driver_pm4_replay PROPERTIES + ENVIRONMENT "ANYPS5_NO_SHADER_CACHE=1;APS5_DUMP_SUBMITS=${agcPm4ReplayCapture}" + FIXTURES_SETUP agc_pm4_replay_capture TIMEOUT 30) + add_test(NAME agc_driver_pm4_replay_tool COMMAND ${CMAKE_COMMAND} -DTOOL=$ "-DCAPTURE=${agcPm4ReplayCapture}" + -P ${CMAKE_CURRENT_SOURCE_DIR}/tests/Pm4ReplayTool.cmake) + set_tests_properties(agc_driver_pm4_replay_tool PROPERTIES + FIXTURES_REQUIRED agc_pm4_replay_capture TIMEOUT 30) + if(WIN32) + set_tests_properties(agc_driver_pm4_replay agc_driver_pm4_replay_tool PROPERTIES ENVIRONMENT_MODIFICATION + "PATH=path_list_prepend:$") + endif() +endif() + add_executable(agc_compute_pipeline_replay EXCLUDE_FROM_ALL tools/ComputePipelineReplay.cpp) target_include_directories(agc_compute_pipeline_replay PRIVATE ${LIBS_INCLUDE_DIR} diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp new file mode 100644 index 0000000000..dc39feeb1a --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp @@ -0,0 +1,18 @@ +#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_DRIVER_SYNCHRONIZATION_LABELSTORES_HPP +#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_DRIVER_SYNCHRONIZATION_LABELSTORES_HPP + +#include +#include + +namespace AgcDriver::DriverDetail { + +inline constexpr std::size_t LabelStoreHistory = 4; +inline constexpr std::size_t LabelStoreBytes = 64; + +inline bool KeepsLabelStore(std::uint64_t address, std::size_t bytes) { + return bytes != 0 && bytes <= LabelStoreBytes && bytes % 4 == 0 && address % 4 == 0; +} + +} + +#endif diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/WaitMemory.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/WaitMemory.hpp index dc1e3b7890..8157279e9e 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/WaitMemory.hpp +++ b/core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/WaitMemory.hpp @@ -1,6 +1,7 @@ #ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_DRIVER_WAITMEMORY_HPP #define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_DRIVER_WAITMEMORY_HPP +#include "prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp" #include "prx/libSceAgcDriver/Graphics/include/Recorder.hpp" #include #include @@ -9,8 +10,6 @@ namespace AgcDriver::DriverDetail { -inline constexpr std::size_t LabelStoreHistory = 4; - inline constexpr const char* LabelRefusalNames[6] = {"none", "trust-off", "queued", "overwritten", "unclosed", "behind-completion"}; inline constexpr std::chrono::microseconds PollTryInterval{1000}; diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/Pm4.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/Pm4.hpp index ce6f1638eb..35c5467efb 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/include/Pm4.hpp +++ b/core/libs/prx/libSceAgcDriver/Execution/include/Pm4.hpp @@ -142,6 +142,53 @@ struct MemoryCopy { std::optional DecodeMemoryCopy(std::span packet); std::uint32_t DmaSource(std::span packet); std::uint32_t DmaDestination(std::span packet); +inline constexpr std::uint32_t DmaSelectGds = 1; +inline constexpr std::uint32_t DmaSourceData = 2; +inline constexpr std::uint32_t DmaDestinationNowhere = 2; +inline std::size_t DmaBytes(std::span packet) { return packet[6] & 0x3ffffffu; } +inline constexpr std::uint32_t CopyDataGpuClockSource = 18; +inline std::uint32_t CopyDataSource(std::span packet) { return ((packet[1] & 0xfu) << 1u) | ((packet[1] >> 30u) & 1u); } +inline bool CopyDataImmediate(std::uint32_t source) { return source >= 10 && source != CopyDataGpuClockSource; } +inline bool CopyDataToRegister(std::span packet) { return ((packet[1] >> 8u) & 0xfu) == 0; } +inline std::size_t CopyDataBytes(std::span packet) { return (packet[1] & 0x10000u) != 0 ? 8 : 4; } +inline constexpr std::uint32_t RewindValid = 0x80000000u; +struct WaitFields { + std::uint32_t function; + std::uint64_t address; + std::uint64_t reference; + std::uint64_t mask; + std::uint32_t poll; + std::size_t bytes; +}; +WaitFields DecodeWait(std::span packet); +struct ConditionalWrite { + std::uint64_t compared; + std::uint64_t destination; + std::uint32_t value; +}; +ConditionalWrite DecodeConditionalWrite(std::span packet); +struct ConstantRamDump { + std::uint32_t offset; + std::uint32_t words; + std::uint64_t destination; +}; +ConstantRamDump DecodeConstantRamDump(std::span packet); +inline constexpr std::uint32_t SampleDumpEvent = 0x39; +inline constexpr std::uint64_t SampleDumpBlocks = 16; +inline constexpr std::uint64_t SampleDumpStride = 16; +inline constexpr std::uint64_t SampleDumpBlockBytes = 8; +inline constexpr std::uint64_t SampleDumpBytes = (SampleDumpBlocks - 1) * SampleDumpStride + SampleDumpBlockBytes; +inline bool SampleDump(std::span packet) { return ((packet[0] >> 8u) & 0xffu) == 0x46u && packet.size() >= 2 && (packet[1] & 0x3fu) == SampleDumpEvent; } +inline std::uint64_t SampleDumpAddress(std::span packet) { return packet[2] | (static_cast(packet[3]) << 32u); } +struct PacketStore { + enum class Value : std::uint8_t { Bytes, Pattern, RunTime }; + std::uint64_t address = 0; + std::uint64_t bytes = 0; + Value value = Value::RunTime; + std::vector data; + bool conditional = false; +}; +std::vector DecodeStores(std::span packet); // A DISPATCH_INDIRECT's arguments: the guest address of its three group-count dwords (no memory // access, so the GPU can read them in place: VulkanDevice::DispatchIndirect), the DISPATCH_DIRECT // packet made by reading them there (through the checked guest memory path, which waits for diff --git a/core/libs/prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp b/core/libs/prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp new file mode 100644 index 0000000000..1db347dcca --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp @@ -0,0 +1,68 @@ +#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_SUBMISSIONCAPTURE_HPP +#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_EXECUTION_INCLUDE_SUBMISSIONCAPTURE_HPP + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace AgcDriver { + +enum class SubmissionKind : std::uint8_t { Submit, Rewind, Suspend }; + +struct CapturedSubmission { + std::uint64_t ordinal = 0; + std::uint32_t queue = 0; + std::uint64_t serial = 0; + std::uint64_t timeUs = 0; + SubmissionKind kind = SubmissionKind::Submit; + std::size_t dwords = 0; + std::vector commands; + std::vector heldAtSubmit; + std::vector> conditionalRanges; + bool heldKnown = true; + bool conditionalKnown = true; +}; + +class SubmissionCapture { +public: + static constexpr std::string_view IndexName = "index.txt"; + static constexpr std::string_view Magic = "aps5-submissions 1"; + static constexpr std::string_view Columns = "ordinal queue type serial time_us dwords kind file held conditional"; + static constexpr std::string_view ColumnsWithoutHeld = "ordinal queue type serial time_us dwords kind file"; + + explicit SubmissionCapture(const std::filesystem::path& directory); + SubmissionCapture(const SubmissionCapture&) = delete; + SubmissionCapture& operator=(const SubmissionCapture&) = delete; + + static SubmissionCapture* Active(); + static CapturedSubmission Prepare(std::uint32_t queue, SubmissionKind kind, std::span commands, const std::map& conditionalEnds); + void Take(CapturedSubmission& submission, std::uint64_t serial, std::uint64_t timeUs, const std::set& heldAtSubmit) noexcept; + void Write(const CapturedSubmission& submission); + + static std::string FileName(const CapturedSubmission& submission); + static std::string_view KindName(SubmissionKind kind); + static std::string_view QueueType(std::uint32_t queue); + static std::vector ReadIndex(const std::filesystem::path& directory); + static std::vector ReadCommands(const std::filesystem::path& directory, const CapturedSubmission& submission); + static std::vector Read(const std::filesystem::path& directory); + +private: + std::filesystem::path directory; + std::mutex mutex; + std::ofstream index; + std::atomic next{0}; +}; + +} + +#endif diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp index c65419b0ee..04f7ce5601 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketExecution.cpp @@ -303,7 +303,7 @@ void Driver::execute(const Submission& submission) { if (condition == 0) nextCursor = submission.conditionalEnds.at(cursor); if (traceGpu) std::fprintf(stderr, "[gpu] %.1f queue 0x%x COND_EXEC at DWORD %zu reads 0x%x at 0x%llx: %s %zu dwords\n", TraceMs(), submission.queue, cursor, condition, static_cast(packet[1] | (static_cast(packet[2]) << 32u)), condition == 0 ? "skips" : "executes", submission.conditionalEnds.at(cursor) - cursor - count); } else if (sampleDump && !wroteOnGpu) { - dumpSampleCounters(packet[2] | (static_cast(packet[3]) << 32u)); + dumpSampleCounters(Pm4::SampleDumpAddress(packet)); } else if (opcode != 0x42 && opcode != 0x46 && opcode != 0x58) { if (!wroteOnGpu) { if (Pm4::IndirectRegisterOpcode(opcode)) Pm4::ExecuteIndirectRegisters(packet, submission.registerLists.at(cursor), queue); diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp index a2ba8abae2..a034cf91f1 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp @@ -170,13 +170,13 @@ bool Driver::preparePacketMemory(const Submission& submission, QueueState& queue static const bool drawDrain = std::getenv("APS5_DRAW_DRAIN") != nullptr; drawPacket = Pm4::DrawOpcode(opcode); - sampleDump = opcode == 0x46 && (packet[1] & 0x3fu) == 0x39u; + sampleDump = Pm4::SampleDump(packet); if (sampleDump && !drainAll) { GuestMemory::TagGpuLockSite(GuestMemory::GpuLockSite::Label); std::lock_guard gpuLock(GuestMemory::GpuMutex()); if (const auto localDevice = device.Load()) { recordDeferredLabels(localDevice.get(), submission.queue); - wroteOnGpu = localDevice->DumpSamplesOnGpu(packet[2] | (static_cast(packet[3]) << 32u)); + wroteOnGpu = localDevice->DumpSamplesOnGpu(Pm4::SampleDumpAddress(packet)); } } @@ -241,9 +241,9 @@ void Driver::dumpSampleCounters(std::uint64_t address) { samples = recorder->SamplesTotal(); } constexpr std::uint64_t ready = 1ull << 63u; - for (std::uint64_t db = 0; db < 16; ++db) { + for (std::uint64_t db = 0; db < Pm4::SampleDumpBlocks; ++db) { const std::uint64_t value = ready | (db == 0 ? samples : 0u); - GuestMemory::Write(address + db * 16u, std::as_bytes(std::span(&value, 1)), 8); + GuestMemory::Write(address + db * Pm4::SampleDumpStride, std::as_bytes(std::span(&value, 1)).first(Pm4::SampleDumpBlockBytes), 8); } } diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/Submission.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/Submission.cpp index a0db6f839f..b673d7eb11 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/Submission.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/Submission.cpp @@ -5,13 +5,33 @@ #include "prx/libSceAgcDriver/Execution/include/Driver/Synchronization/SynchronizationStatistics.hpp" #include "prx/libSceAgcDriver/Execution/include/GuestMemory.hpp" #include "prx/libSceAgcDriver/Execution/include/Pm4.hpp" +#include "prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp" #include "prx/libSceAgcDriver/Execution/include/ThreadPriority.hpp" #include +#include #include #include +#include namespace AgcDriver::DriverDetail { +namespace { + +std::uint64_t captureTime() { + return static_cast(std::llround(TraceMs() * 1000.0)); +} + +void writeCaptured(Driver& driver, SubmissionCapture& capture, const CapturedSubmission& captured) { + try { + capture.Write(captured); + } catch (...) { + driver.ReportFailure(std::current_exception()); + throw; + } +} + +} + void Driver::copyCommands(Submission& submission, const std::uint32_t* guest, std::size_t words) { submission.commands.clear(); submission.conditionalEnds.clear(); @@ -187,14 +207,14 @@ void Driver::reserveOutputs(Submission& submission) { void Driver::executeRewindTail(const Submission& stalled) { PerformanceTimer timing("Driver.Rewind"); std::atomic_ref control(*const_cast(stalled.rewindTail - 1)); - if ((control.load(std::memory_order_acquire) & 0x80000000u) == 0) { + if ((control.load(std::memory_order_acquire) & Pm4::RewindValid) == 0) { noteWaitBlocked(stalled.queue, reinterpret_cast(stalled.rewindTail - 1), true); struct BlockedWait { Driver& driver; std::uint32_t queue; ~BlockedWait() { driver.noteWaitBlocked(queue, 0, false); } } blocked{*this, stalled.queue}; - while ((control.load(std::memory_order_acquire) & 0x80000000u) == 0) { + while ((control.load(std::memory_order_acquire) & Pm4::RewindValid) == 0) { CheckFailure(); checkStopping(); noteAwaitingTitle(stalled.queue, reinterpret_cast(stalled.rewindTail - 1)); @@ -211,6 +231,8 @@ void Driver::executeRewindTail(const Submission& stalled) { if (APS5_ENABLE_TIMING_LOG) tail.validatedAt = std::chrono::steady_clock::now(); waitForFlipRoom(tail); if (APS5_ENABLE_TIMING_LOG) tail.roomReadyAt = std::chrono::steady_clock::now(); + auto* const capture = SubmissionCapture::Active(); + auto captured = capture != nullptr ? SubmissionCapture::Prepare(tail.queue, SubmissionKind::Rewind, tail.commands, tail.conditionalEnds) : CapturedSubmission{}; { std::lock_guard lock(mutex); rethrowFailure(); @@ -219,7 +241,9 @@ void Driver::executeRewindTail(const Submission& stalled) { tail.shaders = shaders; tail.serial = stalled.serial; tail.received = ++eventSerial; + if (capture != nullptr) capture->Take(captured, tail.serial, captureTime(), tail.heldAtSubmit); } + if (capture != nullptr) writeCaptured(*this, *capture, captured); if (APS5_ENABLE_TIMING_LOG) tail.enqueuedAt = tail.dequeuedAt = tail.orderedAt = std::chrono::steady_clock::now(); timing.Finish(); execute(tail); @@ -254,6 +278,8 @@ void Driver::Submit(const Packet* packet, std::uint32_t queue) { static const bool trace = std::getenv("APS5_TRACE_GPU") != nullptr; if (trace) std::fprintf(stderr, "[gpu] %.1f submit queue=0x%x dwords=%zu at %p\n", TraceMs(), queue, submission.commands.size(), static_cast(descriptor.addr)); const auto validated = profile ? std::chrono::steady_clock::now() : start; + auto* const capture = SubmissionCapture::Active(); + auto captured = capture != nullptr ? SubmissionCapture::Prepare(queue, SubmissionKind::Submit, submission.commands, submission.conditionalEnds) : CapturedSubmission{}; { std::lock_guard lock(mutex); rethrowFailure(); @@ -264,6 +290,7 @@ void Driver::Submit(const Packet* packet, std::uint32_t queue) { submission.serial = accepted + 1; submission.received = ++eventSerial; + if (capture != nullptr) capture->Take(captured, submission.serial, captureTime(), submission.heldAtSubmit); if (profile) { const auto now = std::chrono::steady_clock::now(); submission.enqueuedAt = now; @@ -280,11 +307,14 @@ void Driver::Submit(const Packet* packet, std::uint32_t queue) { ++accepted; } changed.notify_all(); + if (capture != nullptr) writeCaptured(*this, *capture, captured); } void Driver::SuspendPoint() { const auto receivedAt = APS5_ENABLE_TIMING_LOG ? std::chrono::steady_clock::now() : std::chrono::steady_clock::time_point{}; require(!onWorkerThread(), "worker cannot suspend itself"); + auto* const capture = SubmissionCapture::Active(); + auto captured = capture != nullptr ? SubmissionCapture::Prepare(0, SubmissionKind::Suspend, {}, {}) : CapturedSubmission{}; std::unique_lock lock(mutex); rethrowFailure(); checkStopping(); @@ -296,10 +326,14 @@ void Driver::SuspendPoint() { boundary.queue = 0; boundary.receivedAt = receivedAt; boundary.enqueuedAt = std::chrono::steady_clock::now(); + if (capture != nullptr) capture->Take(captured, boundary.serial, captureTime(), boundary.heldAtSubmit); enqueue(std::move(boundary)); ++accepted; changed.notify_all(); + if (capture == nullptr) return; + lock.unlock(); + writeCaptured(*this, *capture, captured); } bool Driver::waitFree(const Submission& submission) { @@ -380,8 +414,9 @@ void Driver::noteHeldAtSubmit(Submission& submission, std::size_t cursor) { return; } if ((opcode != 0x3c && opcode != 0x93) || packet.size() < 7 || ((packet[1] >> 4u) & 3u) != 1u) return; - const auto address = packet[2] | (static_cast(packet[3]) << 32u); - const std::size_t bytes = Pm4::WaitAwaitedBytes(packet); + const auto wait = Pm4::DecodeWait(packet); + const auto address = wait.address; + const std::size_t bytes = wait.bytes; if (address % 4 != 0 || !GuestMemory::Accessible(reinterpret_cast(address), bytes)) return; const auto worker = workers.find(submission.queue); for (std::size_t offset = 0; offset < bytes; offset += 4) { diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/LabelHistory.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/LabelHistory.cpp index 696612f794..baa6e78b03 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/LabelHistory.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/LabelHistory.cpp @@ -7,7 +7,7 @@ namespace AgcDriver::DriverDetail { void Driver::noteLabelStore(std::uint64_t address, std::span bytes, std::uint64_t stamp, std::uint32_t queue) { - if (bytes.empty() || bytes.size() > 64 || bytes.size() % 4 != 0 || address % 4 != 0) return; + if (!KeepsLabelStore(address, bytes.size())) return; std::lock_guard lock(labelStoresMutex); for (std::size_t offset = 0; offset < bytes.size(); offset += 4) { auto& history = labelStores[address + offset]; diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/Pm4.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/Pm4.cpp index 2fdd6b930b..b91eadc3a8 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/Pm4.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/Pm4.cpp @@ -114,16 +114,14 @@ std::vector copySource(std::uint64_t source, std::size_t bytes, bool return data; } -constexpr std::uint32_t DmaSelectGds = 1; -constexpr std::uint32_t DmaSelectNowhere = 2; bool gdsRange(std::uint64_t offset, std::size_t bytes) { return offset <= GdsBytes && bytes <= GdsBytes - offset; } std::vector dmaSourceBytes(std::span packet) { - const std::size_t bytes = packet[6] & 0x3ffffffu; - if (dmaSource(packet) != DmaSelectGds) return copySource(address(packet[2], packet[3]), bytes, dmaSource(packet) == 2); + const std::size_t bytes = DmaBytes(packet); + if (dmaSource(packet) != DmaSelectGds) return copySource(address(packet[2], packet[3]), bytes, dmaSource(packet) == DmaSourceData); return copySource(GdsAddress() + packet[2], bytes, false); } @@ -133,14 +131,9 @@ void copyMemory(std::uint64_t source, std::uint64_t destination, std::size_t byt GuestMemory::Write(destination, copySource(source, bytes, immediate)); } -constexpr std::uint32_t CopyDataGpuClockSource = 18; constexpr std::uint32_t CopyDataCachePolicy = (3u << 13u) | (3u << 25u); constexpr std::uint32_t ContextRegisterBase = 0xa000; -bool copyDataToRegister(std::span packet) { - return ((packet[1] >> 8u) & 0xfu) == 0; -} - std::uint32_t copyDataCount(std::span packet) { return (packet[1] & 0x10000u) != 0 ? 2u : 1u; } @@ -150,8 +143,8 @@ std::uint64_t gpuClockCount() { } std::vector copyDataSource(std::span packet, std::size_t bytes) { - const auto source = ((packet[1] & 0xfu) << 1u) | ((packet[1] >> 30u) & 1u); - if (source != CopyDataGpuClockSource) return copySource(address(packet[2], packet[3]), bytes, source >= 10); + const auto source = CopyDataSource(packet); + if (source != CopyDataGpuClockSource) return copySource(address(packet[2], packet[3]), bytes, CopyDataImmediate(source)); const auto value = gpuClockCount(); std::vector data(bytes); for (std::size_t i = 0; i < bytes; ++i) data[i] = static_cast(value >> (i * 8)); @@ -460,9 +453,9 @@ void Validate(std::span packet, std::uint32_t queue) { case 0x40: { size(6); require((packet[1] & ~(0x40110f0fu | CopyDataCachePolicy)) == 0, "COPY_DATA engine or reserved fields are not implemented"); - const auto source = ((packet[1] & 0xfu) << 1u) | ((packet[1] >> 30u) & 1u); + const auto source = CopyDataSource(packet); const auto destination = ((packet[1] >> 8u) & 0xfu) << 1u; - if (copyDataToRegister(packet)) { + if (CopyDataToRegister(packet)) { require(source == 2 || source == 4, "COPY_DATA to a register from a register, GDS, immediate or clock source is not implemented"); require((packet[2] & 3u) == 0, "misaligned COPY_DATA source"); require(packet[5] == 0 && packet[4] >= ContextRegisterBase && packet[4] - ContextRegisterBase <= 0x400u - copyDataCount(packet), "COPY_DATA to a register outside the context registers is not implemented"); @@ -476,12 +469,12 @@ void Validate(std::span packet, std::uint32_t queue) { case 0x50: size(7); require((packet[1] & ~(0xe0300001u | DmaSourceCachePolicy | DmaDestinationCachePolicy)) == 0, "DMA_DATA reserved fields are not implemented"); - require(memorySelector(dmaDestination(packet)) || dmaDestination(packet) == DmaSelectGds || dmaDestination(packet) == DmaSelectNowhere, "DMA_DATA register destination is not implemented"); - require(dmaDestination(packet) != DmaSelectNowhere || memorySelector(dmaSource(packet)), "DMA_DATA prefetch of a register, GDS or immediate source is not implemented"); - require(memorySelector(dmaSource(packet)) || dmaSource(packet) == 2 || dmaSource(packet) == DmaSelectGds, "DMA_DATA register source is not implemented"); - require(dmaSource(packet) != DmaSelectGds || (packet[3] == 0 && gdsRange(packet[2], packet[6] & 0x3ffffffu)), "DMA_DATA GDS source range exceeds the GDS"); - require(dmaDestination(packet) != DmaSelectGds || (packet[5] == 0 && gdsRange(packet[4], packet[6] & 0x3ffffffu)), "DMA_DATA GDS destination range exceeds the GDS"); - require(dmaSource(packet) != 2 || packet[3] == 0, "DMA_DATA immediate exceeds 32 bits"); + require(memorySelector(dmaDestination(packet)) || dmaDestination(packet) == DmaSelectGds || dmaDestination(packet) == DmaDestinationNowhere, "DMA_DATA register destination is not implemented"); + require(dmaDestination(packet) != DmaDestinationNowhere || memorySelector(dmaSource(packet)), "DMA_DATA prefetch of a register, GDS or immediate source is not implemented"); + require(memorySelector(dmaSource(packet)) || dmaSource(packet) == DmaSourceData || dmaSource(packet) == DmaSelectGds, "DMA_DATA register source is not implemented"); + require(dmaSource(packet) != DmaSelectGds || (packet[3] == 0 && gdsRange(packet[2], DmaBytes(packet))), "DMA_DATA GDS source range exceeds the GDS"); + require(dmaDestination(packet) != DmaSelectGds || (packet[5] == 0 && gdsRange(packet[4], DmaBytes(packet))), "DMA_DATA GDS destination range exceeds the GDS"); + require(dmaSource(packet) != DmaSourceData || packet[3] == 0, "DMA_DATA immediate exceeds 32 bits"); break; default: throw std::runtime_error("known packet has no validator"); } @@ -594,28 +587,29 @@ std::optional ResolveStore(std::span packet, co const auto fits = [limit](std::uint64_t destination, std::size_t bytes) { return bytes <= limit && destination % 4 == 0 && bytes % 4 == 0; }; switch ((packet[0] >> 8u) & 0xffu) { case 0x40: { - if (packet.size() < 6 || copyDataToRegister(packet)) return std::nullopt; - const auto source = ((packet[1] & 0xfu) << 1u) | ((packet[1] >> 30u) & 1u); - const std::size_t bytes = (packet[1] & 0x10000u) != 0 ? 8 : 4; + if (packet.size() < 6 || CopyDataToRegister(packet)) return std::nullopt; + const std::size_t bytes = CopyDataBytes(packet); const auto destination = address(packet[4], packet[5]); if (!fits(destination, bytes)) return std::nullopt; return StoreWrite{destination, {}, copyDataSource(packet, bytes)}; } case 0x50: { if (packet.size() < 7 || dmaDestination(packet) == DmaSelectGds) return std::nullopt; - if (dmaDestination(packet) == DmaSelectNowhere) return StoreWrite{address(packet[2], packet[3]), {}, {}}; - const std::size_t bytes = packet[6] & 0x3ffffffu; + if (dmaDestination(packet) == DmaDestinationNowhere) return StoreWrite{address(packet[2], packet[3]), {}, {}}; + const std::size_t bytes = DmaBytes(packet); const auto destination = address(packet[4], packet[5]); if (!fits(destination, bytes)) return std::nullopt; return StoreWrite{destination, {}, dmaSourceBytes(packet)}; } case 0x83: { - if (packet.size() < 5 || packet[1] / 4 > QueueState::ConstantRamWords || packet[2] > QueueState::ConstantRamWords - packet[1] / 4) return std::nullopt; - const auto words = queue.ConstantRam().subspan(packet[1] / 4, packet[2]); + if (packet.size() != 5) return std::nullopt; + const auto dump = DecodeConstantRamDump(packet); + if (dump.offset / 4 > QueueState::ConstantRamWords || dump.words > QueueState::ConstantRamWords - dump.offset / 4) return std::nullopt; + const auto words = queue.ConstantRam().subspan(dump.offset / 4, dump.words); if (words.size_bytes() > limit) return std::nullopt; // A view of the queue's constant RAM: valid until the next CONST_RAM write on this queue, // which is a later packet of the same worker. - return StoreWrite{address(packet[3], packet[4]), std::as_bytes(words), {}}; + return StoreWrite{dump.destination, std::as_bytes(words), {}}; } default: return std::nullopt; } @@ -629,10 +623,87 @@ std::uint32_t DmaDestination(std::span packet) { return dmaDestination(packet); } +WaitFields DecodeWait(std::span packet) { + const auto opcode = packet.empty() ? 0u : (packet[0] >> 8u) & 0xffu; + require((opcode == 0x3cu && packet.size() == 7) || (opcode == 0x93u && packet.size() == 9), "expected WAIT_REG_MEM or WAIT_REG_MEM_64 packet"); + const bool wide = opcode == 0x93u; + return {packet[1] & 7u, address(packet[2], packet[3]), wide ? address(packet[4], packet[5]) : packet[4], wide ? address(packet[6], packet[7]) : packet[5], wide ? packet[8] : packet[6], WaitAwaitedBytes(packet)}; +} + +ConditionalWrite DecodeConditionalWrite(std::span packet) { + require(packet.size() == 9 && ((packet[0] >> 8u) & 0xffu) == 0x45u, "expected COND_WRITE packet"); + return {address(packet[2], packet[3]), address(packet[6], packet[7]), packet[8]}; +} + +ConstantRamDump DecodeConstantRamDump(std::span packet) { + require(packet.size() == 5 && ((packet[0] >> 8u) & 0xffu) == 0x83u, "expected DUMP_CONST_RAM packet"); + return {packet[1], packet[2], address(packet[3], packet[4])}; +} + +std::vector DecodeStores(std::span packet) { + require(!packet.empty(), "truncated PM4 header"); + const auto known = [](std::uint64_t destination, std::span bytes) { + return PacketStore{destination, bytes.size(), PacketStore::Value::Bytes, {bytes.begin(), bytes.end()}, false}; + }; + const auto pattern = [](std::uint64_t destination, std::uint64_t bytes, std::uint32_t value) { + const auto repeated = std::as_bytes(std::span(&value, 1)); + return PacketStore{destination, bytes, PacketStore::Value::Pattern, {repeated.begin(), repeated.end()}, false}; + }; + const auto runTime = [](std::uint64_t destination, std::uint64_t bytes) { return PacketStore{destination, bytes, PacketStore::Value::RunTime, {}, false}; }; + if (FillerPacket(packet[0])) return {}; + switch ((packet[0] >> 8u) & 0xffu) { + case 0x49: { + const auto label = DecodeLabelWrite(packet); + if (!label.has_value()) return {}; + if ((packet[2] >> 29u) == 3) return {runTime(label->address, label->Bytes().size())}; + return {known(label->address, label->Bytes())}; + } + case 0x37: { + const auto label = DecodeLabelWrite(packet); + if (!label.has_value()) return {}; + return {known(label->address, label->Bytes())}; + } + case 0x40: { + require(packet.size() == 6, "invalid COPY_DATA size"); + if (CopyDataToRegister(packet)) return {}; + const auto destination = address(packet[4], packet[5]); + if (CopyDataImmediate(CopyDataSource(packet))) return {pattern(destination, CopyDataBytes(packet), packet[2])}; + return {runTime(destination, CopyDataBytes(packet))}; + } + case 0x50: { + require(packet.size() == 7, "invalid DMA_DATA size"); + const auto bytes = DmaBytes(packet); + if (bytes == 0 || dmaDestination(packet) == DmaDestinationNowhere || dmaDestination(packet) == DmaSelectGds) return {}; + const auto destination = address(packet[4], packet[5]); + if (dmaSource(packet) == DmaSourceData) return {pattern(destination, bytes, packet[2])}; + return {runTime(destination, bytes)}; + } + case 0x45: { + const auto write = DecodeConditionalWrite(packet); + auto store = known(write.destination, std::as_bytes(std::span(&write.value, 1))); + store.conditional = true; + return {store}; + } + case 0x83: { + const auto dump = DecodeConstantRamDump(packet); + if (dump.words == 0) return {}; + return {runTime(dump.destination, static_cast(dump.words) * 4)}; + } + case 0x46: { + if (!SampleDump(packet)) return {}; + require(packet.size() == 4, "invalid EVENT_WRITE sample dump size"); + std::vector stores; + for (std::uint64_t block = 0; block < SampleDumpBlocks; ++block) stores.push_back(runTime(SampleDumpAddress(packet) + block * SampleDumpStride, SampleDumpBlockBytes)); + return stores; + } + default: return {}; + } +} + std::optional DecodeMemoryCopy(std::span packet) { if (packet.size() != 7 || ((packet[0] >> 8u) & 0xffu) != 0x50) return std::nullopt; if (!memorySelector(dmaSource(packet)) || !memorySelector(dmaDestination(packet))) return std::nullopt; - const std::size_t bytes = packet[6] & 0x3ffffffu; + const std::size_t bytes = DmaBytes(packet); const auto source = address(packet[2], packet[3]); const auto destination = address(packet[4], packet[5]); if (bytes == 0 || (source < destination + bytes && destination < source + bytes)) return std::nullopt; @@ -915,16 +986,19 @@ void Execute(std::span packet, QueueState& queue) { case 0x81: std::copy(packet.begin() + 2, packet.end(), queue.WritableConstantRam().begin() + packet[1] / 4); return; - case 0x83: - GuestMemory::Write(address(packet[3], packet[4]), std::as_bytes(queue.ConstantRam().subspan(packet[1] / 4, packet[2])), 4); + case 0x83: { + const auto dump = DecodeConstantRamDump(packet); + GuestMemory::Write(dump.destination, std::as_bytes(queue.ConstantRam().subspan(dump.offset / 4, dump.words)), 4); return; + } case 0x45: { + const auto write = DecodeConditionalWrite(packet); std::uint32_t value = 0; { const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::Wait); - GuestMemory::Read(address(packet[2], packet[3]), std::as_writable_bytes(std::span(&value, 1)), 4); + GuestMemory::Read(write.compared, std::as_writable_bytes(std::span(&value, 1)), 4); } - if (waitCompares(packet, false, value)) GuestMemory::Write(address(packet[6], packet[7]), std::as_bytes(packet.subspan(8, 1)), 4); + if (waitCompares(packet, false, value)) GuestMemory::Write(write.destination, std::as_bytes(std::span(&write.value, 1)), 4); return; } case 0x37: { @@ -935,7 +1009,7 @@ void Execute(std::span packet, QueueState& queue) { return; } case 0x40: { - if (copyDataToRegister(packet)) { + if (CopyDataToRegister(packet)) { std::array values{}; const auto count = copyDataCount(packet); const GuestMemory::ReadSiteScope site(GuestMemory::ReadSite::Registers); @@ -944,15 +1018,15 @@ void Execute(std::span packet, QueueState& queue) { return; } const auto destination = address(packet[4], packet[5]); - const auto data = copyDataSource(packet, (packet[1] & 0x10000u) != 0 ? 8 : 4); + const auto data = copyDataSource(packet, CopyDataBytes(packet)); GuestMemory::CheckRange(reinterpret_cast(destination), data.size(), 1, true); GuestMemory::Write(destination, data); return; } case 0x28: return; case 0x50: { - const std::size_t bytes = packet[6] & 0x3ffffffu; - if (bytes == 0 || dmaDestination(packet) == DmaSelectNowhere) return; + const std::size_t bytes = DmaBytes(packet); + if (bytes == 0 || dmaDestination(packet) == DmaDestinationNowhere) return; auto data = dmaSourceBytes(packet); const auto destination = dmaDestination(packet) == DmaSelectGds ? GdsAddress() + packet[4] : address(packet[4], packet[5]); GuestMemory::CheckRange(reinterpret_cast(destination), bytes, 1, true); diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/SubmissionCapture.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/SubmissionCapture.cpp new file mode 100644 index 0000000000..217e5b0c81 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/Execution/src/SubmissionCapture.cpp @@ -0,0 +1,237 @@ +#include "prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp" +#include "prx/libSceAgcDriver/Execution/include/Pm4.hpp" +#include +#include +#include +#include +#include +#include +#include +#include + +namespace AgcDriver { +namespace { + +static_assert(std::endian::native == std::endian::little, "captured dwords are stored in the guest's little-endian order"); + +void requireCapture(bool condition, const std::string& reason) { + if (!condition) throw std::runtime_error("submission capture: " + reason); +} + +std::vector split(std::string_view text, char separator) { + std::vector result; + while (true) { + const auto end = text.find(separator); + result.push_back(text.substr(0, end)); + if (end == std::string_view::npos) break; + text.remove_prefix(end + 1); + } + return result; +} + +template +bool parse(std::string_view text, TValue& value, int base = 10) { + if (text.empty()) return false; + const auto [last, error] = std::from_chars(text.data(), text.data() + text.size(), value, base); + return error == std::errc{} && last == text.data() + text.size(); +} + +SubmissionKind kindFromName(std::string_view name, const std::string& where) { + for (const auto kind : {SubmissionKind::Submit, SubmissionKind::Rewind, SubmissionKind::Suspend}) { + if (SubmissionCapture::KindName(kind) == name) return kind; + } + throw std::runtime_error("submission capture: " + where + ": unknown kind '" + std::string(name) + "'"); +} + +bool waitPacket(std::uint32_t header) { + const auto opcode = (header >> 8u) & 0xffu; + return (header >> 30u) == 3u && (opcode == 0x3c || opcode == 0x93); +} + +std::string heldField(const CapturedSubmission& submission) { + if (submission.heldAtSubmit.empty()) return "-"; + std::string text; + for (const auto offset : submission.heldAtSubmit) text += (text.empty() ? "" : ",") + std::to_string(offset); + return text; +} + +std::string conditionalField(const CapturedSubmission& submission) { + if (submission.conditionalRanges.empty()) return "-"; + std::string text; + for (const auto& [start, end] : submission.conditionalRanges) text += (text.empty() ? "" : ",") + std::to_string(start) + ":" + std::to_string(end); + return text; +} + +std::vector readHeld(std::string_view field, std::size_t dwords, const std::string& where) { + std::vector result; + if (field == "-") return result; + for (const auto item : split(field, ',')) { + std::size_t offset = 0; + requireCapture(parse(item, offset) && offset < dwords && (result.empty() || offset > result.back()), where + ": malformed held offsets '" + std::string(field) + "'"); + result.push_back(offset); + } + return result; +} + +std::vector> readConditional(std::string_view field, std::size_t dwords, const std::string& where) { + std::vector> result; + if (field == "-") return result; + for (const auto item : split(field, ',')) { + const auto bounds = split(item, ':'); + std::size_t start = 0, end = 0; + requireCapture(bounds.size() == 2 && parse(bounds[0], start) && parse(bounds[1], end) && start < end && end <= dwords && (result.empty() || start > result.back().first), + where + ": malformed conditional ranges '" + std::string(field) + "'"); + result.emplace_back(start, end); + } + return result; +} + +} + +SubmissionCapture::SubmissionCapture(const std::filesystem::path& directory) : directory(directory) { + std::error_code error; + std::filesystem::create_directories(directory, error); + requireCapture(!error && std::filesystem::is_directory(directory), "cannot create the directory " + directory.string()); + index.open(directory / IndexName, std::ios::binary | std::ios::trunc); + requireCapture(index.is_open(), "cannot create " + (directory / IndexName).string()); + index << Magic << '\n' << Columns << '\n'; + index.flush(); + requireCapture(index.good(), "cannot write " + (directory / IndexName).string()); +} + +SubmissionCapture* SubmissionCapture::Active() { + static SubmissionCapture* const capture = [] { + const char* directory = std::getenv("APS5_DUMP_SUBMITS"); + return directory != nullptr && *directory != '\0' ? new SubmissionCapture(directory) : nullptr; + }(); + return capture; +} + +CapturedSubmission SubmissionCapture::Prepare(std::uint32_t queue, SubmissionKind kind, std::span commands, const std::map& conditionalEnds) { + QueueType(queue); + requireCapture(kind != SubmissionKind::Suspend || commands.empty(), "a suspend point has no commands"); + CapturedSubmission submission; + submission.queue = queue; + submission.kind = kind; + submission.dwords = commands.size(); + submission.commands.assign(commands.begin(), commands.end()); + for (std::size_t cursor = 0; cursor < commands.size(); cursor += Pm4::PacketWords(commands[cursor])) { + if (waitPacket(commands[cursor])) submission.heldAtSubmit.push_back(cursor); + } + for (const auto& [start, end] : conditionalEnds) { + requireCapture(start < end && end <= commands.size(), "a conditional range lies outside the commands"); + submission.conditionalRanges.emplace_back(start, end); + } + return submission; +} + +void SubmissionCapture::Take(CapturedSubmission& submission, std::uint64_t serial, std::uint64_t timeUs, const std::set& heldAtSubmit) noexcept { + submission.ordinal = next.fetch_add(1, std::memory_order_relaxed); + submission.serial = serial; + submission.timeUs = timeUs; + std::erase_if(submission.heldAtSubmit, [&](std::size_t offset) { return !heldAtSubmit.contains(offset); }); +} + +void SubmissionCapture::Write(const CapturedSubmission& submission) { + requireCapture(submission.dwords == submission.commands.size(), "dword count does not match the commands"); + const auto name = FileName(submission); + if (!submission.commands.empty()) { + std::ofstream file(directory / name, std::ios::binary | std::ios::trunc); + file.write(reinterpret_cast(submission.commands.data()), static_cast(submission.commands.size() * sizeof(std::uint32_t))); + file.close(); + requireCapture(!file.fail(), "cannot write " + (directory / name).string()); + } + char head[160]; + const int length = std::snprintf(head, sizeof(head), "%llu 0x%x %s %llu %llu %zu %s ", static_cast(submission.ordinal), submission.queue, QueueType(submission.queue).data(), + static_cast(submission.serial), static_cast(submission.timeUs), submission.dwords, KindName(submission.kind).data()); + requireCapture(length > 0 && static_cast(length) < sizeof(head), "index line does not fit"); + const auto line = std::string(head, static_cast(length)) + name + " " + heldField(submission) + " " + conditionalField(submission) + "\n"; + std::lock_guard lock(mutex); + index.write(line.data(), static_cast(line.size())); + index.flush(); + requireCapture(index.good(), "cannot append to " + (directory / IndexName).string()); +} + +std::string SubmissionCapture::FileName(const CapturedSubmission& submission) { + if (submission.dwords == 0) return "-"; + char name[48]; + std::snprintf(name, sizeof(name), "submission_%06llu.bin", static_cast(submission.ordinal)); + return name; +} + +std::string_view SubmissionCapture::KindName(SubmissionKind kind) { + switch (kind) { + case SubmissionKind::Submit: return "submit"; + case SubmissionKind::Rewind: return "rewind"; + case SubmissionKind::Suspend: return "suspend"; + } + throw std::runtime_error("submission capture: unknown submission kind"); +} + +std::string_view SubmissionCapture::QueueType(std::uint32_t queue) { + if (queue == 0) return "graphics"; + if (queue >= 0x20 && queue < 0x58) return "compute"; + char what[64]; + std::snprintf(what, sizeof(what), "queue 0x%x is neither the graphics queue nor a compute queue", queue); + throw std::runtime_error(std::string("submission capture: ") + what); +} + +std::vector SubmissionCapture::ReadIndex(const std::filesystem::path& directory) { + const auto path = directory / IndexName; + std::ifstream input(path, std::ios::binary); + requireCapture(input.is_open(), "cannot open " + path.string()); + std::string line; + requireCapture(std::getline(input, line) && line == Magic, path.string() + " does not start with '" + std::string(Magic) + "'"); + requireCapture(std::getline(input, line) && (line == Columns || line == ColumnsWithoutHeld), path.string() + " has no column line '" + std::string(Columns) + "' or '" + std::string(ColumnsWithoutHeld) + "'"); + const bool complete = line == Columns; + const std::size_t columns = complete ? 10 : 8; + std::vector result; + for (std::size_t number = 3; std::getline(input, line); ++number) { + const auto where = path.string() + " line " + std::to_string(number); + requireCapture(!input.eof(), where + " is incomplete"); + const auto values = split(line, ' '); + requireCapture(values.size() == columns, where + " does not have " + std::to_string(columns) + " fields"); + CapturedSubmission submission; + requireCapture(parse(values[0], submission.ordinal) && values[1].starts_with("0x") && parse(values[1].substr(2), submission.queue, 16) && parse(values[3], submission.serial) + && parse(values[4], submission.timeUs) && parse(values[5], submission.dwords), where + " has a malformed number"); + requireCapture(QueueType(submission.queue) == values[2], where + ": queue type does not match the queue"); + submission.kind = kindFromName(values[6], where); + requireCapture(submission.kind != SubmissionKind::Suspend || submission.dwords == 0, where + ": a suspend point has commands"); + requireCapture(submission.dwords <= std::numeric_limits::max() / sizeof(std::uint32_t), where + ": dword count overflows"); + requireCapture(values[7] == FileName(submission), where + ": file name does not match the ordinal and size"); + submission.heldKnown = complete; + submission.conditionalKnown = complete; + if (complete) { + submission.heldAtSubmit = readHeld(values[8], submission.dwords, where); + submission.conditionalRanges = readConditional(values[9], submission.dwords, where); + } + result.push_back(std::move(submission)); + } + requireCapture(input.eof(), "cannot read " + path.string()); + std::sort(result.begin(), result.end(), [](const auto& left, const auto& right) { return left.ordinal < right.ordinal; }); + const auto repeated = std::adjacent_find(result.begin(), result.end(), [](const auto& left, const auto& right) { return left.ordinal == right.ordinal; }); + requireCapture(repeated == result.end(), path.string() + " lists ordinal " + (repeated == result.end() ? std::string() : std::to_string(repeated->ordinal)) + " twice"); + return result; +} + +std::vector SubmissionCapture::ReadCommands(const std::filesystem::path& directory, const CapturedSubmission& submission) { + if (submission.dwords == 0) return {}; + const auto path = directory / FileName(submission); + std::error_code error; + const auto size = std::filesystem::file_size(path, error); + requireCapture(!error, "cannot read the size of " + path.string()); + requireCapture(size == submission.dwords * sizeof(std::uint32_t), path.string() + " does not hold " + std::to_string(submission.dwords) + " dwords"); + std::vector commands(submission.dwords); + std::ifstream input(path, std::ios::binary); + input.read(reinterpret_cast(commands.data()), static_cast(size)); + requireCapture(input.good() && input.peek() == std::char_traits::eof(), "cannot read " + path.string()); + return commands; +} + +std::vector SubmissionCapture::Read(const std::filesystem::path& directory) { + auto submissions = ReadIndex(directory); + for (auto& submission : submissions) submission.commands = ReadCommands(directory, submission); + return submissions; +} + +} diff --git a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp index 4e72b41c06..148ef9b16a 100644 --- a/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp +++ b/core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp @@ -1909,7 +1909,7 @@ bool VulkanDevice::AfterRecordedWork(std::function action, bool reapFirs bool VulkanDevice::DumpSamplesOnGpu(std::uint64_t address) { if (!state->recorder) return false; auto& recorder = *state->recorder; - constexpr std::size_t bytes = 15 * 16 + 8; + constexpr std::size_t bytes = Pm4::SampleDumpBytes; if (OpportunisticReap()) recorder.Reap(); if (state->CopiedWriterOverlaps(address, bytes) || (Graphics::Recorder::PendingCompletionLabels() != 0 && recorder.CompletionLabelIn(address, bytes))) return false; const auto context = graphicsContext(); diff --git a/core/libs/prx/libSceAgcDriver/tests/Pm4Replay.cpp b/core/libs/prx/libSceAgcDriver/tests/Pm4Replay.cpp new file mode 100644 index 0000000000..98cf833fd3 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tests/Pm4Replay.cpp @@ -0,0 +1,730 @@ +#include "../tools/Pm4Analysis.hpp" +#include "prx/libSceAgcDriver/Execution/include/Driver.hpp" +#include "prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp" +#include "prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp" +#include "prx/libSceAgcDriver/Submit/include/Acb.hpp" +#include "prx/libSceAgcDriver/Submit/include/Dcb.hpp" +#include "prx/libc/include/Shutdown.hpp" +#include "SceTypes.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +extern "C" { +std::uint32_t* APS5_VABI sceAgcDcbWriteData(CommandBuffer*, std::uint8_t, std::uint8_t, std::uint64_t, const void*, std::uint32_t, std::uint8_t, std::uint8_t); +std::uint32_t* APS5_VABI sceAgcAcbWriteData(CommandBuffer*, std::uint8_t, std::uint8_t, std::uint64_t, const void*, std::uint32_t, std::uint8_t, std::uint8_t); +std::uint32_t* APS5_VABI sceAgcDcbWaitRegMem(CommandBuffer*, std::uint8_t, std::uint8_t, std::uint8_t, std::uint8_t, const volatile void*, std::uint64_t, std::uint64_t, std::uint32_t); +std::uint32_t* APS5_VABI sceAgcAcbWaitRegMem(CommandBuffer*, std::uint8_t, std::uint8_t, std::uint8_t, const volatile void*, std::uint64_t, std::uint64_t, std::uint32_t); +std::uint32_t* APS5_VABI sceAgcCbReleaseMem(CommandBuffer*, std::uint8_t, std::uint16_t, std::uint8_t, std::uint8_t, const volatile Label*, std::uint8_t, std::uint64_t, std::uint16_t, std::uint16_t, std::uint8_t, std::uint32_t); +std::uint32_t* APS5_VABI sceAgcDcbDmaData(CommandBuffer*, std::uint8_t, std::uint8_t, std::uint8_t, std::uint64_t, std::uint8_t, std::uint8_t, std::uint64_t, std::uint32_t, std::uint8_t, std::uint8_t, std::uint8_t); +std::uint32_t* APS5_VABI sceAgcDcbCondExec(CommandBuffer*, const volatile std::uint32_t*, std::uint32_t); +std::uint32_t* APS5_VABI sceAgcAcbRewind(CommandBuffer*, std::uint32_t); +int APS5_VABI sceAgcAsyncRewindPatchSetRewindState(std::uint32_t*, std::uint8_t); +} + +namespace { + +using AgcDriver::CapturedSubmission; +using AgcDriver::SubmissionCapture; +using AgcDriver::SubmissionKind; +using AgcDriver::Pm4Analysis::WaitReport; +using AgcDriver::Pm4Analysis::WriterMatch; + +constexpr std::uint8_t GraphicsMemory = 0x0a; +constexpr std::uint8_t ComputeMemory = 5; +constexpr std::uint8_t Equal = 3; +constexpr std::uint8_t Bits32 = 0; +constexpr std::uint8_t Bits64 = 1; +constexpr std::uint8_t ReleaseEvent = 0x14; +constexpr std::uint8_t Data32 = 1; +constexpr std::uint8_t Data64 = 2; +constexpr std::uint8_t DmaMemory = 0; +constexpr std::uint8_t DmaFill = 2; +constexpr std::uint32_t WrittenValue = 0x11; +constexpr std::uint64_t ReleasedValue = 0x0000123400000007ull; +constexpr std::uint32_t FillValue = 0x5a5a5a5au; +constexpr std::uint32_t HeldValue = 9; +constexpr std::uint32_t CpuValue = 5; +constexpr std::uint32_t One = 1; +constexpr std::uint32_t Zero = 0; +constexpr std::uint32_t BeforeRewindValue = 1; +constexpr std::uint32_t AfterRewindValue = 2; +constexpr std::uint32_t WriteDataWords = 5; + +alignas(8) std::uint32_t writtenLabel = 0; +alignas(8) Label releasedLabel{}; +alignas(16) std::array filled{}; +alignas(8) std::uint32_t heldLabel = 0; +alignas(8) std::uint32_t cpuLabel = 0; +alignas(8) std::uint32_t reusedLabel = 0; +alignas(8) std::uint32_t conditionLabel = 0; +alignas(8) std::uint32_t guardedLabel = 0; +alignas(8) std::uint32_t beforeRewind = 0; +alignas(8) std::uint32_t afterRewind = 0; +alignas(8) std::uint32_t syntheticLabel = 0; +alignas(8) std::uint32_t otherLabel = 0; + +void check(bool condition, const std::string& reason) { + if (!condition) throw std::runtime_error(reason); +} + +std::uint64_t at(const volatile void* pointer) { + return static_cast(reinterpret_cast(pointer)); +} + +std::string hex(const volatile void* pointer) { + char text[32]; + std::snprintf(text, sizeof(text), "0x%llx", static_cast(at(pointer))); + return text; +} + +class Commands { +public: + Commands() = default; + Commands(const Commands&) = delete; + Commands& operator=(const Commands&) = delete; + CommandBuffer* Buffer() { return &buffer; } + std::vector Words(std::size_t from = 0) const { return {words.begin() + static_cast(from), words.begin() + (buffer.cursor_up - words.data())}; } + std::size_t Offset(const std::uint32_t* packet) const { return static_cast(packet - words.data()); } + std::size_t Size() const { return static_cast(buffer.cursor_up - words.data()); } + void Submit(std::uint32_t queue) { + Packet packet{words.data(), static_cast(buffer.cursor_up - words.data()), 0, {}}; + check((queue == 0 ? sceAgcDriverSubmitDcb(&packet) : sceAgcDriverSubmitAcb(queue, &packet)) == 0, "submission failed"); + } + +private: + std::array words{}; + CommandBuffer buffer{words.data(), words.data() + words.size(), words.data(), words.data() + words.size(), nullptr, nullptr, 0}; +}; + +std::size_t waitAt(const Commands& commands, const std::uint32_t* firstTag) { + return commands.Offset(firstTag) + 4; +} + +std::vector names(CapturedSubmission submission) { + std::vector result; + for (const auto& packet : AgcDriver::Pm4Analysis::Decode(submission)) result.push_back(packet.name); + return result; +} + +const WaitReport& waitOn(const AgcDriver::Pm4Analysis::Analysis& analysis, const volatile void* address, std::size_t skip = 0) { + for (const auto& report : analysis.waits) { + if (report.wait.kind != AgcDriver::Pm4Analysis::MemoryWait::Kind::Memory || report.wait.fields.address != at(address)) continue; + if (skip-- == 0) return report; + } + throw std::runtime_error("no wait on " + hex(address) + " in the analysis"); +} + +bool listed(const WaitReport& report, WriterMatch::Effect effect, std::uint64_t ordinal) { + return std::any_of(report.writers.begin(), report.writers.end(), [&](const WriterMatch& writer) { return writer.effect == effect && writer.at.ordinal == ordinal; }); +} + +std::string verdict(const WaitReport& report) { + return std::to_string(static_cast(report.verdict)) + " route " + std::to_string(static_cast(report.route)) + ": " + report.reason; +} + +std::pair report(const std::filesystem::path& directory, bool full = false) { + std::FILE* output = std::tmpfile(); + check(output != nullptr, "cannot create a temporary file for the report"); + const auto code = AgcDriver::Pm4Analysis::Report(directory, full, output); + std::string text; + std::rewind(output); + for (int character; (character = std::fgetc(output)) != EOF;) text.push_back(static_cast(character)); + std::fclose(output); + return {code, text}; +} + +std::vector lines(const std::filesystem::path& path) { + std::ifstream input(path, std::ios::binary); + std::vector result; + for (std::string line; std::getline(input, line);) result.push_back(line); + return result; +} + +std::vector fields(const std::string& line) { + std::istringstream input(line); + std::vector result; + for (std::string field; input >> field;) result.push_back(field); + return result; +} + +struct Planned { + std::uint32_t queue = 0; + std::uint64_t serial = 0; + SubmissionKind kind = SubmissionKind::Submit; + std::vector commands; + std::set held; + std::map conditional; +}; + +Planned planned(std::uint32_t queue, std::uint64_t serial, const Commands& commands, std::set held = {}) { + return {queue, serial, SubmissionKind::Submit, commands.Words(), std::move(held), {}}; +} + +void writeCapture(const std::filesystem::path& directory, const std::vector& submissions) { + SubmissionCapture capture(directory); + for (const auto& submission : submissions) { + auto captured = SubmissionCapture::Prepare(submission.queue, submission.kind, submission.commands, submission.conditional); + capture.Take(captured, submission.serial, submission.serial * 1000, submission.held); + capture.Write(captured); + } +} + +void testDriverCapture(const std::filesystem::path& directory) { + heldLabel = HeldValue; + conditionLabel = One; + + Commands graphicsWrites; + sceAgcDcbWriteData(graphicsWrites.Buffer(), GraphicsMemory, 0, at(&writtenLabel), &WrittenValue, 1, 0, 1); + sceAgcDcbDmaData(graphicsWrites.Buffer(), 0, DmaMemory, 0, at(filled.data()), DmaFill, 0, FillValue, sizeof(filled), 0, 1, 0); + graphicsWrites.Submit(0); + + Commands computeRelease; + sceAgcAcbWaitRegMem(computeRelease.Buffer(), Bits32, Equal, 0, &writtenLabel, WrittenValue, 0xffffffffu, 0); + sceAgcCbReleaseMem(computeRelease.Buffer(), ReleaseEvent, 0, 0, 0, &releasedLabel, Data64, ReleasedValue, 0, 0, 0, 0); + computeRelease.Submit(0x20); + + AgcDriverSuspendPoint_nid_postfix(); + + Commands graphicsWaits; + sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits64, Equal, 0, 0, &releasedLabel, ReleasedValue, ~0ull, 0); + sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits32, Equal, 0, 0, &filled[2], FillValue, 0xffffffffu, 0); + const auto heldWait = waitAt(graphicsWaits, sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits32, Equal, 0, 0, &heldLabel, HeldValue, 0xffffffffu, 0)); + const auto cpuWait = waitAt(graphicsWaits, sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits32, Equal, 0, 0, &cpuLabel, CpuValue, 0xffffffffu, 0)); + sceAgcDcbWriteData(graphicsWaits.Buffer(), GraphicsMemory, 0, at(&reusedLabel), &One, 1, 0, 1); + sceAgcDcbWriteData(graphicsWaits.Buffer(), GraphicsMemory, 0, at(&reusedLabel), &Zero, 1, 0, 1); + sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits32, Equal, 0, 0, &reusedLabel, One, 0xffffffffu, 0); + const auto condition = graphicsWaits.Offset(sceAgcDcbCondExec(graphicsWaits.Buffer(), &conditionLabel, WriteDataWords)); + sceAgcDcbWriteData(graphicsWaits.Buffer(), GraphicsMemory, 0, at(&guardedLabel), &One, 1, 0, 1); + sceAgcDcbWaitRegMem(graphicsWaits.Buffer(), Bits32, Equal, 0, 0, &guardedLabel, One, 0xffffffffu, 0); + graphicsWaits.Submit(0); + std::atomic_ref(cpuLabel).store(CpuValue); + + Commands rewound; + sceAgcAcbWriteData(rewound.Buffer(), ComputeMemory, 0, at(&beforeRewind), &BeforeRewindValue, 1, 0, 1); + auto* rewind = sceAgcAcbRewind(rewound.Buffer(), 0); + const auto tail = rewound.Offset(rewind) + 2; + sceAgcAcbWriteData(rewound.Buffer(), ComputeMemory, 0, at(&afterRewind), &AfterRewindValue, 1, 0, 1); + const auto submittedRewound = rewound.Words(); + rewound.Submit(0x21); + sceAgcAsyncRewindPatchSetRewindState(rewind, 1); + AgcDriverWaitIdle_nid_postfix(); + check(std::atomic_ref(beforeRewind).load() == BeforeRewindValue && std::atomic_ref(afterRewind).load() == AfterRewindValue, "the driver did not execute the captured submissions"); + check(std::atomic_ref(guardedLabel).load() == One, "the driver did not execute the WRITE_DATA its COND_EXEC guards"); + + const auto index = lines(directory / SubmissionCapture::IndexName); + check(index.size() == 8 && index[0] == SubmissionCapture::Magic && index[1] == SubmissionCapture::Columns, "the index does not hold its two header lines and one line per submission"); + const auto computeLine = fields(index[3]); + check(computeLine.size() == 10 && computeLine[0] == "1" && computeLine[1] == "0x20" && computeLine[2] == "compute" && computeLine[6] == "submit" && computeLine[7] == "submission_000001.bin" && computeLine[9] == "-", "the index line of the compute submission is " + index[3]); + check(index[4].starts_with("2 0x0 graphics ") && index[4].ends_with(" 0 suspend - - -"), "the index line of the suspend point is " + index[4]); + const auto waitsLine = fields(index[5]); + check(waitsLine.size() == 10 && waitsLine[0] == "3", "the index line of the graphics waits is " + index[5]); + const auto conditionalEnd = condition + 5 + WriteDataWords; + check(waitsLine[9] == std::to_string(condition) + ":" + std::to_string(conditionalEnd), "the index does not list the COND_EXEC range: " + index[5]); + + const auto captured = SubmissionCapture::Read(directory); + check(captured.size() == 6, "the capture does not hold the four submissions, the suspend point and the rewind tail"); + const std::vector queues{0, 0x20, 0, 0, 0x21, 0x21}; + const std::vector kinds{SubmissionKind::Submit, SubmissionKind::Submit, SubmissionKind::Suspend, SubmissionKind::Submit, SubmissionKind::Submit, SubmissionKind::Rewind}; + for (std::size_t i = 0; i < captured.size(); ++i) { + check(captured[i].ordinal == i && captured[i].queue == queues[i] && captured[i].kind == kinds[i], "captured submission " + std::to_string(i) + " has the wrong ordinal, queue or kind"); + check(i == 0 || captured[i].timeUs >= captured[i - 1].timeUs, "capture times go backwards"); + check(i == 0 || i == 5 || captured[i].serial > captured[i - 1].serial, "capture serials are not in submission order"); + } + const auto& held = captured[3].heldAtSubmit; + check(std::find(held.begin(), held.end(), heldWait) != held.end(), "the wait on a label that held its value at submit is not marked held: " + index[5]); + check(std::find(held.begin(), held.end(), cpuWait) == held.end(), "the wait on a label the CPU stores after the submit is marked held: " + index[5]); + check(captured[5].serial == captured[4].serial, "the rewind tail does not carry the serial of its submission"); + check(captured[0].commands == graphicsWrites.Words() && captured[1].commands == computeRelease.Words() && captured[2].commands.empty() && captured[3].commands == graphicsWaits.Words(), "captured dwords differ from the submitted ones"); + check(captured[4].commands == std::vector(submittedRewound.begin(), submittedRewound.begin() + static_cast(tail)), "captured dwords up to the REWIND differ from the submitted ones, whose valid bit was still clear"); + check(captured[5].commands == rewound.Words(tail), "captured dwords of the rewind tail differ from the submitted ones"); + + check(names(captured[0]) == std::vector{"WRITE_DATA", "DMA_DATA"}, "decoded packets of the graphics writes"); + check(names(captured[1]) == std::vector{"SET_UCONFIG_REG", "WAIT_REG_MEM", "SET_UCONFIG_REG", "RELEASE_MEM"}, "decoded packets of the compute release"); + check(names(captured[4]) == std::vector{"WRITE_DATA", "REWIND"} && names(captured[5]) == std::vector{"WRITE_DATA"}, "decoded packets of the rewound submission"); + const auto release = AgcDriver::Pm4Analysis::Decode(captured[1]).back(); + check(release.operands.find("address " + hex(&releasedLabel)) != std::string::npos, "RELEASE_MEM operands do not name its label: " + release.operands); + check(release.stores.size() == 1 && release.stores[0].store.address == at(&releasedLabel) && release.stores[0].store.bytes == 8 && release.stores[0].label, "RELEASE_MEM does not decode to its 8-byte label store"); + const auto fill = AgcDriver::Pm4Analysis::Decode(captured[0]).back(); + check(fill.stores.size() == 1 && fill.stores[0].store.address == at(filled.data()) && fill.stores[0].store.bytes == sizeof(filled) && fill.stores[0].store.value == AgcDriver::Pm4::PacketStore::Value::Pattern && !fill.stores[0].label, "DMA_DATA does not decode to its fill"); + const auto waits = AgcDriver::Pm4Analysis::Decode(captured[3]); + const auto guarded = std::find_if(waits.begin(), waits.end(), [](const auto& packet) { return !packet.stores.empty() && packet.stores[0].store.address == at(&guardedLabel); }); + check(guarded != waits.end() && guarded->conditional && guarded->stores[0].conditional, "the WRITE_DATA inside the COND_EXEC range is not conditional"); + + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + check(analysis.waits.size() == 8, "expected eight waits, got " + std::to_string(analysis.waits.size())); + for (const auto* label : {static_cast(&writtenLabel), static_cast(&releasedLabel)}) { + const auto& crossQueue = waitOn(analysis, label); + check((crossQueue.verdict == WaitReport::Verdict::Released && crossQueue.route == WaitReport::Route::HeldAtSubmit) || (crossQueue.verdict == WaitReport::Verdict::Unknown && crossQueue.crossQueue), + "a wait on a store of another queue is decided by something else than its value at submit: " + verdict(crossQueue)); + } + check(waitOn(analysis, &releasedLabel).wait.fields.bytes == 8, "the wait on the RELEASE_MEM label does not await 8 bytes"); + const auto& filledWait = waitOn(analysis, &filled[2]); + check(filledWait.verdict == WaitReport::Verdict::Released && (filledWait.route == WaitReport::Route::HeldAtSubmit || (filledWait.route == WaitReport::Route::CurrentValue && listed(filledWait, WriterMatch::Effect::CurrentValue, 0))), + "the wait inside the fill is not released by the earlier DMA_DATA on its queue: " + verdict(filledWait)); + const auto& heldReport = waitOn(analysis, &heldLabel); + check(heldReport.verdict == WaitReport::Verdict::Released && heldReport.route == WaitReport::Route::HeldAtSubmit, "the wait whose value held at submit is not released by it: " + verdict(heldReport)); + const auto& cpu = waitOn(analysis, &cpuLabel); + check(cpu.verdict == WaitReport::Verdict::Unreleased && cpu.writerCount == 0 && cpu.reason.starts_with("unreleased by the GPU; a CPU store could release it"), "the wait on a label the CPU stores after the submit is not unreleased: " + verdict(cpu)); + const auto& reused = waitOn(analysis, &reusedLabel); + check(reused.verdict == WaitReport::Verdict::Released && reused.route == WaitReport::Route::LabelHistory && listed(reused, WriterMatch::Effect::LabelHistory, 3), + "the wait after a store of 1 and a store of 0 in its own submission is not released by the label history, as the driver releases it: " + verdict(reused)); + const auto& conditional = waitOn(analysis, &guardedLabel); + check(conditional.verdict == WaitReport::Verdict::Unknown && !conditional.crossQueue && listed(conditional, WriterMatch::Effect::Undecided, 3) && conditional.reason.find("is conditional") != std::string::npos, + "the wait on a store inside a COND_EXEC range is not unknown: " + verdict(conditional)); + const auto rewindWait = std::find_if(analysis.waits.begin(), analysis.waits.end(), [](const auto& report) { return report.wait.kind == AgcDriver::Pm4Analysis::MemoryWait::Kind::Rewind; }); + check(rewindWait != analysis.waits.end() && rewindWait->verdict == WaitReport::Verdict::Released && rewindWait->route == WaitReport::Route::RewindTail && rewindWait->at.ordinal == 4, "the REWIND is not released by its captured tail"); + check(analysis.Count(WaitReport::Verdict::Unreleased) == 1 && analysis.Count(WaitReport::Verdict::Released) + analysis.Count(WaitReport::Verdict::Unknown) == 7, "expected one unreleased wait and seven released or unknown"); + check(report(directory).first == 1, "the report does not fail on an unreleased wait"); +} + +void testWriterPresent(const std::filesystem::path& source, const std::filesystem::path& directory) { + std::vector submissions; + for (const auto& captured : SubmissionCapture::Read(source)) { + Planned copy{captured.queue, captured.serial, captured.kind, captured.commands, {captured.heldAtSubmit.begin(), captured.heldAtSubmit.end()}, {captured.conditionalRanges.begin(), captured.conditionalRanges.end()}}; + submissions.push_back(std::move(copy)); + } + Commands cpuWriter; + sceAgcAcbWriteData(cpuWriter.Buffer(), ComputeMemory, 0, at(&cpuLabel), &CpuValue, 1, 0, 1); + submissions.push_back(planned(0x22, submissions.back().serial + 1, cpuWriter)); + writeCapture(directory, submissions); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + check(analysis.submissions.size() == submissions.size() && analysis.waits.size() == 8, "the rewritten capture does not hold the submissions and their waits"); + check(analysis.Count(WaitReport::Verdict::Unreleased) == 0, "a wait is unreleased although a store in the capture may release it"); + const auto& cpu = waitOn(analysis, &cpuLabel); + check(cpu.verdict == WaitReport::Verdict::Unknown && cpu.crossQueue && cpu.reason.starts_with("unknown: depends on cross-queue timing (stores on queue 0x22 at #6)") && cpu.writers.front().at.queue == 0x22 && cpu.writers.front().order == WriterMatch::Order::OtherQueue, + "the wait on a label another queue stores is not unknown by cross-queue timing: " + verdict(cpu)); + check(report(directory).first == 0, "the report fails without an unreleased wait"); +} + +void testOverwrittenAcrossSubmissions(const std::filesystem::path& directory) { + Commands storeOne, storeZero, waitOne; + sceAgcDcbWriteData(storeOne.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &One, 1, 0, 1); + sceAgcDcbWriteData(storeZero.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &Zero, 1, 0, 1); + sceAgcDcbWaitRegMem(waitOne.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0); + writeCapture(directory, {planned(0, 1, storeOne), planned(0, 2, storeZero), planned(0, 3, waitOne)}); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + const auto& wait = waitOn(analysis, &syntheticLabel); + check(wait.verdict == WaitReport::Verdict::Unknown && !wait.crossQueue && listed(wait, WriterMatch::Effect::Rejected, 1) && listed(wait, WriterMatch::Effect::Overwritten, 0) && wait.reason.find("depends on when it ran") != std::string::npos, + "a store of 1 in an earlier submission, overwritten by a store of 0, does not leave the wait unknown (the label history holds it only if it ran after the waiting submission entered the driver): " + verdict(wait)); +} + +void testLabelHistory(const std::filesystem::path& directory, std::size_t newer) { + constexpr std::array Newer{2, 3, 4, 5, 6, 7, 8, 9}; + Commands graphics; + sceAgcDcbWriteData(graphics.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &One, 1, 0, 1); + for (std::size_t i = 0; i < newer; ++i) sceAgcDcbWriteData(graphics.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &Newer.at(i), 1, 0, 1); + sceAgcDcbWaitRegMem(graphics.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0); + writeCapture(directory, {planned(0, 1, graphics)}); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + const auto& wait = waitOn(analysis, &syntheticLabel); + if (newer < AgcDriver::DriverDetail::LabelStoreHistory) { + check(wait.verdict == WaitReport::Verdict::Released && wait.route == WaitReport::Route::LabelHistory && listed(wait, WriterMatch::Effect::LabelHistory, 0), "a store kept in the label history does not release the wait: " + verdict(wait)); + } else { + check(wait.verdict == WaitReport::Verdict::Unknown && !wait.crossQueue && listed(wait, WriterMatch::Effect::Evicted, 0), "a store pushed out of the label history releases the wait: " + verdict(wait)); + } +} + +void testUnreleasable(const std::filesystem::path& directory) { + constexpr std::uint32_t StoredValue = 3; + Commands waitFirst, writeAfter, releaseOther, waitOther, rewindForever; + sceAgcDcbWaitRegMem(waitFirst.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0); + sceAgcDcbWriteData(writeAfter.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &One, 1, 0, 1); + sceAgcCbReleaseMem(releaseOther.Buffer(), ReleaseEvent, 0, 0, 0, reinterpret_cast(&otherLabel), Data32, StoredValue, 0, 0, 0, 0); + sceAgcDcbWaitRegMem(waitOther.Buffer(), Bits32, Equal, 0, 0, &otherLabel, StoredValue + 1, 0xffffffffu, 0); + sceAgcAcbRewind(rewindForever.Buffer(), 0); + writeCapture(directory, {planned(0, 1, waitFirst), planned(0, 2, writeAfter), planned(0x20, 3, releaseOther), planned(0, 4, waitOther), planned(0x23, 5, rewindForever)}); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + check(analysis.waits.size() == 3 && analysis.Count(WaitReport::Verdict::Unreleased) == 3, "expected three unreleased waits"); + const auto& later = waitOn(analysis, &syntheticLabel); + check(later.writerCount == 1 && listed(later, WriterMatch::Effect::After, 1) && later.writers.front().order == WriterMatch::Order::LaterOnQueue && later.reason.find("are later on its queue and run after it") != std::string::npos, + "a store later on the waiting queue releases the wait: " + verdict(later)); + const auto& rejected = waitOn(analysis, &otherLabel); + check(rejected.writerCount == 1 && listed(rejected, WriterMatch::Effect::OtherQueue, 2) && !rejected.writers.front().accepted && rejected.writers.front().stored == StoredValue && !rejected.crossQueue, + "a store of another queue the compare rejects does not leave the wait unreleased: " + verdict(rejected)); + check(analysis.waits.back().verdict == WaitReport::Verdict::Unreleased && analysis.waits.back().wait.kind == AgcDriver::Pm4Analysis::MemoryWait::Kind::Rewind, "a REWIND without a tail is not reported as unreleased"); + + const auto [code, text] = report(directory); + const auto newest = text.find("UNRELEASED #4 "); + const auto middle = text.find("UNRELEASED #3 "); + const auto oldest = text.find("UNRELEASED #0 "); + check(code == 1 && newest != std::string::npos && middle != std::string::npos && oldest != std::string::npos && newest < middle && middle < oldest, "the report does not list unreleased waits newest first:\n" + text); +} + +std::vector rawWait(std::uint64_t address, std::uint32_t reference) { + return {0xc0053c00u, 0x13u, static_cast(address), static_cast(address >> 32u), reference, 0xffffffffu, 0}; +} + +std::vector rawWrite(std::uint64_t address, std::uint32_t value) { + return {0xc0033700u, 0x100500u, static_cast(address), static_cast(address >> 32u), value}; +} + +std::vector joined(std::vector first, const std::vector& second) { + first.insert(first.end(), second.begin(), second.end()); + return first; +} + +std::vector repeated(std::size_t count, const std::vector& packet) { + std::vector result; + for (std::size_t i = 0; i < count; ++i) result = joined(std::move(result), packet); + return result; +} + +Planned raw(std::uint32_t queue, std::uint64_t serial, std::vector commands, std::set held = {}) { + return {queue, serial, SubmissionKind::Submit, std::move(commands), std::move(held), {}}; +} + +const WaitReport& waitAtAddress(const AgcDriver::Pm4Analysis::Analysis& analysis, std::uint64_t address) { + for (const auto& report : analysis.waits) { + if (report.wait.kind == AgcDriver::Pm4Analysis::MemoryWait::Kind::Memory && report.wait.fields.address == address) return report; + } + throw std::runtime_error("no wait on the address in the analysis"); +} + +void testReviewerCaptures(const std::filesystem::path& directory) { + constexpr std::uint64_t A = 0x10000100; + constexpr std::uint64_t B = 0x10000200; + constexpr std::size_t StaleComputeWait = 5; + const auto stale = [&](std::set held) { return std::vector{raw(0, 1, rawWrite(B, 1)), raw(0x20, 2, joined(joined(rawWrite(A, 1), rawWait(B, 1)), rawWrite(A, 0)), std::move(held)), raw(0, 3, rawWait(A, 1))}; }; + const auto evict = [&](std::size_t newer) { return std::vector{raw(0, 1, joined(rawWait(B, 1), rawWait(A, 1))), raw(0x20, 2, joined(joined(rawWrite(A, 1), repeated(newer, rawWrite(A, 2))), rawWrite(B, 1)))}; }; + const std::vector>> captures{ + {"deadlock", {raw(0, 1, joined(rawWait(A, 1), rawWrite(B, 1))), raw(0x20, 2, joined(rawWait(B, 1), rawWrite(A, 1)))}}, + {"stale", stale({})}, + {"stale-held", stale({StaleComputeWait})}, + {"evict3", evict(3)}, + {"evict4", evict(4)}, + {"evict-own4", {raw(0, 1, joined(joined(rawWrite(A, 1), rawWait(B, 1)), rawWait(A, 1))), raw(0x20, 2, joined(repeated(4, rawWrite(A, 0)), rawWrite(B, 1)))}}, + }; + for (const auto& [name, submissions] : captures) { + const auto path = directory / name; + writeCapture(path, submissions); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(path); + check(analysis.waits.size() == 2, "the reviewer capture " + name + " does not hold its two waits"); + for (const auto& wait : analysis.waits) { + const bool held = wait.wait.heldAtSubmit && wait.verdict == WaitReport::Verdict::Released && wait.route == WaitReport::Route::HeldAtSubmit; + check(held || (wait.verdict == WaitReport::Verdict::Unknown && wait.crossQueue && wait.reason.starts_with("unknown: depends on cross-queue timing (stores on queue ")), + "the reviewer capture " + name + " reports the wait at #" + std::to_string(wait.at.ordinal) + " +" + std::to_string(wait.at.offset) + " as " + verdict(wait)); + } + check(analysis.Count(WaitReport::Verdict::Released) == (name == "stale-held" ? 1u : 0u), "the reviewer capture " + name + " releases a wait the capture does not prove released"); + check(report(path).first == 0, "the report of the reviewer capture " + name + " claims an unreleased wait"); + } +} + +void testProvableRoutes(const std::filesystem::path& directory) { + constexpr std::uint64_t A = 0x10000100; + constexpr std::uint64_t B = 0x10000200; + constexpr std::uint64_t C = 0x10000300; + constexpr std::uint32_t AlwaysCompare = 0x10; + const auto analyzed = [&](const std::string& name, const std::vector& submissions) { + writeCapture(directory / name, submissions); + return AgcDriver::Pm4Analysis::Analyze(directory / name); + }; + auto always = rawWait(C, 1); + always[1] = AlwaysCompare; + + const auto own = analyzed("own-submission", {raw(0, 1, joined(joined(rawWrite(A, 1), rawWait(A, 1)), always))}); + const auto& ownWait = waitAtAddress(own, A); + check(ownWait.verdict == WaitReport::Verdict::Released && ownWait.route == WaitReport::Route::CurrentValue && listed(ownWait, WriterMatch::Effect::CurrentValue, 0), "a store earlier in the waiting submission does not release the wait: " + verdict(ownWait)); + const auto& alwaysWait = waitAtAddress(own, C); + check(alwaysWait.verdict == WaitReport::Verdict::Released && alwaysWait.route == WaitReport::Route::Always, "a wait whose compare always passes is not released: " + verdict(alwaysWait)); + + const auto earlier = analyzed("earlier-submission", {raw(0, 1, rawWrite(A, 1)), raw(0x20, 2, rawWrite(B, 5)), raw(0, 3, rawWait(A, 1))}); + const auto& earlierWait = waitAtAddress(earlier, A); + check(earlierWait.verdict == WaitReport::Verdict::Released && earlierWait.route == WaitReport::Route::CurrentValue && listed(earlierWait, WriterMatch::Effect::CurrentValue, 0), + "a store in an earlier submission on the waiting queue, with no store of another queue to its bytes, does not release the wait: " + verdict(earlierWait)); + + for (const auto& [name, submissions, writer] : std::vector, std::string>>{ + {"other-queue-after", {raw(0, 1, rawWrite(A, 1)), raw(0x20, 2, rawWrite(A, 0)), raw(0, 3, rawWait(A, 1))}, "queue 0x20 at #1"}, + {"other-queue-before", {raw(0x20, 1, rawWrite(A, 0)), raw(0, 2, rawWrite(A, 1)), raw(0, 3, rawWait(A, 1))}, "queue 0x20 at #0"}, + }) { + const auto analysis = analyzed(name, submissions); + const auto& wait = waitAtAddress(analysis, A); + check(wait.verdict == WaitReport::Verdict::Unknown && wait.crossQueue && wait.reason.starts_with("unknown: depends on cross-queue timing (stores on " + writer + ")"), + name + ": a store of another queue to the awaited bytes, which can run between the store on the waiting queue and the wait whatever their submit order, leaves the wait other than unknown: " + verdict(wait)); + } + + const auto history = analyzed("history-other-queue", {raw(0, 1, joined(joined(rawWrite(A, 1), rawWait(B, 1)), rawWait(A, 1))), raw(0x20, 2, joined(repeated(AgcDriver::DriverDetail::LabelStoreHistory - 1, rawWrite(A, 0)), rawWrite(B, 1)))}); + const auto& historyWait = waitAtAddress(history, A); + check(historyWait.verdict == WaitReport::Verdict::Released && historyWait.route == WaitReport::Route::LabelHistory && listed(historyWait, WriterMatch::Effect::LabelHistory, 0), + "a label earlier in the waiting submission with fewer stores of all queues after it than the label history keeps does not release the wait: " + verdict(historyWait)); + check(waitAtAddress(history, B).verdict == WaitReport::Verdict::Unknown && waitAtAddress(history, B).crossQueue, "the wait on the store of another queue is not unknown: " + verdict(waitAtAddress(history, B))); + + Commands notKept; + sceAgcDcbDmaData(notKept.Buffer(), 0, DmaMemory, 0, at(&syntheticLabel), DmaFill, 0, One, sizeof(syntheticLabel), 0, 1, 0); + sceAgcDcbWriteData(notKept.Buffer(), GraphicsMemory, 0, at(&syntheticLabel), &Zero, 1, 0, 1); + sceAgcDcbWaitRegMem(notKept.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0); + const auto fill = analyzed("fill-overwritten", {planned(0, 1, notKept)}); + const auto& fillWait = waitOn(fill, &syntheticLabel); + check(fillWait.verdict == WaitReport::Verdict::Unknown && !fillWait.crossQueue && listed(fillWait, WriterMatch::Effect::Overwritten, 0) && fillWait.reason.find("is a RELEASE_MEM or WRITE_DATA label") != std::string::npos, + "a DMA_DATA fill the label history does not keep, overwritten before the wait, does not leave the wait unknown: " + verdict(fillWait)); +} + +struct Step { + std::uint32_t queue = 0; + std::vector commands; +}; + +using Outcome = std::tuple; +using Outcomes = std::map, Outcome>; + +Outcomes outcomes(const AgcDriver::Pm4Analysis::Analysis& analysis) { + Outcomes result; + std::map seen; + for (const auto& report : analysis.waits) { + const auto address = report.wait.fields.address; + result.emplace(std::pair{address, seen[address]++}, Outcome{report.verdict, report.route, report.crossQueue}); + } + return result; +} + +std::vector concatenatedOrder(const std::vector>& groups, bool reversed) { + std::vector order; + for (std::size_t i = 0; i < groups.size(); ++i) { + const auto group = reversed ? groups.size() - 1 - i : i; + order.insert(order.end(), groups[group].size(), group); + } + return order; +} + +std::vector roundRobinOrder(const std::vector>& groups, std::size_t total, bool reversed) { + std::vector order; + for (std::size_t step = 0; order.size() < total; ++step) { + for (std::size_t i = 0; i < groups.size(); ++i) { + const auto group = reversed ? groups.size() - 1 - i : i; + if (step < groups[group].size()) order.push_back(group); + } + } + return order; +} + +std::vector shuffledOrder(const std::vector>& groups, std::size_t total, std::uint64_t& random) { + std::vector left; + for (const auto& group : groups) left.push_back(group.size()); + std::vector order; + while (order.size() < total) { + random = random * 6364136223846793005ull + 1442695040888963407ull; + const auto group = static_cast(random >> 33u) % groups.size(); + if (left[group] == 0) continue; + --left[group]; + order.push_back(group); + } + return order; +} + +void testPermutedSubmissions(const std::filesystem::path& directory) { + constexpr std::uint64_t Base = 0x10001000; + constexpr std::size_t Submissions = 15; + constexpr std::size_t Shuffles = 4; + constexpr std::size_t Full = AgcDriver::DriverDetail::LabelStoreHistory; + const auto address = [](std::size_t group, std::size_t slot) { return Base + group * 0x100 + slot * 0x10; }; + const std::vector> groups{ + {{0x20, rawWrite(address(0, 1), 1)}, {0x21, joined(joined(rawWrite(address(0, 0), 1), rawWait(address(0, 1), 1)), rawWrite(address(0, 0), 0))}, {0x20, rawWait(address(0, 0), 1)}}, + {{0x22, joined(rawWait(address(1, 0), 1), rawWrite(address(1, 1), 1))}, {0x23, joined(rawWait(address(1, 1), 1), rawWrite(address(1, 0), 1))}}, + {{0x24, rawWrite(address(2, 0), 1)}, {0x25, joined(rawWrite(address(2, 1), 5), rawWait(address(2, 1), 6))}, {0x24, joined(rawWait(address(2, 0), 1), rawWait(address(2, 2), 1))}}, + {{0x26, joined(joined(rawWrite(address(3, 0), 1), rawWrite(address(3, 0), 0)), rawWait(address(3, 0), 1))}, {0x27, rawWrite(address(3, 1), 3)}, {0x26, rawWait(address(3, 1), 4)}}, + {{0x28, joined(joined(rawWrite(address(4, 0), 1), rawWait(address(4, 1), 1)), rawWait(address(4, 0), 1))}, {0x29, joined(repeated(Full - 1, rawWrite(address(4, 0), 0)), rawWrite(address(4, 1), 1))}, + {0x2a, joined(joined(rawWrite(address(4, 2), 1), rawWait(address(4, 3), 1)), rawWait(address(4, 2), 1))}, {0x2b, joined(repeated(Full, rawWrite(address(4, 2), 0)), rawWrite(address(4, 3), 1))}}, + }; + using Verdict = WaitReport::Verdict; + using Route = WaitReport::Route; + const Outcome unknown{Verdict::Unknown, Route::None, true}; + const Outcome unreleased{Verdict::Unreleased, Route::None, false}; + const Outcomes expected{ + {{address(0, 1), 0}, unknown}, {{address(0, 0), 0}, unknown}, + {{address(1, 0), 0}, unknown}, {{address(1, 1), 0}, unknown}, + {{address(2, 0), 0}, {Verdict::Released, Route::CurrentValue, false}}, {{address(2, 1), 0}, unreleased}, {{address(2, 2), 0}, unreleased}, + {{address(3, 0), 0}, {Verdict::Released, Route::LabelHistory, false}}, {{address(3, 1), 0}, unreleased}, + {{address(4, 0), 0}, {Verdict::Released, Route::LabelHistory, false}}, {{address(4, 1), 0}, unknown}, {{address(4, 2), 0}, unknown}, {{address(4, 3), 0}, unknown}, + }; + std::vector> orders{concatenatedOrder(groups, false), concatenatedOrder(groups, true), roundRobinOrder(groups, Submissions, false), roundRobinOrder(groups, Submissions, true)}; + std::uint64_t random = 0x2545f4914f6cdd1dull; + for (std::size_t shuffle = 0; shuffle < Shuffles; ++shuffle) orders.push_back(shuffledOrder(groups, Submissions, random)); + for (std::size_t permutation = 0; permutation < orders.size(); ++permutation) { + check(orders[permutation].size() == Submissions, "a submission order does not hold the 15 submissions"); + std::vector next(groups.size(), 0); + std::vector submissions; + std::string described; + for (const auto group : orders[permutation]) { + const auto& step = groups[group][next[group]++]; + submissions.push_back(raw(step.queue, submissions.size() + 1, step.commands)); + described += std::to_string(group); + } + const auto path = directory / ("order-" + std::to_string(permutation)); + writeCapture(path, submissions); + const auto result = outcomes(AgcDriver::Pm4Analysis::Analyze(path)); + check(result.size() == expected.size(), "the capture in submission order " + described + " does not hold the 13 waits"); + for (const auto& [key, outcome] : result) { + check(expected.contains(key) && expected.at(key) == outcome, "submission order " + described + " gives the wait on " + hex(reinterpret_cast(static_cast(key.first))) + " the verdict " + std::to_string(static_cast(std::get<0>(outcome))) + " route " + + std::to_string(static_cast(std::get<1>(outcome))) + " cross-queue " + std::to_string(std::get<2>(outcome))); + } + } +} + +template +void expectFailure(TAction action, const std::string& text) { + try { + action(); + } catch (const std::runtime_error& error) { + check(std::string(error.what()).find(text) != std::string::npos, "expected a failure naming '" + text + "', got: " + error.what()); + return; + } + throw std::runtime_error("expected a failure naming '" + text + "'"); +} + +void testFailedWrite(const std::filesystem::path& directory) { + const auto blocker = directory / "submission_000006.bin"; + std::filesystem::create_directory(blocker); + Commands store; + sceAgcDcbWriteData(store.Buffer(), GraphicsMemory, 0, at(&otherLabel), &One, 1, 0, 1); + expectFailure([&] { store.Submit(0); }, "cannot write"); + expectFailure([&] { store.Submit(0); }, "cannot write"); + std::filesystem::remove(blocker); + check(lines(directory / SubmissionCapture::IndexName).size() == 8, "the index lists a submission whose file was not written"); +} + +void testHeldAtSubmit(const std::filesystem::path& directory) { + Commands waitHeld; + const auto wait = waitAt(waitHeld, sceAgcDcbWaitRegMem(waitHeld.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0)); + writeCapture(directory, {planned(0, 1, waitHeld, {wait})}); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + const auto& held = waitOn(analysis, &syntheticLabel); + check(held.verdict == WaitReport::Verdict::Released && held.route == WaitReport::Route::HeldAtSubmit, "a wait whose value held at submit is not released: " + verdict(held)); + check(report(directory).first == 0, "the report fails on a wait released at submit"); +} + +void testRewindValidWithoutTail(const std::filesystem::path& directory) { + Commands rewound; + sceAgcAcbWriteData(rewound.Buffer(), ComputeMemory, 0, at(&beforeRewind), &BeforeRewindValue, 1, 0, 1); + auto* rewind = sceAgcAcbRewind(rewound.Buffer(), 0); + sceAgcAsyncRewindPatchSetRewindState(rewind, 1); + writeCapture(directory, {planned(0x20, 1, rewound)}); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + const auto rewindWait = std::find_if(analysis.waits.begin(), analysis.waits.end(), [](const auto& report) { return report.wait.kind == AgcDriver::Pm4Analysis::MemoryWait::Kind::Rewind; }); + check(rewindWait != analysis.waits.end(), "the capture of a REWIND valid at submit lists no REWIND"); + check(rewindWait->verdict == WaitReport::Verdict::Unknown && rewindWait->reason.find("reads it again") != std::string::npos, + "a REWIND valid at submit without a captured tail is not unknown: " + verdict(*rewindWait)); +} + +void replaceField(const std::filesystem::path& directory, std::size_t field, const std::string& value) { + auto index = lines(directory / SubmissionCapture::IndexName); + check(index.size() == 3, "the capture to corrupt does not hold one submission"); + auto values = fields(index[2]); + values.at(field) = value; + std::ofstream output(directory / SubmissionCapture::IndexName, std::ios::binary | std::ios::trunc); + output << index[0] << '\n' << index[1] << '\n'; + for (std::size_t i = 0; i < values.size(); ++i) output << (i == 0 ? "" : " ") << values[i]; + output << '\n'; + check(output.good(), "cannot rewrite the index to corrupt"); +} + +void dropHeldColumns(const std::filesystem::path& directory) { + const auto index = lines(directory / SubmissionCapture::IndexName); + std::ofstream output(directory / SubmissionCapture::IndexName, std::ios::binary | std::ios::trunc); + output << index.at(0) << '\n' << SubmissionCapture::ColumnsWithoutHeld << '\n'; + for (std::size_t line = 2; line < index.size(); ++line) { + const auto values = fields(index[line]); + check(values.size() == 10, "the index line to rewrite does not have 10 fields: " + index[line]); + for (std::size_t i = 0; i < 8; ++i) output << (i == 0 ? "" : " ") << values[i]; + output << '\n'; + } + check(output.good(), "cannot rewrite the index without its held and conditional columns"); +} + +void testIndexWithoutHeld(const std::filesystem::path& directory) { + Commands graphics, compute; + sceAgcDcbWaitRegMem(graphics.Buffer(), Bits32, Equal, 0, 0, &cpuLabel, CpuValue, 0xffffffffu, 0); + sceAgcDcbWriteData(graphics.Buffer(), GraphicsMemory, 0, at(&otherLabel), &One, 1, 0, 1); + sceAgcDcbCondExec(graphics.Buffer(), &conditionLabel, WriteDataWords); + sceAgcDcbWriteData(graphics.Buffer(), GraphicsMemory, 0, at(&guardedLabel), &One, 1, 0, 1); + sceAgcDcbWaitRegMem(graphics.Buffer(), Bits32, Equal, 0, 0, &guardedLabel, One, 0xffffffffu, 0); + sceAgcAcbWaitRegMem(compute.Buffer(), Bits32, Equal, 0, &otherLabel, One, 0xffffffffu, 0); + writeCapture(directory, {planned(0, 1, graphics), planned(0x20, 2, compute)}); + dropHeldColumns(directory); + const auto captured = SubmissionCapture::ReadIndex(directory); + check(captured.size() == 2 && !captured[0].heldKnown && !captured[0].conditionalKnown, "an index without the held and conditional columns is read as if it had them"); + const auto analysis = AgcDriver::Pm4Analysis::Analyze(directory); + check(analysis.waits.size() == 3, "expected three waits in the capture without held waits"); + const auto& cpu = waitOn(analysis, &cpuLabel); + check(cpu.verdict == WaitReport::Verdict::Unknown && !cpu.crossQueue && cpu.reason.starts_with("unknown: the index has no held column"), "a wait no store releases is not unknown when the index does not say whether it was held at submit: " + verdict(cpu)); + const auto& behind = waitOn(analysis, &otherLabel); + check(behind.verdict == WaitReport::Verdict::Unknown && behind.crossQueue, "a store of another queue decides a wait offline: " + verdict(behind)); + const auto& guarded = waitOn(analysis, &guardedLabel); + check(guarded.verdict == WaitReport::Verdict::Unknown && !guarded.crossQueue && guarded.wait.conditional && guarded.reason.find("is conditional") != std::string::npos, "a store after a COND_EXEC whose range the index does not hold is not conditional: " + verdict(guarded)); + const auto [code, text] = report(directory); + check(code == 0 && text.find("unknown 3 (cross-queue timing 1), unreleased 0\n") != std::string::npos && text.find("the index has no held and conditional columns") != std::string::npos && text.find("UNKNOWN #0 ") != std::string::npos, + "the report does not show the waits of unknown outcome:\n" + text); +} + +void testCorrupt(const std::filesystem::path& directory) { + constexpr std::size_t HeldField = 8; + constexpr std::size_t ConditionalField = 9; + Commands waitOnly; + sceAgcDcbWaitRegMem(waitOnly.Buffer(), Bits32, Equal, 0, 0, &syntheticLabel, One, 0xffffffffu, 0); + const auto expectThrow = [&](const std::vector& submissions, std::size_t field, const std::string& value, const std::string& what) { + std::filesystem::remove_all(directory); + writeCapture(directory, submissions); + if (!value.empty()) replaceField(directory, field, value); + bool threw = false; + try { + AgcDriver::Pm4Analysis::Analyze(directory); + } catch (const std::runtime_error&) { + threw = true; + } + check(threw, "the analysis accepts " + what); + }; + expectThrow({{0, 1, SubmissionKind::Submit, {0xc0044700u, 0x00000528u, static_cast(at(&syntheticLabel)), 0x40000000u, 1, 0}, {}, {}}}, 0, {}, "an EVENT_WRITE_EOP the driver rejects"); + expectThrow({planned(0, 1, waitOnly)}, HeldField, "0", "a wait held at submit that marks no wait packet"); + expectThrow({planned(0, 1, waitOnly)}, ConditionalField, "4:11", "a conditional range that starts at no COND_EXEC"); +} + +} + +int main() { + std::filesystem::path scratch; + try { + const char* captured = std::getenv("APS5_DUMP_SUBMITS"); + check(captured != nullptr && *captured != '\0', "set APS5_DUMP_SUBMITS to the capture directory of this test"); + const std::filesystem::path directory = captured; + std::filesystem::remove_all(directory); + scratch = std::filesystem::temp_directory_path() / ("anyps5-pm4-replay-" + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())); + testDriverCapture(directory); + testWriterPresent(directory, scratch / "writer-present"); + testOverwrittenAcrossSubmissions(scratch / "overwritten"); + testLabelHistory(scratch / "history-kept", AgcDriver::DriverDetail::LabelStoreHistory - 1); + testLabelHistory(scratch / "history-evicted", AgcDriver::DriverDetail::LabelStoreHistory); + testUnreleasable(scratch / "unreleasable"); + testReviewerCaptures(scratch / "reviewed"); + testProvableRoutes(scratch / "routes"); + testPermutedSubmissions(scratch / "permuted"); + testHeldAtSubmit(scratch / "held"); + testRewindValidWithoutTail(scratch / "rewind-valid"); + testIndexWithoutHeld(scratch / "without-held"); + testCorrupt(scratch / "corrupt"); + std::filesystem::remove_all(scratch); + testFailedWrite(directory); + expectFailure([] { LibcRunShutdown_nid_postfix(); }, "cannot write"); + std::puts("PM4 submission capture, decode and wait release tests passed"); + return 0; + } catch (const std::exception& error) { + std::fprintf(stderr, "%s\n", error.what()); + std::error_code ignored; + if (!scratch.empty()) std::filesystem::remove_all(scratch, ignored); + try { + LibcRunShutdown_nid_postfix(); + } catch (const std::exception& shutdown) { + std::fprintf(stderr, "shutdown: %s\n", shutdown.what()); + } + return 1; + } +} diff --git a/core/libs/prx/libSceAgcDriver/tests/Pm4ReplayTool.cmake b/core/libs/prx/libSceAgcDriver/tests/Pm4ReplayTool.cmake new file mode 100644 index 0000000000..bc30109c2f --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tests/Pm4ReplayTool.cmake @@ -0,0 +1,29 @@ +foreach(variable TOOL CAPTURE) + if(NOT DEFINED ${variable}) + message(FATAL_ERROR "${variable} is not set") + endif() +endforeach() + +execute_process(COMMAND "${TOOL}" "${CAPTURE}" RESULT_VARIABLE code OUTPUT_VARIABLE output ERROR_VARIABLE errors) +if(NOT code STREQUAL "1") + message(FATAL_ERROR "agc_pm4_replay exited with ${code} instead of 1 on a capture with an unreleased wait:\n${output}${errors}") +endif() +if(NOT output MATCHES "waits 8: released [4-6] \\([^)]*\\), unknown [1-3] \\(cross-queue timing [0-2]\\), unreleased 1\n") + message(FATAL_ERROR "agc_pm4_replay does not report seven released or unknown waits and one unreleased:\n${output}") +endif() +if(NOT output MATCHES "the tool does not model cross-queue timing") + message(FATAL_ERROR "agc_pm4_replay does not say that it does not model cross-queue timing:\n${output}") +endif() +if(NOT output MATCHES "unreleased waits, then unknown waits, newest first:\n\nUNRELEASED #3 queue 0x0 [^\n]*\n unreleased by the GPU; a CPU store could release it") + message(FATAL_ERROR "agc_pm4_replay does not list the unreleased wait of submission #3:\n${output}") +endif() + +execute_process(COMMAND "${TOOL}" "${CAPTURE}" --full RESULT_VARIABLE code OUTPUT_VARIABLE output ERROR_VARIABLE errors) +if(NOT code STREQUAL "1" OR NOT output MATCHES "submissions in execution order per queue:") + message(FATAL_ERROR "agc_pm4_replay --full exited with ${code} or printed no submissions:\n${output}${errors}") +endif() + +execute_process(COMMAND "${TOOL}" "${CAPTURE}/missing" RESULT_VARIABLE code OUTPUT_VARIABLE output ERROR_VARIABLE errors) +if(NOT code STREQUAL "2" OR NOT errors MATCHES "cannot open") + message(FATAL_ERROR "agc_pm4_replay exited with ${code} instead of 2 on a missing capture:\n${output}${errors}") +endif() diff --git a/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.cpp b/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.cpp new file mode 100644 index 0000000000..e16f862818 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.cpp @@ -0,0 +1,765 @@ +#include "Pm4Analysis.hpp" +#include "prx/libSceAgcDriver/Execution/include/Driver/Synchronization/LabelStores.hpp" +#include +#include +#include +#include +#include +#include +#include +#include + +namespace AgcDriver::Pm4Analysis { +namespace { + +constexpr std::uint64_t SmallStoreBytes = 4096; +constexpr std::size_t ShownValues = 4; +constexpr std::size_t MarkerCharacters = 48; +constexpr std::size_t ListedWaits = 32; +constexpr std::size_t LastSubmissionsPerQueue = 2; +constexpr std::size_t ShownPackets = 48; +constexpr std::size_t ContextBefore = 8; +constexpr std::size_t ContextAfter = 4; +constexpr const char* CompareNames[8] = {"always", "<", "<=", "==", "!=", ">=", ">", "reserved"}; + +std::string format(const char* text, ...) { + va_list arguments; + va_start(arguments, text); + va_list measured; + va_copy(measured, arguments); + const int length = std::vsnprintf(nullptr, 0, text, measured); + va_end(measured); + std::string result(length < 0 ? 0 : static_cast(length), '\0'); + const int written = length < 0 ? length : std::vsnprintf(result.data(), result.size() + 1, text, arguments); + va_end(arguments); + if (length < 0 || written != length) throw std::runtime_error("PM4 analysis: cannot format a line"); + return result; +} + +void require(bool condition, const std::string& reason) { + if (!condition) throw std::runtime_error("PM4 analysis: " + reason); +} + +std::uint64_t address(std::uint32_t low, std::uint32_t high) { + return low | (static_cast(high) << 32u); +} + +std::string hex(std::uint64_t value) { + return format("0x%llx", static_cast(value)); +} + +std::string values(std::span words) { + std::string text; + for (std::size_t i = 0; i < words.size() && i < ShownValues; ++i) text += format(i == 0 ? "0x%x" : " 0x%x", words[i]); + if (words.size() > ShownValues) text += format(" (+%zu)", words.size() - ShownValues); + return text; +} + +std::string marker(std::span payload) { + std::string text; + for (const auto byte : std::as_bytes(payload)) { + const auto character = std::to_integer(byte); + if (character == 0 || text.size() == MarkerCharacters) break; + text.push_back(character >= 0x20 && character < 0x7f ? static_cast(character) : '?'); + } + return "\"" + text + "\""; +} + +std::string describeNop(std::span p) { + const auto at = [&](std::size_t index) { return index < p.size() ? p[index] : 0u; }; + switch ((p[0] >> 2u) & 0x3fu) { + case 0: return format("%zu dwords", p.size() - 1); + case 0x06: return format("output %u index %d", at(1), static_cast(at(2))); + case 0x0b: return marker(p.subspan(1)); + case 0x17: return format("output %u index %d mode %u argument %s", at(1), static_cast(at(2)), at(3), hex(address(at(4), at(5))).c_str()); + case 0x1a: return format("operation %u", at(1)); + case 0x09: case 0x0c: return {}; + default: return values(p.subspan(1)); + } +} + +std::string describeDmaEnd(std::uint32_t select, std::uint32_t low, std::uint32_t high, bool destination) { + if (select == Pm4::DmaSelectGds) return format("GDS offset 0x%x", low); + if (destination && select == Pm4::DmaDestinationNowhere) return "nothing (prefetch)"; + return hex(address(low, high)); +} + +std::string describe(std::span p) { + const auto opcode = (p[0] >> 8u) & 0xffu; + const auto at = [&](std::size_t index) { return index < p.size() ? p[index] : 0u; }; + switch (opcode) { + case 0x10: return describeNop(p); + case 0x3c: case 0x93: { + const auto wait = Pm4::DecodeWait(p); + return format("[%s] (%zu bytes) %s 0x%llx mask 0x%llx poll %u", hex(wait.address).c_str(), wait.bytes, CompareNames[wait.function], static_cast(wait.reference), static_cast(wait.mask), wait.poll); + } + case 0x49: return format("event 0x%x data select %u interrupt %u address %s data %s", p[1] & 0x3fu, p[2] >> 29u, (p[2] >> 24u) & 7u, hex(address(p[3], p[4])).c_str(), hex(address(p[5], p[6])).c_str()); + case 0x37: return format("dst %u address %s%s: %s", (p[1] >> 8u) & 0xfu, hex(address(p[2], p[3])).c_str(), (p[1] & 0x10000u) != 0 ? " (one address)" : "", values(p.subspan(4)).c_str()); + case 0x40: { + const auto source = Pm4::CopyDataSource(p); + if (Pm4::CopyDataToRegister(p)) return format("source select %u source %s to register 0x%x", source, hex(address(p[2], p[3])).c_str(), p[4]); + return format("source select %u source %s to %s (%zu bytes)", source, hex(address(p[2], p[3])).c_str(), hex(address(p[4], p[5])).c_str(), Pm4::CopyDataBytes(p)); + } + case 0x50: { + const auto from = Pm4::DmaSource(p) == Pm4::DmaSourceData ? format("fill 0x%08x", p[2]) : "copy from " + describeDmaEnd(Pm4::DmaSource(p), p[2], p[3], false); + return format("%s to %s (%zu bytes)", from.c_str(), describeDmaEnd(Pm4::DmaDestination(p), p[4], p[5], true).c_str(), Pm4::DmaBytes(p)); + } + case 0x46: { + auto text = format("event 0x%x index %u", p[1] & 0x3fu, (p[1] >> 8u) & 7u); + if (Pm4::SampleDump(p)) text += " samples to " + hex(Pm4::SampleDumpAddress(p)); + return text; + } + case 0x45: { + const auto write = Pm4::DecodeConditionalWrite(p); + return format("[%s] %s 0x%x mask 0x%x: store 0x%x to %s", hex(write.compared).c_str(), CompareNames[p[1] & 7u], p[4], p[5], write.value, hex(write.destination).c_str()); + } + case 0x83: { + const auto dump = Pm4::DecodeConstantRamDump(p); + return format("offset 0x%x %u dwords to %s", dump.offset, dump.words, hex(dump.destination).c_str()); + } + case 0x59: return (p[1] & Pm4::RewindValid) != 0 ? "valid" : "not valid"; + case 0x68: case 0x69: case 0x76: case 0x78: case 0x79: case 0x7a: + if (Pm4::IsTagMarker(p)) return format("tag 0x%08x", p[2]); + return format("0x%x = %s", p[1], values(p.subspan(2)).c_str()); + case 0x11: return format("index %u base %s", at(1), hex(address(at(2), at(3))).c_str()); + case 0x12: case 0x13: case 0x2a: case 0x2f: case 0x42: return format("%u", at(1)); + case 0x15: return format("groups %ux%ux%u initiator 0x%x", at(1), at(2), at(3), at(4)); + case 0x16: return p.size() == 4 ? format("arguments %s initiator 0x%x", hex(address(at(1), at(2))).c_str(), at(3)) : format("offset 0x%x initiator 0x%x", at(1), at(2)); + case 0x20: return format("control 0x%x address %s", at(1), hex(address(at(2), at(3))).c_str()); + case 0x22: return format("[%s] guards %zu dwords", hex(address(at(1), at(2))).c_str(), Pm4::ConditionalWords(p)); + case 0x24: case 0x25: return format("offset 0x%x locations 0x%x 0x%x initiator 0x%x", at(1), at(2), at(3), at(4)); + case 0x2c: case 0x38: return format("offset 0x%x control 0x%x count %u count address %s stride %u initiator 0x%x", at(1), at(4), at(5), hex(address(at(6), at(7))).c_str(), at(8), at(9)); + case 0x26: return format("address %s", hex(address(at(1), at(2))).c_str()); + case 0x27: return format("%u indices at %s max %u initiator 0x%x", at(4), hex(address(at(2), at(3))).c_str(), at(1), at(5)); + case 0x2d: return format("%u vertices initiator 0x%x", at(1), at(2)); + case 0x35: return format("%u indices from %u max %u initiator 0x%x", at(3), at(2), at(1), at(4)); + case 0x3f: return format("predicated, address %s %u dwords", hex(address(at(1), at(2) & 0xffffu)).c_str(), at(3) & 0xfffffu); + case 0x58: { + const auto base = ((static_cast(at(5)) << 32u) | at(4)) << 8u; + const auto bytes = ((static_cast(at(3)) << 32u) | at(2)) << 8u; + return format("control 0x%x base %s bytes %s", at(1), hex(base).c_str(), hex(bytes).c_str()); + } + case 0x63: case 0x64: case 0x9f: + return at(3) == 0x80000000u ? format("%u pairs at %s", at(4), hex(address(at(1), at(2))).c_str()) : format("0x%x x%u from %s", at(3), at(4), hex(address(at(1), at(2))).c_str()); + case 0x81: return format("offset 0x%x %zu dwords", at(1), p.size() > 2 ? p.size() - 2 : 0); + default: return values(p.subspan(1)); + } +} + +bool conditionalStart(std::uint32_t header) { + const auto opcode = (header >> 8u) & 0xffu; + return opcode == 0x22 || (opcode == 0x3f && Pm4::Predicated(header)); +} + +struct Located { + std::size_t submission = 0; + std::size_t offset = 0; + std::uint32_t header = 0; +}; + +struct Store { + Located at; + MemoryStore store; +}; + +struct Waiter { + Located at; + MemoryWait wait; +}; + +using Key = std::tuple; + +struct QueueStores { + std::vector byExecution; + std::vector byEntry; +}; + +struct RangeStores { + std::map queues; + std::size_t count = 0; +}; + +struct Accepting { + std::map> prefix; + std::size_t total = 0; +}; + +using CompareKey = std::tuple; +using Listed = std::vector>; + +class Matcher { +public: + Matcher(const std::vector& submissions, const std::vector& stores) : submissions(submissions), stores(stores) { + for (std::size_t index = 0; index < stores.size(); ++index) (stores[index].store.store.bytes <= SmallStoreBytes ? small : large).push_back(index); + std::sort(small.begin(), small.end(), [&](std::size_t left, std::size_t right) { return stores[left].store.store.address < stores[right].store.store.address; }); + } + + void Match(WaitReport& report, const Waiter& waiter) { + const auto& fields = report.wait.fields; + const auto& range = rangeFor(fields.address, fields.bytes); + const auto& accepting = acceptingFor(range, report.wait); + report.writerCount = range.count; + const auto queue = submissions[waiter.at.submission].queue; + const auto found = range.queues.find(queue); + const bool stored = found != range.queues.end(); + const auto& own = stored ? found->second.byExecution : empty; + const auto before = lowerExecution(own, execution(waiter.at)); + const auto others = range.count - own.size(); + const auto ownAccepting = stored ? accepting.prefix.at(queue)[before] : 0; + const auto otherAccepting = accepting.total - (stored ? accepting.prefix.at(queue).back() : 0); + Listed listed; + + std::optional lastValue; + bool lastAccepts = false; + if (before != 0) { + lastValue = settledValue(own[before - 1], fields); + lastAccepts = lastValue.has_value() && Pm4::WaitComparesValue(report.wait.packet, *lastValue); + if (lastAccepts && others == 0) { + setReleased(report, WaitReport::Route::CurrentValue, + format("released: %s, the last store before it on its queue, leaves 0x%llx at %s, which the compare accepts, and no other queue stores to the awaited bytes", where(own[before - 1]).c_str(), static_cast(*lastValue), hex(fields.address).c_str())); + listed.emplace_back(own[before - 1], WriterMatch::Effect::CurrentValue); + report.writers = writers(listed, waiter); + return; + } + } + + std::optional kept; + std::size_t newer = 0; + for (auto position = before; position != 0;) { + const auto index = own[--position]; + if (stores[index].at.submission != waiter.at.submission) break; + if (!keptAccepting(index, report.wait)) continue; + kept = index; + newer = before - 1 - position + others; + break; + } + if (kept.has_value() && newer < DriverDetail::LabelStoreHistory) { + const auto value = storedBytes(stores[*kept].store.store, fields.address, fields.bytes); + setReleased(report, WaitReport::Route::LabelHistory, + format("released: %s stores 0x%llx after the waiting submission entered the driver, and at most %zu other stores to the awaited bytes can run between it and the wait, so the driver's label history, which keeps the last %zu stores per dword over all queues, still holds it when the wait runs", + where(*kept).c_str(), static_cast(*value), newer, DriverDetail::LabelStoreHistory)); + listed.emplace_back(*kept, WriterMatch::Effect::LabelHistory); + report.writers = writers(listed, waiter); + return; + } + + if (kept.has_value()) listed.emplace_back(*kept, WriterMatch::Effect::Evicted); + if (before != 0 && own[before - 1] != kept) listed.emplace_back(own[before - 1], lastAccepts ? WriterMatch::Effect::Overwritten : lastValue.has_value() ? WriterMatch::Effect::Rejected : WriterMatch::Effect::Undecided); + if (stored) { + const auto& prefix = accepting.prefix.at(queue); + const auto end = before == 0 ? 0 : before - 1; + if (const auto earlier = lastAcceptingBefore(prefix, end); earlier.has_value() && own[*earlier] != kept) { + listed.emplace_back(own[*earlier], settledValue(own[*earlier], fields).has_value() ? WriterMatch::Effect::Overwritten : WriterMatch::Effect::Undecided); + } + if (const auto later = firstAcceptingFrom(prefix, before); later.has_value()) listed.emplace_back(own[*later], WriterMatch::Effect::After); + } + const auto queues = noteOthers(range, queue, submissions[waiter.at.submission].ordinal, listed); + + const auto awaited = format("%s+%zu", hex(fields.address).c_str(), fields.bytes); + if (ownAccepting + otherAccepting == 0) { + report.verdict = WaitReport::Verdict::Unreleased; + report.reason = unreleasedReason(range, awaited, own.size() - before); + if (!submissions[waiter.at.submission].heldKnown) { + report.verdict = WaitReport::Verdict::Unknown; + report.reason = "unknown: the index has no held column, so its value may have passed the compare at submit; otherwise " + report.reason; + } + } else if (others != 0) { + report.verdict = WaitReport::Verdict::Unknown; + report.crossQueue = true; + report.reason = format("unknown: depends on cross-queue timing (stores on %s): %zu of the %zu stores to %s on other queues write a value the compare may accept, and the capture does not fix when a store of another queue runs relative to the wait", + queues.c_str(), otherAccepting, others, awaited.c_str()); + if (lastAccepts) report.reason += format("; %s, the last store before it on its queue, leaves a value the compare accepts, but a store of another queue can replace it", where(own[before - 1]).c_str()); + if (kept.has_value()) report.reason += format("; %s, a label earlier in the waiting submission that the compare accepts, can be pushed out of the driver's label history by %zu stores to the awaited bytes, as many as it keeps per dword", where(*kept).c_str(), newer); + } else { + report.verdict = WaitReport::Verdict::Unknown; + report.reason = "unknown: " + sameQueueReason(own[before - 1], fields, kept, newer); + } + report.writers = writers(listed, waiter); + } + +private: + static constexpr std::size_t ListedPerQueue = 4; + + const std::vector& submissions; + const std::vector& stores; + const std::vector empty; + std::vector small; + std::vector large; + std::map, RangeStores> ranges; + std::map accepting; + + Key execution(const Located& at) const { + const auto& submission = submissions[at.submission]; + return {submission.serial, submission.ordinal, at.offset}; + } + + Key entry(const Located& at) const { + return {submissions[at.submission].ordinal, at.offset, 0}; + } + + std::size_t lowerExecution(const std::vector& list, const Key& key) const { + return static_cast(std::lower_bound(list.begin(), list.end(), key, [&](std::size_t index, const Key& value) { return execution(stores[index].at) < value; }) - list.begin()); + } + + static bool covers(const Store& store, std::uint64_t at, std::size_t bytes) { + return at >= store.store.store.address && at - store.store.store.address <= store.store.store.bytes && bytes <= store.store.store.bytes - (at - store.store.store.address); + } + + static std::optional storedBytes(const Pm4::PacketStore& store, std::uint64_t at, std::size_t bytes) { + if (store.value == Pm4::PacketStore::Value::RunTime || at < store.address || at - store.address > store.bytes || bytes > store.bytes - (at - store.address)) return std::nullopt; + std::uint64_t value = 0; + for (std::size_t i = 0; i < bytes; ++i) { + const auto index = static_cast(at - store.address) + i; + const auto byte = store.value == Pm4::PacketStore::Value::Bytes ? store.data.at(index) : store.data.at(index % store.data.size()); + value |= std::to_integer(byte) << (8u * i); + } + return value; + } + + std::optional settledValue(std::size_t index, const Pm4::WaitFields& fields) const { + if (stores[index].store.conditional) return std::nullopt; + return storedBytes(stores[index].store.store, fields.address, fields.bytes); + } + + bool mayAccept(std::size_t index, const MemoryWait& wait) const { + const auto value = storedBytes(stores[index].store.store, wait.fields.address, wait.fields.bytes); + return !value.has_value() || Pm4::WaitComparesValue(wait.packet, *value); + } + + bool keptAccepting(std::size_t index, const MemoryWait& wait) const { + const auto& store = stores[index]; + const auto& fields = wait.fields; + if (!store.store.label || store.store.conditional || !DriverDetail::KeepsLabelStore(store.store.store.address, store.store.store.bytes)) return false; + if (!covers(store, fields.address, fields.bytes) || (fields.address - store.store.store.address) % 4 != 0) return false; + const auto value = storedBytes(store.store.store, fields.address, fields.bytes); + return value.has_value() && Pm4::WaitComparesValue(wait.packet, *value); + } + + static std::optional lastAcceptingBefore(const std::vector& prefix, std::size_t end) { + if (prefix[end] == 0) return std::nullopt; + return static_cast(std::lower_bound(prefix.begin(), prefix.begin() + static_cast(end) + 1, prefix[end]) - prefix.begin()) - 1; + } + + static std::optional firstAcceptingFrom(const std::vector& prefix, std::size_t from) { + if (prefix.back() == prefix[from]) return std::nullopt; + return static_cast(std::upper_bound(prefix.begin() + static_cast(from), prefix.end(), prefix[from]) - prefix.begin()) - 1; + } + + std::string where(std::size_t index) const { + const auto& at = stores[index].at; + return format("#%llu +0x%04zx %s", static_cast(submissions[at.submission].ordinal), at.offset, Pm4::Name(at.header).c_str()); + } + + static void setReleased(WaitReport& report, WaitReport::Route route, std::string reason) { + report.verdict = WaitReport::Verdict::Released; + report.route = route; + report.reason = std::move(reason); + } + + std::string noteOthers(const RangeStores& range, std::uint32_t queue, std::uint64_t entered, Listed& listed) const { + std::string text; + for (const auto& [storeQueue, lists] : range.queues) { + if (storeQueue == queue) continue; + const auto& list = lists.byEntry; + const auto ordinal = [&](std::size_t position) { return submissions[stores[list[position]].at.submission].ordinal; }; + auto high = static_cast(std::lower_bound(list.begin(), list.end(), entered, [&](std::size_t index, std::uint64_t value) { return submissions[stores[index].at.submission].ordinal < value; }) - list.begin()); + auto low = high; + std::vector nearest; + while (nearest.size() < ListedPerQueue && (low != 0 || high != list.size())) { + const bool takeLow = high == list.size() || (low != 0 && entered - ordinal(low - 1) <= ordinal(high) - entered); + nearest.push_back(takeLow ? list[--low] : list[high++]); + } + std::sort(nearest.begin(), nearest.end(), [&](std::size_t left, std::size_t right) { return entry(stores[left].at) < entry(stores[right].at); }); + std::vector ordinals; + for (const auto index : nearest) { + listed.emplace_back(index, WriterMatch::Effect::OtherQueue); + const auto at = submissions[stores[index].at.submission].ordinal; + if (ordinals.empty() || ordinals.back() != at) ordinals.push_back(at); + } + text += format("%squeue 0x%x at", text.empty() ? "" : "; ", storeQueue); + for (std::size_t i = 0; i < ordinals.size(); ++i) text += format("%s #%llu", i == 0 ? "" : ",", static_cast(ordinals[i])); + if (list.size() > nearest.size()) text += format(" (+%zu more)", list.size() - nearest.size()); + } + return text; + } + + std::string sameQueueReason(std::size_t last, const Pm4::WaitFields& fields, const std::optional& kept, std::size_t newer) const { + const auto& store = stores[last]; + const auto lastStore = where(last) + ", the last store before it on its queue,"; + if (store.store.conditional) return lastStore + " is conditional (COND_WRITE, inside a COND_EXEC range or a predicated packet), so the value the wait reads depends on run time"; + if (store.store.store.value == Pm4::PacketStore::Value::RunTime) return lastStore + " stores a value known only when the GPU runs it"; + const auto value = storedBytes(store.store.store, fields.address, fields.bytes); + if (!value.has_value()) return lastStore + " writes only part of the awaited bytes"; + auto text = format("%s leaves 0x%llx, which the compare rejects, and the capture does not prove that the driver's label history still holds an earlier store the compare accepts when the wait runs", lastStore.c_str(), static_cast(*value)); + if (kept.has_value()) return text + format(": %s, a label earlier in the waiting submission that the compare accepts, can be pushed out of it by %zu newer stores to the awaited bytes", where(*kept).c_str(), newer); + return text + ": no store the compare accepts is a RELEASE_MEM or WRITE_DATA label the history keeps earlier in the waiting submission, and whether an earlier one ran after the submission entered the driver depends on when it ran"; + } + + static std::string unreleasedReason(const RangeStores& range, const std::string& awaited, std::size_t later) { + const std::string prefix = "unreleased by the GPU; a CPU store could release it (CPU and shader stores are not captured): its value at submit fails the compare and "; + if (range.count == 0) return prefix + "no packet in the capture writes " + awaited; + if (range.count == later) return prefix + format("the %zu stores to %s in the capture are later on its queue and run after it", range.count, awaited.c_str()); + auto text = prefix + format("none of the %zu stores to %s in the capture that can run before it writes a value the compare accepts", range.count - later, awaited.c_str()); + if (later != 0) text += format(" (%zu more are later on its queue and run after it)", later); + return text; + } + + std::vector writers(Listed& listed, const Waiter& waiter) const { + std::stable_sort(listed.begin(), listed.end(), [](const auto& left, const auto& right) { return left.second < right.second; }); + std::vector result; + std::set seen; + const auto& waiting = submissions[waiter.at.submission]; + for (const auto& [index, effect] : listed) { + if (result.size() == Analysis::ListedWriters) break; + if (!seen.insert(index).second) continue; + const auto& at = stores[index].at; + const auto& submission = submissions[at.submission]; + WriterMatch match; + match.at = {submission.ordinal, submission.queue, submission.serial, at.offset, Pm4::Name(at.header)}; + if (submission.queue == waiting.queue) match.order = execution(at) < execution(waiter.at) ? WriterMatch::Order::EarlierOnQueue : WriterMatch::Order::LaterOnQueue; + match.effect = effect; + match.stored = storedBytes(stores[index].store.store, waiter.wait.fields.address, waiter.wait.fields.bytes); + match.accepted = mayAccept(index, waiter.wait); + result.push_back(std::move(match)); + } + return result; + } + + const Accepting& acceptingFor(const RangeStores& range, const MemoryWait& wait) { + const auto& fields = wait.fields; + const auto [found, inserted] = accepting.try_emplace({fields.address, fields.bytes, fields.function, fields.reference, fields.mask}); + if (!inserted) return found->second; + auto& result = found->second; + for (const auto& [queue, lists] : range.queues) { + auto& prefix = result.prefix[queue]; + prefix.push_back(0); + for (const auto index : lists.byExecution) prefix.push_back(prefix.back() + (mayAccept(index, wait) ? 1 : 0)); + result.total += prefix.back(); + } + return result; + } + + const RangeStores& rangeFor(std::uint64_t at, std::size_t bytes) { + const auto [found, inserted] = ranges.try_emplace({at, bytes}); + if (!inserted) return found->second; + auto& range = found->second; + const auto overlaps = [&](std::size_t index) { + const auto& store = stores[index].store.store; + return store.address < at + bytes && at < store.address + store.bytes; + }; + std::vector matches; + const auto first = std::lower_bound(small.begin(), small.end(), at >= SmallStoreBytes ? at - SmallStoreBytes + 1 : 0, [&](std::size_t index, std::uint64_t value) { return stores[index].store.store.address < value; }); + for (auto it = first; it != small.end() && stores[*it].store.store.address < at + bytes; ++it) { + if (overlaps(*it)) matches.push_back(*it); + } + for (const auto index : large) { + if (overlaps(index)) matches.push_back(index); + } + range.count = matches.size(); + for (const auto index : matches) { + auto& lists = range.queues[submissions[stores[index].at.submission].queue]; + lists.byExecution.push_back(index); + lists.byEntry.push_back(index); + } + for (auto& [queue, lists] : range.queues) { + std::sort(lists.byExecution.begin(), lists.byExecution.end(), [&](std::size_t left, std::size_t right) { return execution(stores[left].at) < execution(stores[right].at); }); + std::sort(lists.byEntry.begin(), lists.byEntry.end(), [&](std::size_t left, std::size_t right) { return entry(stores[left].at) < entry(stores[right].at); }); + } + return range; + } +}; + +void matchRewind(WaitReport& report, const std::map, std::vector>& segments) { + const auto& ordinals = segments.at({report.at.queue, report.at.serial}); + const auto self = std::find(ordinals.begin(), ordinals.end(), report.at.ordinal); + if (self != ordinals.end() && self + 1 != ordinals.end()) { + report.verdict = WaitReport::Verdict::Released; + report.route = WaitReport::Route::RewindTail; + report.reason = format("released: its tail entered the driver as #%llu", static_cast(*(self + 1))); + return; + } + if (report.wait.released) { + report.verdict = WaitReport::Verdict::Unknown; + report.route = WaitReport::Route::None; + report.reason = "unknown: the valid bit was set when the submission entered the driver, but the driver reads it again when the queue reaches the REWIND and the capture has no tail"; + return; + } + report.verdict = WaitReport::Verdict::Unreleased; + report.reason = "unreleased: no rewind tail in the capture, so the title had not set the REWIND valid bit when the capture ended"; +} + +const char* verdictName(WaitReport::Verdict verdict) { + switch (verdict) { + case WaitReport::Verdict::Released: return "released"; + case WaitReport::Verdict::Unknown: return "UNKNOWN"; + case WaitReport::Verdict::Unreleased: return "UNRELEASED"; + } + throw std::runtime_error("PM4 analysis: unknown verdict"); +} + +std::string describeWriter(const WriterMatch& match) { + const char* order = match.order == WriterMatch::Order::EarlierOnQueue ? "earlier on this queue" : match.order == WriterMatch::Order::LaterOnQueue ? "later on this queue" : "on another queue"; + std::string text = format("#%llu queue 0x%x serial %llu +0x%04zx %s, %s", static_cast(match.at.ordinal), match.at.queue, static_cast(match.at.serial), match.at.offset, match.at.name.c_str(), order); + text += match.stored.has_value() ? format(", stores 0x%llx", static_cast(*match.stored)) : ", stores part of the awaited bytes or a value known only at run time"; + switch (match.effect) { + case WriterMatch::Effect::CurrentValue: return text + ": the value the wait reads, releases it"; + case WriterMatch::Effect::LabelHistory: return text + ": still in the driver's label history when the wait runs, releases it"; + case WriterMatch::Effect::OtherQueue: return text + (match.accepted ? ": the compare may accept it" : ": the compare rejects it") + ", and the capture does not fix when it runs relative to the wait"; + case WriterMatch::Effect::Undecided: return text + ": conditional, or its value is known only at run time"; + case WriterMatch::Effect::Overwritten: return text + ": the compare accepts it, but another store to the awaited bytes can replace it before the wait"; + case WriterMatch::Effect::Evicted: return text + ": the compare accepts it, but newer stores to the awaited bytes can push it out of the driver's label history"; + case WriterMatch::Effect::Rejected: return text + ": the compare rejects it"; + case WriterMatch::Effect::After: return text + (match.accepted ? ": the compare may accept it" : "") + ", but it runs after the wait"; + } + throw std::runtime_error("PM4 analysis: unknown writer effect"); +} + +std::string waitLine(const WaitReport& report) { + std::string text = format("#%llu queue 0x%x serial %llu +0x%04zx %s", static_cast(report.at.ordinal), report.at.queue, static_cast(report.at.serial), report.at.offset, report.at.name.c_str()); + if (report.wait.kind == MemoryWait::Kind::Memory) { + const auto& fields = report.wait.fields; + text += format(" [%s] (%zu bytes) %s 0x%llx mask 0x%llx", hex(fields.address).c_str(), fields.bytes, CompareNames[fields.function], static_cast(fields.reference), static_cast(fields.mask)); + } + if (report.wait.conditional) text += " (conditional)"; + return text; +} + +std::string submissionLine(const CapturedSubmission& submission) { + const char* kind = submission.kind == SubmissionKind::Rewind ? " (rewind tail)" : submission.kind == SubmissionKind::Suspend ? " (suspend point)" : ""; + return format("#%llu queue 0x%x %s serial %llu at %llu.%03llu ms: %zu dwords%s", static_cast(submission.ordinal), submission.queue, SubmissionCapture::QueueType(submission.queue).data(), static_cast(submission.serial), + static_cast(submission.timeUs / 1000), static_cast(submission.timeUs % 1000), submission.dwords, kind); +} + +void printPackets(std::FILE* output, const std::vector& packets, std::size_t from, std::size_t to, std::size_t marked) { + for (std::size_t i = from; i < to && i < packets.size(); ++i) { + const auto& packet = packets[i]; + std::fprintf(output, " %s+0x%04zx %-26s %s%s\n", packet.offset == marked ? "> " : " ", packet.offset, packet.name.c_str(), packet.operands.c_str(), packet.conditional && (!packet.stores.empty() || packet.wait.has_value()) ? " (conditional)" : ""); + } +} + +std::vector decodePackets(const CapturedSubmission& submission, bool text) { + const std::span commands = submission.commands; + require(commands.size() == submission.dwords, "the commands do not hold the dword count of the index"); + std::vector packets; + std::vector boundaries; + std::vector guarded; + auto range = submission.conditionalRanges.begin(); + auto held = submission.heldAtSubmit.begin(); + for (std::size_t cursor = 0; cursor < commands.size();) { + std::erase_if(guarded, [cursor](std::size_t end) { return end <= cursor; }); + boundaries.push_back(cursor); + const auto header = commands[cursor]; + DecodedPacket packet; + packet.offset = cursor; + packet.header = header; + if (text) packet.name = Pm4::Name(header); + if (Pm4::FillerPacket(header)) { + std::size_t run = 1; + while (cursor + run < commands.size() && Pm4::FillerPacket(commands[cursor + run])) ++run; + for (std::size_t inner = 1; inner < run; ++inner) boundaries.push_back(cursor + inner); + packet.words = run; + if (text) packet.operands = format("x%zu", run); + packets.push_back(std::move(packet)); + cursor += run; + continue; + } + if ((header >> 30u) != 3u) require(false, format("header 0x%08x at DWORD %zu is not a type-3 packet", header, cursor)); + const auto words = Pm4::PacketWords(header); + if (words > commands.size() - cursor) require(false, format("%s at DWORD %zu needs %zu dwords, %zu remain", Pm4::Name(header).c_str(), cursor, words, commands.size() - cursor)); + const auto span = commands.subspan(cursor, words); + try { + Pm4::Validate(span, submission.queue); + } catch (const std::exception& error) { + throw std::runtime_error(format("PM4 analysis: %s at DWORD %zu is a packet the driver rejects, so the capture is corrupt: ", Pm4::Name(header).c_str(), cursor) + error.what()); + } + packet.words = words; + packet.conditional = !guarded.empty() || Pm4::Predicated(header); + if (conditionalStart(header) && !submission.conditionalKnown) { + guarded.push_back(commands.size()); + } else if (conditionalStart(header)) { + if (range == submission.conditionalRanges.end() || range->first != cursor) require(false, format("%s at DWORD %zu has no conditional range in the index", Pm4::Name(header).c_str(), cursor)); + if (range->second < cursor + words) require(false, format("the conditional range of %s at DWORD %zu ends inside it", Pm4::Name(header).c_str(), cursor)); + guarded.push_back(range->second); + ++range; + } + if (range != submission.conditionalRanges.end() && range->first <= cursor) require(false, format("a conditional range of the index starts at DWORD %zu, which is no COND_EXEC or predicated INDIRECT_BUFFER", range->first)); + const auto opcode = (header >> 8u) & 0xffu; + if (text) packet.operands = describe(span); + for (auto& store : Pm4::DecodeStores(span)) { + const bool conditional = packet.conditional || store.conditional; + packet.stores.push_back({std::move(store), conditional, opcode == 0x49 || opcode == 0x37}); + } + const bool heldHere = held != submission.heldAtSubmit.end() && *held == cursor; + if (opcode == 0x3c || opcode == 0x93) { + MemoryWait wait; + wait.fields = Pm4::DecodeWait(span); + wait.heldAtSubmit = heldHere; + wait.conditional = packet.conditional; + wait.packet.assign(span.begin(), span.end()); + packet.wait = std::move(wait); + } else if (opcode == 0x59) { + MemoryWait wait; + wait.kind = MemoryWait::Kind::Rewind; + wait.released = (span[1] & Pm4::RewindValid) != 0; + wait.packet.assign(span.begin(), span.end()); + packet.wait = std::move(wait); + } + if (heldHere && (!packet.wait.has_value() || packet.wait->kind != MemoryWait::Kind::Memory)) require(false, format("the index marks DWORD %zu as a wait held at submit, but %s is no WAIT_REG_MEM", cursor, Pm4::Name(header).c_str())); + if (heldHere) ++held; + if (held != submission.heldAtSubmit.end() && *held <= cursor) require(false, format("the index marks DWORD %zu as a wait held at submit, which is inside a packet", *held)); + packets.push_back(std::move(packet)); + cursor += words; + } + require(range == submission.conditionalRanges.end(), "a conditional range of the index starts past the commands"); + require(held == submission.heldAtSubmit.end(), "a wait held at submit lies past the commands"); + for (const auto& [start, end] : submission.conditionalRanges) { + if (end != commands.size() && !std::binary_search(boundaries.begin(), boundaries.end(), end)) require(false, format("the conditional range at DWORD %zu ends inside a packet", start)); + } + return packets; +} + +std::vector decodeSubmission(const std::filesystem::path& directory, CapturedSubmission submission, bool text) { + try { + submission.commands = SubmissionCapture::ReadCommands(directory, submission); + return decodePackets(submission, text); + } catch (const std::exception& error) { + throw std::runtime_error(format("submission #%llu: ", static_cast(submission.ordinal)) + error.what()); + } +} + +} + +std::vector Decode(const CapturedSubmission& submission) { + return decodePackets(submission, true); +} + +std::size_t Analysis::Count(WaitReport::Verdict verdict) const { + return static_cast(std::count_if(waits.begin(), waits.end(), [verdict](const WaitReport& report) { return report.verdict == verdict; })); +} + +std::size_t Analysis::CrossQueue() const { + return static_cast(std::count_if(waits.begin(), waits.end(), [](const WaitReport& report) { return report.crossQueue; })); +} + +Analysis Analyze(const std::filesystem::path& directory) { + Analysis analysis; + analysis.submissions = SubmissionCapture::ReadIndex(directory); + const auto& submissions = analysis.submissions; + std::map, std::vector> segments; + for (const auto& submission : submissions) { + if (submission.kind != SubmissionKind::Suspend) segments[{submission.queue, submission.serial}].push_back(submission.ordinal); + } + std::vector stores; + std::vector waiters; + for (std::size_t index = 0; index < submissions.size(); ++index) { + if (submissions[index].dwords == 0) continue; + auto packets = decodeSubmission(directory, submissions[index], false); + analysis.packets += packets.size(); + for (auto& packet : packets) { + for (auto& store : packet.stores) stores.push_back({{index, packet.offset, packet.header}, std::move(store)}); + if (packet.wait.has_value()) waiters.push_back({{index, packet.offset, packet.header}, std::move(*packet.wait)}); + } + } + analysis.stores = stores.size(); + Matcher matcher(submissions, stores); + for (const auto& waiter : waiters) { + WaitReport report; + const auto& submission = submissions[waiter.at.submission]; + report.at = {submission.ordinal, submission.queue, submission.serial, waiter.at.offset, Pm4::Name(waiter.at.header)}; + report.wait = waiter.wait; + if (report.wait.kind == MemoryWait::Kind::Rewind) { + matchRewind(report, segments); + } else if (report.wait.fields.function == 0) { + report.verdict = WaitReport::Verdict::Released; + report.route = WaitReport::Route::Always; + report.reason = "released: the compare function always passes"; + } else if (report.wait.heldAtSubmit) { + report.verdict = WaitReport::Verdict::Released; + report.route = WaitReport::Route::HeldAtSubmit; + report.reason = "released: its value passed the compare when the submission entered the driver (held at submit)"; + } else { + matcher.Match(report, waiter); + } + analysis.waits.push_back(std::move(report)); + } + return analysis; +} + +int Report(const std::filesystem::path& directory, bool full, std::FILE* output) { + const auto analysis = Analyze(directory); + const auto& submissions = analysis.submissions; + std::map perQueue; + std::size_t tails = 0, suspends = 0, dwords = 0; + for (const auto& submission : submissions) { + dwords += submission.dwords; + if (submission.kind == SubmissionKind::Suspend) ++suspends; + else ++perQueue[submission.queue]; + if (submission.kind == SubmissionKind::Rewind) ++tails; + } + std::fprintf(output, "capture %s: submissions %zu (rewind tails %zu, suspend points %zu), dwords %zu, packets %zu, memory stores %zu\n", directory.string().c_str(), submissions.size(), tails, suspends, dwords, analysis.packets, analysis.stores); + for (const auto& [queue, count] : perQueue) std::fprintf(output, " queue 0x%x %s: submissions %zu\n", queue, SubmissionCapture::QueueType(queue).data(), count); + std::map routes; + for (const auto& report : analysis.waits) { + if (report.verdict == WaitReport::Verdict::Released) ++routes[report.route]; + } + const auto unreleased = analysis.Count(WaitReport::Verdict::Unreleased); + const auto unknown = analysis.Count(WaitReport::Verdict::Unknown); + std::fprintf(output, "waits %zu: released %zu (compare always %zu, held at submit %zu, current value %zu, label history %zu, rewind tail %zu), unknown %zu (cross-queue timing %zu), unreleased %zu\n", analysis.waits.size(), analysis.Count(WaitReport::Verdict::Released), + routes[WaitReport::Route::Always], routes[WaitReport::Route::HeldAtSubmit], routes[WaitReport::Route::CurrentValue], routes[WaitReport::Route::LabelHistory], routes[WaitReport::Route::RewindTail], unknown, analysis.CrossQueue(), unreleased); + std::fprintf(output, "(released only where the capture proves it; the tool does not model cross-queue timing, so a wait that stores of another queue decide is unknown)\n"); + std::fprintf(output, "(stores are the PM4 packets of the capture; CPU and shader stores are not captured)\n"); + if (std::any_of(submissions.begin(), submissions.end(), [](const CapturedSubmission& submission) { return !submission.heldKnown || !submission.conditionalKnown; })) { + std::fprintf(output, "(the index has no held and conditional columns: a wait no store of the capture can release is unknown instead of unreleased, and every packet after a COND_EXEC or predicated INDIRECT_BUFFER is conditional to the end of its submission)\n"); + } + + std::map byOrdinal; + for (std::size_t index = 0; index < submissions.size(); ++index) byOrdinal.emplace(submissions[index].ordinal, index); + std::vector shown; + if (full) { + for (const auto& report : analysis.waits) shown.push_back(&report); + } else { + for (const auto verdict : {WaitReport::Verdict::Unreleased, WaitReport::Verdict::Unknown}) { + for (auto it = analysis.waits.rbegin(); it != analysis.waits.rend() && shown.size() < ListedWaits; ++it) { + if (it->verdict == verdict) shown.push_back(&*it); + } + } + if (!shown.empty()) std::fprintf(output, "\nunreleased waits, then unknown waits, newest first:\n"); + } + for (const auto* report : shown) { + std::fprintf(output, "\n%s %s\n %s\n", verdictName(report->verdict), waitLine(*report).c_str(), report->reason.c_str()); + for (const auto& writer : report->writers) std::fprintf(output, " writer %s\n", describeWriter(writer).c_str()); + if (report->writerCount > report->writers.size()) std::fprintf(output, " (%zu stores to the awaited bytes in the capture)\n", report->writerCount); + if (full) continue; + const auto& submission = submissions[byOrdinal.at(report->at.ordinal)]; + const auto packets = decodeSubmission(directory, submission, true); + const auto at = static_cast(std::find_if(packets.begin(), packets.end(), [&](const DecodedPacket& packet) { return packet.offset == report->at.offset; }) - packets.begin()); + std::fprintf(output, " in %s\n", submissionLine(submission).c_str()); + printPackets(output, packets, at > ContextBefore ? at - ContextBefore : 0, at + ContextAfter + 1, report->at.offset); + } + if (!full && unreleased + unknown > shown.size()) std::fprintf(output, "\n(+%zu more unreleased and unknown waits; --full lists every wait)\n", unreleased + unknown - shown.size()); + + std::vector order; + for (std::size_t index = 0; index < submissions.size(); ++index) { + if (submissions[index].kind != SubmissionKind::Suspend) order.push_back(index); + } + std::sort(order.begin(), order.end(), [&](std::size_t left, std::size_t right) { + return std::tie(submissions[left].queue, submissions[left].serial, submissions[left].ordinal) < std::tie(submissions[right].queue, submissions[right].serial, submissions[right].ordinal); + }); + std::fprintf(output, full ? "\nsubmissions in execution order per queue:\n" : "\nlast submissions of each queue (--full prints every submission and packet):\n"); + for (std::size_t begin = 0; begin < order.size();) { + auto end = begin; + while (end < order.size() && submissions[order[end]].queue == submissions[order[begin]].queue) ++end; + for (auto index = full || end - begin <= LastSubmissionsPerQueue ? begin : end - LastSubmissionsPerQueue; index < end; ++index) { + const auto& submission = submissions[order[index]]; + std::fprintf(output, "%s\n", submissionLine(submission).c_str()); + const auto packets = submission.dwords == 0 ? std::vector{} : decodeSubmission(directory, submission, true); + const auto limit = full ? packets.size() : std::min(packets.size(), ShownPackets); + printPackets(output, packets, 0, limit, std::numeric_limits::max()); + if (limit < packets.size()) std::fprintf(output, " (+%zu packets)\n", packets.size() - limit); + } + begin = end; + } + return unreleased == 0 ? 0 : 1; +} + +} diff --git a/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.hpp b/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.hpp new file mode 100644 index 0000000000..cff6c6da5c --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tools/Pm4Analysis.hpp @@ -0,0 +1,91 @@ +#ifndef CORE_LIBS_PRX_LIBSCEAGCDRIVER_TOOLS_PM4ANALYSIS_HPP +#define CORE_LIBS_PRX_LIBSCEAGCDRIVER_TOOLS_PM4ANALYSIS_HPP + +#include "prx/libSceAgcDriver/Execution/include/Pm4.hpp" +#include "prx/libSceAgcDriver/Execution/include/SubmissionCapture.hpp" +#include +#include +#include +#include +#include +#include +#include + +namespace AgcDriver::Pm4Analysis { + +struct MemoryStore { + Pm4::PacketStore store; + bool conditional = false; + bool label = false; +}; + +struct MemoryWait { + enum class Kind : std::uint8_t { Memory, Rewind }; + Kind kind = Kind::Memory; + Pm4::WaitFields fields{}; + bool heldAtSubmit = false; + bool conditional = false; + bool released = false; + std::vector packet; +}; + +struct DecodedPacket { + std::size_t offset = 0; + std::size_t words = 0; + std::uint32_t header = 0; + std::string name; + std::string operands; + bool conditional = false; + std::vector stores; + std::optional wait; +}; + +std::vector Decode(const CapturedSubmission& submission); + +struct PacketLocation { + std::uint64_t ordinal = 0; + std::uint32_t queue = 0; + std::uint64_t serial = 0; + std::size_t offset = 0; + std::string name; +}; + +struct WriterMatch { + enum class Order : std::uint8_t { EarlierOnQueue, LaterOnQueue, OtherQueue }; + enum class Effect : std::uint8_t { CurrentValue, LabelHistory, Evicted, Overwritten, Undecided, OtherQueue, Rejected, After }; + PacketLocation at; + Order order = Order::OtherQueue; + Effect effect = Effect::OtherQueue; + std::optional stored; + bool accepted = false; +}; + +struct WaitReport { + enum class Verdict : std::uint8_t { Released, Unknown, Unreleased }; + enum class Route : std::uint8_t { None, Always, HeldAtSubmit, CurrentValue, LabelHistory, RewindTail }; + PacketLocation at; + MemoryWait wait; + Verdict verdict = Verdict::Unreleased; + Route route = Route::None; + bool crossQueue = false; + std::string reason; + std::size_t writerCount = 0; + std::vector writers; +}; + +struct Analysis { + static constexpr std::size_t ListedWriters = 8; + std::vector submissions; + std::size_t packets = 0; + std::size_t stores = 0; + std::vector waits; + std::size_t Count(WaitReport::Verdict verdict) const; + std::size_t CrossQueue() const; +}; + +Analysis Analyze(const std::filesystem::path& directory); +int Report(const std::filesystem::path& directory, bool full, std::FILE* output); + +} + +#endif diff --git a/core/libs/prx/libSceAgcDriver/tools/Pm4Replay.cpp b/core/libs/prx/libSceAgcDriver/tools/Pm4Replay.cpp new file mode 100644 index 0000000000..7c44c8da28 --- /dev/null +++ b/core/libs/prx/libSceAgcDriver/tools/Pm4Replay.cpp @@ -0,0 +1,35 @@ +#include "Pm4Analysis.hpp" +#include +#include +#include +#include + +namespace { + +int usage() { + std::fprintf(stderr, "usage: agc_pm4_replay [--full]\n" + " decodes the submissions APS5_DUMP_SUBMITS= captured and reports each wait as released when\n" + " the capture alone proves it, unreleased when no store in the capture can release it and unknown\n" + " otherwise (cross-queue timing is not modelled); exit code 1 when a wait is unreleased, 2 on errors\n"); + return 2; +} + +} + +int main(int argc, char** argv) { + std::filesystem::path directory; + bool full = false; + for (int index = 1; index < argc; ++index) { + const std::string_view argument = argv[index]; + if (argument == "--full") full = true; + else if (!argument.starts_with("-") && directory.empty()) directory = argument; + else return usage(); + } + if (directory.empty()) return usage(); + try { + return AgcDriver::Pm4Analysis::Report(directory, full, stdout); + } catch (const std::exception& error) { + std::fprintf(stderr, "agc_pm4_replay: %s\n", error.what()); + return 2; + } +} diff --git a/docs/dev/BUILD.md b/docs/dev/BUILD.md index 82b5576701..2687b2c651 100644 --- a/docs/dev/BUILD.md +++ b/docs/dev/BUILD.md @@ -118,6 +118,19 @@ Each invocation creates one pipeline. With no `--cache-in`, its application pipe `--fixture` compiles a tiny synthetic compute shader with specialization, sparse descriptor binding and push constant metadata; `--write-fixture DIRECTORY` saves it without opening Vulkan. It performs no GPU work and does not reproduce a title's compile workload. With `BUILD_TESTING=ON`, `agc_compute_pipeline_capture` checks serialization and rejected inputs. With SPIRV-Tools also enabled, `agc_compute_pipeline_replay` compiles the fixture with fresh and imported application caches, verifies cache rejection, and checks required subgroup support. `agc_compute_pipeline_capture_run` verifies the existing branch-past-endpgm compute fixture's output with capture disabled and enabled, then replays the captured pipelines and checks their identities and metadata. Each test has a 30-second timeout. Runtime tests skip when Vulkan is absent; set `ANYPS5_REQUIRE_VULKAN=1` to require execution. There is no performance pass threshold. +## PM4 submission replay + +Set `APS5_DUMP_SUBMITS` to a directory before starting a title to save every submission the AGC driver accepts, as it enters the driver. `index.txt` gets one line per submission: ordinal, queue, `graphics` or `compute`, serial, microseconds on the clock of the `APS5_TRACE_GPU` lines, dword count, kind, file name, the dword offsets of the waits whose value passed their compare when the submission entered the driver (`held`, which the driver then does not wait for) and the COND_EXEC and predicated indirect buffer ranges as `start:end` dword offsets (`-` for none). `submission_.bin` holds the dwords the driver copied, with indirect buffers inlined. A REWIND tail is listed as `rewind` with the serial of its submission, a suspend point as `suspend` without a file. The index is truncated on the first submission. A failed write puts the driver in its failed state: the call that made the submission throws after the submission was queued, and every later driver call throws the same error. Unset, the switch costs one test per submission; set, one copy of the commands per submission before the driver lock, the ordinal, time and held waits under it without allocating, and the files written after it. A capture holds command buffers only, no guest memory. An index written before the held and conditional columns (column line `ordinal queue type serial time_us dwords kind file`) is still read, with both unknown. + +`agc_pm4_replay [--full]` decodes the capture offline with the driver's packet decoder, validator and store decode (`Pm4::DecodeStores`) and reports each WAIT_REG_MEM, WAIT_REG_MEM_64 and REWIND as released, unreleased or unknown. It does not model cross-queue timing: a wait is released only when the capture alone proves that the driver releases it, and unknown, with the reason, whenever its outcome depends on when the stores of another queue run, since a store of another queue can run before, between or after the stores of the waiting queue whatever order the submissions entered the driver in. A memory wait is released when its compare always passes; when it was held at submit; when the last store to the awaited bytes before it on its queue (in that submission or an earlier one) covers them with a value the compare accepts, is not conditional, and no other queue stores to any of those bytes anywhere in the capture (current value); or when a RELEASE_MEM or WRITE_DATA label earlier in the waiting submission, one the driver keeps in its label history (4-byte aligned, at most 64 bytes, not conditional), stores a value the compare accepts and fewer than 4 other stores to the awaited bytes can run between it and the wait (label history). The driver keeps the last 4 label stores per dword over all queues (`LabelHistory.cpp`) and finds one there only if it ran after the waiting submission entered the driver, which the capture proves only for a store earlier in that submission. The stores counted against the history are those between the label and the wait on the waiting queue and every store of another queue to the awaited bytes, whatever its submit order; stores later on the waiting queue run after the wait. A REWIND is released when its tail is in the capture. One that was valid at submit without a tail in the capture is unknown, because the driver reads the valid bit again when the queue reaches the REWIND. A memory wait is unreleased when it was not held at submit and no store in the capture that can run before it may write a value the compare accepts (a store of part of the awaited bytes or of a value known only at run time may): the GPU does not release it, but a CPU store could, and so could a shader store, since the capture holds neither. A REWIND without a tail that was not valid at submit is unreleased. Every other wait is unknown: `unknown: depends on cross-queue timing (stores on queue X at #...)` when another queue stores to the awaited bytes; otherwise when the last store before it on its queue is conditional (COND_WRITE, inside a COND_EXEC range or a predicated packet), stores a value known only at run time or only part of the awaited bytes, or overwrites an accepted store that the capture does not prove is still in the label history (one in an earlier submission, one the history does not keep, or one with 4 or more stores after it). When the index has no held column, a wait that would be unreleased is unknown, since its value may have passed the compare at submit; without the conditional column every packet after a COND_EXEC or predicated INDIRECT_BUFFER is conditional to the end of its submission. The report counts the waits per outcome and route and the unknown waits that depend on cross-queue timing, says when the index lacks the columns, lists up to 32 unreleased and then unknown waits, newest first, each with the packets around it and the stores to the awaited bytes that decide it (for another queue, the 4 nearest the waiting submission in submit order), then the last submissions of each queue. `--full` lists every wait and prints every packet. A packet the driver rejects means the capture is corrupt and fails the analysis. The exit code is 1 when a wait is unreleased and 2 on errors. + +```sh +cmake --build build --target agc_pm4_replay +build/core/libs/prx/libSceAgcDriver/agc_pm4_replay submits +``` + +On Windows use `agc_pm4_replay.exe` and prepend `build/core/libs/libs/unpatched` to the process `PATH`. With `BUILD_TESTING=ON`, `agc_driver_pm4_replay` captures submissions made with the libSceAgc command builders through the driver and checks the index, the held waits, the conditional ranges, the dwords, the decode and the outcome of each wait, then checks hand-made captures: each release route; a store of another queue to the awaited bytes that entered before or after the store of the waiting queue (unknown); the label history limit, with the stores after the label on the waiting queue or on another queue; a fill the history does not keep; a store in an earlier submission overwritten later; later stores, rejected values and a missing rewind tail (unreleased); the six captures of the reviews (a two-queue deadlock, a store of 0 behind a released wait on another queue, with and without that wait held at submit, and labels pushed out of the history by stores of another queue), none of whose waits may be released except the one held at submit; five groups of waits in eight submission orders with identical outcomes; an index without the held and conditional columns; corrupt captures; then a failed write. `agc_driver_pm4_replay_tool` runs the tool on the driver capture and checks its exit codes and report. + ## Pipeline statistics Set `APS5_PIPELINE_STATS=1` to capture and print driver statistics for each newly created graphics or compute pipeline. This requires `VK_KHR_pipeline_executable_properties` and `pipelineExecutableInfo`; an unsupported device fails with an error. Statistic names and units are driver-specific. Capturing statistics can increase pipeline compilation cost. The setting is disabled by default.