Read-only addresses are not part of the dispatch key, so the fresh build after a refused
move replaced the template that serves the other address, and the two addresses
rebuilt each other. A refused dispatch now uses or builds an entry under the exact key
(read-only bases included) and leaves the movable template alone. The [dispmove] line
counts the rebuilds after a refusal, the builds inserted beside the template and the
dispatches served by an exact entry. Restores the MarkGpuWrites comment.
The content key of a compute stage leaves the base address and size of its
read-only guest buffers out, as a draw's key does. A template hit whose such
buffers moved is proved the way a draw's is (ShaderResources::MovedDispatchBuffers:
read-only, read in place when built, in reach, nothing pending over the new range)
and binds snapshots of the new ranges in a descriptor set of its own, kept with the
batch that records the dispatch. A refused proof builds the dispatch fresh, as
before. APS5_NO_DISPATCH_MOVE=1 turns the path off, APS5_DISPMOVE_MAX_KIB caps the
bytes snapshotted per dispatch.
A template whose storage image lies over memory that a depth surface covers is no
longer proved from stamps (fastRevalidate): cachedStorageTexture answers such a
binding with the plane's colour mirror, copied in from the depth image for every use,
and nothing stamps the appearance of a depth surface after the build. The full walk
decides, and a stale image can no longer be copied over the depth image.
Tests: Graphics.cpp (verdicts of the proof), execution DispatchMove (real device:
moved, exact, shortened, stored-over, own-output and written-range refusals) and
DispatchMoveDepth (a depth surface appearing over a template's plain storage image,
planes inside and outside the write watch, imports watched and unwatched).
On Windows the atexit handlers of a DLL run after the process has ended
its other threads, so the logger thread can be gone with lines still
queued and the unbounded Flush waits forever (guest_shader_link timed
out on 4 of the last 7 completed main builds since #6371). Run the bounded
FlushAtEnd there instead.
Co-Authored-By: Claude <noreply@anthropic.com>
APS5_SPANNED_WAVE is now read once by the driver, which sets
SpirvTarget::spannedWaves only on a device with 32-lane compute subgroup
control and computeFullSubgroups. The recompiler no longer reads the
environment. The field is part of the cache key and of the request
serialization (version 32), so a cached or replayed module keeps the mode
it was built in.
A spanned module numbers its lanes SubgroupId * 32 +
SubgroupLocalInvocationId, which needs two full 32-lane subgroups. The
artifact records this as fullSubgroups (disk cache and pipeline identity
included), and the driver creates those pipelines with
REQUIRE_FULL_SUBGROUPS. Compute pipeline captures accept the flag with a
required subgroup size.
The two-lane fallback now catches only SpannedWaveUnsupported, thrown for
a shuffle wider than 32 bits; other emit errors surface as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With pipelined draws, a DISPATCH_DIRECT on queue 0 whose dispatch-cache entry hits with a
recipe runs as an ordered commit item instead of draining the pipeline first. Misses, reads
that overlap writes still pending in the pipeline, verified data hits and writes that cannot
be ranged still drain. A dispatch that is resolved ahead (or adopted from a group resolved
ahead) and copy kernels keep the drained path; with the switch on, queue 0 does not resolve
dispatch groups ahead, since that drains the pipeline at every group. Off by default.
agc_driver_pipelined_dispatch counts drains per opcode (each dispatch drains with the switch
off); agc_driver_pipelined_dispatch_item checks that repeated dispatches do not drain, and that
a dispatch whose descriptor table a pending fill retargets drains and reads the new buffer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A graphics pipeline the store builds or a compute pipeline a dispatch builds is
appended to pipelines.journal next to the shader cache entries, as the
materialized result of each stage (the specialized SPIR-V, bindings and push
constants) plus the recipe the pipeline was built from: the stages'
identities, vertex input layout, descriptor set layout keys and the state
fields the Pipeline constructor reads. When a VulkanDevice is created with a
window (a headless device waits for its first draw or dispatch, since the
window's device usually replaces it first) the journal is loaded and its
pipelines are built on a pool of threads while the title loads, so the first
draw or dispatch of a configuration seen on an earlier run finds its pipeline
resident instead of compiling it on the queue worker.
The pipeline stores key on RecompileResult::Identity, a hash of the module
words, specialization constants, subgroup width and bindings that is the same
on every run (SharedSpirv::ContentHash, computed once per shared module),
rather than on the process-lifetime variant ids; a generated stage is keyed
like any other. CachedPipeline builds outside the store's mutex and the
library parts and link are built outside the library store's mutex, so replay
threads and queue workers build in parallel; a concurrent build of the same
key keeps the first inserted. A replayed pipeline enters the store at its
least recently used end and a full store takes none, and its optimized relink
waits for the first draw that binds it, so the title's own pipelines come
first in both. The compute pipeline creation moves into
VulkanDevice::createComputePipeline, shared by dispatches and the replay. The
store key also covers the color component mapping, the depth target's sample
count and a depth bias clamp, which the constructor reads.
The file names the device it was written for (the properties the driver's
pipeline cache file is named by) and starts over for another; a torn tail is
cut at a record boundary; two runs sharing the directory take turns on a lock
file and keep each other's records; a read failure never starts the file
over. A device cancels its replay before it is replaced or destroyed, and a
replaced device destroys its pipelines then rather than at the driver's stop.
APS5_NO_PIPELINE_JOURNAL=1 disables it; APS5_PIPELINE_JOURNAL_THREADS,
APS5_PIPELINE_JOURNAL_ENTRIES (the first recorded), APS5_PIPELINE_STORE_ENTRIES
and APS5_PIPELINE_JOURNAL_MAX_MIB bound it; APS5_PROFILE_DRAW prints a
[pipeline-journal] summary. APS5_VERIFY_SET_LAYOUTS=1 now also checks a
dispatch's set layout against the one described from its shader. The lavapipe
test suite, which shares one cache directory, runs with the journal off except
for the journal's own tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P6hw3kRAuQdfcU9YCZgaoK
A GPU hang or a crash in a title leaves no record of what the queues
were last given: APS5_TRACE_GPU prints every submission (too much to
leave on), and the WAIT_REG_MEM timeout message shows only the packet
before the wait on the waiting queue.
Each queue now keeps a ring of its last 16 submissions
(APS5_PM4_RING=<n>, 0 to 4096, 0 disables): the TraceMs time in tenths
of a millisecond, serial, size, guest address and a copy of the first
512 dwords after indirect buffers are inlined, recorded under the
driver lock next to the serial, REWIND tails included. The ring of a
queue and the failure report registration are set up at the top of
Submit, before any output is reserved, so recording under the lock
does not allocate. Disabled, Submit only tests a static flag; enabled,
it costs one bounded copy per submission.
DumpSubmissionRings decodes the kept dwords with Pm4::PacketWords and
the new Pm4::KnownName, which returns the decoder's names as views of
static strings (empty for an unknown opcode, printed as UNKNOWN_0x%02x);
Pm4::Name now builds its std::string from it. Every line is formatted
with integer and string conversions only into a 512-byte stack buffer
and written with one WriteFile / write(2) on the stderr handle, so the
dump does not allocate from the C runtime heap, lines from other
threads do not interleave and the CRT stream lock is not taken (the
rule of #3871). The header gives the number of queues and the ring
size, and each queue line the submissions recorded and kept.
libc gets a small registry of failure reports
(LibcRegisterFailureReport_nid_no_patch, Unregister, Run); Run runs
the registered reports the first time it is called and returns at once
after that. The Windows crash report runs it after the thread dump of
an unhandled exception, after SIGABRT (which also covers the
APS5_HANG_DUMP_SECS timed dump, since it aborts) and after an invalid
CRT parameter, so an abort after a reported exception does not print
the rings twice. The driver registers its dump at the first submission
and unregisters at static destruction; the dump there waits a bounded
time for each ring lock. What the crash-path dump cannot do (a ring
held by the crashed thread, rings corrupted by the crash, Linux) is
recorded in technical debt.
APS5_STALL_WATCHDOG_SECS=<n> (0 to 86400, 0 disables) makes a waiter
that has not progressed for n seconds dump the rings once per wait: a
WAIT_REG_MEM (progress as for APS5_GPU_WAIT_TIMEOUT_MS: another packet
runs or completes), a REWIND waiting for the title, and a compute
submission held behind queue 0. The last one prints under the driver
lock, so submissions and the other queues wait for it (USAGE.md says
so). Both variables are read at the top of Submit, so an invalid value
of either throws at the first submission and never later inside a GPU
worker. The order wait behind queue 0 releases its order hold on scope
exit, so a throwing dump cannot leave it counted.
No packet semantics change; packet sizes and names are those of the
existing decoder (PM4 type-3 header layout as in Mesa
src/amd/common/sid.h). The queue bound of Submit and of the rings is
one constant, SubmissionQueueEnd.
Tests: agc_driver_pm4_tests gets ten modes, each its own ctest:
agc_driver_pm4_ring (APS5_PM4_RING=4, six submissions on queue 0 and
two on 0x20: the dump holds #3..#6 and #1..#2 in order, nothing
older), agc_driver_pm4_ring_empty (one line), _ring_disabled (one
line), _ring_invalid and agc_driver_pm4_stall_watchdog_invalid
(APS5_STALL_WATCHDOG_SECS=1x: the submission throws and is not
executed), _ring_report (LibcRunFailureReports_nid_no_patch dumps the
rings), _ring_report_heap (Windows: with ucrtbase.dll's HeapAlloc and
HeapReAlloc imports counting calls from the reporting thread, the
failure report of a submission holding INDEX_BUFFER_SIZE names it and
allocates nothing, while Pm4::Name of the same header is seen
allocating), and with APS5_STALL_WATCHDOG_SECS=1 a WAIT_REG_MEM
(agc_driver_pm4_stall_watchdog), a REWIND (_stall_watchdog_rewind) and
a compute submission held behind a queue 0 rendering wait
(_stall_watchdog_order), each held for about 2.5 s, dump once and
complete when released. The PM4 catalog test checks KnownName against
every opcode and custom name. windows_backtrace_failure_report checks
that abort() runs the registered reports once and not an unregistered
one; windows_backtrace_failure_report_once that abort() does not run
them again after an earlier run. With the dump decoding through
Pm4::Name, agc_driver_pm4_ring_report_heap fails with "the failure
report allocated from the C runtime heap"; with Run resetting its
flag, windows_backtrace_failure_report_once prints the report twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reapplies #5791 (reverted by #6552). agc_driver_mesh_index_cache now skips
on llvmpipe like agc_driver_mesh does: llvmpipe aborts compiling these mesh
programs (LLVM ERROR: Cannot emit physreg copy), which is what turned main's
linux job red.
Co-Authored-By: Claude <noreply@anthropic.com>
agc_driver_mesh_index_cache aborts on the linux job (lavapipe, LLVM 20.1.2:
"LLVM ERROR: Cannot emit physreg copy instruction") on every main push build
since the merge (first red: 38777177302). The PR was merged on a
windows-only run after a linux-red batch was split.
Co-Authored-By: Claude <noreply@anthropic.com>
Lines queued behind the logger were lost when the process ended through
abort(), std::terminate, the Windows crash report or a guest _Exit,
because only driver failure, Driver::stop and a normal exit flushed the
queue.
libc keeps one end flush (LibcSetEndFlush_nid_no_patch) and runs it from
those paths. Its first registration chains a SIGABRT handler in front of
the previous one, so every abort() and std::terminate in the process is
covered, on Linux as well; with the default disposition the handler
restores it and raises again, so raise(SIGABRT) still ends the process.
Guest abort() and _Exit, and ReportCrash for the first fatal exception,
run the flush directly. AsyncLog registers a bounded flush (FlushFor, 2 s)
that polls two counters and takes no lock, so a dying process cannot hang
on a blocked stream or on the queue lock; after its first timeout it
returns at once. Flush counts queued and finished entries instead of
watching the queue and the writer flag, so it waits for the lines queued
before the call and not for a moment in which no thread is logging.
agc_async_log_fatal runs the test executable as a child that queues lines
behind a held stream and ends in abort(), std::terminate(), raise(SIGABRT)
(POSIX) or an access violation (Windows), and once more with the stream
never released. The lines of the first four must arrive in order, the last
child must end after the bound, and every child has a watchdog.
agc_async_log gains the bounded flush case.
Flush must not return while the logger still waits for a stream that
another thread holds (the regression reported on the pull request: it gave
up after two seconds), and a full queue must block its caller instead of
dropping a line. AsyncLog::Capacity becomes public so the test sizes its
load from it.
Texture creation logged one stdout line per texture and per mip through
WriteLogLine, which formatted with the mingw printf and flushed every
line on the queue worker (about 786k lines in a 25 minute cave run).
AsyncLog keeps a bounded queue and a logger thread. AGC_LOG_DEFERRED
captures numeric arguments on the caller and formats them on the logger
thread; WriteLogLine hands finished lines to a per-module sink in
libc/LogMacros.hpp that AgcDriver installs, so all AgcDriver stdout
lines keep their order and are written one flush per batch. A full queue
blocks the caller, so no line is dropped. Flush runs on driver failure,
shutdown and at exit. APS5_LOG_SYNC=1 restores the synchronous path.
Identical "deferred write-back failed" notes are printed three times per
10 s and then counted into one "repeated N more times" line. A
"[log] 10 s" line reports lines, bytes, caller time and logger time
(APS5_LOG_STATS=0 hides it).
The lazy mode of agc_driver_prepared_shaders only checked dispatch output and failure reporting, which eager preparation also produces. Read the registered snapshot through Driver::RegisteredShaders: registration leaves it deferred with no prepared artifact, and the first dispatch prepares it.
A draw whose user-data words differ only in pointers the capture read through
looks up an alias key built from the contents behind them, shifts the cached
variant's runs by the pointer delta, validates them by value and patches the
forwarded shader-data words.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dispatch look-ahead stopped before every DISPATCH_INDIRECT whose
arguments an earlier dispatch of the run may write, or whose earlier
dispatches have unknown writes. That stop protects a CPU read of the
arguments at the resolve, which only the thread-dimensions, misaligned and
APS5_NO_GPU_INDIRECT paths make. A GPU-read indirect dispatch never reads
its arguments on the CPU, so the stop left it and every dispatch behind it
to be captured when the worker reached them, after the GPU mutex waits
between dispatches. That is the late read the group capture exists to
avoid: by then the title had re-armed the gate and reused its table slots,
and the capture read descriptors the title had not written yet. The
look-ahead now stops only before an indirect dispatch whose arguments it
would read on the CPU; the path choice is shared with dispatchIndirect.
AgcDriverDispatchesAdoptedAhead_nid_postfix counts the dispatches adopted
from the look-ahead so a test can tell a gate-time capture from a late one.
Two tests check through it that both dispatches of a gated group are
captured at the gate: one whose writer stores the indirect arguments
through a registered binding, and one whose writer stores them through a
binding misaligned in its host import.
On Windows the write watch is dropped in whole 64 KiB blocks
(WriteWatchCoverage::Exclude), but the unwatch hook was given the page
range of the import. An image with results pending in the same block as
an imported page, outside that page, lost the watch without its guest
bytes saved, and storing its results threw.
The hook now gets the span of blocks Exclude drops (ExcludedSpan, the same
block math), so every image with results pending in those blocks saves its
bytes first.
Use checked host stdio for BMP output and stop requesting dumps after the first write failure. Keep pixel validation, allocation and non-I/O failures on the existing fatal path.
Co-Authored-By: Codex <noreply@openai.com>
A pattern fill reads its count and pattern from guest memory on the queue worker. With pipelined
draws it read them while earlier commit items that write them (labels, DMA stores, draws) were
still queued, so it stored the stale value (agc_driver_pattern_buffer_fill under load). It now
drains the draw pipeline first when a queued item writes one of its input ranges.
The same fill held the GPU lock from the input read through DrawPipeline::Enqueue. With the
pipeline full, Enqueue waited for the committer while the committer's next item waited for the GPU
lock: a deadlock. The input lock is released before the fill is queued, and Enqueue throws when it
would wait for room while the calling thread holds the GPU lock.
(cherry picked from commit c653e9ea1d)
With APS5_PIPELINED_DRAWS=1, queue 0's worker decodes, captures and recompiles a draw and hands it to
a committer thread that records it (Graphics::Draw under the GPU lock), so the two run in parallel
instead of one after the other. Work the committer has not reached yet is tracked by the guest ranges
it will write; a later packet that reads or writes them, or one the pipeline cannot order, drains it.
Ordered as commit items instead of drains:
- labels (WRITE_DATA, RELEASE_MEM) and same-queue waits on them; APS5_PIPELINE_DRAIN_LABELS=1 drains;
- the end of a queue-0 submission and a released REWIND tail; APS5_PIPELINE_DRAIN_SUBMISSIONS=1 drains;
- the title's buffer fill kernels; APS5_PIPELINE_DRAIN_FILLS=1 drains;
- DMA_DATA immediates and copies (APS5_PIPELINE_DMA=1).
A draw's write ranges include the storage images it writes and its exact color and depth target
ranges (APS5_EXACT_DRAW_WRITES=1), not estimates.
Draw-path costs the pipeline exposed:
- ShaderMemory moves, so a commit owns its capture's pages;
- a submission made before the worker's current collect epoch began keeps the epoch, so a worker
behind a queue of submissions walks a surface once per catch-up (APS5_SUBMISSION_EPOCH_EACH=1 bumps
at every submission); a thread remembers the ranges it walked for the whole epoch;
- the pipeline signals only threads that wait; the pending-image and unit-shadow registries take a
spinning slim lock instead of std::mutex (winpthreads contention was kernel events); ScanPending skips
the registry walk when no pending image can overlap;
- fewer heap allocations per recorded draw (reused descriptor sets and scratch vectors, buffer pool
size classes kept).
GTA V Performance mode, prologue bank interior, RTX 5080: 11.7 -> about 20 fps together with the cheaper
captures below it.
Off by default here; the follow-up turns it on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Gjon Gojcevic <7807870+gjonn@users.noreply.github.com>
The per-draw capture of a shader's memory (the SRT walk and the guest pages it reads) cost about
15 us per draw packet in GTA V Performance mode; it now costs about 6.6.
- The SRT evaluator memoizes per plan value in a thread's reused, stamped array instead of a hash
table allocated and grown on every walk (evaluate 5.3 -> 3.6 us per capture).
- ExtractPlan also stores the plan's values as 32-byte records with argument ids, and the walk
evaluates that compact copy: the IR walk spent most of its time in cache misses (3.6 -> 2.7 us).
The evaluator is a template over the node view, so both walks share every opcode's semantics, memo,
cycle and failure rules. A plan that cannot be expressed compactly keeps the IR walk.
APS5_VERIFY_SRT_COMPACT=1 walks both and throws on any difference.
- ShaderMemory's valid/read/recent marks are arrays of 64-bit words scanned with countr_zero instead
of bitsets tested bit by bit (about 1 us per draw packet).
- Captured pages come from a per-thread pool, are found through a short vector with a last-page
shortcut, and a page no recorded GPU work writes gets the flush hook a whole-page read would make,
then has each 64-byte block copied when a word of it is first read (about 0.3 us per packet).
Pages with pending GPU writes and unmapped pages are read as before.
- APS5_PROFILE_DRAW splits the capture time into SRT evaluation and materialization, and prints the
[capture] line once per 10 s instead of every 500 captures (it cost about 2 us per capture).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Gjon Gojcevic <7807870+gjonn@users.noreply.github.com>
Since 39fdc48b5 a graphics program outside every registered shader was
read raw only while the registry was empty, and any RegisterShader after
the first raw draw threw. A title that registers some shaders through
sceAgcCreateShader and also draws programs it never registers stops at
its first such draw with "graphics program does not belong to a
compatible registered shader": oops-sdk homebrew registers its display
shaders while radeonsi (oops-mesa) submits raw programs.
Graphics programs now resolve as compute dispatches already do: an
address inside a registered shader resolves to that registration (a
registration of another type still throws), any other address is read
raw. RegisterShader throws only when the new shader's code overlaps a
graphics program that was drawn raw, where the program would get a
second identity (the draw cache revalidates raw programs by reading them
raw), instead of after any raw draw.
The drawn range is recorded once per raw snapshot: a flag on the
snapshot keeps later decodes of the same program off the lock. Recorded
ranges are merged, so the overlap check looks only at the predecessor of
the queried end.
Tests: agc_driver_prepared_graphics (a draw whose programs are registered
as compute shaders is refused, where it used an address outside every
registration before), agc_driver (ProgramSnapshot reads raw beside registered shaders
and refuses a registered program of another type; nested, overlapping
and adjacent raw ranges), agc_driver_prepared_unregistered_graphics
(raw-drawn ranges recorded exactly, unrelated code not reported).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A draw stage whose captured words differ from a cached variant only at pure
data positions (read by the shader as data, never as an address, descriptor
or decode input) is now a hit: the live words are patched into the cached
compiled result, as the dispatch cache's EqualData hits do. The positions are
recorded at insert from the stage's resource capture, minus the words the
draw decode reads. Patched draws take no draw recipe.
APS5_NO_DRAW_STAGE_DATA_HITS=1 turns it off; APS5_VERIFY_DRAW_DATA_HITS=1
builds the stage anyway and throws on any difference. The draw-cache profile
line counts stage data hits and refreshed words.
Co-Authored-By: Claude <noreply@anthropic.com>
Since #5216 a key gets its entry at its second sight, so a shape whose
user words change on every draw never got an entry and its data-only hits
never started. A key whose shape was seen before but has no entry is now
admitted at its first sight. A shape that already has an entry serves
data-only hits from it, and its new keys are still admitted at their
second sight, so per-draw keys don't churn the cache. Shapes are
remembered in a second sightings table of the same size.
agc_driver_draw_cache_reuse draws three times with new pixel user words
over a cached shape (three data-only hits, the vertex stages reused, no
insert), then three keys of a new shape (the second one is admitted), and
runs again with APS5_VERIFY_DRAW_DATA_HITS=1.
Co-Authored-By: Claude <noreply@anthropic.com>
The draw key now has a shape: the key without the user words the stages read.
A draw whose key misses but whose shape has an entry copies that entry's decode,
re-reads only the user words (and merged-stage pointers), and reuses each
compiled stage whose captured reads still hold, compiling the rest.
APS5_NO_DRAW_DATA_HITS=1 disables it; APS5_VERIFY_DRAW_DATA_HITS=1 also decodes
and compiles in full and throws when the results differ.
A shape keeps its previous entries and only points at its newest key, so
draws of one shape that alternate between stable user words keep their own
key hits. The shape is passed only when data hits are on, so
APS5_NO_DRAW_DATA_HITS=1 keeps the previous behaviour exactly.
Rebased onto main a8b032b2 by CouchRiv (the conflicts with #639 resolved; built and tested on Linux with an RTX 4070 Ti and RADV, and on Windows with an RTX 5080), and onto main 6ac63bf0c by gjonn (main's free-function decoder with its static-ABI mode, its incremental register key as the shape key).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: CouchRiv <270190108+CouchRiv@users.noreply.github.com>
Co-Authored-By: Gjon Gojcevic <7807870+gjonn@users.noreply.github.com>
A dispatch now leaves its initiator and, for DISPATCH_DIRECT, its group
counts in the shader registers, where the CP puts them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AlxcBTW