The data-hit verification keeps this branch's pending-span query on its ShaderMemory and takes #5958's CaptureInvocation, which retries a capture with converted samples.
Keeps this branch's tagVariantRuns declaration next to main's dumpRequest suffix parameter in Driver.hpp, and reads #6232's pointer-chain test words from its table's snapshot, where this branch keeps them.
#7527's selected-record limit test reads a record's descriptor words through the
table snapshot's shared words list; this branch keeps a snapshot's words per column,
so the test reads them from its column instead.
10_11_11_UNORM (format 30) images are read and written through an
R32_UINT view, so the hardware sampler cannot filter them. image_sample
of a 2D or 2D array view now fetches the one or four texels at LOD 0,
decodes them and blends them in the shader, with bilinear weights on
the 8-bit grid, for point or bilinear filtering with wrap or
clamp-to-edge addressing.
A shader gets the converted sample mode only after a converted
descriptor reaches one of its sampled images: that capture throws
ConvertedSampleRequired and the driver prepares the program again
with GuestContext::convertedSamples.
The R10_G11_B11 float format is sampled natively since #7271 and keeps
its refusal on the converted path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An S# that forces unnormalized coordinates was refused when it samples a
T# the shader declares 3D, 2D-array, cube or multisampled, since Vulkan
allows unnormalized coordinates only with a single-level 1D or 2D view.
A null T# reads zeros at any coordinate, and a T# whose channels all
select constants is emitted as those constants without a sample, so when
the S# samples no other T# its coordinates cannot change a result. Such
an S# is now marked in its binding (samplerCoordinatesIgnored) and the
driver creates its sampler without FORCE_UNNORMALIZED. The flag is part
of the binding content key and of the shader disk cache, whose format
version is raised to 38. An S# that also samples a real T# while a null
T# of such a shape shares it is still refused. PPSA01317 pairs an
unnormalized clamped bilinear S# with a null 3D T# and a 3D T# whose
channels all select 0 for image_sample_lz in a compute shader during
boot.
AI-assisted: yes
Main's #6402 stores to image tables: a table with a written entry takes
the fresh path, as it does on main, and so does one without a sampled
table binding, whose entries the fresh path rejects. The shared place
step uses the binding as a pointer. Main's #6614 frees descriptor sets
from shared pools one by one, while this branch keeps only dedicated
pools, so both stay. #7101's bulk key read counts its unmapped-page
skips as unmapped, and its key words sit next to this branch's write
stamp and compared runs.
Main's #7101 reads a key buffer's words in one call where this branch
names the reason a narrowing is skipped. Keep main's bulk read and
count both of its unmapped-page skips as unmapped.
Texture format 43 holds unsigned floats of 10 bits (R, bit 0), 11 bits
(G, bit 10) and 11 bits (B, bit 21). VK_FORMAT_B10G11R11_UFLOAT_PACK32
holds R11 at bit 0, G11 at bit 11 and B10 at bit 22, so storing the
guest R field in the Vulkan B field and the guest B field in the Vulkan R
field moves every texel without changing a bit of any component. A
sampled texture of format 43 is now uploaded that way (a repack pass over
the detiled words, TextureRepack.comp) and viewed with R and B swapped
back (TextureComponentChannel), so the shader samples it natively: point
and bilinear filtering, every mip level and LOD, and cube faces
work as for any other float texture. A texture whose results a storage
image still holds is copied from that image (vkCmdCopyImageToBuffer, the
repack pass, vkCmdCopyBufferToImage) and copied again whenever the
image changed, like the depth and block copies. The recompiler treats a
sampled format-43 image as a float image without conversion; storage
reads and writes keep the packed R32_UINT path, and a 3D texture whose
results are still in a storage image is flushed and uploaded instead.
PPSA01317 samples a mipmapped format-43 texture at a LOD other than 0
and a format-43 cube map during boot.
AI-assisted: yes
Each image-table snapshot and resolution built node-based indexes and
dropped them at the end of the call: an unordered_map of distinct
descriptor words in TableSnapshotter, and in ResolveImageTables a map of
poison records, a set of expected poison codes, a map of bound elements
and an unordered_map of records per table. A table of N distinct entries
cost several allocations per entry on every dispatch.
The words, element and poison indexes are now open-addressing tables over
the vectors they index, and the per-table records a dense array by code.
Their slots live in per-thread scratch stamped per use, so a call reuses
the previous call's storage without clearing it; scratch above 2 MiB is
released after the call. Expected poison codes are a bit vector.
AI-assisted: yes
Narrowing a table column read its key buffer one key at a time, with a
page lookup and a single-word read per key; PPSA05684 scans about 20,000
keys per table snapshot. The scan now checks each page once and reads the
keys with the bulk reader, and maps a key to its record without dividing
when the multiplication cannot wrap.
Gameplay, image-table time per snapshot: about 6.5 ms before, 5 ms after.
AI-assisted: yes
An IMAGE_STORE whose T# comes from a runtime-indexed image table used to
poison the whole table as an unsupported operation. Written, non-atomic
table images now get storage modes, a STORAGE_IMAGE table binding (260,
the sampled image table moves to 261 and keeps the variable count), and
storage views of tracked storage textures at the same element indices.
Every bound entry is marked written, so each dispatch writes back all of
the table's storage images. Atomic, packed, BY2/BY4 and dynamic-mip table
stores still fault. Tables with written entries are not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
image_store with D16 data to a storage image of a SINT format threw
"stores 16-bit data to an image of a SINT format", both when the
descriptor is decoded and in the SPIR-V image emitter, and the runtime
image modes left out the SINT conversions for such images. God of War
Ragnarok (PPSA08330) stops on it in compute shader 0x1052a17900 right
after its MFSR setup.
Each 16-bit half is now converted like the D16 format buffer stores
(EmitD16StoreComponent): sign-extended, then saturated to the
component width as for 32-bit data. The buffer form was measured on
RDNA2 in #331; the image form is not measured (TechnicalDebt).
agc_driver_image_store_sint now stores D16 data with DMASK 0xf and 0x5
to all eight SINT formats and after an image_atomic_add on 32_SINT,
instead of expecting the refusal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves#4943's draw relocation against this branch: tagVariantRuns, which now builds a draw variant's runs and words, also records the captured words it shares with a table's supplied blocks, per region as before. A variant whose runs relocation splits keeps sharing for the draw cache's word compare, and skips the vouched validation, which already requires one run per region.
Moves a few include lines and the ResolveImageTablesShared declaration by a line or two, so they no longer share an insertion point with other open pull requests. No behaviour change.
Brings the series onto main 4e9a2a871 (#1193, #1201, #1202, #6060,
#6333 and the descriptor word tracking of 682b07eb0).
Resolutions:
- ShaderMemory: the column memo's supplied and spilled blocks now sit
on #1193's page vector, page pool and word masks. Pages are found by
binary search, since a spilled block can add many pages, and a block
placed into a lazily read page fills the rest of the 16-word blocks it
touches, so a later lazy read never overwrites placed words.
- ResourceMaterializer and SharedValues: the selected-record path and
the words index carry availableDwordMask, as the snapshot hash does.
A selected record's dwords past the end of its buffer count as read
zeros, which is what a scalar buffer load returns there, so a key past
the heap still selects a null image without reading the heap.
- DescriptorCache: kept table sets live beside #1201's recycled sets.
- Tests and CMake: both sides' tests are kept.
Every unwritten reuse handed the column's block to ShaderMemory::supplyWords, which checked the
block's runs (each in order, aligned, not overlapping, adding up to its words) and asked
GuestMemory::Accessible once per group of pages: one pass over 32,768 runs per capture for
Village's heap, about 200 us in the Village-like bench.
The column memo checks the runs once when it keeps them and marks the block ordered, so a supply
takes the block's span and word count as they are. Accessibility is asked once for the span's
pages; only when that fails (a page between records unmapped, which a stride wider than a page
can leave) are the runs' pages asked group by group as before.
The supply drops to about 3 us. The table vouch test checks that a strided table is kept as one
ordered block over its records' span.
A cache hit compared every captured region against guest memory, the table's words included:
about 1 MiB and 32,768 regions for Village's heap, 77-81 us per compare. When a capture shared
a column's words by reference (the memo's block), the variant records that block for those
words, and an unwritten block now vouches for them instead: one ShaderMemory::Vouched call per
block (the layout, no pending write, the flush hook, collect and UnchangedSince over the block's
span), then validateCaptured skips the regions the block holds. Regions noted pending are always
compared, and so is everything when the block is gone or refused.
The block keeps its stamp in a shared cell instead of the memo, so a compare that finds the
column unchanged after a store elsewhere in its span renews the stamp for every variant sharing
the block, not just for the thread that compared it. The cell is a sequence lock: a reader that
races a renewal gets no stamp and compares.
APS5_NO_VOUCHED_VALIDATION=1 turns the vouch off. ShaderMemory::Vouches counts vouched and
refused blocks and the words they covered.
Tests: VouchSharedRegions over a strided table (only the table's regions, one block, one call),
refused for a pending GPU write into a record or between records, a changed mapping, CPU, driver
and GPU-result stores, a store between records until a compare renews the stamp, and a released
block; and agc_driver_image_table_vouched_lookups, which dispatches over a write-watched heap
(APS5_WRITE_WATCH_IMPORTS=watch, as on Windows) and checks that hits are vouched for and that a
CPU store, a CP WRITE_DATA pending ahead of the dispatch and a driver store each refuse the vouch
and capture the table again.
An unwritten column was vouched for run by run: ShaderMemory::unchangedSince, with its pending
write query, flush hook, collect and tracker lock, once per contiguous run of the words the
column read. A heap whose records are wider than the descriptor they hold, like Resident Evil
Village's T#s of 8 dwords at a 48-byte stride, makes every record its own run: 32,768 calls per
reuse there. Each capture also took a new stamp before it looked at the memo, so a hit paid the
pending query, flush and collect twice.
The memo's words now carry the range their runs span, and a reuse vouches for that range in one
call. The range covers more than the runs (the bytes between records), so a store between two
records refuses the vouch and the column is compared instead: still exact, never stale. A hit
is tried before the capture stamps the column; the stamp is taken only when the memo cannot be
reused unwritten. ShaderMemory::Vouches counts the vouch calls.
The test reads a strided table of 4,096 records and checks that its reuse vouches in one call,
that a store between records compares and keeps the column, and that a store into a record
reads it again.
A result memo miss resolves every image table of the snapshot again: ResolveImageTables decodes each
distinct T#, builds the element and poison indices and fills one map record per key, which is most
of a miss for a large table. A miss whose table did not change is common: another buffer, flat SRT
word or user data word changes the result key while the column is the one the column memo shares.
ResolveImageTablesShared keeps the last resolutions and hands one out by reference when the variant
(its id), the reducing state of the direct samplers paired with table images and every column match:
base, size, records, keys, fault, outside, and the column's codes and distinct T#s by identity. The
memo holds those shared values, so a block it compares cannot be freed and reused for other words;
a column read again is a new block and resolves again even when its words are equal.
It keeps at most 4 resolutions per variant and 256 in all within 8 Mi words, counting the columns
their keys keep alive, least recently used first out, under one mutex. A resolution is immutable
once made and is shared between threads read-only. APS5_NO_TABLE_RESOLUTION_REUSE=1 turns it off.
ImageTableReuseCounts gains the resolutions made and the ones shared.
When the write tracker vouched for an unwritten image table column, the capture still copied each of
its words into ShaderMemory's pages (supplyWords) so they would be in the captured regions, and
Regions() then found them again bit by bit. On a 32,768-record column that is a 1 MiB copy per
capture (about 120 us here) and a scan of 262,144 read bits per Regions() call (about 230 us), and
the draw path calls TakeRecentRegions() and Regions() once each per stage.
The memo now keeps a column's words in one immutable block (SrtGuestWords), and supplyWords hands
that block over. ShaderMemory keeps a reference and reports its runs page by page, the regions a
read of the same words reports, merged in address order with the pages and the initial regions; a
second stage that supplies the same block only marks it recent again. The words reach pages only
when they must: a later read in the same ShaderMemory that touches them first moves the whole block
into its pages (spilled), so a word is never reported twice, and a block that overlaps pages already
fetched or another block's runs is copied as before. A spilled block stays referenced until the
ShaderMemory goes, so the regions already taken from it stay valid. ShaderMemory::Supplies() counts
the words shared, copied and spilled.
A capture that reused a memoized image table column still copied it: each record's state and code
and the distinct T#s went from the memo into the snapshot, the snapshot rebuilt its codes from the
states, and the column's per-record vectors were allocated before the memo was even looked up. Every
Recompile then hashed the codes and the distinct T#s word by word for the result memo. On a
32,768-record column of distinct T#s that was about 155 us of copying per capture and 410 us of
hashing per Recompile here, for a column nothing wrote.
A column is now built once into immutable shared values (SharedValues: its codes, its distinct T#s
and its read ranges) and the memo keeps that column. A capture that reuses it takes a reference, so
two captures of an unchanged table share one column and copy nothing, and the per-record vectors
are allocated only when the column is walked. The codes index the column's own distinct T#s
(ImageTableColumnSnapshot::words) instead of a list shared by the shader's columns, so every column
of a shader is shared whole, not only the first. A column read again after a write is a new one:
the snapshots holding the old one keep it unchanged. The result memo hashes each shared value once
and keeps the hash in it; equal contents hash equal whether they are shared or not.
ImageTableReuseCounts gains the reuses that shared a column and the columns built.
A static, read-only image slot whose T# has TYPE 1 is now bound as a null
image, like TYPE 0 since #5473 and #4779. On gfx1010, image_sample_lz,
image_sample_b, image_load and image_get_resinfo through a format-22
TYPE 1 T# return zeros without a fault, and gfx1035 reads it as zeros
too. Written, atomic and table slots still throw, and the reserved
types 2-7 stay refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since 682b07eb0, an image whose dword 3 wasn't captured is poisoned
(OutsideSnapshot). A record selected past the end of its heap reads
nothing there, so it was poisoned instead of selecting a null image, and
agc_shader_bindless and agc_driver_image_table failed after the merge of
main. A scalar buffer load returns zeros past the end of the buffer, so
those dwords now count as read zeros, and the dwords the selection reads
count as captured. Every other path keeps main's rule.
The per-draw capture of a shader's memory (the SRT walk and the guest pages it reads) cost about
15 us per draw packet in GTA V Performance mode; it now costs about 6.6.
- The SRT evaluator memoizes per plan value in a thread's reused, stamped array instead of a hash
table allocated and grown on every walk (evaluate 5.3 -> 3.6 us per capture).
- ExtractPlan also stores the plan's values as 32-byte records with argument ids, and the walk
evaluates that compact copy: the IR walk spent most of its time in cache misses (3.6 -> 2.7 us).
The evaluator is a template over the node view, so both walks share every opcode's semantics, memo,
cycle and failure rules. A plan that cannot be expressed compactly keeps the IR walk.
APS5_VERIFY_SRT_COMPACT=1 walks both and throws on any difference.
- ShaderMemory's valid/read/recent marks are arrays of 64-bit words scanned with countr_zero instead
of bitsets tested bit by bit (about 1 us per draw packet).
- Captured pages come from a per-thread pool, are found through a short vector with a last-page
shortcut, and a page no recorded GPU work writes gets the flush hook a whole-page read would make,
then has each 64-byte block copied when a word of it is first read (about 0.3 us per packet).
Pages with pending GPU writes and unmapped pages are read as before.
- APS5_PROFILE_DRAW splits the capture time into SRT evaluation and materialization, and prints the
[capture] line once per 10 s instead of every 500 captures (it cost about 2 us per capture).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Gjon Gojcevic <7807870+gjonn@users.noreply.github.com>
A kept column the runtime cannot vouch for is read and compared with its kept words. When they
differ, the fresh walk read the same runs again, so a capture that follows every change to a table
read each word twice: 1,048,576 words per capture for a 65,536-record column, 12% slower than
before the reuse with the write watch and 16% slower outside it.
The compare now leaves its words with the snapshotter, and the fresh walk takes them instead of
reading when its plan has the same runs: words the capture has already read read the same again.
Such a capture reads each word once, as before the reuse, and the 65,536-record column costs 7%
(12% outside the write watch) more than before the reuse; the rest is keeping the column. With 16
distinct T#s it costs 5 to 8% less. agc_shader_table_snapshot_reuse expects a changed table to be
read exactly once per capture.
Every capture of a shader with an image table read every record of each column and looked each
record up among the snapshot's distinct descriptors. A 32,768-record column of distinct T#s cost
6.1 ms per capture here (2.3 ms with 16 distinct T#s), and Resident Evil Village's title captures
such columns on nearly every draw.
TableSnapshotter now keeps the last 16 column snapshots per thread (at most 4 Mi words in all,
least recently used first out), keyed by heap base and size, record shape and key count: the runs
it read, their words, each record's state and code, and the distinct T#s. A narrowed, selected or
pointer-indirect column, or one with an unmapped record, is not kept. A capture of a kept column
takes the codes and the distinct T#s from it with no per-record work. When they are the snapshot's
first words they are taken as they are, and the words index is built only if a later column needs
it, so the result is the snapshot a fresh walk would produce, codes and words order included.
The runtime gets three optional callbacks: stampWrites gives a stamp for a range before any of it is
read, unchangedSince tells whether nothing wrote the range since, and supplyWords hands words that
equal guest memory to the capture as read, so they stay in the captured regions the draw cache
compares. ShaderMemory implements them with the write tracker. The stamp is CollectWrites's
generation, refused when this ShaderMemory already fetched a page of the range or recorded GPU work
writes it. A range is unchanged while no block of it is stamped after the stamp (collected CPU
stores, the driver's Write, WriteChanged and StoreOwnBytes, MarkWritten for GPU writes into
imports), no recorded GPU work writes it and the guest allocation generation is the stamp's. When
every run is vouched for, the column is not read at all; otherwise its runs are read and the
snapshot is reused only when they are equal word for word.
The 32,768-record column now costs 0.48 ms per capture with no word read when unwritten, and about
0.9 ms outside the write watch. APS5_NO_TABLE_SNAPSHOT_REUSE=1 turns the reuse off. The [image-table]
profile line counts the reuses, and ResourceMaterializer::TableReuse returns them with the words the
snapshotter read.
The [image-table] line under APS5_PROFILE_DRAW now splits the skipped
narrowings by reason: a value key that cannot be evaluated, a key
descriptor that is not a buffer, a key buffer with more positions than
APS5_IMAGE_TABLE_KEY_SCAN (with the most positions seen), pending GPU
writes over the key buffer, and an unmapped key page. Selected records
never narrow, so the line counts them apart.
Since #757, a T# loaded from a buffer heap at a runtime offset is an
image table column, and every capture snapshots the whole heap, up to
65,536 records, even when user data or an SRT read fixes the offset.
When the T#'s eight dwords and its offset are runtime values the SRT
walker evaluates, the column is now selected: the offset is kept as a
descriptor source, the shader indexes entry 0, and the snapshot reads
only the record at the evaluated offset, with the heap base moved to
that record.
The column keeps a table's semantics. An invalid T#, an unmapped record
or an offset that carries past 32 bits is poison that faults only when a
lane selects it; an offset the walker cannot read faults the table
instead of throwing; the record's mode is picked per entry, so no
pipeline variant is added; and a dispatch that writes the record faults
it as written by its dispatch. Dwords past the heap read zeros, as the
scalar loads would, and a record past the key cap is read rather than
poisoned.
A column stays a whole table when its offset is chosen per lane, or when
the shader may write the memory the offset is read from at run time: the
buffer it is read from (by V#), or any memory when the read or a write
goes through a pointer or a GPU-only descriptor. Samplers are unchanged:
an S# the walker evaluates is already a direct sampler. The plan stores
the selection, so the disk cache format is 36.
Five agc_shader_memory and agc_shader_bindless checks asserted the old
planning and now assert the selected record:
- "image table SRT key": a key read through the SRT narrowed a 4-key
table to records 0 and 2. It now selects record 2 alone, captures no
other record, and a changed key moves the capture without changing the
module or its disk key.
- "image table key union" and "image table late key": two keys of one
heap shared one column with two value key domains. Each key now
selects its own record, as two one-record tables.
- "image table unkeyed access": a read keyed through the SRT shared a
whole column with a read keyed per lane. The first now selects its
record; the second keeps the whole, unnarrowed table.
- "uniform table image": the T# at a user data offset was one key of a
full table. It is now the one selected record, and the record the user
data does not select is not read.
A value key domain only arises for a key the walker evaluates, so it no
longer narrows an image column; key buffer domains (keys loaded per
lane) still do.
Adrià Franch