A 64-lane program that doesn't stop invocations reads one dword per lane at
0xfff00 + low over a table whose first range is [0x100000, 0x100010): one
lane below the first range, one past its end, one at its last dword and the
others at its start. The unmapped lanes must read zero with the unmapped
fault recorded, the others their dword, and every read kind must match the
anchored program.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver builds tables whose ranges ascend by begin but can overlap
(a per-build region read over the address space's ranges). A lookup
selects the last range beginning at or below an address, so the anchor's
range now ends at the next range's begin when that comes first, and the
driver requires only ascending begins (BdaAbi::RangesOrdered).
An anchor is also not used for a range longer than 2^62 bytes, whose
offset arithmetic could wrap with a positive immediate. Anchors are
counted per read site after adjacent loads are grouped and reads with
no used dwords are dropped, so a program does not search for one
effective read; read eligibility shares RoutesApertures.
Tests: the anchored test programs can replace the lookup path by a
sentinel, which shows which reads load directly; the anchor table has an
overlapping range; a recompiled compute program reads through an anchor
on the GPU.
AI-assisted: yes
Address-based reads whose 64-bit base is a pair of user-data registers
search the BDA table at every read, although the base is one value for
the whole draw. Where a program has two or more such reads of one base
(at most four bases, those with the most reads), the range holding the
base is now searched once at the program's entry, by the subgroup's
elected lane when the program already uses subgroup ballots. Each read
then takes one branch: when that range holds all of its bytes, and for
dword reads the device address is 4-aligned, it loads them directly;
otherwise it runs the lookup path as before, so data, faults, invocation
stops and wraparound are unchanged. Ranges ending above
0xffffffff00000000, whose base plus a 32-bit offset could wrap, and
ranges whose device addresses wrap are not used.
The direct path relies on the table being ascending and disjoint, which
the driver builds and now requires (BdaAbi::RangesOrdered).
AI-assisted: yes
EmitBdaWrite, used for buffer stores through GPU-selected descriptors, recorded an
Unaligned fault and dropped every 32-bit store whose address is not 4-byte aligned.
EmitBdaStore and the BDA reads already take the byte path for unaligned addresses, and
raw buffer accesses use the exact bytes, as RDNA does in unaligned mode. The unaligned
case now stores the four bytes one by one like the 8- and 16-bit stores of the same
function.
Sword of the Sea hit this about once per frame from one compute store, so four bytes
were lost and a fault was reported each time.
The BDA store execution table now also runs through EmitBdaWrite; its unaligned dword
cases failed before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shader store through the BDA table records its 4 KiB page in the fault
buffer, and the driver marks those pages written after the use. The page
number was truncated to 32 bits, so a store above 16 TiB marked a page at
another address. And the driver scanned all 4096 slots after every use,
even when nothing was stored.
Slots now hold a 128 KiB chunk (its number plus one, 32 bits for any
address below 512 TiB) and a mask of its 4 KiB pages. A shader that
claims a slot appends it to a list, with a count. The driver reads the
count, decodes only the claimed slots, marks each run of pages once and
clears only those slots. A writable BDA range past 512 TiB is refused
when the table is built.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A global_store_dwordx2/x3/x4 was split into one StoreAddressU32 per
dword, each with its own write-pointer lookup and its own 4-byte store.
Across a wave those are strided partial writes, which are very slow to
host memory: RADV's clear_htile_mask kernel (one dwordx4 load and store
per thread over 640 KiB) took ~9 ms. Multi-dword stores are now one
StoreAddressU32xN; the emitter probes the whole span once and, when it
is mapped writable and 4-byte aligned, stores it as one vector (16- or
8-byte aligned) or as consecutive dwords, noting each written page.
Anything else takes the per-dword path as before, so faults and
range-spanning stores behave as they did. APS5_BDA_DWORD_STORES=1
restores per-dword stores.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every BDA dword read site still emitted in the entry function the add
of its first dword address with the overflow fault, a probe_bda_pointer
call, the wide loads, and one read_bda_dword_bytes call per extracted
dword. NVIDIA's pipeline compile time grows with every call left in the
entry function, so large compute programs with a few hundred such reads
spend 47-77 s in vkCreateComputePipelines.
A read site is now one call to a DontInline read_bda_span function, in
a stop and a coherent variant, taking the address, the immediate offset,
the mask of extracted dwords and the pc, which does the same steps in
the same order. Fault reasons, addresses, pcs, Volatile, the stop
behaviour and the APS5_BDA_BYTE_READS path are unchanged.
A BDA dword read whose span the probe rejects falls back to four byte
lookups with a branch each, and every fault was recorded inline with a
compare-exchange and seven stores. Both were copied into every read
site and both wave64 lane halves, so one large compute program grows
to about a million SPIR-V words and spends about 95 s in
vkCreateComputePipelines.
Faults are now recorded by one DontInline function, record_bda_fault,
and the byte fallback of a dword is read_bda_dword_bytes, in a stop
and a coherent variant, returning the value and whether the invocation
stops. Read order, fault reasons, pcs, Volatile and the stop behaviour
are unchanged, and so is the probe path.
