Since host imports group adjacent ranges into aligned blocks (#6457), an
import can span both 64 KiB blocks that an earlier import left unwatched
and the metadata of resident multisampled DCC targets.
ImportWatched with staysWatched took the whole import out of the write
watch when any part of it was not watched before. It now walks and
compares each run of watched blocks of the import on its own, and
unwatches only the blocks that were not watched or whose walk failed.
A resident DCC target took the generation of its metadata from a walk
of the metadata's pages alone. CPU stores to the rest of the metadata's
64 KiB block made before that walk were stamped later by the import's
walk, so the block read as written, and the import window marked the
metadata page as possibly stored over: the next draw refused the
target. The generations now come from a walk of the whole blocks.
agc_write_tracking and agc_driver_msaa_dcc_clear_* gain regression
tests for both.
Fixes#8103
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A MarkWritten whose fresh collect fails (a no-access or unwatchable
resident page inside the 64 KiB block span) took the span out of the
write watch without running the unwatch hook, so pending storage-image
results over it kept no saved guest bytes and their write-back refused:
storing storage image results pending over memory outside the write
watch without the guest bytes saved before them is not implemented
(#7017, Palworld PPSA23126 among others).
The hook now runs over the block span before the blocks leave the
watch, as Unwatch and the import fallback do, with the tracker lock
released for it like #6308 does for imports. The debt entry drops the
failed collect from the unsaved cases.
The check makes a collect fail over a no-access arena page and requires
the hook to have covered the block; it needs the WindowsMappings
protection failure, so it runs in the Windows-only section of the write
tracking test.
GuestMemory::Rewatch takes ranges that left the write watch back into
it. Only whole tracker blocks whose pages can be collected return:
their write-watch marks are collected and dropped, and every block is
stamped as a CPU store with a new generation, so a snapshot or a proof
made before the block left the watch reads as changed, and the blocks'
untracked hints stop applying. It does nothing outside Windows.
Since #6060 a settled Windows host import of private write-watched
memory stays watched, and its bytes are vouched by the write tracker's
stamps instead of compared. A GPU write that runs from such an import
into a neighbouring import that left the watch reaches the tracker as
one MarkWritten over a partly watched range. The test imports two
blocks that stay watched next to one that leaves the watch, and checks
that a driver store and an own store across the boundary stamp the
watched block they wrote, are seen over exactly its stored bytes, and
leave the other watched block unchanged.
stampWatchedPart stamped a block when the store's part of it started or
ended in watched memory. That is exact on Windows, where the watch
excludes whole 64 KiB blocks and the arena's host regions are 64 KiB
holes. On Linux and macOS the watch unregisters page ranges, so a
block can be watched only between two unwatched ranges, and a store
across all three left its watched pages with their old stamps.
Off Windows the block is now stamped when the store's part of it
overlaps any watched page (GuestWriteWatchOverlaps). A store that
overlaps no watched page at all returns after that one lookup, where
it made two coverage lookups per block before.
The test watches only the middle pages of a block: a driver store and
an own store around them must stamp it, a store into an unwatched end
must not.
GuestWriteWatchCovers says whether a range is watched in full. The
Linux and macOS watches unregister page ranges, so a range can hold
watched pages between unwatched ones, and code that must act on every
watched byte of a store has had no way to find them short of asking
page by page. GuestWriteWatchOverlaps answers with one lookup in the
registered ranges, like Covers.
The macOS write-watch test and agc_write_tracking on Linux check it
around watched pages between unwatched ones, at both edges and for an
empty range.
A driver store and an own store across a watched block and an unwatched
one must stamp the watched block and no other. On Linux, a store across
the halves of a split block must stamp it, and a store into its unwatched
half alone must not.
UnchangedSince and UnchangedSinceAll answered by the stamps of the 64 KiB blocks a range
touches, so a CPU or driver store anywhere in a shared block failed every unchanged proof
over it. When a block's stamp is newer, they now look at what wrote the block: the CPU page
stamps over the range and the block's driver-store pieces, as StoredOver does.
For that answer to see every driver stamp, the walk after an own store stamps the pages it
finds dirty in a page map of their own. The unchanged proofs count those pages, so a guest
store beside the stored bytes in the same page is not hidden. StoredOver counts them only in
a block without driver pieces: where pieces exist it answers as before. They take no
driver-piece slot and do not count for the complete coverage StoredAllSince proves. On
Windows, a private mapping made over a block that has driver pieces marks them as
forgotten instead of dropping them.
StoredOver no longer answers stored over for a block whose newer stamp comes only from CPU
stores to other pages.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second queue that settles imports is always created on Windows, and a device without one keeps
the comparisons. The other-family settle path is tested through the recorder test device built
with its settle queue on another family (agc_import_watch_other_family) instead of
APS5_SETTLE_OTHER_FAMILY.
The settle read keeps a private import's window open for one round trip on the settle queue, so a
store from another thread can land inside it. CheckImportWindowOtherThread: a store of new bytes from
another thread within the window counts as written, and a same-value one keeps its possible CPU
touch. The debt entry notes the longer window.
An import that leaves its range watched copies the whole range first, so
that each page the import resolves can be compared with its bytes when
the window closes (#3874). Ghost of Tsushima (PPSA02225) maps and
imports 4.9 GB of direct memory at start-up, and that copy is the
largest allocation the title makes.
Pages that are all zero are no longer copied. Each page of the range
gets a slot, the index of its copy, or none for a page that was all
zero, and such a page is compared with a zero page. Every page is
compared with the same bytes as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three ways to break the untracked hint passed every test:
- hinted() without its coverage check walks, and resets, the marks of
blocks that fresh backing put back under the watch;
- walkHints stamping the walk's first block instead of the marked page's
block hides a store from an image over another block;
- compareUntracked without NewestStamp in a hinted key misses a GPU write
into an import, which sets no mark and is only recorded by MarkWritten.
CheckUntrackedHint now walks the whole range after a store into its
second block and reads each block's hint on its own. It also recommits
fresh backing over the hinted range: the range must have no hint, and a
CPU store must still reach its collect. RunHintedImport hides a store's
mark and records it with MarkWritten, and the image must still be
uploaded at the next epoch. Each of the three changes above now fails
agc_write_tracking or agc_driver_dcc_keys_first_write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows every direct host import leaves the write watch, and a storage
image over it compares its whole surface with its guest-byte copy at every
collect epoch. #4432 measured 7-8.6 s of every 10 s in those compares on a
GTA V scene, with 98% of the compared surfaces unchanged.
Tracker blocks that leave the watch now keep their write-watch marks as a
hint. Leaving the watch gives a block a fresh hint stamp, and
UntrackedHint walks a range of such blocks, at most once per collect epoch
and thread, stamping the blocks whose pages a CPU store marked; it
returns the newest stamp of the range. compareUntracked puts that stamp in
its memo key in place of the collect epoch, so an image is compared again
after a mark in its range, a driver store, a mapping change or a recorded
GPU write. The byte compare still decides whether anything changed: a mark
the driver makes without a CPU store costs one compare and drops nothing.
Nothing else reads the hint. Collects, Watched, UnchangedSince,
ChangedBlocks and NewestStamp still treat those blocks as untracked, and
Linux is unchanged.
agc_driver_dcc_keys_first_write refreshes an imported storage image over
50 collect epochs with no store: it compared the image 50 times before and
0 times now. A CPU store is still uploaded on the next refresh, and a store
of the bytes already there costs one compare and keeps the pending
results. agc_write_tracking checks the hint itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sampled texture over a partially resident surface (a texture array whose
heap the guest commits on demand) never got a write generation: CollectWrites
fails on a range with reserved pages, so every cache hit compared the whole
committed part with its snapshot. CollectCommittedWrites collects the
committed runs one by one and returns the runs with the generation; the cache
entry records them, and a hit skips the compare only when the committed runs
are the same and UnchangedCommittedSince holds over them. A commit, decommit,
view mapping or unmapping inside the surface changes the runs or stamps the
new pages, so it is compared as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second import-window case of agc_write_tracking makes one page
unreadable with VirtualProtect or mprotect and expects an import that
stays watched to throw before it copies its range. On Windows CheckRange
answers from its table of committed arena pages, which is kept current
by the registry's invalidator. The first case had checked the same
allocation, so the table still said the page was readable, the check
passed and the copy faulted on the page.
ProtectPage now reports the change through GuestAllocationsInvalidate,
as the registry does for a guest protection change and as
tests/BdaResources.cpp does after VirtualProtect.
A host import that leaves its range watched can resolve the write
protection of the imported pages without storing anything, as NVIDIA's
Linux driver does for every page. ImportWatched stamped those pages as a
possible CPU store (BlockMaybeWritten), and only a storage image tracked
by 64 KiB unit could clear the stamp, by comparing with a copy of the
guest bytes it was uploaded from. Since #3372 a fast-cleared upload
therefore reads its whole surface from guest memory wherever such an
import is possible, although nothing but that compare uses the copy.
ImportWatched now takes whether the import leaves its range watched. If
it does, the range is copied before the stores made so far are
collected, and after the import each page whose write protection was
resolved is compared with that copy. A page with other bytes is stamped
as a CPU store. A page with the same bytes keeps only the CPU-side stamp
the label rules read, so UnchangedSince holds for it for every consumer:
storage images tracked by layer, snapshots and caches as well.
If it does not (a host-pointer import on a device whose imports are
compared, a direct import on Windows), nothing is copied or compared:
the pages resolved in the window get the CPU-side stamp and the range
leaves the watch before the tracker is unlocked. Where host-pointer and
dma-buf imports differ in this, the dma-buf import that follows a failed
host-pointer import is a window of its own, so each window knows its
kind before it starts.
BlockMaybeWritten, the written-stamp table, WatchedImportsStampPages and
the unit compare in StorageTexture::Refresh are gone. The fast-clear
upload skips the guest-byte copy whenever its range is watched, as
before #3372.
On Windows the guest arena is reserved while libc loads, around whatever
the process has already mapped in its interval. About one start in forty
has nothing mapped there, so the arena has no host hole:
agc_driver_graphics then fails with "write-watched arena contained no
host exclusion to test" and agc_write_tracking prints that its host-hole
checks were skipped.
agc_write_tracking now starts itself suspended, reserves one allocation
granule in the middle of the child's arena interval before the child's
loader runs, and the child runs the host-hole checks on that range. The
range is a host hole on every start, so the checks no longer depend on
the address layout. The four queries agc_driver_graphics made on an
enumerated hole are among them and are removed there.
The memo stamp held TrackerGeneration, which every collect and driver store
advances, so it almost never matched. Driver stores into uncovered ranges
record their generation per 64 KiB block, and NewestStamp of the surface
replaces TrackerGeneration in the stamp.
On Windows agc_write_tracking has failed since #3594 with
write tracking test failed: an unwatched range reads as not stored over
from the check in CheckSharedBlock:
if (!Watched(outside, unwatched.size())) Require(StoredOver(outside, unwatched.size(), TrackerGeneration()), "an unwatched range reads as not stored over");
The host stack array lies in a host hole of the guest arena. #3594 made
Watched, collectWrites, UnchangedSince, UnchangedSinceAll and storeOwn
reject ranges overlapping such holes, but StoredOver, MarkWritten,
WrittenSince, ChangedBlocks and ImportWatched still accepted them through
WriteTracker::covers(), so StoredOver proved a range not stored over while
Watched called it unwatched (#4087).
The exclusion now lives in WriteTracker::covers() on Windows, which every
query already asks, and the per-function checks are removed. Host holes
are padded to the allocation granularity and the tracker blocks are 64 KiB
from the arena base, so no block is shared between a hole and arena
memory, and ranges outside the holes get the same answers as before.
A new case enumerates the arena for a host hole and checks every query
against it; on the previous code StoredOver, MarkWritten, StoreOwnBytes,
WrittenSince and ChangedBlocks fail it.
Fixes#4087
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Scan version stamps within contiguous leaves. On Linux, collect nearby
pages within queue epochs only after confirming full watched coverage.
Keep the requested range on Windows. Select the range before collection
and do not retry a failed walk.
Test sparse-leaf boundaries, write histories, partial unwatches and epochs.
Fedora 44 Release tests pass with default, disabled and shared caching.
Isolated upstream comparison reduces the saved collect replay by 11-12%.
Windows was not tested.
AI-assisted: yes.
StorageTexture::writeBackLayers keeps the CPU's bytes for an edge unit when StoredOver
reports that the CPU wrote it since the unit's generation. That check read one generation
per 64 KiB block, so a store to another allocation sharing the block counted as a store
over this surface and the whole block was kept, discarding GPU results written into it.
The driver path in the same check already compares range pieces; the CPU path now does
the same. Each collect records the generation of every page it found dirty, and StoredOver
tests only the pages its range overlaps. A block with no page record keeps the previous
whole-block answer.
Measured on an RTX 3080 Ti, main at df16c4c2:
- agc_write_tracking stores at base + Block + 8, outside the queried range but inside the
shared block. That case fails on main ("a CPU store outside the shared range counts for
it") and passes here.
- Full suite 408 of 408 (Linux, CachyOS, GCC, Release, BUILD_TESTING=ON).
This does not cover the other half of #865: the report's finding 2, where the stamps fall
on the surface's own pages and the bytes under them are unchanged, still keeps the block.
No title was run here, so the fix is offered for the report's own repro.
A store the driver makes with the CPU into watched guest memory (a
GuestMemory::Write, a changed-bytes write-back, a completion action's label
store, the copy HLE's CPU path) faults its pages dirty in the write watch and
stamps the range with MarkWritten. The faults stayed behind: collects are
memoized per packet, so the next walk over those pages came later and stamped
them again, as a CPU write made after every check in between.
PPSA21564 fast-clears its normal G-buffer every frame with a register DCC key
fill on the GPU. While that fill is still pending, the first draw into the
target materializes the clear with a CPU store of the clear texels and a
refresh of the resident image. A dozen draws later a walk found the store's
faults, the target read as changed, and the image was uploaded again from
guest memory, dropping the draws recorded in between. Lighting then sampled
cleared normals and the level's moon and rocks went unlit in about half the
frames.
The driver's stores now consume those faults themselves: a walk before the
store stamps earlier CPU writes as such, a walk after it takes the store's
own faults as a driver store, and the range is stamped as before.
On Windows the arena's range also holds host regions that were already in
use when it was reserved (heaps, thread stacks). They are not write-watched,
so a store into one is stamped as before, without a walk.
Write stamps are per 64 KiB block, so a write-back of one surface stamps the
whole block it shares with the surface packed right after it. A pending
image of that neighbour read the stamp as a store over its own bytes: its
write-back kept the guest bytes of the shared block in place of its results
and its next refresh re-uploaded them, stale. PPSA21564's half-resolution
fog texture (960x540 RGBA16F, linear, starting 0xb800 bytes before a block
end right after another 960x540 surface) lost its first 6 rows that way,
which the fog composite upsampled into an orange band across the top 12
rows of the scene in one of its cinematics.
The tracker now remembers the byte ranges of the driver's own stores
(MarkWritten) in the blocks they cover only in part, and StoredOver says
whether a store reached a given range since a generation (a CPU store still
counts for its whole block). The write-back keep rule and the refresh drop
rule ask it for the blocks a surface covers only in part.
AnyPS5 Contributors