Under -DANYPS5_ENABLE_ASAN=ON with ASAN_OPTIONS=detect_leaks=0, 14 of
the 849 tests outside the AGC driver, shader and VideoOut suites failed
on main for reasons that come from the ASan build itself, not from the
code they test:
- libc copies the main executable's file-backed writable pages before
remapping them anonymously. In an instrumented test executable those
pages hold ASan's redzones around globals, so the memcpy reported a
global-buffer-overflow at load (patched_libraries,
guest_memory_lifetime). The ASan build copies the pages word by word
without instrumentation.
- Five tests check that a local variable is on the thread's stack.
With detect_stack_use_after_return, which is on by default on Linux,
instrumented locals live on ASan's fake stack. These tests now run
with it off. guest_kernel_is_stack also bounded the worker stack by
the application area without the ASan build's arena offset.
- The relinker module tests and loadexec_arguments load instrumented
libraries into a Python process or a relinked guest, so the ASan
runtime is not first in the initial library list. They now run with
verify_asan_link_order=0. Preloading libasan instead put its
__cxa_throw interceptor ahead of libc.prx's and broke the unwind of
an exception that LoadExec throws and catches.
- guest_allocator_failures limits RLIMIT_DATA to 128 MiB and then
commits 300 GiB. ASan maps 37.5 GiB of shadow for that commit, which
the limit refuses, and aborts. The ASan build skips that section.
- fiber checks the address of a local against the fiber's context too,
so it runs without fake stacks as well. Its CheckActiveContext then
reads the fiber stack word by word above that local, through the
redzones of the frames on that stack, and is no longer instrumented.
- agc_multisample_guest_access passed aligned_alloc a size that is not
a multiple of the alignment, which C11 does not allow and ASan
rejects. It now rounds the size up.
add_asan_test_options appends to ASAN_OPTIONS for one test in an ASan
build and does nothing otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unreal Engine walks the frame-pointer chain until it reads a return
address of 0 and never checks the frame pointer itself. Guest threads
started with whatever rbp the host code held, so the walk ran into host
start-up frames (glibc's saved rbp 0x21 in a LEGO Batman core dump,
address 8 in Returnal on Windows) and faulted. Fibers ended their chain
with a null frame, and makecontext started its function with rbp 0, so
the same walker read address 8 there too.
Guest threads now start through a thunk that points rbp at a
{saved rbp 0, return address 0} record, as the relinker's entry stub
does for the main thread; the fiber trampolines push the same record,
and the makecontext start function is built with frame pointers so its
frame, over the zero return address makecontext stores, ends the chain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since 794714983 every fiber ran on an mmap'd host stack so that host calls
cannot overrun the small contexts PPSA01341 packs next to each other. A
title that derives state from the stack pointer then reads the wrong
memory: PPSA05684's job fibers have 512 KiB contexts at 1 MiB-aligned
addresses and find their job record by masking rsp with ~0xfffff, so the
job system lost track of a GameInit job and the game waited forever
right after loading big4r/actor76.txt. Its 64 KiB fibers fault later on
a host stack too.
A fiber now gets a host stack only when its context is smaller than
64 KiB; a larger context is the stack again, as on the console. The
limit sits between the 16 KiB contexts that need protection and the
64 KiB ones that need the guest stack (technical debt entry). The fiber
test runs a 512 KiB fiber and checks that its stack is in its context;
it fails without this change.
AI-assisted: yes
Aps5FiberMain kept the fiber's entry address in rbp when it called the
entry, so the guest entry saved that code address as the caller's frame.
A guest walking frame pointers from inside a fiber read machine code as
a frame record and faulted. libSceFiber is now built with frame
pointers: Aps5FiberMain sets up a frame whose saved rbp is the zero of
the initial fiber frame, and the chain ends there.
#4265 gives each fiber an mmap'd host stack on Linux and leaves Windows
as it was: there a fiber still runs host calls inside the context the
title passed to sceFiberInitialize. Demon's Souls packs its 16 KiB job
fiber contexts back to back. A job that calls
sceAgcCreateInterpolantMapping reaches shader preparation on the fiber
(Driver::ResolveShaderAbi, PrepareRegistered, the RDNA decoder,
BuildShaderStageInputInfo), goes more than 16 KiB deep and overwrites
the frames at the top of the context below it. On the loading screen
after the opening cinematic the fiber that owns that context then
failed the title's stack check, or returned to 0x7fffffff.
On Windows each fiber now gets a host stack from VirtualAlloc, the
context size plus 1 MiB, committed like the 1 MiB host margin of guest
threads, with a PAGE_NOACCESS page at the bottom. While the fiber runs,
the TIB stack base and limit and the deallocation stack describe that
host stack, as Windows does for its own fibers, so stack probes and
GetCurrentThreadStackLimits see the stack the fiber runs on. A host
stack that cannot be allocated, protected or released throws instead
of falling back to the context.
agc_driver_gpu_map_on_fiber: GetCurrentThreadStackLimits on a fiber now
returns its host stack, which has room for the import, so on Windows a
GPU mapping made on a fiber imports on the fiber's stack instead of a
host thread. The fiber's context is still not written.
Test: on Windows, fiber checks that the TIB stack bounds inside a fiber
cover its host stack, at least the context plus 256 KiB. With #4265
alone it fails at "the thread's stack bounds are the fiber's host
stack", and without that check at "a host call from a 16 KiB fiber
writes nothing below its context"; with this change fiber and
agc_driver_gpu_map_on_fiber pass (Windows Server 2025, RTX 3060,
WinLibs GCC 15.2.0, Release).
Demon's Souls (PPSA01342) on that host: at the loading screen the two
16 KiB job fibers above the overwritten one now use 17,592 and 16,936
bytes of their host stacks, the title's stack check no longer fails
there, and the load goes on until a separate Vulkan allocation failure
(vkAllocateMemory of a 4 MiB host-visible buffer, out of device memory).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host code that a guest fiber calls runs on the fiber's stack. Demon's
Souls (PPSA01341) gives its job fibers 16 KiB contexts packed next to each
other, and a fiber that reaches shader preparation through
Driver::ResolveGraphicsStagesAbi goes about 21 KiB deep when SPIRV-Tools
validates the module. That overwrites the saved frame of the fiber below,
and the next switch to that fiber pops garbage into rbp, rbx, r12-r15 and
rip (a jump to 0x1a in the observed crash).
Fibers now run on an mmap'd host stack of the context size plus 1 MiB,
with a guard page at the bottom, the same margin guest threads get since
#2647. The guest context is no longer written. Context size checks fill
and scan the top context-size bytes of the host stack, so
size_context_margin keeps its meaning. If the mapping fails the fiber
runs in its context as before. Windows is unchanged: it pins and
publishes the guest context as the stack, and that path is not covered
here.
The fiber test now runs a 256 KiB host call from a 16 KiB fiber placed at
the top of a guard-filled area and checks nothing below the context was
written. It fails on main.
A title calls our exports on stacks it sized for the console. With lazy
binding the first call through a PLT slot runs the dynamic linker's symbol
lookup on that stack: on the 2 KiB fiber context Astro's Playroom uses it
wrote below the context into the neighbouring heap chunks.
sceFiberStartContextSizeCheck and sceFiberStopContextSizeCheck toggle a
mode in which new fibers get their context filled with a pattern, so
sceFiberGetInfo can report the unused stack margin.
sceFiberGetThreadFramePointerAddress returns the frame pointer of the
thread that called sceFiberRun.
Co-Authored-By: Claude <noreply@anthropic.com>
Dean