HotSpot makes its stack guard zones with an RW protect of the lowest
pages of the thread's stack, then PROT_NONE, as it commits them on
native Windows (os::pd_create_stack_guard_pages, then os::guard_memory).
Below a host stack's guard page such a protect was only recorded, so the
pages stayed reserved: they could not be written after the RW protect,
and the zones never held.
A protect of reserved pages that starts at the bottom of the reserve and
stops short of the guard page now commits and protects them. The guard
page and the reserve above the range stay as they are, so the stack still
grows through the guard down to it, and the stack limit in the TEB still
follows the committed stack.
Windows moves the guard down by committing the pages below it as
read/write guard pages, committed or not. The zones therefore hold until
the stack is within one guard region of them (three pages on GitHub's
runners), and the pages the guard reaches then come in read/write. The
test checks both, and the debt entry says so.
HotSpot makes its stack guard zones this way: an RW protect of the
lowest 48 KiB of the thread's stack, then PROT_NONE. Both calls succeed,
the guard page stays above them, and the stack still grows through it.
Every case now also checks that the stack limit in the TEB follows the
committed stack.
Committing the reserved pages of a host thread's stack before protecting
it, as the stack branch did for a range the arena doesn't own, also
cleared the guard page's PAGE_GUARD. The thread was left with no guard
and no reserve: an overflow ran into whatever lay below the stack instead
of raising a stack overflow, and the stack limit in the thread's TEB,
which Windows moves only when the guard page is hit, stopped following
the stack. A range that began inside the reserve also left the reserved
pages below it with no guard above them, so the stack could no longer
grow into them.
The stack branch now leaves the guard page and the reserved pages below
it as they are, and commits and protects only what lies above them;
reserved pages above the guard are still committed first. The protection
is recorded for the whole range, and a range of only reserved and guard
pages changes nothing on the host. The test protects the main thread's
whole stack and a reserved thread's whole stack and upper part, then
checks that the guard page is still there and that the stack grows
through it, also after a PROT_NONE protect of its lowest page. The debt
file records that those pages come in read/write as the stack grows.
Guest threads run on host stacks that Windows reserves (0x210000 bytes
by default) and commits on demand. Resident Evil Village protects one
thread's whole host stack with 0xc2 at startup while its lowest 0xd000
bytes are still only reserved, so the Windows mprotect emulation threw.
When such a stack sat inside the arena bounds as a host region, the call
took the arena commit path and failed to split a placeholder that was
never one.
Commit the reserved pages of a host range before protecting it, and only
use the arena commit for ranges the arena owns.
The Windows path from #3788 changed the protection of any committed private host region outside the guest registry, including a range crossing a guest stack's lower bound. Ranges inside a live guest thread stack keep the path from #2982.
A stack base need not be page aligned (it is not on Linux), so rounding the
range down first could leave the stack and miss the stack path. The range the
guest asked for is now checked against the guest stack, and the rounded
pages are protected, as FreeBSD rounds the address down.
A thread that exits with part of its stack protected left those pages to the
next thread that reused the stack. Protected stack pages are recorded, and a
thread's stack is made readable and writable again when it is unregistered at
exit.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows a guest thread runs on its host thread's stack, which is not a
registered guest allocation, so mprotect on any of its pages failed with
EINVAL. A guest that guards the bottom of its own stack (Dolphin's JIT
read-protects 64 KiB there to catch stack overflows) got an error where the
PS5 changes the protection.
A range that lies inside a live guest thread stack, with no GPU access
requested, now takes the native protection directly. Ranges that cross the
stack's bounds are still rejected.
AI-assisted (Claude Code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Yunus Shaikh