A protect below the main thread's stack mapping mapped only the
protected pages. Linux keeps a growing stack one guard gap (1 MiB by
default) above a readable mapping, so after a read-only protect, or a
protect later restored to read/write, at the bottom of the reported
stack the stack could no longer grow into the megabyte above it and a
deep enough call faulted where the console has stack pages. The pages
between the protect and the top of the stack are now mapped as well,
so the stack never has to grow past them.
glibc reports the main thread's stack as reaching down to the end of
the mapping below it. The pages a protect maps under the main stack
become that mapping, so after the first protect sceKernelIsStack
reported a stack starting above them, and a second protect of the same
pages, such as restoring them to read/write, failed as outside any
stack. Each thread now queries its host stack once, which also stops
every protect on the main thread from reading /proc/self/maps.
MapUnmappedPages probed every 4 KiB page of the protected range with
msync before mapping it. Under an unlimited stack limit the main
thread's reported stack reaches down to the heap, tens of TiB, so a
guest protect over the whole stack ran for hours. Mapped and unmapped
runs are now found by halving the remaining range, and the unmapped
ones are mapped without reserving swap, as the stack itself grows.
The stack branch now records the reserve and guard below a protected
range with a separate RecordProtection call, and keeps the call that
records the protected range itself as main has it. The ranges recorded
are the same: RecordProtection replaces what it covers, so the two calls
together record the whole requested range, as before. #6184 adds a line
next to that call, and the two no longer touch.
HotSpot makes its stack guard zones with an RW protect of the lowest
pages of the thread's stack, then PROT_NONE, as it commits them on
native Windows (os::pd_create_stack_guard_pages, then os::guard_memory).
Below a host stack's guard page such a protect was only recorded, so the
pages stayed reserved: they could not be written after the RW protect,
and the zones never held.
A protect of reserved pages that starts at the bottom of the reserve and
stops short of the guard page now commits and protects them. The guard
page and the reserve above the range stay as they are, so the stack still
grows through the guard down to it, and the stack limit in the TEB still
follows the committed stack.
Windows moves the guard down by committing the pages below it as
read/write guard pages, committed or not. The zones therefore hold until
the stack is within one guard region of them (three pages on GitHub's
runners), and the pages the guard reaches then come in read/write. The
test checks both, and the debt entry says so.
HotSpot makes its stack guard zones this way: an RW protect of the
lowest 48 KiB of the thread's stack, then PROT_NONE. Both calls succeed,
the guard page stays above them, and the stack still grows through it.
Every case now also checks that the stack limit in the TEB follows the
committed stack.
Committing the reserved pages of a host thread's stack before protecting
it, as the stack branch did for a range the arena doesn't own, also
cleared the guard page's PAGE_GUARD. The thread was left with no guard
and no reserve: an overflow ran into whatever lay below the stack instead
of raising a stack overflow, and the stack limit in the thread's TEB,
which Windows moves only when the guard page is hit, stopped following
the stack. A range that began inside the reserve also left the reserved
pages below it with no guard above them, so the stack could no longer
grow into them.
The stack branch now leaves the guard page and the reserved pages below
it as they are, and commits and protects only what lies above them;
reserved pages above the guard are still committed first. The protection
is recorded for the whole range, and a range of only reserved and guard
pages changes nothing on the host. The test protects the main thread's
whole stack and a reserved thread's whole stack and upper part, then
checks that the guard page is still there and that the stack grows
through it, also after a PROT_NONE protect of its lowest page. The debt
file records that those pages come in read/write as the stack grows.
On Linux the guest's main runs on the host main thread's stack, which
the kernel maps only as it grows. sceKernelIsStack reports the whole
range that pthread_getattr_np gives, which is sized by RLIMIT_STACK and
is mostly not mapped yet. A protection of that range failed with ENOMEM,
so mprotect returned EINVAL and sceKernelMprotect stopped the title.
When the stack branch's mprotect fails with ENOMEM, the range's unmapped
pages are now mapped read-write at their own addresses
(MAP_FIXED_NOREPLACE) and the protection is applied again. This matches
how the reserved pages of a host stack are committed on Windows.
Yunus Shaikh