A protect below the main thread's stack mapping mapped only the
protected pages. Linux keeps a growing stack one guard gap (1 MiB by
default) above a readable mapping, so after a read-only protect, or a
protect later restored to read/write, at the bottom of the reported
stack the stack could no longer grow into the megabyte above it and a
deep enough call faulted where the console has stack pages. The pages
between the protect and the top of the stack are now mapped as well,
so the stack never has to grow past them.
MapUnmappedPages probed every 4 KiB page of the protected range with
msync before mapping it. Under an unlimited stack limit the main
thread's reported stack reaches down to the heap, tens of TiB, so a
guest protect over the whole stack ran for hours. Mapped and unmapped
runs are now found by halving the remaining range, and the unmapped
ones are mapped without reserving swap, as the stack itself grows.
The stack branch now records the reserve and guard below a protected
range with a separate RecordProtection call, and keeps the call that
records the protected range itself as main has it. The ranges recorded
are the same: RecordProtection replaces what it covers, so the two calls
together record the whole requested range, as before. #6184 adds a line
next to that call, and the two no longer touch.
HotSpot makes its stack guard zones with an RW protect of the lowest
pages of the thread's stack, then PROT_NONE, as it commits them on
native Windows (os::pd_create_stack_guard_pages, then os::guard_memory).
Below a host stack's guard page such a protect was only recorded, so the
pages stayed reserved: they could not be written after the RW protect,
and the zones never held.
A protect of reserved pages that starts at the bottom of the reserve and
stops short of the guard page now commits and protects them. The guard
page and the reserve above the range stay as they are, so the stack still
grows through the guard down to it, and the stack limit in the TEB still
follows the committed stack.
Windows moves the guard down by committing the pages below it as
read/write guard pages, committed or not. The zones therefore hold until
the stack is within one guard region of them (three pages on GitHub's
runners), and the pages the guard reaches then come in read/write. The
test checks both, and the debt entry says so.
Committing the reserved pages of a host thread's stack before protecting
it, as the stack branch did for a range the arena doesn't own, also
cleared the guard page's PAGE_GUARD. The thread was left with no guard
and no reserve: an overflow ran into whatever lay below the stack instead
of raising a stack overflow, and the stack limit in the thread's TEB,
which Windows moves only when the guard page is hit, stopped following
the stack. A range that began inside the reserve also left the reserved
pages below it with no guard above them, so the stack could no longer
grow into them.
The stack branch now leaves the guard page and the reserved pages below
it as they are, and commits and protects only what lies above them;
reserved pages above the guard are still committed first. The protection
is recorded for the whole range, and a range of only reserved and guard
pages changes nothing on the host. The test protects the main thread's
whole stack and a reserved thread's whole stack and upper part, then
checks that the guard page is still there and that the stack grows
through it, also after a PROT_NONE protect of its lowest page. The debt
file records that those pages come in read/write as the stack grows.
On Linux the guest's main runs on the host main thread's stack, which
the kernel maps only as it grows. sceKernelIsStack reports the whole
range that pthread_getattr_np gives, which is sized by RLIMIT_STACK and
is mostly not mapped yet. A protection of that range failed with ENOMEM,
so mprotect returned EINVAL and sceKernelMprotect stopped the title.
When the stack branch's mprotect fails with ENOMEM, the range's unmapped
pages are now mapped read-write at their own addresses
(MAP_FIXED_NOREPLACE) and the protection is applied again. This matches
how the reserved pages of a host stack are committed on Windows.
Follows mlockall (#4386) for XPSemu (PPSA97358, QEMU os-posix.c), which
imports munlockall (NpLBpgVV7PU) from libkernel.prx. Predicted stop:
FAIL: unresolved ELF import NpLBpgVV7PU from libkernel.prx
The NID is computed with tools/nid_names.py and matches aerolib.
Follows FreeBSD 11.4 sys/vm/vm_mmap.c sys_munlockall
(8c639945a44588f6cfdddfb057bb5aad88735da0): checks PRIV_VM_MUNLOCK,
clears wire state and returns 0. To match, munlockall unlocks ranges
locked by mlock (through VirtualUnlock on Windows, tolerating
ERROR_NOT_LOCKED, and munlockall on Linux).
A mapping without MAP_FIXED treated its address as the lowest place it
could go, and a hint at or above the end of the guest arena threw on the
host. FreeBSD 11.4's vm_mmap_object places such mappings with
vm_map_find_min, which takes the address as a hint only: it tries at or
above it, then again from its minimum address.
GuestArenaAllocateFromHint does the same in the arena: at or above the
hint when it lies below the arena's end and a range fits there, otherwise
from the arena's start. MapPlaced and the Windows mmap_aligned use it for
hinted mappings; GuestArenaAllocateAtOrAbove keeps its meaning for the
callers that need a range above an address.
PS5CEMU-HAR (PPSA99360) reserves its JIT range with
sceKernelReserveVirtualRange and a hint near its own code, a host address
above the arena here, and aborted with "mapping address hint is above the
guest address space arena" when launching a 3DS title.
Follow FreeBSD 11's sys_mincore by rounding unaligned address down to page boundary rather than returning EINVAL. Update guest memory test and document unmodeled residency/referenced/modified bits in libkernel technical debt.
Implement mincore (+hFjyohmOU0) in libkernel following FreeBSD 11's sys_mincore (sys/vm/vm_mmap.c). Validate page alignment, return EFAULT for null vector, check range bounds and residency via sceKernelVirtualQuery, and report MINCORE_INCORE (1) for resident pages. Required by Java 25 LAN / DNS (PPSA99869).
WindowsMappings keys each physical page's state (moved into the section,
saved bytes of an unmapped resident page) by the section handle value.
sceKernelReleaseDirectMemory frees the backing before it unmaps the views,
so ForgetSection ran while those views still existed and kept their slots.
After the views went, the slots stayed under a handle value that Windows
hands out again, and the next direct allocation whose section got that
value started with the old page state: its first mapping went to the
section instead of private memory, or received the old allocation's bytes.
Views now share ownership of their slot, ForgetSection drops every slot
of the section at once, and JIT shared memory forgets its section before
closing it too.
Guest threads run on host stacks that Windows reserves (0x210000 bytes
by default) and commits on demand. Resident Evil Village protects one
thread's whole host stack with 0xc2 at startup while its lowest 0xd000
bytes are still only reserved, so the Windows mprotect emulation threw.
When such a stack sat inside the arena bounds as a host region, the call
took the arena commit path and failed to split a placeholder that was
never one.
Commit the reserved pages of a host range before protecting it, and only
use the arena commit for ranges the arena owns.
Charge each recorded protection ranges 2 MiB granules to the gpu
pool when the range has GPU-access protection bits and to the cpu
pool otherwise, so the available counts reflect committed usage
instead of staying at the 2048-entry budget. Follows KytyPS5s
counting model.
Map fixed private anonymous ranges through the existing guest arena. Preserve occupied ranges for MAP_EXCL and use FreeBSD 11.4 errors.
Cover replacement, exclusivity, range validation, rounding, and memory access.
Map made every singly mapped section page private resident memory. A JIT
or shm_open section is file backed and committed: its bytes live in the
file, and its alias descriptors map duplicated handles, which the physical
page slots do not match, so a resident copy diverged from the file, and
committing its view on a demote failed with access denied
(guest_jit_shared_memory). Only direct memory backings, mapped through the
new GuestArenaMapDirect, now become resident; other sections keep main's
shared views. Shared pages are committed only while their section page is
reserved, as on main.
The socket host write tests map their direct memory a second time, so the
write-protected shared path they check is still the one they exercise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
A direct memory page mapped at one guest address now lives in private
MEM_WRITE_WATCH memory at that address, and its writes are collected with
GetWriteWatch like flexible memory: no fault is raised. A page mapped at a
second address moves its whole resident run into the section, behind a
read-only fence while it is copied, and every address becomes a view of it,
tracked by write protection. Unmapped resident pages that are not blank are
kept in a private reservation until they are mapped again, not in the
section.
Ported onto main's 4 KiB subpage tracking of shared views: resident runs
keep a per-page collected bit and enter the fresh ranges, so Fresh and
Armed collections select them like shared views; a demoted run carries its
collected bits into the views and reports every page once. MapAlias throws
through refuse() when a resident run extends outside the aliased range, and
the caller keeps main's fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1W4jdNSKTV7xwuS6xQPJV
Read the optional aligned budget from param.json and apply it consistently to configured, available and mapped flexible memory. Add parser and allocation regression coverage.
The allocation tracker threw a runtime_error (a hole in the protected range) from inside
sceKernelMprotect, which terminated the title. mprotect already turned the exception into
EINVAL. sceKernelMprotect and sceKernelMtypeprotect now check that the range is registered
first and return EINVAL; the retype of direct mappings is skipped on failure.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
sceKernelVirtualQuery, sceKernelMunmap, sceKernelReleaseDirectMemory and the
reservation paths walked a copy of the whole allocation registry or of every
direct mapping under one lock on each call. The registry now answers lookups
from its ordered map, applies range replacements as patches instead of
rebuilding the map, and direct mappings carry an index by physical offset so a
release finds only the mappings it covers. Minecraft (PPSA17221) holds about
7,000 direct mappings and spent its world generation in these walks.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mlockall (x7g7Ebeo8-U, aerolib) follows FreeBSD 11.4's sys_mlockall
(sys/vm/vm_mmap.c:1067-1131, freebsd-src
8c639945a44588f6cfdddfb057bb5aad88735da0) for the flag check: 0 or
bits outside MCL_CURRENT (1) and MCL_FUTURE (2) fail with EINVAL. The
title process is taken to pass priv_check(PRIV_VM_MLOCK), as mlock
already does.
Valid flags throw. MCL_FUTURE would need every later guest and host
mapping to be locked; nothing in the mapping paths does that, and
passing it to the host would make later host allocations fail above
the host RLIMIT_MEMLOCK. MCL_CURRENT would have to lock the whole
process, which on AnyPS5 also holds host stacks, heaps, modules and
driver mappings (sceKernelVirtualQuery reports them as guest memory),
with ENOMEM above a guest RLIMIT_MEMLOCK that is not known (getrlimit
throws for it). mlock pins host pages (VirtualLock, mlock) range by
range and keeps no record, so there is no state to extend. Recorded
in TechnicalDebt.
XPSemu (PPSA97358) calls it only with -overcommit mem-lock=on
(ZiZc3/XPSemu 8eec07b4, os-posix.c:330-355, reached from
system/vl.c:831-835):
int flags = MCL_CURRENT | MCL_FUTURE;
...
ret = mlockall(flags);
if (ret < 0) {
error_report("mlockall: %s", strerror(errno));
}
and exits on failure, so that option ends in the throw; without it
the title is not affected. Predicted stop removed (registry index 201,
not run):
FAIL: unresolved ELF import x7g7Ebeo8-U from libkernel.prx
guest_mlock checks EINVAL for 0, 4, 5 and -1 and the throw for 1, 2
and 3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 87f4aa331b02d098fe27516809495447ff6a9fa6)
The libc.prx heap that Animal Well (PPSA22520 01.000.011, SDK 9.04)
bundles maps each arena with sceKernelMapFlexibleMemory(&addr, len, 3,
0x8000). It threw "Unsupported memory mapping flags 0x8000" before the
first frame. The flag is ignored as sceKernelMapNamedFlexibleMemoryInternal
already does; other unknown flags still throw.
PPSA30601 (Resonance) stops at the throwing stub. The four int* arguments follow
OpenOrbis' libkernel.h, the Orbital type database and SharpProspero; the values
follow KytyPS5 (2048 entries of 2 MiB per side) without its accounting of used
entries, listed in TechnicalDebt.md. A null pointer returns EFAULT.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review of #3787 asked for no retries after an error in Jit.cpp.
Linux: memfd_create is called once with MFD_CLOEXEC. MFD_EXEC was
tried first and dropped on EINVAL; the flag only sets the file's execute
permission bits (Linux 6.3 and later), not whether the object maps
PROT_EXEC, which is decided by the mapping protection, as for the direct
memory memfd in DirectMemory.cpp. A one-shot choice from the headers
would fail on older kernels, so the flag is dropped; TechnicalDebt
records it.
Windows: the delete-on-close backing file is named from the process id,
a process-wide atomic counter and the performance counter, so CREATE_NEW
cannot meet an existing name, and any CreateFileW failure throws with
the Win32 error instead of looping up to 64 times on ERROR_FILE_EXISTS.
The descriptor stays a CRT descriptor on the file, so close, dup and
dup2 keep working as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The three JIT shared memory exports (avvJ3J0H0EY, MR221Mwo0Pc,
YKT49TOLQWs) are absent imports of three homebrew titles that cannot
finish loading without them. Predicted next stops (registry order):
- VLC PPSA85300 after getdtablesize (idx 354):
"FAIL: unresolved ELF import YKT49TOLQWs from libkernel.prx"
(Map idx 452, Create idx 462)
- XPSemu PPSA97358 after tcsendbreak (idx 214):
"FAIL: unresolved ELF import avvJ3J0H0EY from libkernel.prx"
(Create idx 398, Map idx 399)
- RetroArch PPSA99169:
"FAIL: unresolved ELF import avvJ3J0H0EY from libkernel.prx"
(Create 476, Alias 477, Map 478)
XPSemu and VLC (ps5/compat/mmap.c, __wrap_mmap) create an object for
every anonymous PROT_EXEC mapping (QEMU's TCG buffer) and map it with
the same protection, falling back to flexible memory on any non-zero
return. RetroArch's eboot runs a probe: Create("ps5-platform-probe",
0x10000, 7, &fd), Alias(fd, 3, &alias), Map(fd, 5, &address), then logs
"jit create=0x%08x alias=0x%08x map=0x%08x".
Signatures from Orbital cc408367 tools/ida/db_types.json (name, len,
maxProt, fdOut; fd, maxProt, fdOut; fd, prot, startOut), matching
ps4sdk 4df9d001 include/sce/kernel.h, ps5-jar-loader 8034177e
LibKernel.java and the three titles' call sites. SharpProspero 5dfdb4b0
declares the alias as returning the descriptor; the RetroArch call site
passes an out pointer.
The object is a host anonymous file (memfd on Linux, a delete-on-close
temporary file mapped as a PAGE_EXECUTE_READWRITE section on Windows),
the alias a duplicate descriptor, and each map maps the whole object
through the direct memory placement path, so a write through one view
is visible through the others. Bad lengths and protection bits are
EINVAL, protection past the descriptor's maximum is EACCES (FreeBSD 11
vm_map_protect), unknown descriptors are EBADF; close, dup and dup2
keep the descriptor table in step. Unverified details are recorded in
TechnicalDebt.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Yunus Shaikh