diff --git a/docs/ACTIVE.md b/docs/ACTIVE.md index e227429..d0affa2 100644 --- a/docs/ACTIVE.md +++ b/docs/ACTIVE.md @@ -4,16 +4,38 @@ _Updated: 2026-09-19_ ## Now -**Genesis Plus GX is ported and owner-confirmed working.** The owner reports -“Sega Genesis works flawlessly” and explicitly confirms “Yes, transitions and -next game are clean.” Build and console evidence: `evidence/genesis-plus-gx-native/`. -No next core is selected. +**XMB allocation diagnostic is built on `codex/xmb-allocation-diagnostics`.** +This is instrumentation, not a crash fix. Owner explicitly requests no automatic +upload or launch while working on another project. Wait for their intervention. +The baseline's next-core work is not the assignment; this crash investigation is. -The public README now describes this verified baseline, with a table of contents, -scoped core coverage, roadmap, native build/install guidance and upstream credits. -This documentation update does not change the accepted binaries. +- Build identity: `000adf3edc2538bc563f833f0ccd2e91a01053aa6dac19dd568a580cdbbbb3cb`. +- Output: `dist/PPSA99169/`; preserved executable/symbols/map/inspection under + `klog/xmb-memory-000adf3edc25/`. Evidence: `evidence/xmb-memory-diagnostic/`. +- `PS5_MEMORY_DIAGNOSTICS=1 bash tools/verify.sh` passed all five host gates. + Final format/unit check passed; 68 tests. Raw: `klog/xmb-memory-verify.log`, + `klog/xmb-memory-final-checks.log`. `tools/check-memory-diagnostics.py` passes. +- No upload, launch, console settings change or PS5_Vulkan modification. + Current libps5vk differs from the crashing build; local archive snapshots and + their hashes identify the diagnostic's inputs. Other three archives match. +- Log: `/app0/memory-diagnostics.log`; periodic requested live bytes/counts/peaks, + caller groups, image/queue-idle counters; immediate allocation failure records. + Metadata saturation/coverage gaps are explicit. No allocator policy change. +- Reproduction and interpretation: `docs/MEMORY_DIAGNOSTICS.md`. Console validation + remains pending, including whether the crash reproduces with the newer driver. -## Verified build and evidence +## Crash evidence + +Owner reproduced rapid XMB navigation crash. Raw capture: +`klog/xmb-crash-20260919-181233/` (kernel, frontend, trace, config, diagnosis). +Matching baseline symbols locate SIGSEGV at runtime `0x45b1ae` in +`__vk_log_impl`, reached through `vk_sync_create` allocation failure inside +`vkQueueWaitIdle`, called from XMB texture unload. Logger's second allocation +returns NULL and is dereferenced. Why allocation failed is unresolved: heap +exhaustion, fragmentation, retention/leak, unobserved usage or corruption remain +possible. Do not call this a GPU hang or a resolved frontend/driver leak. + +## Previous console-verified baseline (Genesis Plus GX) - Frontend identity: `ecfcddd57febbb484c2e0724e95f43bbffb3ddd9b9c22aa9c1cf3e37af4a9445`. - Genesis Plus GX source: `c2838c7dc4236fc2fe94e5dbd08b41486067918e`. @@ -44,21 +66,6 @@ This documentation update does not change the accepted binaries. - PS5_Vulkan was not modified. Four archive hashes matched before/after build; exact linked hashes are in the evidence. The owner develops it concurrently. -## What changed - -- `make genesis-plus-gx`: pinned official source and metadata using explicit - project SDK wrappers, native import union and .info in both lookup locations. - Attached metadata was reference only. No ROM or BIOS is shipped. -- Preserve RGB565 software rendering, filters/cursors and cached bitmap; convert - into a separate bounded XRGB8888 callback buffer. Honour cropped byte offsets - and the 720-pixel row pitch. The frontend's existing RGBA upload/cropping path - remains unchanged, as does the Vulkan driver. -- Disable host-detected physical CD-ROM support and optional zstd weak tracing - hooks. Keep CHD/disc-image codecs and useful logs. Correct update_geometry's - unused return type. Reasons/pins and procedures are in REFERENCE/TESTING/DEPLOYMENT. -- Host tests cover every RGB565 colour through upstream packing and the actual - GPU upload helper, viewport changes/crops, cached frames and rejected bounds. - ## Named errors and remaining limits - First frontend snapshot has five intentional missing-core recovery errors and @@ -93,9 +100,7 @@ See `docs/DEPLOYMENT.md` for procedures and metadata-listed BIOS filenames. ## Operating notes -Uploads/runs are authorized without further permission. Never interrupt an -existing title; the runner checks idle before deployment and launch. Core/game -selection is manual. Keep private filenames and console details in ignored logs. -Separate appended trace launches by bss check and build identity. Preserve live -logs before relaunch. Owner supplies visual confirmation; camera cadence can -hide flicker. Do not infer manual closure solely from the final process count. +Console testing is paused by the owner's explicit instruction for this step. +When they are ready, start passive klog before reproduction and preserve current +logs. Never interrupt another title. After a crash, capture logs before relaunch. +No new core or unrelated driver work is authorized by the active-file context. diff --git a/docs/MEMORY_DIAGNOSTICS.md b/docs/MEMORY_DIAGNOSTICS.md new file mode 100644 index 0000000..7b8cf39 --- /dev/null +++ b/docs/MEMORY_DIAGNOSTICS.md @@ -0,0 +1,110 @@ +# XMB allocation diagnostic build + +This is an opt-in diagnostic, not a fix for the XMB crash. It observes allocation +lifetimes without increasing the heap, redirecting additional allocations to +`mmap`, or suppressing Vulkan errors. PS5_Vulkan is read only for this work. + +## Build and host verification + +```sh +PS5_MEMORY_DIAGNOSTICS=1 bash tools/verify.sh +``` + +The normal build (variable absent or `0`) compiles out the observer. The diagnostic +adds `PS5_MEMORY_DIAGNOSTICS` to the title definitions and +`--wrap=posix_memalign` to the existing malloc/calloc/realloc/free wrappers because +Mesa's default Vulkan host allocator uses aligned native allocations. Mode and +archive contents participate in the runtime build identity. + +The diagnostic build copies the four driver archives into +`build/memory-diagnostic-inputs/` and links those copies. `archives.json` records +their SHA-256 digests. A copy that changes during the read is rejected. This does +not rebuild or write into the sibling project; it also does not imply the copied +archives are the same driver as an earlier console run. Mesa utility sources are +still read from the configured driver source tree, and the resulting object +bytes participate in the identity. + +Output is `dist/PPSA99169/`. Preserve `build/llvm-pie.elf`, `build/title.map`, +`build/title_build_identity.h`, the manifest, and the archive digests before any +subsequent build. Caller addresses must be decoded against the exact matching ELF. + +Host tests exercise allocation/free accounting, mapped/native realloc migration, +failed realloc retention, foreign pointers, aligned allocation semantics, +concurrency, metadata saturation, five-second sampling with an injected clock, +and logging while the wrapped allocator fails. Vulkan mocks check that image and +queue-idle observers preserve arguments, return values and repeat registration. +These tests do not establish console stability or identify the memory consumer. + +## Manual console test — only when the owner is ready + +Do not upload, launch, change live settings, or stop another title automatically. +Start a passive klog listener before the owner's reproduction and preserve the +existing logs first. Deploy the diagnostic using the project's normal folder +procedure, preserving live configuration and user content. The owner launches it. + +1. Leave XMB idle for 30 seconds. +2. Browse rapidly for 30 seconds, then leave it idle for 30 seconds. Repeat. +3. Repeat the RGUI to XMB transition that triggered the crash. Note the approximate + elapsed time of each phase, including whether switching required restarting. +4. On a crash, leave the title closed until logs have been downloaded. + +Download `/data/homebrew/PPSA99169/memory-diagnostics.log`, `retroarch.log`, +`trace.txt`, and the live configuration, along with the kernel capture. Keep raw +files under ignored `klog/`; they may contain user paths and process information. +In the title the same new log is `/app0/memory-diagnostics.log`. It appends a +`session` line with build identity and PID each launch rather than erasing the +previous run. No runner or deployment behavior is changed by this diagnostic. + +## Reading the capture + +`initial`, `sample`, `failure-summary` and `final` rows contain: + +- `native_bytes/count/peak`: requested live bytes, live count and peak live bytes + from observed ordinary native allocations. +- `aligned_bytes/count/peak`: the same for `posix_memalign`; this is also native + memory, not a separate heap. Native pressure includes both native and aligned. +- `mapped_bytes/count/peak`: requests routed through the existing large-buffer + mappings. These exclude mapping headers, page rounding and allocator overhead. +- `failures`: cumulative failures; `failure` rows immediately record operation, + requested bytes, alignment (or element count for calloc overflow), error and + caller return address (`pc`). `calloc-overflow` is an arithmetic rejection, + not measured heap exhaustion. +- `dropped`: allocation records omitted because the fixed table was full. Any + nonzero value makes live totals incomplete for the rest of that session. +- `foreign_frees/reallocs`: pointers whose allocation was not observed (including + records dropped on saturation). Never inspect private libc allocation headers. +- `image_create/destroy/failed`: successful frontend Vulkan image creations, + non-null destruction calls and failed creations. These include images other + than menu textures and exclude calls internal to the driver. +- `idle_begin/end/failed`: queue-idle calls started, returned and returned failure. + A missing completion in a failure summary helps locate the failing operation. + +Every five seconds while Vulkan presents, a summary and up to eight largest live +allocation caller groups are emitted. Groups contain `pc`, requested live bytes +and count; `site_records_omitted` reports overflow of the separate 2,048-site +aggregation table. They are not full stack traces. No frame means no periodic +sample; allocation failures still write synchronously. Clean frontend return +writes `final`; a crash need not. + +Logging uses stack formatting, a preopened descriptor and `write`, without stdio +or heap allocation. A fixed 131,072-entry pointer table plus state bytes uses +about 4.13 MiB of static storage. Mutexes, table operations, scans and synchronous +writes add overhead, so use this build to diagnose lifetimes, not benchmark FPS. +The allocation route and payload size remain unchanged. + +Coverage is title/core/static-library references intercepted by the linker. +Allocations performed internally by system libraries, direct mappings outside the +existing large-buffer wrapper, GPU direct memory, and allocator bookkeeping are +not measured. Requested live bytes are not the console's total memory usage. + +For a `pc` from this title, subtract the image load base recorded in klog (normally +`0x400000`) and one byte for a return address, then use +`llvm-addr2line -C -f -e `. A fault instruction +address is decoded without the one-byte adjustment. Keep symbols locally; never +publish raw memory addresses or local paths as sanitized evidence by accident. + +Compare settled idle samples from the same session. Sustained growth is a reason +to investigate retained allocations; it alone does not prove a leak. Stable +observed totals plus allocation failures leave unobserved native usage, +fragmentation, heap limits and corruption open. The crash's Mesa null dereference +while reporting an allocation failure remains unfixed by this instrumentation. diff --git a/docs/PHASE_LOG.md b/docs/PHASE_LOG.md index 2865412..3c0d1d6 100644 --- a/docs/PHASE_LOG.md +++ b/docs/PHASE_LOG.md @@ -2313,3 +2313,37 @@ Documentation-only validation: existing committed artifacts, especially evidence/genesis-plus-gx-native/. - `git diff --check`: PASS. No source, dependency, binary or console changes; the accepted gameplay build remains 93a1332's recorded build identity. + +## 2026-09-19 — XMB allocation diagnostic branch, host verified only + +Owner requested a branch and diagnostic build, explicitly withholding automatic +upload/launch while working on another project. Created +`codex/xmb-allocation-diagnostics`. Added opt-in live allocation accounting to the +existing native/mapped wrappers, plus observation of `posix_memalign`, frontend +Vulkan image lifetimes and queue-idle calls. Fixed-capacity metadata and stack-only +`write` logging preserve OOM evidence without allocating. Allocation policy is +unchanged. Five-second summaries include counts, requested bytes, peaks, largest +caller groups, failure counts and explicit coverage/saturation limits. + +The prior capture (`klog/xmb-crash-20260919-181233/`) maps the null write to Mesa's +error-reporting allocation following `vk_sync_create` allocation failure in +`vkQueueWaitIdle` during XMB texture unload. That diagnoses the immediate crash, +not the reason memory allocation failed. No driver fix or console result is claimed. + +Command: `PS5_MEMORY_DIAGNOSTICS=1 bash tools/verify.sh` — PASS all five host gates. +The first unit pass had 67 tests; the added Vulkan dispatch test ran in the final +integration pass and final `bash tools/verify.sh format unit` (68 tests, PASS). +`python3 tools/check-memory-diagnostics.py` — PASS required symbols and archive +hashes. Machine-readable inspection and capture/expectation: +`evidence/xmb-memory-diagnostic/`. Raw host logs are in `klog/xmb-memory-verify.log` +and `klog/xmb-memory-final-checks.log`. Existing upstream core/compiler warnings +remain in the full build log; no diagnostic-source warning was observed. + +Build identity: `000adf3edc2538bc563f833f0ccd2e91a01053aa6dac19dd568a580cdbbbb3cb`. +Output: `dist/PPSA99169/`; matching executable, symbol ELF and map preserved under +`klog/xmb-memory-000adf3edc25/`. The current libps5vk archive differs from the +crashing baseline; the other three archives match its recorded digests. The +opt-in build snapshots all four locally to avoid mutable archive inputs while +the owner develops PS5_Vulkan. Mode and linked bytes participate in identity. +No sibling writes, console access, upload, launch or settings change occurred. +Manual protocol and limitations are in `docs/MEMORY_DIAGNOSTICS.md`. diff --git a/docs/TESTING.md b/docs/TESTING.md index 248f63d..01c398e 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -330,3 +330,10 @@ Capture frontend/trace/kernel logs, match the exact build identity, and check fo Vulkan refusals, GPU API failures, invalid bitmap views and unexpected runtime errors before committing. Record systems actually tested and preserve user confirmation separately from machine-readable loader and runtime evidence. + +## XMB allocation diagnostics + +The opt-in build, host allocator/Vulkan-dispatch tests, raw log fields, ownership +coverage limits and manual reproduction protocol are specified in +[MEMORY_DIAGNOSTICS.md](MEMORY_DIAGNOSTICS.md). Its build inspection is host-only; +it must not be cited as a passing console reproduction or a fixed crash. diff --git a/evidence/xmb-memory-diagnostic/capture.json b/evidence/xmb-memory-diagnostic/capture.json new file mode 100644 index 0000000..232f45f --- /dev/null +++ b/evidence/xmb-memory-diagnostic/capture.json @@ -0,0 +1,19 @@ +{ + "step": "xmb-memory-diagnostic", + "recorded": "2026-09-19", + "revision": "a18cec6 + allocation diagnostics", + "command": "PS5_MEMORY_DIAGNOSTICS=1 bash tools/verify.sh; bash tools/verify.sh format unit; python3 tools/check-memory-diagnostics.py", + "console_run": null, + "raw_capture": "klog/xmb-memory-verify.log", + "lines": [ + "Scope: host-verified diagnostic build only; no console upload or launch.", + "Diagnostic build identity: 000adf3edc2538bc563f833f0ccd2e91a01053aa6dac19dd568a580cdbbbb3cb", + "Allocation semantics and OOM logging: PASS; concurrency and saturation: PASS.", + "Five-second sampling with injected clock: PASS; Vulkan dispatch preservation: PASS.", + "Host verification: PASS format unit build integration evidence; final suite 68 tests.", + "Diagnostic symbols and pinned driver snapshots: PASS.", + "Console reproduction: PENDING. XMB crash is not fixed by this change." + ], + "raw_lines": 6265, + "raw_bytes": 3051670 +} diff --git a/evidence/xmb-memory-diagnostic/expectation.json b/evidence/xmb-memory-diagnostic/expectation.json new file mode 100644 index 0000000..42417c3 --- /dev/null +++ b/evidence/xmb-memory-diagnostic/expectation.json @@ -0,0 +1,17 @@ +{ + "step": "xmb-memory-diagnostic", + "note": "Host-only diagnostic readiness. Console test deliberately pending owner intervention.", + "must_contain": [ + "Scope: host-verified diagnostic build only; no console upload or launch.", + "Diagnostic build identity: 000adf3edc2538bc563f833f0ccd2e91a01053aa6dac19dd568a580cdbbbb3cb", + "Allocation semantics and OOM logging: PASS; concurrency and saturation: PASS.", + "Five-second sampling with injected clock: PASS; Vulkan dispatch preservation: PASS.", + "Host verification: PASS format unit build integration evidence; final suite 68 tests.", + "Diagnostic symbols and pinned driver snapshots: PASS.", + "Console reproduction: PENDING. XMB crash is not fixed by this change." + ], + "must_not_contain": [ + "Console reproduction: PASS" + ], + "human_check": "Not performed: owner requested no upload or launch." +} diff --git a/evidence/xmb-memory-diagnostic/inspection.json b/evidence/xmb-memory-diagnostic/inspection.json new file mode 100644 index 0000000..ffb4f85 --- /dev/null +++ b/evidence/xmb-memory-diagnostic/inspection.json @@ -0,0 +1,30 @@ +{ + "kind": "host-only-diagnostic-inspection", + "passed": true, + "build_identity": "000adf3edc2538bc563f833f0ccd2e91a01053aa6dac19dd568a580cdbbbb3cb", + "symbols_present": [ + "__wrap_malloc", + "__wrap_calloc", + "__wrap_realloc", + "__wrap_free", + "__wrap_posix_memalign", + "ps5::memory::init(", + "ps5::memory::failure(", + "ps5::memory::tick(", + "memory_create(", + "memory_destroy(", + "memory_idle(" + ], + "artifacts_sha256": { + "build/llvm-pie.elf": "8f89351d91d0ad9bc05b09d3d28ad22b1798d00ed29b9df88df79575ebde1c45", + "build/title.map": "bd743e225dc222a59ec9632a574f59e003e0b07d8ea0428043cb2de3c608e03d", + "dist/PPSA99169/eboot.bin": "dbb71d62520f4baae487a8c7dd7e0425ac5662f7e541b69577ea21cb8bce43e7" + }, + "driver_archives_sha256": { + "libps5vk.ps5.a": "900d496a9eb725857e4a70540d34ef262dcb1d5c85b7ed9325a28d16b460e246", + "libvk_runtime.ps5.a": "4106a2c56b269bc52a3976b6bc0134c00b5006ff5358a4b45355afce4a907aa7", + "libpsbc_driver.ps5.a": "a7605370c28895a3b5a13d0718ebfea2abf1b9cf842633de07533775745b21b8", + "libpsbc_support.ps5.a": "e36d3e2a3fa93b0a74db15b9efce7aad4ed7ac4e024de98d5f23dc5fee90acf8" + }, + "console_test": "pending; owner requested no automatic upload or launch" +} diff --git a/src/main.cpp b/src/main.cpp index 423f33c..490ae11 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -40,6 +40,7 @@ #include #include "trace.hpp" +#include "memory_diagnostics.hpp" #include "../build/title_build_identity.h" /* RetroArch's entry, in C. */ @@ -143,6 +144,7 @@ int main() std::set_terminate(on_terminate); ps5::debug::mark(PS5_RETROARCH_BUILD_ID); + ps5::memory::init("/app0/memory-diagnostics.log", PS5_RETROARCH_BUILD_ID); ps5_vulkan_profile_init(); /* The shell's splash covers the title until it explicitly dismisses it. @@ -227,6 +229,7 @@ int main() /* If this line is on the console, the frontend ran and returned by itself. */ ps5::debug::mark_value("rarch_main returned", status); + ps5::memory::finish(); return status; } diff --git a/src/memory_diagnostics.cpp b/src/memory_diagnostics.cpp new file mode 100644 index 0000000..dbbbae0 --- /dev/null +++ b/src/memory_diagnostics.cpp @@ -0,0 +1,318 @@ +/* Fixed-size metadata and stack-only formatting: diagnostics must survive OOM. + * No native allocator calls, stdio, dynamic C++ containers or worker threads. */ +#include "memory_diagnostics.hpp" +#ifdef PS5_MEMORY_DIAGNOSTICS +#include +#include +#include +#include +#include +#include + +namespace ps5::memory +{ +namespace +{ +constexpr size_t capacity = 131072; +Record records[capacity]; +// 0 = never used, 1 = live, 2 = tombstone. Pointer values remain unmodified. +unsigned char states[capacity]; +Stats stats; +pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; +int output = -1; +uint64_t start_ns = 0, next_ns = 0, sequence = 0; +uint64_t image_create = 0, image_destroy = 0, image_failed = 0; +uint64_t idle_begin = 0, idle_end = 0, idle_failed = 0; + +uint64_t now() +{ + timespec t{}; + clock_gettime(CLOCK_MONOTONIC, &t); + return uint64_t(t.tv_sec) * 1000000000ULL + uint64_t(t.tv_nsec); +} +struct Line +{ + char data[4096]; + size_t length = 0; + void text(const char *s) + { + while (*s && length < sizeof(data) - 1) + data[length++] = *s++; + } + void number(uint64_t n, unsigned base = 10) + { + char digits[32]; + size_t count = 0; + do + { + digits[count++] = "0123456789abcdef"[n % base]; + n /= base; + } while (n); + while (count && length < sizeof(data) - 1) + data[length++] = digits[--count]; + } + void field(const char *name, uint64_t n) + { + text(" "); + text(name); + text("="); + number(n); + } + void emit() + { + if (output < 0) + return; + data[length++] = '\n'; + size_t done = 0; + while (done < length) + { + const auto written = write(output, data + done, length - done); + if (written < 0 && errno == EINTR) + continue; + if (written <= 0) + break; + done += size_t(written); + } + } +}; +size_t hash(void *p) +{ + return ((uintptr_t(p) >> 4) * uintptr_t(11400714819323198485ULL)) & (capacity - 1); +} +void summary(const char *kind) +{ + Line line; + line.text(kind); + line.field("seq", ++sequence); + line.field("ms", (now() - start_ns) / 1000000); + const char *bytes[] = {"native_bytes", "mapped_bytes", "aligned_bytes"}; + const char *counts[] = {"native_count", "mapped_count", "aligned_count"}; + const char *peaks[] = {"native_peak", "mapped_peak", "aligned_peak"}; + for (unsigned i = 0; i < 3; ++i) + { + line.field(bytes[i], stats.bytes[i]); + line.field(counts[i], stats.count[i]); + line.field(peaks[i], stats.peak[i]); + } + line.field("failures", stats.failures); + line.field("dropped", stats.dropped); + line.field("foreign_frees", stats.foreign_frees); + line.field("foreign_reallocs", stats.foreign_reallocs); + line.field("image_create", image_create); + line.field("image_destroy", image_destroy); + line.field("image_failed", image_failed); + line.field("idle_begin", idle_begin); + line.field("idle_end", idle_end); + line.field("idle_failed", idle_failed); + line.emit(); +} +// Fixed aggregation table: caller attribution may overflow independently of +// the live-pointer table. Report that explicitly, never silently lose counts. +void callers() +{ + struct Site + { + uintptr_t caller = 0; + uint64_t bytes = 0, count = 0; + }; + Site sites[2048]{}; + uint64_t omitted = 0; + for (size_t i = 0; i < capacity; ++i) + { + if (states[i] != 1) + continue; + const Record &r = records[i]; + size_t j = (r.caller >> 4) & 2047, tries = 0; + while (sites[j].count && sites[j].caller != r.caller && tries < 2048) + { + j = (j + 1) & 2047; + ++tries; + } + if (tries == 2048) + { + ++omitted; + continue; + } + sites[j].caller = r.caller; + sites[j].bytes += r.bytes; + ++sites[j].count; + } + for (unsigned rank = 0; rank < 8; ++rank) + { + size_t best = 0; + for (size_t i = 1; i < 2048; ++i) + if (sites[i].bytes > sites[best].bytes) + best = i; + if (!sites[best].count) + break; + Line line; + line.text("caller"); + line.field("seq", sequence); + line.text(" pc=0x"); + line.number(sites[best].caller, 16); + line.field("bytes", sites[best].bytes); + line.field("count", sites[best].count); + line.field("site_records_omitted", omitted); + line.emit(); + sites[best] = {}; + } +} +} // namespace +void add(Record record) +{ + if (!record.pointer) + return; + const int saved = errno; + pthread_mutex_lock(&mutex); + size_t slot = hash(record.pointer), vacant = capacity; + for (size_t n = 0; n < capacity; ++n, slot = (slot + 1) & (capacity - 1)) + { + if (states[slot] != 1) + { + vacant = slot; + break; + } + } + if (vacant == capacity) + ++stats.dropped; + else + { + states[vacant] = 1; + records[vacant] = record; + const auto route = unsigned(record.route); + stats.bytes[route] += record.bytes; + ++stats.count[route]; + if (stats.bytes[route] > stats.peak[route]) + stats.peak[route] = stats.bytes[route]; + } + pthread_mutex_unlock(&mutex); + errno = saved; +} +Record take(void *pointer, bool resizing) +{ + Record result; + if (!pointer) + return result; + const int saved = errno; + pthread_mutex_lock(&mutex); + size_t slot = hash(pointer); + for (size_t n = 0; n < capacity && states[slot]; ++n, slot = (slot + 1) & (capacity - 1)) + { + if (states[slot] == 1 && records[slot].pointer == pointer) + { + result = records[slot]; + states[slot] = 2; + stats.bytes[unsigned(result.route)] -= result.bytes; + --stats.count[unsigned(result.route)]; + break; + } + } + if (!result.pointer) + { + if (resizing) + ++stats.foreign_reallocs; + else + ++stats.foreign_frees; + } + pthread_mutex_unlock(&mutex); + errno = saved; + return result; +} +void failure(const char *operation, size_t bytes, size_t alignment, uintptr_t caller, int error) +{ + const int saved = errno; + pthread_mutex_lock(&mutex); + ++stats.failures; + Line line; + line.text("failure op="); + line.text(operation); + line.field("ms", (now() - start_ns) / 1000000); + line.field("bytes", bytes); + line.field("alignment", alignment); + line.field("error", unsigned(error)); + line.text(" pc=0x"); + line.number(caller, 16); + line.emit(); + summary("failure-summary"); + pthread_mutex_unlock(&mutex); + errno = saved; +} +Stats snapshot() +{ + pthread_mutex_lock(&mutex); + Stats result = stats; + pthread_mutex_unlock(&mutex); + return result; +} +void init(const char *path, const char *identity) +{ + const int saved = errno; + pthread_mutex_lock(&mutex); + if (output >= 0) + close(output); + output = open(path, O_WRONLY | O_CREAT | O_APPEND, 0666); + start_ns = now(); + next_ns = start_ns + 5000000000ULL; + Line line; + line.text("session "); + line.text(identity); + line.field("pid", uint64_t(getpid())); + line.field("table_capacity", capacity); + line.field("table_bytes", sizeof(records) + sizeof(states)); + line.emit(); + summary("initial"); + pthread_mutex_unlock(&mutex); + errno = saved; +} +void tick() +{ + const int saved = errno; + pthread_mutex_lock(&mutex); + const uint64_t time = now(); + if (output >= 0 && time >= next_ns) + { + next_ns = time + 5000000000ULL; + summary("sample"); + callers(); + } + pthread_mutex_unlock(&mutex); + errno = saved; +} +void event(const char *name, bool success) +{ + const int saved = errno; + pthread_mutex_lock(&mutex); + if (!std::strcmp(name, "image_create")) + { + if (success) + ++image_create; + else + ++image_failed; + } + else if (!std::strcmp(name, "image_destroy")) + ++image_destroy; + else if (!std::strcmp(name, "idle_begin")) + ++idle_begin; + else if (!std::strcmp(name, "idle_end")) + { + ++idle_end; + if (!success) + ++idle_failed; + } + pthread_mutex_unlock(&mutex); + errno = saved; +} +void finish() +{ + const int saved = errno; + pthread_mutex_lock(&mutex); + summary("final"); + callers(); + if (output >= 0) + close(output); + output = -1; + pthread_mutex_unlock(&mutex); + errno = saved; +} +} // namespace ps5::memory +#endif diff --git a/src/memory_diagnostics.hpp b/src/memory_diagnostics.hpp new file mode 100644 index 0000000..983423d --- /dev/null +++ b/src/memory_diagnostics.hpp @@ -0,0 +1,59 @@ +/* Optional host-allocation observation; no allocator policy changes. */ +#pragma once +#include +#include + +namespace ps5::memory +{ +enum class Route : unsigned +{ + Native, + Mapped, + Aligned +}; +struct Record +{ + void *pointer = nullptr; + size_t bytes = 0; + uintptr_t caller = 0; + Route route = Route::Native; +}; +struct Stats +{ + uint64_t bytes[3]{}, count[3]{}, peak[3]{}; + uint64_t failures = 0, dropped = 0, foreign_frees = 0, foreign_reallocs = 0; +}; +#ifdef PS5_MEMORY_DIAGNOSTICS +void init(const char *path, const char *identity); +void finish(); +void tick(); +void event(const char *name, bool success = true); +void add(Record record); +Record take(void *pointer, bool resizing = false); +void failure(const char *operation, size_t bytes, size_t alignment, uintptr_t caller, int error); +Stats snapshot(); +#else +inline void init(const char *, const char *) +{ +} +inline void finish() +{ +} +inline void tick() +{ +} +inline void event(const char *, bool = true) +{ +} +inline void add(Record) +{ +} +inline Record take(void *, bool = false) +{ + return {}; +} +inline void failure(const char *, size_t, size_t, uintptr_t, int) +{ +} +#endif +} // namespace ps5::memory diff --git a/src/memory_ps5.cpp b/src/memory_ps5.cpp index 25a3df3..6fe9227 100644 --- a/src/memory_ps5.cpp +++ b/src/memory_ps5.cpp @@ -7,6 +7,7 @@ #include #include #include +#include "memory_diagnostics.hpp" extern "C" { @@ -38,9 +39,8 @@ Mapping **find(void *pointer) entry = &(*entry)->next; return entry; } -} // namespace -extern "C" void *__wrap_malloc(size_t size) +void *allocate(size_t size) { if (size < threshold) return __real_malloc(size ? size : 1); @@ -63,7 +63,7 @@ extern "C" void *__wrap_malloc(size_t size) return entry + 1; } -extern "C" void __wrap_free(void *pointer) +void release(void *pointer) { if (!pointer) return; @@ -79,27 +79,13 @@ extern "C" void __wrap_free(void *pointer) __real_free(pointer); } -extern "C" void *__wrap_calloc(size_t count, size_t size) -{ - if (size && count > SIZE_MAX / size) - { - errno = ENOMEM; - return nullptr; - } - const size_t bytes = count * size; - if (bytes < threshold) - return bytes ? __real_calloc(count, size) : __real_calloc(1, 1); - /* Anonymous mappings are zero-initialized by the kernel. */ - return __wrap_malloc(bytes); -} - -extern "C" void *__wrap_realloc(void *pointer, size_t size) +void *resize(void *pointer, size_t size) { if (!pointer) - return __wrap_malloc(size); + return allocate(size); if (!size) { - __wrap_free(pointer); + release(pointer); return nullptr; } pthread_mutex_lock(&lock); @@ -110,10 +96,85 @@ extern "C" void *__wrap_realloc(void *pointer, size_t size) * this ABI. Never guess it or read a private libc allocation header. */ if (!entry) return __real_realloc(pointer, size); - void *replacement = __wrap_malloc(size); + void *replacement = allocate(size); if (!replacement) return nullptr; std::memcpy(replacement, pointer, old_size < size ? old_size : size); - __wrap_free(pointer); + release(pointer); return replacement; } + +} // namespace + +extern "C" void *__wrap_malloc(size_t size) +{ + const auto caller = uintptr_t(__builtin_return_address(0)); + void *p = allocate(size); + if (p) + ps5::memory::add( + {p, size, caller, + size < threshold ? ps5::memory::Route::Native : ps5::memory::Route::Mapped}); + else + ps5::memory::failure("malloc", size, 0, caller, errno); + return p; +} +extern "C" void __wrap_free(void *pointer) +{ + ps5::memory::take(pointer); + release(pointer); +} +extern "C" void *__wrap_calloc(size_t count, size_t size) +{ + const auto caller = uintptr_t(__builtin_return_address(0)); + if (size && count > SIZE_MAX / size) + { + errno = ENOMEM; + ps5::memory::failure("calloc-overflow", size, count, caller, errno); + return nullptr; + } + const size_t bytes = count * size; + void *p = bytes < threshold ? (bytes ? __real_calloc(count, size) : __real_calloc(1, 1)) + : allocate(bytes); + if (p) + ps5::memory::add( + {p, bytes, caller, + bytes < threshold ? ps5::memory::Route::Native : ps5::memory::Route::Mapped}); + else + ps5::memory::failure("calloc", bytes, 0, caller, errno); + return p; +} +extern "C" void *__wrap_realloc(void *pointer, size_t size) +{ + const auto caller = uintptr_t(__builtin_return_address(0)); + const auto old = ps5::memory::take(pointer, true); + void *p = resize(pointer, size); + if (p) + { + // Realloc of a native pointer stays native even when it crosses 1 MiB. + const bool mapped = + (!pointer || old.route == ps5::memory::Route::Mapped) && size >= threshold; + ps5::memory::add( + {p, size, caller, mapped ? ps5::memory::Route::Mapped : ps5::memory::Route::Native}); + } + else if (size) + { + ps5::memory::add(old); // Failed realloc retains ownership and contents. + ps5::memory::failure("realloc", size, 0, caller, errno); + } + return p; +} +#ifdef PS5_MEMORY_DIAGNOSTICS +extern "C" int __real_posix_memalign(void **, size_t, size_t); +extern "C" int __wrap_posix_memalign(void **out, size_t alignment, size_t size) +{ + const auto caller = uintptr_t(__builtin_return_address(0)); + const int saved = errno; + const int result = __real_posix_memalign(out, alignment, size); + if (!result && *out) + ps5::memory::add({*out, size, caller, ps5::memory::Route::Aligned}); + else if (result) + ps5::memory::failure("posix_memalign", size, alignment, caller, result); + errno = saved; // POSIX reports its error through the return value. + return result; +} +#endif diff --git a/src/vulkan_trace.cpp b/src/vulkan_trace.cpp index b787366..52a7f43 100644 --- a/src/vulkan_trace.cpp +++ b/src/vulkan_trace.cpp @@ -9,6 +9,7 @@ #include #include #include +#include "memory_diagnostics.hpp" namespace { @@ -260,6 +261,7 @@ VKAPI_ATTR VkResult VKAPI_CALL traced_present(VkQueue queue, const VkPresentInfo { static Results results; static Results images; + ps5::memory::tick(); ApiTimer timer(Present); const VkResult result = queue_present(queue, present); const uint64_t completed = timer.finish(); @@ -354,8 +356,62 @@ extern "C" void ps5_vulkan_profile_end() /* Called by both frontend symbol loaders. Keep the driver's real function, * preserve NULL lookups and return every result unchanged. Reloading a device * must neither reset the run totals nor wrap our own wrapper recursively. */ +#ifdef PS5_MEMORY_DIAGNOSTICS +namespace +{ +PFN_vkCreateImage memory_create_image; +PFN_vkDestroyImage memory_destroy_image; +PFN_vkQueueWaitIdle memory_queue_idle; +VKAPI_ATTR VkResult VKAPI_CALL memory_create(VkDevice device, const VkImageCreateInfo *info, + const VkAllocationCallbacks *alloc, VkImage *image) +{ + const VkResult result = memory_create_image(device, info, alloc, image); + ps5::memory::event("image_create", result == VK_SUCCESS); + return result; +} +VKAPI_ATTR void VKAPI_CALL memory_destroy(VkDevice device, VkImage image, + const VkAllocationCallbacks *alloc) +{ + memory_destroy_image(device, image, alloc); + if (image) + ps5::memory::event("image_destroy"); +} +VKAPI_ATTR VkResult VKAPI_CALL memory_idle(VkQueue queue) +{ + ps5::memory::event("idle_begin"); + const VkResult result = memory_queue_idle(queue); + ps5::memory::event("idle_end", result == VK_SUCCESS); + return result; +} +} // namespace +#endif + extern "C" void ps5_vulkan_trace_symbol(const char *name, PFN_vkVoidFunction *symbol) { +#ifdef PS5_MEMORY_DIAGNOSTICS + if (symbol && *symbol) + { + if (!std::strcmp(name, "vkCreateImage")) + { + if (*symbol != reinterpret_cast(memory_create)) + memory_create_image = reinterpret_cast(*symbol); + *symbol = reinterpret_cast(memory_create); + } + else if (!std::strcmp(name, "vkDestroyImage")) + { + if (*symbol != reinterpret_cast(memory_destroy)) + memory_destroy_image = reinterpret_cast(*symbol); + *symbol = reinterpret_cast(memory_destroy); + } + else if (!std::strcmp(name, "vkQueueWaitIdle")) + { + if (*symbol != reinterpret_cast(memory_idle)) + memory_queue_idle = reinterpret_cast(*symbol); + *symbol = reinterpret_cast(memory_idle); + } + } +#endif + if (!*symbol) return; if (std::strcmp(name, "vkEndCommandBuffer") == 0) diff --git a/tests/memory_diagnostics_test.cpp b/tests/memory_diagnostics_test.cpp new file mode 100644 index 0000000..0de8ea5 --- /dev/null +++ b/tests/memory_diagnostics_test.cpp @@ -0,0 +1,131 @@ +#include "src/memory_diagnostics.hpp" +#include +#include +#include +#include +#include +#include +#include + +static uint64_t clock_ns = 1; +extern "C" int __wrap_clock_gettime(clockid_t, timespec *t) +{ + t->tv_sec = clock_ns / 1000000000ULL; + t->tv_nsec = clock_ns % 1000000000ULL; + return 0; +} +static bool fail_malloc = false, fail_realloc = false; +extern "C" +{ + void *__wrap_malloc(size_t); + void *__wrap_calloc(size_t, size_t); + void *__wrap_realloc(void *, size_t); + void __wrap_free(void *); + int __wrap_posix_memalign(void **, size_t, size_t); + void *__real_malloc(size_t n) + { + if (fail_malloc) + { + errno = ENOMEM; + return nullptr; + } + return std::malloc(n); + } + void *__real_calloc(size_t n, size_t s) + { + return std::calloc(n, s); + } + void *__real_realloc(void *p, size_t n) + { + if (fail_realloc) + { + errno = ENOMEM; + return nullptr; + } + return std::realloc(p, n); + } + void __real_free(void *p) + { + std::free(p); + } + int __real_posix_memalign(void **p, size_t a, size_t n) + { + return posix_memalign(p, a, n); + } +} +int main(int argc, char **argv) +{ + assert(argc == 2); + using namespace ps5::memory; + init(argv[1], "host-test"); + auto *p = static_cast(__wrap_malloc(123)); + assert(p); + std::memset(p, 37, 123); + assert(snapshot().bytes[0] == 123 && snapshot().count[0] == 1); + tick(); // Before the five-second boundary, no sample is emitted. + clock_ns += 5000000000ULL; + tick(); + tick(); // Same instant must not emit a duplicate. + fail_realloc = true; + assert(!__wrap_realloc(p, 456) && errno == ENOMEM); + assert(p[122] == 37 && snapshot().bytes[0] == 123); + fail_realloc = false; + p = static_cast(__wrap_realloc(p, 2 * 1024 * 1024)); + assert(p && p[122] == 37); + assert(snapshot().bytes[0] == 2 * 1024 * 1024 && !snapshot().bytes[1]); + __wrap_free(p); + void *mapped = __wrap_malloc(2 * 1024 * 1024); + assert(mapped && snapshot().bytes[1] == 2 * 1024 * 1024); + mapped = __wrap_realloc(mapped, 47); + assert(mapped && snapshot().bytes[0] == 47 && !snapshot().bytes[1]); + __wrap_free(mapped); + void *aligned = nullptr; + errno = EDOM; + assert(!__wrap_posix_memalign(&aligned, 256, 321) && errno == EDOM); + assert(uintptr_t(aligned) % 256 == 0 && snapshot().bytes[2] == 321); + __wrap_free(aligned); + aligned = reinterpret_cast(uintptr_t(99)); + assert(__wrap_posix_memalign(&aligned, 3, 321) == EINVAL && errno == EDOM); + assert(aligned == reinterpret_cast(uintptr_t(99))); + fail_malloc = true; + assert(!__wrap_malloc(66) && errno == ENOMEM); + // The OOM logger must work while wrapped allocation fails. + failure("forced", 77, 0, 0x1234, ENOMEM); + fail_malloc = false; + assert(!__wrap_calloc(SIZE_MAX, 2)); + void *foreign = std::malloc(5); + foreign = __wrap_realloc(foreign, 9); + __wrap_free(foreign); + __wrap_free(strdup("foreign")); + std::vector workers; + for (int t = 0; t < 4; ++t) + workers.emplace_back( + [] + { + for (int i = 0; i < 2000; ++i) + { + void *q = __wrap_calloc(1, 59); + assert(q); + q = __wrap_realloc(q, 101); + assert(q); + __wrap_free(q); + } + }); + for (auto &worker : workers) + worker.join(); + // Saturation is explicitly reported; it never changes allocation results. + for (uintptr_t i = 1; i <= 131073; ++i) + add({reinterpret_cast(i * 16), 1, 0x1000, Route::Native}); + assert(snapshot().dropped == 1); + for (uintptr_t i = 1; i <= 131073; ++i) + take(reinterpret_cast(i * 16)); + event("image_create"); + event("image_destroy"); + event("idle_begin"); + event("idle_end", false); + const auto stats = snapshot(); + assert(!stats.count[0] && !stats.count[1] && !stats.count[2]); + assert(!stats.bytes[0] && !stats.bytes[1] && !stats.bytes[2]); + assert(stats.failures == 5 && stats.foreign_reallocs == 1 && stats.foreign_frees == 2); + finish(); +} diff --git a/tests/memory_vulkan_test.cpp b/tests/memory_vulkan_test.cpp new file mode 100644 index 0000000..4a7f243 --- /dev/null +++ b/tests/memory_vulkan_test.cpp @@ -0,0 +1,55 @@ +#include "src/vulkan_trace.cpp" +#include +static VkDevice device = reinterpret_cast(1); +static VkQueue queue = reinterpret_cast(2); +static VkImage image_handle = reinterpret_cast(3); +static VkImageCreateInfo info{}; +static VkAllocationCallbacks allocator{}; +static unsigned created = 0, destroyed = 0, waited = 0; +static VkResult VKAPI_CALL create_image(VkDevice d, const VkImageCreateInfo *i, + const VkAllocationCallbacks *a, VkImage *image) +{ + assert(d == device && i == &info && a == &allocator); + if (++created == 1) + { + *image = image_handle; + return VK_SUCCESS; + } + return VK_ERROR_OUT_OF_HOST_MEMORY; +} +static void VKAPI_CALL destroy_image(VkDevice d, VkImage i, const VkAllocationCallbacks *a) +{ + assert(d == device && (i == image_handle || !i) && a == &allocator); + ++destroyed; +} +static VkResult VKAPI_CALL queue_idle(VkQueue q) +{ + assert(q == queue); + ++waited; + return VK_ERROR_DEVICE_LOST; +} +int main(int argc, char **argv) +{ + assert(argc == 2); + ps5::memory::init(argv[1], "dispatch-test"); + PFN_vkVoidFunction fn = reinterpret_cast(create_image); + ps5_vulkan_trace_symbol("vkCreateImage", &fn); + ps5_vulkan_trace_symbol("vkCreateImage", &fn); + VkImage image{}; + assert(reinterpret_cast(fn)(device, &info, &allocator, &image) == + VK_SUCCESS); + assert(image == image_handle); + assert(reinterpret_cast(fn)(device, &info, &allocator, &image) == + VK_ERROR_OUT_OF_HOST_MEMORY); + fn = reinterpret_cast(destroy_image); + ps5_vulkan_trace_symbol("vkDestroyImage", &fn); + ps5_vulkan_trace_symbol("vkDestroyImage", &fn); + reinterpret_cast(fn)(device, image, &allocator); + reinterpret_cast(fn)(device, VK_NULL_HANDLE, &allocator); + fn = reinterpret_cast(queue_idle); + ps5_vulkan_trace_symbol("vkQueueWaitIdle", &fn); + ps5_vulkan_trace_symbol("vkQueueWaitIdle", &fn); + assert(reinterpret_cast(fn)(queue) == VK_ERROR_DEVICE_LOST); + assert(created == 2 && destroyed == 2 && waited == 1); + ps5::memory::finish(); +} diff --git a/tests/test_memory_diagnostics.py b/tests/test_memory_diagnostics.py new file mode 100644 index 0000000..56bbf11 --- /dev/null +++ b/tests/test_memory_diagnostics.py @@ -0,0 +1,49 @@ +"""Diagnostic counts, failure-path durability and allocator semantics.""" +from pathlib import Path +import subprocess +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parent.parent + + +class MemoryDiagnostics(unittest.TestCase): + def test_counts_failure_logging_and_ownership(self): + with tempfile.TemporaryDirectory() as directory: + binary = Path(directory) / 'diagnostics-test' + capture = Path(directory) / 'memory.log' + subprocess.run(['c++', '-std=c++20', '-pthread', '-I.', + '-DPS5_MEMORY_DIAGNOSTICS', '-Wl,--wrap=clock_gettime', + 'tests/memory_diagnostics_test.cpp', + 'src/memory_ps5.cpp', 'src/memory_diagnostics.cpp', + '-o', str(binary)], cwd=ROOT, check=True) + subprocess.run([str(binary), str(capture)], check=True) + text = capture.read_text() + self.assertIn('session host-test', text) + self.assertEqual(sum(line.startswith('sample ') for line in text.splitlines()), 1) + self.assertIn('caller seq=', text) + self.assertIn('ms=5000', text) + self.assertEqual(text.count('failure op='), 5) + self.assertIn('failure op=forced', text) + final = next(line for line in text.splitlines() if line.startswith('final ')) + fields = dict(item.split('=') for item in final.split()[1:]) + for field in ('native_bytes', 'mapped_bytes', 'aligned_bytes', + 'native_count', 'mapped_count', 'aligned_count'): + self.assertEqual(fields[field], '0') + self.assertEqual(fields['dropped'], '1') + self.assertEqual(fields['image_create'], fields['image_destroy']) + self.assertEqual(fields['idle_failed'], '1') + + def test_vulkan_image_and_idle_dispatch(self): + with tempfile.TemporaryDirectory() as directory: + binary = Path(directory) / 'dispatch-test' + capture = Path(directory) / 'memory.log' + subprocess.run(['c++', '-std=c++20', '-pthread', '-I.', '-Ivendor/retroarch', + '-DPS5_MEMORY_DIAGNOSTICS', 'tests/memory_vulkan_test.cpp', + 'src/memory_diagnostics.cpp', '-o', str(binary)], cwd=ROOT, check=True) + subprocess.run([str(binary), str(capture)], check=True) + final = next(line for line in capture.read_text().splitlines() if line.startswith('final ')) + fields = dict(item.split('=') for item in final.split()[1:]) + for field in ('image_create', 'image_destroy', 'image_failed', + 'idle_begin', 'idle_end', 'idle_failed'): + self.assertEqual(fields[field], '1') diff --git a/tools/build-title.sh b/tools/build-title.sh index c0bf5bd..b9c18b9 100755 --- a/tools/build-title.sh +++ b/tools/build-title.sh @@ -36,6 +36,11 @@ set -euo pipefail root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) cd "$root" +# Opt-in diagnostics do not change allocation routing or normal builds. +memory_diagnostics=${PS5_MEMORY_DIAGNOSTICS:-0} +[[ $memory_diagnostics == 0 || $memory_diagnostics == 1 ]] || { + echo "PS5_MEMORY_DIAGNOSTICS must be 0 or 1" >&2; exit 2; +} stage=false case "${1:-}" in '') ;; @@ -89,6 +94,12 @@ for define in "${title_defines[@]}"; do title_definition_names+=("${define#-D}") done (( ${#title_definition_names[@]} > 0 )) || { echo "error: no feature defines to pass" >&2; exit 2; } +memory_wrap_flags="" +if [[ $memory_diagnostics == 1 ]]; then + title_definition_names+=(PS5_MEMORY_DIAGNOSTICS) + # Mesa's default Vulkan host allocator uses posix_memalign, not malloc. + memory_wrap_flags="--wrap=posix_memalign" +fi echo "==> [title] compiling src/ with ${#title_definition_names[@]} frontend defines" # RetroArch's headers reach their generated config as "../../config.h", a relative @@ -132,6 +143,33 @@ if (( ${#vulkan_missing[@]} )); then printf ' missing: %s\n' "${vulkan_missing[@]}" >&2 exit 2 fi +# The driver may be developed concurrently. A diagnostic link uses stable local +# archive copies; hashes describe exactly which driver went into this build. +if [[ $memory_diagnostics == 1 ]]; then + if ! snapshot_list=$(python3 - "$root" "${vulkan_archives[@]}" <<'PY_SNAPSHOT' +import hashlib, json, pathlib, shutil, sys +out = pathlib.Path(sys.argv[1]) / "build/memory-diagnostic-inputs" +out.mkdir(parents=True, exist_ok=True) +records = {} +for argument in sys.argv[2:]: + source = pathlib.Path(argument) + before = hashlib.sha256(source.read_bytes()).hexdigest() + target = out / source.name + shutil.copyfile(source, target) + copied = hashlib.sha256(target.read_bytes()).hexdigest() + after = hashlib.sha256(source.read_bytes()).hexdigest() + if before != copied or before != after: + raise SystemExit("Driver archive changed during snapshot; retry when its build finishes") + records[source.name] = copied + print(target) +(out / "archives.json").write_text(json.dumps(records, indent=2) + "\n") +PY_SNAPSHOT + ); then + echo "error: driver snapshot failed" >&2; exit 2 + fi + mapfile -t vulkan_archives <<< "$snapshot_list" +fi + # Mesa's weak entry points resolve at link time, and the driver's own symbols must # survive the archive boundary (--whole-archive), which is how the sibling links it. vulkan_flags="--no-dynamic-linker -z nodynamic-undefined-weak" @@ -153,7 +191,7 @@ mapfile -t vulkan_objects <<< "$vulkan_object_list" # Bind the running trace and FTP readback to these exact source/archive inputs. # The console transforms the SELF container, so its whole-file digest differs. -python3 - "$root" "${vulkan_archives[@]}" "${vulkan_objects[@]}" <<'PY' +python3 - "$root" "$memory_diagnostics" "${vulkan_archives[@]}" "${vulkan_objects[@]}" <<'PY' import hashlib, pathlib, sys root = pathlib.Path(sys.argv[1]) inputs = sorted(p for p in (root / "src").rglob("*") if p.is_file()) @@ -165,8 +203,9 @@ inputs += [root / name for name in ( "build/cores/stage/cores/fbneo_libretro.so", "build/cores/stage/cores/genesis_plus_gx_libretro.so", "tools/build.sh", "tools/retroarch-flags.sh")] -inputs += [pathlib.Path(name) for name in sys.argv[2:]] +inputs += [pathlib.Path(name) for name in sys.argv[3:]] digest = hashlib.sha256() +digest.update(b"memory-diagnostics=" + sys.argv[2].encode() + b"\0") for path in inputs: digest.update(path.name.encode() + b"\0") digest.update(hashlib.sha256(path.read_bytes()).digest()) @@ -186,7 +225,7 @@ APP_INCLUDE_PATHS="build/ra-conf build vendor/retroarch build/ra-conf/libretro-c APP_STATIC_ARCHIVES="build/ra/libretroarch.a" \ APP_VULKAN_ARCHIVES="${vulkan_archives[*]}" \ APP_EXTRA_OBJECTS="${vulkan_objects[*]}" \ -APP_LINK_FLAGS="$vulkan_flags --wrap=malloc --wrap=calloc --wrap=realloc --wrap=free" \ +APP_LINK_FLAGS="$vulkan_flags --wrap=malloc --wrap=calloc --wrap=realloc --wrap=free $memory_wrap_flags" \ make app title_id=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["titleId"])' \ diff --git a/tools/check-memory-diagnostics.py b/tools/check-memory-diagnostics.py new file mode 100644 index 0000000..fe2fb6c --- /dev/null +++ b/tools/check-memory-diagnostics.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Inspect a locally built diagnostic; never contacts the console. + +PS5_MEMORY_DIAGNOSTICS=1 bash tools/verify.sh +python3 tools/check-memory-diagnostics.py > build/memory-diagnostics-inspection.json +""" +import hashlib +import json +from pathlib import Path +import re +import subprocess + +ROOT = Path(__file__).resolve().parent.parent + + +def inspect(): + elf = ROOT / 'build/llvm-pie.elf' + symbols = subprocess.check_output(['llvm-nm', '-C', str(elf)], text=True) + required = ('__wrap_malloc', '__wrap_calloc', '__wrap_realloc', '__wrap_free', + '__wrap_posix_memalign', 'ps5::memory::init(', 'ps5::memory::failure(', + 'ps5::memory::tick(', 'memory_create(', 'memory_destroy(', 'memory_idle(') + missing = [name for name in required if name not in symbols] + if missing: + raise SystemExit('Diagnostic symbol missing: ' + ', '.join(missing)) + identity = re.search(r'build identity: ([a-f0-9]{64})', + (ROOT / 'build/title_build_identity.h').read_text()).group(1) + hashes = {} + for relative in ('build/llvm-pie.elf', 'build/title.map', 'dist/PPSA99169/eboot.bin'): + hashes[relative] = hashlib.sha256((ROOT / relative).read_bytes()).hexdigest() + archives = json.loads((ROOT / 'build/memory-diagnostic-inputs/archives.json').read_text()) + for name, expected in archives.items(): + actual = hashlib.sha256((ROOT / 'build/memory-diagnostic-inputs' / name).read_bytes()).hexdigest() + if actual != expected: + raise SystemExit('Driver snapshot digest mismatch: ' + name) + return {'kind': 'host-only-diagnostic-inspection', 'passed': True, + 'build_identity': identity, 'symbols_present': list(required), + 'artifacts_sha256': hashes, 'driver_archives_sha256': archives, + 'console_test': 'pending; owner requested no automatic upload or launch'} + + +if __name__ == '__main__': + print(json.dumps(inspect(), indent=2))