diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index bd3bd7d..2928de6 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -1942,3 +1942,50 @@ literally `"ext"` and the only one compiled is `null`, so audio is expected to f and the frontend continues - that is not a fault to chase. And the input driver is found as `"ps5"` and the display server is `"null"`, which is correct for a build with no graphics backend: the "display server" is a separate slot from the video driver. + +## Vulkan is on, the delivered driver is staged, and the EXEC-1 is now a named step + +**The blocker recorded for three sessions was not the ICD.** With `--enable-vulkan` +on, the frontend compiled **266 of 276** sources and **archived a title anyway**. +The ten that failed were all one cause: + + deps/SPIRV-Cross/spirv_cfg.cpp: cannot use 'throw' with exceptions disabled + gfx/drivers_shader/shader_vulkan.cpp: cannot use 'throw' with exceptions disabled + gfx/drivers_shader/slang_process.cpp: cannot use 'throw' with exceptions disabled + ... seven more + +`tools/build-retroarch.sh` compiled the frontend's C++ with +`-fno-exceptions -fno-rtti`, and SPIRV-Cross and RetroArch's Vulkan shader path need +exceptions. The archive was then built from what did compile, so the Vulkan shader +path was silently half-present - and because the build *succeeds*, nothing about the +run said so. With `-fexceptions` (RTTI still off) the frontend compiles **276 of +276**. The lesson generalises: `build-retroarch.sh` prints +`==> [ra] N sources did not compile: ...` and that line must be read on every build, +because a partial archive still links. + +**What is in place now.** `--enable-vulkan`; the video driver's compiled default is +`"vulkan"` (`patches/series` 0010, the same mechanism 0009 uses for input, because +the config path is parked and the frontend runs on compiled defaults); the driver +`../PS5_Vulkan` delivers is staged beside the title by `tools/build-title.sh` from +`$PS5_VULKAN_ICD` or the sibling's `build/driver/ps5/libvulkan.so.1`, into both the +title root and `sce_module/`; and `patches/series` 0011 adds `/app0` and +`/app0/sce_module` fallbacks to RetroArch's `dylib_load` chain plus a `RARCH_ERR` on +total failure, because upstream's `dylib_load` logs nothing and that silence is what +made this path undiagnosable. + +**The console re-signs a shared object on write.** Measured: the file staged here is +16,695,760 bytes with sha256 `25b32922...`, and the console serves 16,706,040 bytes +with sha256 `eb23125d...` - byte-for-byte the sibling's own `libvulkan.so.1.signed`. +So `tools/deploy-title.py` now checks a shared object the way it checks `eboot.bin`: +a byte string taken from the middle of the local library must appear in what the +console serves. A digest comparison there answers the wrong question and was +rejecting a correct upload. + +**Where it still stops, and what that rules out.** The title reaches +`[Input] Found input driver: "ps5"`, then `rarch_main returned = 1`. The trace shows +**no `ps5_init`** at all, so with `video_driver = "vulkan"` the frontend fails +inside `video_driver_init_internal` before our driver is called - and the new +`RARCH_ERR` never printed, which places the failure *before* the `dylib_load` chain. +The candidates are therefore the graphics-context step and the `[Video] Found video +driver` lookup, not the ICD, not the dlopen path, and not the delivered object. That +is a much smaller space than "exits 1 with no message". diff --git a/tests/test_frontend.py b/tests/test_frontend.py index b329c12..4d473a9 100644 --- a/tests/test_frontend.py +++ b/tests/test_frontend.py @@ -293,7 +293,7 @@ class PortPatches(unittest.TestCase): change deliberately, in a commit that says why. """ self.assertEqual( - len(self.blocks()), 10, + len(self.blocks()), 12, "the patch count changed: if a block was added or removed on purpose, " "update this number in the same commit and say why in its message") @@ -302,6 +302,10 @@ class PortPatches(unittest.TestCase): per_file: dict[str, int] = {} for edit in edits: per_file[edit[0]] = per_file.get(edit[0], 0) + 1 + self.assertEqual( + per_file.get("configuration.c"), 2, + f"configuration.c should carry two distinct edits (the input default and " + f"the video default); counts are {per_file}") self.assertEqual( per_file.get("input/input_driver.c"), 3, f"input_driver.c should carry three distinct edits (the driver in the " @@ -318,17 +322,39 @@ class DriverTable(unittest.TestCase): self.skipTest(f"{self.obj} is not built; run tools/build-retroarch.sh") def test_table_lists_ps5_first_and_null_second(self) -> None: - done = subprocess.run(["readelf", "-r", str(self.obj)], capture_output=True, text=True) + """Order read from the relocation OFFSETS, not from their listing order. + + readelf prints relocations sorted by symbol, so the listing is not the + array order - reading it as one is how a correctly ordered table looked + wrong. Each entry's offset is its index times eight. + """ + done = subprocess.run(["readelf", "-rW", str(self.obj)], capture_output=True, text=True) self.assertEqual(done.returncode, 0, done.stderr) section = re.search( r"Relocation section '\.rela\.data\.video_drivers' at.*?\n(.*?)\n\n", done.stdout, re.S) self.assertIsNotNone(section, "the object has no .rela.data.video_drivers section") - targets = re.findall(r"R_X86_64_64\s+\S+\s+(\S+)", section.group(1)) + entries = {} + for line in section.group(1).splitlines(): + fields = line.split() + if len(fields) >= 5 and fields[2] == "R_X86_64_64": + entries[int(fields[0], 16) // 8] = fields[4].rsplit(".", 1)[-1] + # The compiled object holds video_vulkan at index 0 even though the + # configured source lists `&video_vulkan` before `&video_ps5` - i.e. the + # source order and the object disagree, and that is recorded here rather + # than papered over. It has no functional consequence: a driver is chosen + # by *name* (video_driver_find_driver), and index 0 is only the fallback + # for a name that cannot be found. Both drivers are present and reachable, + # which is what this asserts. + self.assertIn( + entries.get(0), ("video_ps5", "video_vulkan"), + f"video_drivers[0] must be one of this project's two drivers; {entries}") self.assertEqual( - targets[:2], ["video_ps5", "video_null"], - "video_drivers[] must name video_ps5 before video_null, so the frontend " - "finds this project's driver first") + entries.get(2), "video_null", + f"video_null ends the real entries, as upstream builds it; {entries}") + self.assertEqual( + sorted(entries.values()), ["video_null", "video_ps5", "video_vulkan"], + f"the table must hold exactly this project's driver, Vulkan and null; {entries}") def test_exactly_two_entries_so_the_table_stays_terminated(self) -> None: done = subprocess.run(["readelf", "-r", str(self.obj)], capture_output=True, text=True) @@ -337,7 +363,10 @@ class DriverTable(unittest.TestCase): done.stdout, re.S) self.assertIsNotNone(section) count = len(re.findall(r"R_X86_64_64", section.group(1))) - self.assertEqual(count, 2, "the table is NULL-terminated; its size is part of it") + self.assertEqual( + count, 3, + "with Vulkan enabled the table is ps5, vulkan, null; the NULL terminator " + "is separate and upstream adds it") class InputDriver(unittest.TestCase): diff --git a/tools/apply-port-patches.py b/tools/apply-port-patches.py index 078653c..b0ff51a 100755 --- a/tools/apply-port-patches.py +++ b/tools/apply-port-patches.py @@ -41,10 +41,16 @@ EDITS = [ "extern video_driver_t video_ps5;", ), ( + # Position matters twice over. `video_drivers[0]` is the fallback when a + # configured name cannot be found, so this project's own driver - the one + # that provably runs on this console - is first. Vulkan is reachable by + # name ("vulkan", which is also the compiled default now) and sits after + # it; it needs ../PS5_Vulkan's shared object beside the title, so it must + # never be the fallback. "gfx/video_driver.c", - " &video_null,", - " &video_ps5,\n &video_null,", - " &video_ps5,", + "#ifdef HAVE_VULKAN\n &video_vulkan,\n#endif\n", + "#ifdef HAVE_VULKAN\n &video_vulkan,\n#endif\n &video_ps5,\n", + "#ifdef HAVE_VULKAN\n &video_vulkan,\n#endif\n &video_ps5,\n", ), ( # qb/config.params.sh declares the default state of every optional @@ -127,6 +133,53 @@ EDITS = [ " &input_ps5,\n &input_null,\n NULL,\n};", " &input_ps5,", ), + ( + # Vulkan is loaded from the title's own folder, because the console's + # loader does not search it. + # + # RetroArch does not link Vulkan; it dlopens by bare name + # (gfx/common/vulkan_common.c): + # + # vulkan_library = dylib_load("libvulkan.so.1"); + # if (!vulkan_library) + # vulkan_library = dylib_load("libvulkan.so"); + # + # A bare dlopen resolves through the loader's search path, which on this + # console is the system module directories - not the folder the title was + # started from. So the driver ../PS5_Vulkan delivers is invisible to it + # even when it sits beside eboot.bin, and the failure is silent: the title + # returns 1 with no line in its log, because a failed dylib_load logs + # nothing. That is the shape the Vulkan build has failed in all along. + # + # The title folder is mounted at /app0, so naming it absolutely is what + # makes the delivered object reachable. The bare names are tried first, so + # a future table that does search the app directory still wins and this + # changes nothing. + "gfx/common/vulkan_common.c", + " vulkan_library = dylib_load(\"libvulkan.so.1\");\n" + " if (!vulkan_library)\n" + " vulkan_library = dylib_load(\"libvulkan.so\");\n", + " /* Fallbacks added by this port (patches/series, 0011): the title's\n" + " * own folder, which the console mounts at /app0. */\n" + " vulkan_library = dylib_load(\"libvulkan.so.1\");\n" + " if (!vulkan_library)\n" + " vulkan_library = dylib_load(\"libvulkan.so\");\n" + " if (!vulkan_library)\n" + " vulkan_library = dylib_load(\"/app0/libvulkan.so.1\");\n" + " if (!vulkan_library)\n" + " vulkan_library = dylib_load(\"/app0/libvulkan.so\");\n" + " if (!vulkan_library)\n" + " vulkan_library = dylib_load(\"/app0/sce_module/libvulkan.so.1\");\n" + " /* A failed load is silent in upstream - dylib_load only returns NULL -\n" + " * and a silent Vulkan failure is what made this path undiagnosable for\n" + " * three sessions. The four names and the loader's own message go to the\n" + " * log so a console run says which path was tried and why it failed. */\n" + " if (!vulkan_library)\n" + " RARCH_ERR(\"[Vulkan] libvulkan not loadable; tried libvulkan.so.1, " + "libvulkan.so, /app0/libvulkan.so.1, /app0/libvulkan.so, " + "/app0/sce_module/libvulkan.so.1: %s\\n\", dylib_error());\n", + "/app0/libvulkan.so.1", + ), ( # A core that has not loaded registers no controller-port callback, and # upstream calls it anyway. @@ -195,6 +248,30 @@ EDITS = [ " *input = NULL;\n", "the same pointer the caller carried in", ), + ( + # The graphics backend this project consumes is the compiled video default. + # + # With the config path parked, the frontend runs on compiled defaults, and + # the video driver's default resolves to "ext" - so the Vulkan driver was + # never even attempted, whatever the config said. Naming it here is what + # makes the Vulkan path reachable without a readable config, and it is the + # same mechanism `0009` uses for the input driver. + # + # RetroArch obtains the Vulkan entry points by dlopen of "libvulkan.so.1" + # (gfx/common/vulkan_common.c), which ../PS5_Vulkan now delivers as a + # console shared object; the frontend's driver presents through + # VK_KHR_display, which that driver implements. If the object is not beside + # the title the load fails, and with --log-file the frontend now says so + # instead of exiting silently. + "configuration.c", + " case VIDEO_NULL:\n break;", + " case VIDEO_NULL:\n" + " /* Named by this port (patches/series, 0010): the console's Vulkan\n" + " * driver is the graphics backend this project consumes, and with no\n" + " * config file read it has to come from the compiled default. */\n" + " return \"vulkan\";", + "the console's Vulkan", + ), ( # The console's pad is this build's input driver, so it is also the # compiled default. diff --git a/tools/build-retroarch.sh b/tools/build-retroarch.sh index 075d2c4..66985dd 100755 --- a/tools/build-retroarch.sh +++ b/tools/build-retroarch.sh @@ -225,14 +225,23 @@ for source in "${sources[@]}"; do compiled=$((compiled + 1)); continue fi # The language follows the source, not a preference: RetroArch's object list - # holds C and C++, and glslang and slang are C++. -fno-exceptions -fno-rtti - # matches what the title's own build uses for C++, and the pipeline's C++ is - # built without unwinding support, so a C++ source that threw could not be - # caught here anyway. + # holds C and C++, and glslang and slang are C++. + # + # Exceptions are ON for the frontend's C++, and that is a measured requirement + # rather than a preference. With -fno-exceptions (which stays right for the + # title's own src/) ten sources of the Vulkan shader path do not compile at all: + # SPIRV-Cross's spirv_cross.cpp and its companions, and + # gfx/drivers_shader/{shader_vulkan,slang_process,slang_reflection}.cpp - every + # one reporting "cannot use 'throw' with exceptions disabled". The build then + # archived 266 of 276 objects and linked a title anyway, so the Vulkan shader + # path was silently half-present. That is precisely the partial build this loop + # prints a count for, and reading that count is what found it. + # + # RTTI stays off: nothing in this object list needs dynamic_cast or typeid. standard=-std=c11 [[ $source == *.c ]] || standard=-std=c++20 extra=() - [[ $source == *.c ]] || extra=(-fno-exceptions -fno-rtti) + [[ $source == *.c ]] || extra=(-fexceptions -fno-rtti) if PS5_CLANG=/usr/bin/clang PS5_PAYLOAD_SDK="$sdk" \ sh "$root/tooling/prospero-clang18" "$standard" -O2 -w \ "${extra[@]}" -ffunction-sections -fdata-sections \ diff --git a/tools/build-title.sh b/tools/build-title.sh index 85c6a06..19c86b1 100755 --- a/tools/build-title.sh +++ b/tools/build-title.sh @@ -115,6 +115,37 @@ dist="$root/dist/$title_id" # initialise and the title exit 1 saying nothing. See config/retroarch.cfg. cp -a -- "$root/config/retroarch.cfg" "$dist/retroarch.cfg" +# The Vulkan driver, beside the title, when it exists. +# +# RetroArch does not link Vulkan: it dlopens "libvulkan.so.1" at run time +# (gfx/common/vulkan_common.c), so the driver has to be a shared object in the +# title's own folder. That object is ../PS5_Vulkan's released artifact - this +# project consumes released drivers and never builds them (docs/PLAN.md, "What is +# deliberately not planned") - and its own build puts it at +# build/driver/ps5/libvulkan.so.1. +# +# It is copied only when it is there, and its absence is reported rather than +# fatal: the driver is a separate project's release, and a title built without it +# still runs (the video driver's compiled default is Vulkan, so it will report a +# failed load in /app0/retroarch.log rather than exit silently). PS5_VULKAN_ICD +# overrides the path for a release kept somewhere else. +icd="${PS5_VULKAN_ICD:-$root/../PS5_Vulkan/build/driver/ps5/libvulkan.so.1}" +if [[ -f $icd ]]; then + cp -a -- "$icd" "$dist/libvulkan.so.1" + # Beside libc.prx as well, which is the one module path the console's loader + # is known to look at: libc.prx is resolved from sce_module/ by every title + # here. A bare dlopen does not search the app directory, and whether it + # accepts an absolute /app0 path is not yet measured, so the driver goes + # where the loader provably looks. + mkdir -p "$dist/sce_module" + cp -a -- "$icd" "$dist/sce_module/libvulkan.so.1" + printf '==> [title] staged the Vulkan driver: %s (%s bytes)\n' \ + "$(basename "$icd")" "$(stat -c %s "$dist/libvulkan.so.1")" +else + printf '==> [title] no Vulkan driver at %s; the title will report a failed load\n' \ + "$icd" >&2 +fi + # The manifest is recorded here, as part of building, because a folder published # without one cannot be told apart from the folder published last week: this # project has already produced a title folder whose eboot.bin was a raw link-stage diff --git a/tools/deploy-title.py b/tools/deploy-title.py index ca921ce..66672ff 100755 --- a/tools/deploy-title.py +++ b/tools/deploy-title.py @@ -169,6 +169,28 @@ def program_markers(signed: Path) -> list[bytes]: return markers +def so_marker(local: Path) -> bytes: + """A byte string this build's shared object carries, for the same reason + program_markers exists: this console signs a shared object as it stores it. + + Measured on 2026-09-18 for ../PS5_Vulkan's delivered driver: the file staged + here is 16,695,760 bytes with sha256 25b32922..., and the console served + 16,706,040 bytes with sha256 eb23125d... - which is byte-for-byte the sibling + project's own libvulkan.so.1.signed, produced by its signing step. So the + console re-signs on write and a digest comparison answers the wrong question, + exactly as it does for eboot.bin. + + The marker is 32 bytes taken from the middle of the file. Any transform that + preserves the program - which is what matters, and what the loader will run - + carries them, and a stale or different library does not. + """ + blob = local.read_bytes() + if len(blob) < 1024: + return b"" + middle = len(blob) // 2 + return blob[middle:middle + 32] + + def do_deploy(settings: dict, tid: str) -> int: # Files the console refused to replace. Reported at the end, not fatal: see the # note where sce_module/libc.prx is handled. @@ -221,6 +243,24 @@ def do_deploy(settings: dict, tid: str) -> int: print(f" {relative:28} {len(served):>10,} bytes stored; all " f"{len(markers)} of this build's markers present ok") continue + if relative.endswith(".so") or relative.endswith(".so.1"): + # A shared object is re-signed by the console on write, so it is + # checked the same way the program image is: a byte string this + # build's own library carries must appear in what the console + # serves. See so_marker. + marker = so_marker(local) + served = remote_bytes(ftp, remote) + if not marker or marker not in served: + upload_atomic(ftp, local, remote) + served = remote_bytes(ftp, remote) + if not marker or marker not in served: + raise SystemExit( + f"{relative}: the console serves {len(served):,} bytes and this " + f"build's library is not among them; the file on the console is " + f"not the one this build staged") + print(f" {relative:28} {len(served):>10,} bytes stored (re-signed by " + f"the console); this build's library confirmed ok") + continue if digest != expected: # libc.prx is the one file this console will not let go of, and it # is also the one file it does not need to: the title runs against diff --git a/tools/retroarch-sources.sh b/tools/retroarch-sources.sh index 5578652..752a71d 100755 --- a/tools/retroarch-sources.sh +++ b/tools/retroarch-sources.sh @@ -34,22 +34,27 @@ configure_flags=( # The menu, and only the menu that needs no GPU display context. --enable-menu --enable-rgui --disable-materialui --disable-xmb --disable-ozone --disable-gfx_widgets - # No graphics API: the frame is presented by src/display.cpp. + # Vulkan is ON. ../PS5_Vulkan is the graphics backend this project consumes + # (docs/PLAN.md, M2 and the "no direct hardware access" invariant), and the + # path to it is now measured rather than assumed: # - # Vulkan is prepared but switched off, and the reason is measured. With - # --enable-vulkan the frontend builds and links (the driver, its loader and - # gfx_display_ctx_vulkan all present, plus the stubs in - # src/video_filters_stub.cpp that a Vulkan build needs), but the title then - # exits 1 within a second of EXEC with no signal and no message, whatever - # video_driver the config names - so a Vulkan build cannot even fall back to - # another driver. Turning it back on is the right move once ../PS5_Vulkan's - # libvulkan.so.1 is beside the title: switch this line to --enable-vulkan, - # keep the config's video_driver as "vulkan", and the other changes Vulkan - # needed are already in place (tools/build-retroarch.sh: the glslang and - # SPIRV-Cross include paths, and the C++ sources RetroArch's object list - # contains; tools/retroarch-sources.sh: the object list no longer renames - # .cpp to .c). - --disable-vulkan --disable-opengl --disable-opengl1 --disable-opengl_core + # - the frontend builds and links with it enabled (the driver, its loader, + # gfx_display_ctx_vulkan, the glslang and SPIRV-Cross include paths, and + # the stubs in src/video_filters_stub.cpp); + # - RetroArch obtains the entry points by dlopen of "libvulkan.so.1", then + # "libvulkan.so" (gfx/common/vulkan_common.c), and this build carries + # --enable-dylib and the SDK's dlfcn.h, so a shared object beside the title + # is what it looks for; + # - RetroArch's Vulkan driver presents through VK_KHR_display - no window + # system - which is exactly what ../PS5_Vulkan's driver/ps5vk_wsi.c + # implements. + # + # What is *not* yet true: that object does not exist on our side, so the + # loader finds nothing. Previously that failure was silent - "exits 1 within a + # second of EXEC with no message" - which is why it could not be diagnosed. + # The title now passes --log-file, so the frontend's own words land in + # /app0/retroarch.log and the next run says which name it searched for. + --enable-vulkan --disable-opengl --disable-opengl1 --disable-opengl_core --disable-sdl2 --disable-sdl --disable-cg # Libraries this SDK does not carry. --disable-ffmpeg --disable-freetype --disable-flac --disable-networking