fix(relinker): take pointers into data between PS4 unwind functions as data

The code analysis of the Windows and macOS outputs followed every
relocation and lea target inside the code range as code. PS4 images keep
read-only data between unwind functions, and strings and tables there
decoded into garbage, failing the conversion. Outside the unwind functions
such a target is now code only when it is 16-byte aligned, starts less
than 256 bytes after an unwind function and decodes to a jump, return or
trap; lea targets go through the same rule instead of the graph's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

core/relinker/relinker/src/analysis/CodeInstructionCollector.cpp

@@ -96,8 +96,11 @@ std::set<Domain::VirtualAddress> CodeInstructionCollector::Collect(const std::ve
         range(offset, size);
         return offset;
     };
+    std::vector<std::pair<std::uint64_t, std::uint64_t>> sceFunctions;
     if (sceBase) {
         const auto unwindFunctions = UnusedNidFilter::ReadExceptionFunctions(bytes, headers, {}, {});
+        for (const auto& function : unwindFunctions) sceFunctions.emplace_back(function.Begin, function.End);
+        std::sort(sceFunctions.begin(), sceFunctions.end());
         for (auto& header : codeHeaders) {
             if (header.Type != 1 || (header.Flags & 1) == 0) continue;
             range(header.Offset, header.FileSize);
@@ -118,10 +121,54 @@ std::set<Domain::VirtualAddress> CodeInstructionCollector::Collect(const std::ve
     std::map<std::uint64_t, std::uint64_t> tables;
     const auto inTable = [&](std::uint64_t address) { const auto after = tables.upper_bound(address); return after != tables.begin() && address < std::prev(after)->second; };
     const auto addRoot = [&](std::uint64_t address) { if (isCode(address) && !inTable(address)) roots.insert(address); };
+    const auto insideSceFunction = [&](std::uint64_t address) {
+        const auto after = std::upper_bound(sceFunctions.begin(), sceFunctions.end(), address, [](std::uint64_t value, const std::pair<std::uint64_t, std::uint64_t>& function) { return value < function.first; });
+        return after != sceFunctions.begin() && address < std::prev(after)->second;
+    };
+    const auto nearSceFunctionEnd = [&](std::uint64_t address) {
+        const auto after = std::upper_bound(sceFunctions.begin(), sceFunctions.end(), address, [](std::uint64_t value, const std::pair<std::uint64_t, std::uint64_t>& function) { return value < function.first; });
+        if (after == sceFunctions.begin()) return false;
+        const auto end = std::prev(after)->second;
+        return address >= end && address - end < 256;
+    };
+    const auto codeAt = [&](std::uint64_t address) {
+        for (const auto& header : codeHeaders)
+            if (header.Type == 1 && (header.Flags & 1) != 0 && address >= header.MappedAddress && address - header.MappedAddress < header.FileSize)
+                return std::span<const std::uint8_t>(bytes.data() + header.Offset + (address - header.MappedAddress), header.FileSize - (address - header.MappedAddress));
+        return std::span<const std::uint8_t>();
+    };
+    const auto decodesToTransfer = [&](std::uint64_t address) {
+        const Codegen::X64InstructionDecoder probe;
+        for (std::size_t count = 0; count < 256; ++count) {
+            const auto code = codeAt(address);
+            if (code.empty()) return false;
+            Codegen::DecodedInstructionInfo info;
+            try {
+                info = probe.DecodeInstruction(code.data(), code.size());
+            } catch (const Codegen::CodegenException&) {
+                return false;
+            }
+            if (info.Length == 0 || info.Length > code.size()) return false;
+            switch (info.FlowKind) {
+                case Codegen::ControlFlowKind::UnconditionalJump:
+                case Codegen::ControlFlowKind::Return:
+                case Codegen::ControlFlowKind::IndirectJump:
+                case Codegen::ControlFlowKind::Trap:
+                    return true;
+                default:
+                    break;
+            }
+            address += info.Length;
+        }
+        return false;
+    };
+    const auto addPointerRoot = [&](std::uint64_t address) {
+        if (sceFunctions.empty() || insideSceFunction(address) || (address % 16 == 0 && nearSceFunctionEnd(address) && decodesToTransfer(address))) addRoot(address);
+    };
     const auto addAddressTakenRoot = [&](std::uint64_t address, const std::uint8_t* code, const Codegen::DecodedInstructionInfo& info) {
         if (!info.HasRipRelativeDisp || code[info.OpcodeOffset] != 0x8d || !(info.RexPrefix & 8)) return;
         const auto displacement = std::bit_cast<std::int32_t>(Io::ReadU32(bytes, fileOffset(address, info.Length) + info.RipRelativeDispOffset));
-        addRoot(address + info.Length + static_cast<std::int64_t>(displacement));
+        addPointerRoot(address + info.Length + static_cast<std::int64_t>(displacement));
     };
     const auto addFunction = [&](std::uint64_t begin, std::uint64_t size, bool symbolAlias) {
         if (size == 0) return;
@@ -226,7 +273,7 @@ std::set<Domain::VirtualAddress> CodeInstructionCollector::Collect(const std::ve
                     && addend <= static_cast<std::uint64_t>(std::numeric_limits<std::int64_t>::max())
                     && Io::ReadU64(bytes, symbol + 8) == addend;
                 if ((index != 0 && !zeroSectionSymbol && !matchingDefinedSymbol) || !pointers.Values.emplace(slot, addend).second) throw Domain::RelinkerException("Code analysis: invalid relative relocation", slot);
-                addRoot(addend);
+                addPointerRoot(addend);
             } else if (type == 1 || type == 6 || type == 7) {
                 if (index != 0 && index >= symbolCount) throw Domain::RelinkerException("Code analysis: invalid relocation symbol", slot);
                 const auto symbol = symbols + index * 24;
@@ -234,7 +281,7 @@ std::set<Domain::VirtualAddress> CodeInstructionCollector::Collect(const std::ve
                 else {
                     const auto target = Io::ReadU64(bytes, symbol + 8) + addend;
                     if (!pointers.Values.emplace(slot, target).second) throw Domain::RelinkerException("Code analysis: duplicate pointer relocation", slot);
-                    addRoot(target);
+                    addPointerRoot(target);
                 }
             }
         }
@@ -405,7 +452,7 @@ std::set<Domain::VirtualAddress> CodeInstructionCollector::Collect(const std::ve
             for (const auto root : roots) if (root >= header.MappedAddress && root - header.MappedAddress < header.FileSize) entries.push_back(root);
             if (entries.empty()) continue;
             const std::span<const std::uint8_t> text(bytes.data() + header.Offset, header.FileSize);
-            const auto graph = UnusedNidFilter::BuildControlFlowGraph(text, header.MappedAddress, entries.front(), entries, pointers, true, sectionReachable[i], [&](std::uint64_t count) { report(6000 + 2500 * count / std::max<std::uint64_t>(scanBytes, 1)); }, inTable);
+            const auto graph = UnusedNidFilter::BuildControlFlowGraph(text, header.MappedAddress, entries.front(), entries, pointers, sceFunctions.empty(), sectionReachable[i], [&](std::uint64_t count) { report(6000 + 2500 * count / std::max<std::uint64_t>(scanBytes, 1)); }, inTable);
             for (const auto address : graph->ReachableVaddrs()) {
                 if (!decodedInstructions.insert(address).second) continue;
                 instructions.insert(address);

core/relinker/relinker/tests/test_sce_code_extent.py

@@ -22,10 +22,11 @@ POINTER = 0xB00
 INTERPRETER = b"/libexec/ld-elf.so.1\0"
 OVERLAPPING = bytes.fromhex("b8 64 c3 90 90 72 fa c3")
 UNSUPPORTED_TLS = bytes.fromhex("64 2d 65 6c 66 2e c3")
+INVALID = bytes.fromhex("41 41 41 41 41 41 66 66 66 66 2e 0f 1f 84 00 00 00 00 00")
 
 
 def fixture(sce_block=True, plt_push_offset=8, gap=0, interpreter_pointer=False, data_pointer=False, gap_entry=False,
-            plt_at=None, early_function=False, function_tls=False):
+            plt_at=None, early_function=False, function_tls=False, pointer_target=None, gap_bytes=None):
     plt = FUNCTION + FUNCTION_SIZE + gap if plt_at is None else plt_at
     entry = FUNCTION + FUNCTION_SIZE if gap_entry else FUNCTION
     data = plt + 0x20 if plt > FUNCTION else FUNCTION + FUNCTION_SIZE
@@ -40,7 +41,7 @@ def fixture(sce_block=True, plt_push_offset=8, gap=0, interpreter_pointer=False,
         (1, 1),
         (0x61000035, 0x600), (0x61000037, 16),
         (0x61000039, 0x620), (0x6100003B, 24), (0x6100003F, 48),
-        (0x6100002F, 0x700), (0x61000031, 24 if interpreter_pointer else 0), (0x61000033, 24),
+        (0x6100002F, 0x700), (0x61000031, 24 if interpreter_pointer or pointer_target is not None else 0), (0x61000033, 24),
         (0x61000027, GOT), (0x61000029, 0x720), (0x6100002B, 7), (0x6100002D, 24),
         (0, 0),
     ]
@@ -82,6 +83,10 @@ def fixture(sce_block=True, plt_push_offset=8, gap=0, interpreter_pointer=False,
         image[FUNCTION:FUNCTION + 3] = b"\x48\x8d\x05"
         struct.pack_into("<iB", image, FUNCTION + 3, data + 0x30 - (FUNCTION + 7), 0xC3)
         image[data + 0x30:data + 0x30 + len(OVERLAPPING)] = OVERLAPPING
+    if pointer_target is not None:
+        struct.pack_into("<QQq", image, 0x700, POINTER, 8, pointer_target)
+    if gap_bytes is not None:
+        image[gap_bytes[0]:gap_bytes[0] + len(gap_bytes[1])] = gap_bytes[1]
     if gap_entry:
         image[entry:entry + len(UNSUPPORTED_TLS)] = UNSUPPORTED_TLS
     if function_tls:
@@ -138,6 +143,9 @@ def main():
             run(relinker, work, "sce-interpreter-pointer-" + target, fixture(gap=0x60, interpreter_pointer=True),
                 target=target)
             run(relinker, work, "sce-data-pointer-after-plt-" + target, fixture(data_pointer=True), target=target)
+            for name, address in (("unaligned", 0x1A4), ("aligned", 0x1B0)):
+                run(relinker, work, "sce-pointer-to-" + name + "-gap-data-" + target,
+                    fixture(early_function=True, pointer_target=address, gap_bytes=(address, INVALID)), target=target)
             run(relinker, work, "sce-code-before-plt-" + target, fixture(gap=0x40, gap_entry=True),
                 "Unsupported " + ("Windows" if target == "windows" else "macOS") + " guest TLS instruction", target)
             run(relinker, work, "sce-plt-before-code-" + target, fixture(plt_at=PLT_BEFORE_CODE), target=target)

docs/dev/debt/relinker.md

@@ -20,6 +20,8 @@
 - In executables with a `PT_SCE_DYNLIBDATA` segment, the PS4 layout, the executable segment also holds read-only data after the code (CUSA00900: 37 MiB after its last function). The [relinker pipeline](../../../core/relinker/relinker/src/pipeline/RelinkerPipeline.cpp) therefore scans for syscalls and import call sites only up to the end of the PLT that follows the last function of the unwind table; code after the PLT without an unwind entry is not scanned. `--to-intel` and the `unused-filter=1` and `unused-filter=2` analyses are not limited this way.
 - The [syscall scan of `sce_module` modules](../../../core/relinker/relinker/src/guest/GuestModuleBuilder.cpp) with a `PT_SCE_DYNLIBDATA` segment stops at the same end, or, when the PLT comes before the code, at the end of the last function of the unwind table; code after it is not scanned.
 - In the same executables and modules, the bytes between unwind functions are 16-byte slots of function tails and padding, and a tail can begin inside an instruction. The [instruction scanner](../../../core/relinker/codegen/src/InstructionScanner.cpp) of the syscall and call-site scans therefore resumes at the next byte when bytes outside every unwind function do not decode, and ends a decoding outside them at the start of the next unwind function. Code without an unwind entry that does not decode from its first byte is skipped by these scans instead of failing the relink; inside an unwind function a decode failure still fails it.
+- Read-only data can also sit between the unwind functions of such images, with whole tables in a gap. There, a `syscall`, `sysenter`, `int 0x80` or `sysret` pattern fails the [syscall scan](../../../core/relinker/relinker/src/analysis/SyscallScanner.cpp) only when a constant syscall number is loaded right before it; the others are skipped and counted in one message. A real syscall without an unwind entry whose number is set further back is therefore not refused, and runs on the host.
+- In the [code analysis](../../../core/relinker/relinker/src/analysis/CodeInstructionCollector.cpp) of the Windows and macOS outputs for such images, a relocation target or a `lea` target outside every unwind function is followed as code only when it is 16-byte aligned, starts less than 256 bytes after the end of an unwind function, and decodes to a jump, return or trap. That is where the thunks and tails without unwind entries of the measured PS4 titles sit; every other such target is taken as data. Code without an unwind entry that is reached only through a pointer and sits elsewhere is not analyzed, so a TLS access in it is not rewritten.
 - In executables and guest modules with a `PT_SCE_DYNLIBDATA` segment, the [code analysis](../../../core/relinker/relinker/src/analysis/CodeInstructionCollector.cpp) of the Windows and macOS outputs follows code only from the end of `PT_INTERP`, or of the `/libexec/ld-elf.so.1` string that begins the segment when no `PT_INTERP` covers it, as in OpenOrbis modules, to the end of the PLT after the last function of the unwind table, or to the end of that function when the PLT comes before the code, as in the `libSceFios2.prx` and `libc.prx` that CUSA02636 bundles (PLT at 0xc8); the pipeline's scans of executables above stop at this same end, also when the PLT comes before the code. A guest module with PLT relocations and no PLT before or after its unwind functions, or with a PLT that overlaps one, fails as the executable does. Code after that end is not analyzed, such as the `DT_FINI` routine without an unwind entry that follows the last unwind function in both modules and that the Windows and macOS outputs call when the program exits: a TLS access in it is not rewritten, the ELF entry point, `DT_INIT`, `DT_FINI`, a branch or a pointer into it is ignored, and a function symbol or init/fini array entry pointing into it fails the conversion.
 - A relocation against a dynamic symbol entry that is all zero, like the null symbol, takes the value 0 plus its addend, as the gABI defines for `STN_UNDEF` ([pipeline](../../../core/relinker/relinker/src/pipeline/RelinkerPipeline.cpp)). The Linux output imports such an entry as an unnamed weak symbol, which the host loader resolves to 0; the Windows and macOS outputs write the value into the slot ([relocation builder](../../../core/relinker/elfpatcher/src/windows/WindowsRelocationBuilder.cpp)). What the console's loader does with such an entry is not measured. OliSe Player (PPSA01153) has seven of them, and its code calls through those slots only when they are not 0 ([#4437](https://github.com/boykopovar/AnyPS5/issues/4437)).
 - In macOS output, a guest instruction with a GS segment prefix, including one with no memory operand (`65 90`, `65 48 89 c8`), which does not touch GS, becomes `UD2` and NOP padding of its own length ([macOS image writer](../../../core/relinker/elfpatcher/src/macos/MacOsImage.cpp)): the host owns GS, which holds the pthread TSD base the guest TLS stubs read, and the guest GS base that FreeBSD's `sysarch(AMD64_SET_GSBASE)` would set is not emulated. A converted module runs until such an instruction is reached, which then faults. Prospero Win (PPSA99995) has six in its Wine `ntdll`.