ps5-dump-forge/vendor

Vendored crates

ps5upload-fpkg (FPKG builder and readers) and ps5upload-pkg (UFS2 reader, package inspection), taken from ps5upload with a small local patch series on top.

Upstream	https://github.com/phantomptr/ps5upload
Tag / commit	v7.0.22 / 01afdf8 ("Release v7.0.22")
Path upstream	engine/crates/ps5upload-fpkg, engine/crates/ps5upload-pkg
License	GPL-3.0-or-later (see LICENSE-ps5upload, and the note below)
Edition	2021, as upstream, on purpose: it keeps the diff to upstream small

The trees here are upstream plus every patch in patches/ except 0021 (lzma-rust2, below), in order. Nothing else differs.

LICENSE-UnRAR.txt and LICENSE-UnRAR-exception.md are not vendored code: UnRAR's license and the GPLv3 §7 permission for the UnRAR source that unrar_sys (crates.io) builds into every target. The release scripts copy both into each package.

Patches

Paths in each patch are relative to this folder (a/ps5upload-fpkg/...). Each patch carries its own tests. Upstream took eleven earlier patches by v7.0.22 (reader empty dirs, reader bounds, Kraken descriptor bounds, SourceTree: Send, .Trashes, strict names, no silent UFS2 drops, header bounds, fakelib2, exFAT cluster runs, public imports_ampr); they were dropped, not rebased.

Patch	What it does
0001-cargo-pins.patch	Pins edition, license and version in both Cargo.tomls instead of inheriting them from upstream's workspace, and lz4_flex (upstream 0.13, for its own LZ4 packer) to Forge's 0.14 with the same features, so the lock holds one copy.
0002-build-from-source-tree.patch	Splits the build into build::prepare(tree, request, control) -> Prepared and build::write_package(prepared, tree, out_file, out_path, progress, control), which writes into a caller-owned file (no .partial, no rename) and verifies it. Prepared exposes the effective manifest (ManifestEntry { path, size, origin: Origin }), the container-only artwork, empty dirs, content id, warnings, estimated size (now covering the container's payloads, per-block tables, per-file/per-directory records and the param.json fields pfsimage.xml repeats), generated_bytes and read_range (the packaged bytes of any manifest file). Files read whole (param.json, container payloads) are bounded to 256 MiB together, by declared size, before any is read. build, build_controlled and build_in_memory now run on the two.
0003-production-request.patch	BuildRequest::production(source, output_dir): PlaintextNoAuth, Kraken, Balanced, Stored metadata, default chunks, and env_overrides: false, so no PS5UPLOAD_FPKG_* variable is read, neither in the constructor nor during the build (PARAM_FILE, SPOOL_DIR, DRM_TYPE, KRAKEN_STORE). Upstream's kraken::Level::Stored (every block raw, the fallback for consoles that refuse a compressed package) is the one route to a stored image: KRAKEN_STORE=1 maps to it only with env_overrides. BuildRequest::new is unchanged, except that a PARAM_FILE replacement is held to the same 256 MiB bound.
0004-compression-thread-cap.patch	BuildRequest::threads: Option<usize> (None/Some(0) = every core, larger caps clamp to the core count) sizes the Kraken compression pool. The output is the same for every cap.
0005-cancellation.patch	Upstream has Error::Cancelled; the builder, stream writer and compressor now return it (they returned a Format error); verify::verify_streaming_controlled(path, passcode, progress, cancel) (verify_streaming wraps it) and verify::verify_file_controlled(PkgFile, ...) check the flag on every block read, the sweep and the CRC table; write_package verifies through the caller's handle, not by reopening the path, and write_package and build check the flag again once a verification passes, so a stop asked for during its last read is never published. prepare checks it after each stage callback, per file (SELF headers, module marks) and between container payload reads; compression checks it per block in the producer and in the pool; the streaming writer checks it per data and metadata block, between sections and before the final sync. Not covered: source::readiness's bounded AMPR scan (at most ~80 MiB read).
0006-ufs2-range-reads.patch	UFS2 read_range reads blocks that lie back to back on disk as one run (one seek and one read, bounded by the requested length; a hole, a jump or a bad pointer ends the run and is met next, as before), and block_ptr reads a whole pointer block once and keeps the last one per indirection depth, keyed by fragment, instead of a seek and an 8-byte read per level per block. Found on hardware: a .ffpkg → .exfat conversion of an 87 GB image ran at ~105 MB/s on a PS5, ~6 syscalls per 64 KiB block. An 8 MiB read of a contiguous file is now 2 reads at 64 KiB blocks (single indirect; was 256) and 3 at 16 KiB blocks in the double-indirect range (was 1536). Bytes and errors match a block at a time, except that a pointer block cut short by a truncated image now fails even when the entries a range needs are present (read_file already did). Every byte offset read_range and block_ptr compute from a pointer is checked (fragment × size, + offset, + length) and must end inside the declared image size, else BlockOutOfRange: with a forged fs_size, a huge pointer's saturated offset wrapped to byte 0 in release builds (and panicked with overflow checks on).
0007-kraken-sparse-extents.patch	kraken_image::describe reads the boundary table's kind byte: a 0x40 boundary before the mount's end opens a sparse extent (nothing stored, no record) up to the next boundary, returned as zero blocks of at most 256 KiB (stored_len 0), which decode_described decodes to zeros; a block stored in no bytes without the whole sparse sentinel (stored_at, even_len 0, no LZ flags, no mode bits) is refused there. A real record whose stored span is empty is refused, and so is a boundary table whose mount end disagrees with the ublock count (ceil(mount / 256 KiB)) or that has a boundary past the mount's end, so a tiny descriptor cannot ask for millions of zero blocks. The walk is O(n log n) in boundaries: the boundary and sparse tables are sorted and looked up by binary search (with linear scans 100,000 boundaries took 3.9 s, extrapolated ~1.5 h for the ~5.6M a 32 MiB descriptor holds; 4M now take 0.24 s). Found in a third-party package: a 256 KiB hole between two files, which shifted every later block. layout is assemble plus its unchanged read-back check.
0008-kraken-entropy-halves.patch	decode_described decodes a non-LZ half stored shorter than its logical size as one bare entropy array (kraken::decode_entropy_half: raw or Huffman, consumed exactly), as a real Oodle encoder writes them (a single-symbol table for a zero half; full tables otherwise; 1,330 of 717,081 blocks in a third-party package); such halves used to go to the LZ decoder. A block of at most 128 KiB with bytes past its even half is refused. Also: decode_half requires the excess streams to meet, and a single-symbol Huffman table must fill its body exactly.
0009-pic-png-entries.patch	pic1.png (0x1006) and pic2.png (0x2040) join PRESENTATION, CONTAINER_ONLY and the system general digest (ids::SYSTEM_DIGEST_IDS, id order): a folder that has them builds a package that carries them as container entries, like pic0.png, and a reader that merges the container gets them back under sce_sys/. A third-party package carries both, outside its image, and its system digest matches only with both in it (before, our verifier reported that digest as wrong). Ids and names from that package's name table.
0010-kraken-decode-speed.patch	Decoder speed, same bytes: a match copies eight bytes at a time and delta literals add eight at a time (every distance is at least 8, so a word never reads a byte not yet written); the Huffman reader refills a whole word while eight bytes of its stream are left (byte by byte at the end, as before) and fills its output by stream rotation instead of i % 3; decode_described decodes from borrowed halves (kraken::decode_parts) instead of copying each into a Half. With the reader's parallel decode this took a 188 GB third-party package from 403 to 1,440 MB/s (4 MiB reads, 14 cores); every one of its 521 files hashes as before.
0011-parallel-write-and-sweep.patch	The write stage of a compressed build reads its spooled image back 8 MiB at a time and digests (and, for a native image, encrypts) the blocks on every core (stream::digest_blocks), each block's SHA3 taken once for both imagedigs and the block list (it was taken twice, one block at a time). The verifier's outer-block sweep (verify::failing_blocks) and its playgo-chunk.crc recomputation (si::chunk_crc_table_controlled) read 8 MiB at a time and work on every core. Same bytes, digests and verdicts. Found on an 89 GB build: after a ~4 min compress, the write and verify stages ran on one core.
0012-sampled-self-check.patch	BuildControl::sample: Some(seed) makes the built package's self-check (verify::verify_file_sampled) read only a seeded sample of its outer blocks and of the playgo-chunk.crc table: the first, the last and 1% of the other 8 MiB batches, at most 128 (1 GiB), SplitMix64 as in Forge's fast verification. Every header, container, digest-table, inode and flat-path check still runs. None (the default, and verify_file_controlled) sweeps every block as before. For a caller that checks the package's files itself: Forge's fast verification, which made a .pkg re-read all of itself twice.
0013-kraken-skip-hopeless-search.patch	The Kraken encoder skips its match search for a half it could not keep compressed (it keeps one only when it saves 2%), storing it raw at once: the half's byte entropy and that of its delta literals (each byte less the one 8 back, the distance literal mode 0 starts with) are both over 7.9 bits, and an estimate of what it could save (order-0 coding plus the matches a one-probe hash scan finds, priced by the encoder's own gain) is under 1%. A heuristic, not a proof: it can store raw a half the search would have kept (one whose savings come from deltas at a distance only a match sets, or from matches the probe misses); such a half is valid, only larger. Every level; the format is unchanged (raw halves are what the encoder already wrote for incompressible data). A test keeps a delta-coded half (≈8-bit bytes, no repeats, half its size as delta literals) searched and compressed at every level. Found on a 9.2 GB game with large compressed audio: 47% of the fast level's CPU went to the 34,071 of 74,725 halves that end up raw; the scan skipped 31,188 of them, the compress stage went from 33 to 24 s (14 cores) and 4 halves the search kept (each saving 2.0–2.6%) are now raw, +12 KB; at the balanced level 1 of a sampled 9,339 halves changed, +2.8 KB.
0014-crc32c-slicing-by-8.patch	crc32c works eight bytes at a time (slicing by 8, tables built at compile time from the one-byte table) instead of one: ~435 to ~2,250 MB/s on an Apple M3 core. Same values (a test checks every length and alignment against a byte at a time). A package's playgo-chunk.crc covers every byte of it, in the write stage and again in a full verification: on an 8.5 GB package the full verify_streaming went from 5.1 to 3.1 s and the write stage from 4.4 to 2.6 s.
0015-kraken-file-sha3-thread.patch	kraken_image::compress takes each file's SHA3 on a thread of its own (hash_files), fed the blocks in file order through a bounded channel (32 blocks, at most 8 MiB; the block buffers move, nothing is copied), instead of on the thread that reads the source, keeps the compression pool fed and lays out the blocks, where it took ~40% of that thread's time. Same digests, in the same order (a test compares them with each file's SHA3); an error or a panic in the pipeline closes the channel, which ends the hasher. On a 9.2 GB game converted from its .pkg (14 cores, fast level, Forge's 4 MiB read-ahead): compress 28.2 to 26.2 s.
0016-ampr-index-packed-source.patch	Upstream rebuilds ampr_emu.index in every package from the packaged files (a stale one crashes the game). An LZ4-packed source (source::lz4_packed: a root ampr_assets.index starting AMPRPAK4) is the exception: its index also lists the files inside the packs, and ampr_emu checks every record it opens against the pack manifest (file id = record order, path, size; a mismatch is EIO), so any change breaks the game. Its own index is packaged byte for byte (Origin::Source, as v6.1.2 did) and the log says so. A packed source without a readable index is refused by prepare, and readiness says so instead of calling the index stale (a readable one is reported kept as it is: it must match the packs). It reads the index whole only within the builder's 256 MiB bound for whole reads (upstream's readiness read it unbounded). lz4_packed returns Result<bool>: a listed manifest whose first 8 bytes cannot be read fails prepare with that read's error (and is a failed readiness check), never "not packed", which would regenerate the index and drop the packed records.
0017-ampr-scan-once.patch	A .pkg build scanned eboot.bin for libSceAmpr (source::imports_ampr: up to 16 MiB of the module, its headers and the load segment holding its dynamic section, up to ~80 MiB in all) three times: in the caller's readiness, in prepare's own readiness, and again where prepare decides on ampr_emu.index. From a folder that is cached reads; from an archive source (measured on a 7,169-file .rar) each scan re-reads the module through the decoder. Readiness now carries the answer (imports_ampr: bool), source::readiness_with(tree, Some(answer)) skips the scan, and BuildRequest::imports_ampr: Option<bool> hands a caller's answer to prepare (None, set by every constructor, scans once); prepare takes its index decision from its own readiness. One test: told Some(false), an AMPR module gets no generated index.
0018-prepare-progress.patch	BuildControl::files: Option<&mut dyn FnMut(Stage, u64, u64)> counts prepare's two per-file passes as (stage, done, total), in files visited: the SELF header read of every file in Stage::Check and the 4-byte module mark of every planned file in Stage::Plan, the files a pass skips without a read included (fakelib/ and under 32 bytes in check; generated and under 4 bytes in plan, now one loop over the plan with PlannedFile::may_be_module, which Plan::mark_modules shares), so done ends at total. Called at a pass's first file, every 256 files and at its last (done == total), so a quarter of a million files make about a thousand calls; the last only when not stopped (the flag is checked first). Found on a 7,169-file .rar source: the two passes took ~33 s and a caller had nothing to show but the stage. One test: 600 extra files, all too short for check's read and half for plan's, both passes counted 0, 256, 512, ... and their total, and a stop during plan ends its count without the total.
0019-trim-tests.patch	Tests only, at the user's request to keep only the critical ones: 187 of 342 #[test] functions go (ps5upload-fpkg 293 to 139, ps5upload-pkg 49 to 16), with the helpers, fixtures (tests/fixtures/ampr_pack, ampr_trace, pfs) and examples/exfat_read_bench.rs only they used. Gone: the package-metadata parser's tests in ps5upload-pkg/src/lib.rs beyond its round trips and real samples (Forge reads only ufs2), tests of helpers, labels, messages, estimates and progress, variants of a property another test keeps, tests of code Forge does not use (the ffpfsc and exFAT writers, the ampr_pack LZ4 packer, the PFS reader, PS3 packages, package kinds, the remote sources), and scale.rs's slow 117 MiB run. Kept: byte layouts against Sony's samples and reference vectors, write-read round trips, most bound and "error, never panic" tests (15 go, among them exFAT's short image, bogus cluster shift and name past its fragments), hardware and third-party findings, crypto, digests and CRCs, and every test a row above names. No code changes.
0020-hold-metadata-region-once.patch	A streaming build held the inner image's metadata region (64 KiB per directory, plus one per ~390 files) four times through the compress stage: Metadata::plain, image_tail (for MetaCodec::Stored, the package's codec, a copy of plain), BlockSource's on-disk blocks (the tail again, one 64 KiB Vec per block) and a clone of plain handed to kraken_image::compress (cloned twice more for the install metadata). It is now held once: Metadata::image_tail() returns the codec's container, or plain for a stored region (the field is private); BlockSource::block copies a metadata block out of it into its reused buffer, zero-padding a short last block as before; BlockSource::metadata_region borrows. The container's presentation artwork (pic*.png/.dds, up to 16 MiB each) was also held twice, in Prepared and in a clone for StreamRequest, which now borrows it (extras: &[ExtraEntry]). Same bytes: the block source test now covers both codecs. The Kraken pipeline keeps threads × 2 blocks in flight, not × 8 (still two per worker; 42 MiB less at 14 threads). Found on a 19 GB game (7,169 files, 897 folders): a 57 MiB region and 76 MiB of artwork, a 590 MiB heap peak while compressing, 794 MiB at the job's peak; now 478.
0022-kraken-encoder-no-alloc-churn.patch	The Kraken encoder allocates less per block, same bytes: huffman_lengths merges with two queues on stack arrays (the leaves sorted once by weight and index, merged nodes coming out in that order, so the merges are the heap's; a test checks against a heap), huff::encoded_len gives encode's length without coding, so a half picks raw or delta literals by size and codes only the winner (it coded both), and the stream arrays, the Huffman streams and the offset bit writers are reserved whole instead of growing. Found on a PS5, whose libc serializes every malloc, realloc and free across threads: 4 compression threads on 32 MiB of a game's image ran at 13.1 MB/s, entropy coding taking 15x as long per block as on one thread; now 27.3 MB/s (one thread: 14.9). Same output on four 32 MiB samples (a .pak volume and three stretches of a 96 GB image).
0023-package-without-eboot.patch	build no longer refuses a source without eboot.bin ("the source has no eboot.bin"): by user decision such a package (additional content, an update) is built, cnt_write::content_class already giving it the additional-content class (0x21) when param.json names neither applicationCategoryType nor applicationDrmType (one that names a category gets 0x20 or 0x26). Unverified on the console.

ampr_emu/ (next to the vendored crates, not patched Rust) holds the unmodified ampr_emu 0.4.2.1 runtime binaries and source archive that ps5-dump-forge-lz4 embeds; see its README.md.

lzma-rust2

lzma-rust2/ is the crates.io package lzma-rust2 0.22.0 (the version sevenz-rust2 0.24 resolves to; Apache-2.0, its LICENSE inside), as published, minus .cargo-ok, .cargo_vcs_info.json, .github/ and its Cargo.lock (the workspace lock pins its dependencies), plus one patch. The root Cargo.toml puts it in [patch.crates-io] and leaves it out of the workspace (exclude), so --workspace lints and tests don't cover it. Drop it once upstream bounds the decoder itself.

Patch

What it does

0021-lzma2-mt-bound-units-out.patch

Lzma2ReaderMt kept dispatching work units while it waited for the next one in order, keeping every unit decoded ahead of it: one lagging worker let decoded data pile up without bound. It now dispatches only while fewer than two units per worker are out (dispatched, not yet returned), and a dropped reader drops its result receiver and joins its workers (each finishes its current unit), so no detached worker still holds buffers when the next reader starts. That bounds what sevenz.rs sizes its decode threads by (a dictionary and six runs per worker, inside a 256 MiB budget). Found on a solid 13.4 GB 7z: one thread took 143 s, 14 took 22 s at a 314 MiB peak.

License file

Upstream's LICENSE at 01afdf8 is not the license text: it holds the GPLv3 title and preamble opening, then the placeholder "[Full GPLv3 text would go here - truncated for brevity]", then the usual "either version 3 of the License, or (at your option) any later version" notice. LICENSE-ps5upload here is the full, unmodified GPL-3.0 text instead (the same file as this repository's LICENSE). The grant stays GPL-3.0-or-later, as upstream's notice and both Cargo.tomls say.

Checking and regenerating

The patches apply, in order, to a pristine upstream checkout and reproduce this folder exactly. From this folder:

git clone https://github.com/phantomptr/ps5upload /tmp/ps5upload
git -C /tmp/ps5upload checkout v7.0.22
rm -rf /tmp/series && mkdir /tmp/series
cp -R /tmp/ps5upload/engine/crates/ps5upload-fpkg /tmp/ps5upload/engine/crates/ps5upload-pkg /tmp/series/
(cd /tmp/series && git init -q && git add -A && git commit -qm pristine)
for p in patches/*.patch; do
    case $p in *0021-*) continue ;; esac
    (cd /tmp/series && git apply "$OLDPWD/$p" && git add -A && \
        git commit -qm "$(basename "$p" .patch | cut -d- -f2-)")
done
diff -r /tmp/series/ps5upload-fpkg ps5upload-fpkg && diff -r /tmp/series/ps5upload-pkg ps5upload-pkg

/tmp/series now holds the series as one commit per patch. To change patch N, copy the edited crates over /tmp/series, git commit --fixup <commit of patch N>, then GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash <pristine commit>; to add a patch, commit it on top. Then write every patch again, each against its parent:

cd /tmp/series && n=0
for c in $(git rev-list --reverse HEAD~$(($(git rev-list --count HEAD) - 1))..HEAD); do
    n=$((n + 1)); [ $n = 21 ] && n=22
    git diff --binary --histogram "$c~1" "$c" > "$OLDPWD/patches/$(printf %04d $n)-$(git log -1 --format=%s "$c" | tr ' ' -).patch"
done

Every patch state builds and passes its own tests; check with the gates below at each commit when the series changes. Gates: cargo fmt -p ps5upload-fpkg -p ps5upload-pkg --check, cargo clippy -p ps5upload-fpkg -p ps5upload-pkg --all-targets with no warnings, and cargo test -p ps5upload-fpkg -p ps5upload-pkg --release.