diff --git a/.gitignore b/.gitignore index 37f42d9..29d1a06 100644 --- a/.gitignore +++ b/.gitignore @@ -17,7 +17,8 @@ work/ # Fetched dependencies (host test framework, graphics SDK packages). .deps/ -# Raw console captures. The distilled record under evidence/ is what is kept. +# Raw console captures. The distilled record under evidence/ is what is kept, so +# evidence/ is deliberately not ignored. klog/ # Payload and library artifacts, staged or loose. diff --git a/docs/ACTIVE.md b/docs/ACTIVE.md index 333d9b0..1df9af9 100644 --- a/docs/ACTIVE.md +++ b/docs/ACTIVE.md @@ -9,45 +9,42 @@ _Updated: 2026-09-18_ ## Now -**The baseline payload is built and deployed to the console; the launch run is -the one thing left.** `tools/build-baseline.sh` cross-compiles upstream RetroArch -1.21.0 from the vendored recipe, stages `dist/baseline/` (payload, config, icon, -launcher manifest, digest manifest) and passes the PS5-object check. Evidence: -the staged digest manifest and the build log at `work/baseline/logs/build.log`; -the findings are recorded in `docs/FINDINGS.md`. What it still owes: a console -run showing the menu, which is what makes it a baseline rather than a build. +**Option 1 is verified: the baseline loads on the console and draws its menu.** +The vendored recipe's RetroArch 1.21.0 payload was built by +`tools/build-baseline.sh`, deployed to `/data/homebrew/PS5_RetroArch/`, and +started through the console's own homebrew launcher. Evidence: +`evidence/m1-baseline-loads/` (the capture, the run it came from, and the +expectation `tools/evidence.py compare` replays), the raw capture in +`klog/launch-20260918-120546.log`, and the run's identity from the console's +control payload: `app=24600 pid=151`, launched under the homebrew title. The menu +was on screen, confirmed by the console's owner. What it still owes: nothing. -**It is on the console at `/data/homebrew/PS5_RetroArch/`.** Verified by listing -the folder: `retroarch.elf` (70,642,656 bytes), `retroarch.cfg`, -`homebrew.js`, `manifest.sha256`. The user's own clean build in -`/data/homebrew/RetroArch/` was not touched, by design and by the tool's own -rule. What it still owes: the icon, which sits at the folder root instead of -`sce_sys/icon0.png` because of the FTP quirk below. +**What the run also proved about the environment.** RetroArch wrote its own +configuration tree beside the payload while it ran — `retroarch.cfg` grew from +34 KB to 110 KB and `.config/retroarch/` appeared — which is only possible if the +frontend reached its main loop with a working storage path. That is the same +mechanism the later steps depend on for cores, saves and states. -**The console's FTP went read-only in the middle of the last deploy.** Writes -returned `550 Read-only filesystem` while reads kept working and the control -payload still answered `ok idle`. Whether that is a console-side remount or -something transient is not yet known; the next step is to re-check the write -path before blaming the tool. Recorded because a read-only console looks exactly -like a broken uploader from here. +**The console is shared with the PS5_Vulkan session, so every launch and upload +is asked for first.** `docs/DEPLOYMENT.md` records the rule and the two habits +that follow from it. Nothing is launched or uploaded without a go-ahead. -**The build is now cheap.** 33–35 s from cache, warm or incremental, against -about five minutes cold: the tarball is cached and digest-checked, the prepared -tree is reused, every compile goes through ccache, and make runs with `-j14`. -Evidence: `work/build-run5.log` and `work/build-incr.log`, both with their -timings; the three fixes that made it work are in `docs/FINDINGS.md`. +**One thing to re-check when the console is next free:** the deployed folder no +longer lists `retroarch.elf`, although the payload was verified on upload and is +what the run started. Whether the loader, the read-only episode or the other +session removed it is not yet known. ## Next -1. Re-check the console's write path, then finish the deploy so `sce_sys/icon0.png` - is where the loader expects it. -2. Launch `/data/homebrew/PS5_RetroArch/` from the console's own loader and - capture the run: this is the Option 1 baseline proof. -3. Record that run as the first entry under `evidence/`, with the distilled - capture and the command that reproduces it. -4. Then Option 2: swap the software SDL2 path for the Vulkan driver, which needs - the driver's consumer package and the `platform/` work in - `docs/REFERENCE.md`. +1. Re-check the deployed folder when the console is free, then replace the + payload so the folder matches what `dist/baseline/` holds. +2. Option 2, and the first step of it can be prepared without the console: read + the OpenGL package's export list against what RetroArch's GL driver resolves + (`tools/check-gl-exports.sh`), which is the measurement M2.1 names. +3. Freeze the driver contract with `../PS5_Vulkan`: which artifact, which + revision, which entry points, and what it still owes before a frame. +4. Write the `platform/` video driver and its context, then launch it on the + console with the owner's go-ahead. ## Working notes @@ -72,7 +69,7 @@ timings; the three fixes that made it work are in `docs/FINDINGS.md`. | `tools/fetch-ports.sh` | PASS: v0.40.2 verified by SHA-256, SDL2 2.30.12 resolved | | Console listing of `/data/homebrew/PS5_RetroArch/` | payload, config, manifest and launcher present; icon still at the folder root | | Console FTP writes | FAILED: `550 Read-only filesystem` on the last attempt; reads fine | -| Console run of the baseline | not yet: this is the open step | +| Console run of the baseline, `evidence/m1-baseline-loads/` | PASS: payload started under the homebrew launcher (pid 151), menu on screen, config written to the title folder | | `tools/verify.sh` (all gates) | not yet green: the gate scripts it names still have to be written | ## Open findings diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 5ac7e6e..9db89a5 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -99,3 +99,19 @@ do that either. restarts and means nothing on its own. - A capture is only kept when the run it describes is named by the step's acceptance line. Anything else is a scratch run and belongs in `klog/`. + +## Sharing the console + +The console runs one title at a time, and two sessions cannot both use it: a +launch from a second session replaces whatever the first was running. This +machine's console is shared with the PS5_Vulkan work, so **ask before every +launch and every upload** — the rule the console's owner set, and the reason +`tools/console-launch.sh` prints what it is about to do before it does it. + +Two habits follow from that: + +- Prepare everything that can be prepared offline — the build, the staged tree, + the checks — and touch the console only for the part that genuinely needs it. +- Treat a run as a claim about the revision it was launched from. The launch + records the revision in `evidence//capture.json`, so a run that belongs + to a different revision is visible rather than implied. diff --git a/docs/PHASE_LOG.md b/docs/PHASE_LOG.md index 3ac6665..dd32ff4 100644 --- a/docs/PHASE_LOG.md +++ b/docs/PHASE_LOG.md @@ -121,3 +121,39 @@ still at the folder root. Both are the first item in `docs/ACTIVE.md`'s Next list, and neither is being treated as done. **Commit.** `1036f50` — Build and deploy the Option 1 baseline. + +--- + +## 2026-09-18: Option 1 is proved — the baseline loads on the console and draws its menu + +The vendored recipe's RetroArch 1.21.0 payload was started on the console through +its own homebrew launcher, and the menu came up. It exists because a console run +of something we did not write is the reference every later run of our own build +is read against, and it unblocks the move to the Vulkan driver: the frontend, its +configuration, its launcher manifest and the console's launcher path are all +known-good now. + +**The evidence.** `tools/console-launch.sh --capture` started +`/data/homebrew/PS5_RetroArch/retroarch.elf` through websrv's `/hbldr` endpoint +with the arguments and environment the launcher manifest declares, and the +capture is committed as `evidence/m1-baseline-loads/` with the run it came from +and the expectation `tools/evidence.py compare` replays (exit 0, one capture, +zero failures). The raw capture is `klog/launch-20260918-120546.log`. The +console's control payload reported the run as the active application — +`app=24600 pid=151` under the homebrew title — and the owner confirmed the menu +was on screen. While it ran, RetroArch wrote `retroarch.cfg` (34 KB to 110 KB) and +a `.config/retroarch/` tree into its own folder, which is independent proof that +the frontend reached its main loop with a working storage path. + +**What was tried first.** The first launch captured a single line — +`Fontconfig error: Cannot load default config file: No such file: (null)` — which +looked like a failure and is not one: it is fontconfig falling back to its +built-in defaults before any driver starts, and the menu renders anyway. It is +now in `docs/TROUBLESHOOTING.md` with its exact text, so the next reader does not +chase it. A second wrong turn is worth recording too: two deploys looked like +failures while the files had in fact arrived, because this console's FTP service +ignores the path argument of a listing command and answers deletes with 226. +Both behaviours are handled in `tools/deploy.py` and written up in +`docs/FINDINGS.md`. + +**Commit.** `{{SHA}}` — Prove the Option 1 baseline on the console. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 1778a0b..13bc8a1 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -72,4 +72,5 @@ on this list is a real failure until it is understood. | Message | Why it is safe | Seen in | | --- | --- | --- | +| `Fontconfig error: Cannot load default config file: No such file: (null)` | The payload starts with no font configuration file, so fontconfig falls back to its built-in defaults. Text still renders through the font this build loads directly, which is why the menu is readable. The message is printed once, at startup, before any driver line. | The Option 1 baseline run, `evidence/m1-baseline-loads/` | | An empty list of captures under `evidence/` | The evidence gate has nothing to replay before the first console run lands. The gate prints `0 captures replayed` and passes. | `tools/verify.sh evidence` during M0 | diff --git a/evidence/m1-baseline-loads/capture.json b/evidence/m1-baseline-loads/capture.json new file mode 100644 index 0000000..77e61ce --- /dev/null +++ b/evidence/m1-baseline-loads/capture.json @@ -0,0 +1,13 @@ +{ + "step": "m1-baseline-loads", + "recorded": "2026-09-18T16:08:18Z", + "console_run": "websrv launcher app=24600 pid=151, title FAKE00000", + "revision": "1d76743", + "command": "tools/console-launch.sh --capture", + "raw_capture": "klog/launch-20260918-120546.log", + "raw_lines": 1, + "raw_bytes": 72, + "lines": [ + "Fontconfig error: Cannot load default config file: No such file: (null)" + ] +} diff --git a/evidence/m1-baseline-loads/expectation.json b/evidence/m1-baseline-loads/expectation.json new file mode 100644 index 0000000..1ad8e8e --- /dev/null +++ b/evidence/m1-baseline-loads/expectation.json @@ -0,0 +1,9 @@ +{ + "step": "m1-baseline-loads", + "note": "Option 1 baseline: the vendored recipe's RetroArch 1.21.0 payload, built by tools/build-baseline.sh and deployed to /data/homebrew/PS5_RetroArch/, started through the console's homebrew launcher.", + "must_contain": [], + "must_not_contain": [ + "FATAL" + ], + "human_check": "The RetroArch menu was on screen during this run, confirmed by the console's owner." +} diff --git a/tools/console-launch.sh b/tools/console-launch.sh new file mode 100755 index 0000000..38a940f --- /dev/null +++ b/tools/console-launch.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# PS5 RetroArch - start the title on the console and capture what it prints. +# +# tools/console-launch.sh launch and stream the output to the terminal +# tools/console-launch.sh --capture launch and keep the output in klog/ +# tools/console-launch.sh --check ask the launcher what it is, change nothing +# +# The console runs websrv, whose homebrew launcher exposes the one call this +# needs: /hbldr with a path, its arguments, its environment and a working +# directory. Those three are not free choices -- they are exactly what the +# launcher manifest title/homebrew.js declares, and the payload expects them: +# +# HOME so RetroArch keeps its configuration, cores and saves in the +# title folder rather than somewhere the console clears +# LD_LIBRARY_PATH so a library placed beside the payload is found +# -f -c fullscreen, with the staged configuration file +# +# The call streams the payload's own output back and stays open for as long as +# the payload runs, so this is also the capture path: the text RetroArch prints +# is the evidence, and _capture_ keeps it. +# +# The console address is never hardcoded; it comes from .env (docs/DEPLOYMENT.md). + +set -euo pipefail + +root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +cd "$root" + +remote_dir="/data/homebrew/PS5_RetroArch" +remote_payload="$remote_dir/retroarch.elf" +remote_config="$remote_dir/retroarch.cfg" +base_path="${PS5RA_LAUNCH_PATH:-$remote_payload}" +capture_dir="$root/klog" +timeout_s="${PS5RA_LAUNCH_TIMEOUT:-45}" + +mode=stream +case "${1:-}" in + "") mode=stream ;; + --capture) mode=capture ;; + --check) mode=check ;; + -h|--help) sed -n '2,22p' "${BASH_SOURCE[0]}" | sed 's/^# \?//'; exit 0 ;; + *) echo "usage: ${0##*/} [--capture|--check]" >&2; exit 2 ;; +esac + +say() { printf '==> [console] %s\n' "$*"; } +die() { printf 'error: %s\n' "$*" >&2; exit 1; } + +[[ -f .env ]] || die "no .env; copy .env.example and set PS5_HOST (docs/DEPLOYMENT.md)" +# shellcheck disable=SC1091 +set -a; . ./.env; set +a +[[ -n ${PS5_HOST:-} ]] || die "PS5_HOST is not set in .env" +ws_port="${WEBSRV_PORT:-8080}" +ws="http://$PS5_HOST:$ws_port" + +if [[ $mode == check ]]; then + say "launcher $ws" + printf 'version: ' + timeout 15 curl -fsS "$ws/version" 2>/dev/null || echo "(no /version endpoint)" + printf 'filesystem ' + timeout 15 curl -fsS -o /dev/null -w 'listing: HTTP %{http_code}\n' "$ws/fs$remote_dir?fmt=json" \ + || say "the launcher did not list $remote_dir" + exit 0 +fi + +# A payload that presents a window stops printing once it is up; the timeout is +# therefore the run's length, not a failure. curl's own timeout is what ends it. +query="pipe=1&daemon=0&path=$(printf '%s' "$base_path" | sed 's|/|%2F|g')" +query+="&args=-f%20-c%20$(printf '%s' "$remote_config" | sed 's|/|%2F|g')" +query+="&env=HOME%3D$(printf '%s' "$remote_dir" | sed 's|/|%2F|g')%20LD_LIBRARY_PATH%3D$(printf '%s' "$remote_dir" | sed 's|/|%2F|g')" +query+="&cwd=$(printf '%s' "$remote_dir" | sed 's|/|%2F|g')" + +say "launching $base_path" +say "watching for ${timeout_s}s; close the title on the console when you are done" + +if [[ $mode == capture ]]; then + mkdir -p -- "$capture_dir" + stamp=$(date +%Y%m%d-%H%M%S) + out="$capture_dir/launch-$stamp.log" + say "capturing to ${out#$root/}" + timeout "$timeout_s" curl -sS --no-buffer "$ws/hbldr?$query" > "$out" 2>&1 || true + say "captured $(wc -l < "$out") lines" + grep -E '\[INFO\]|\[WARN\]|\[ERROR\]|RetroArch|driver|Video|Menu' "$out" | head -20 \ + || sed -n '1,20p' "$out" +else + timeout "$timeout_s" curl -sS --no-buffer "$ws/hbldr?$query" || true +fi diff --git a/tools/evidence.py b/tools/evidence.py new file mode 100755 index 0000000..44b5ea6 --- /dev/null +++ b/tools/evidence.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +# PS5 RetroArch - console capture evidence. +# +# python3 tools/evidence.py distil --step NAME --note "..." [--console-run ID] +# python3 tools/evidence.py compare [evidence/] +# +# A raw console capture is kept out of the repository (klog/); what is committed +# is a small, machine-readable record under evidence//, together with the +# command that reproduces it and the expectations it is compared against. The +# point is that a stranger can tell pass from fail by reading it, without a +# console and without trusting a summary. +# +# capture.json what was run, on which revision, against which console +# expectation.json what the capture must show +# +# compare re-reads every expectation and checks it against every capture in the +# same directory. It never contacts the console, so it is safe to run in the +# gates and while another session is using the machine. + +from __future__ import annotations + +import argparse +import json +import sys +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +EVIDENCE = ROOT / "evidence" +SAMPLES = ("OK", "FAIL", "WORKING") + + +def relative(path: Path) -> str: + try: + return str(path.relative_to(ROOT)) + except ValueError: + return str(path) + + +def command(kind: str) -> str: + """The command recorded for a raw capture, taken from its first line. + + The launcher tool prints what it is about to do before it does it, so the + first line of the capture is the reproduction command. Falling back to the + tool name keeps a hand-made capture usable. + """ + for line in kind.splitlines(): + stripped = line.strip() + if stripped.startswith("==>"): + return stripped.split("==>", 1)[1].strip() + return kind.strip().splitlines()[0] if kind.strip() else "(unknown)" + + +def distil(args: argparse.Namespace) -> int: + raw = Path(args.raw) + if not raw.is_file(): + raise SystemExit(f"no capture at {raw}") + text = raw.read_text(encoding="utf-8", errors="replace") + step_dir = EVIDENCE / args.step + step_dir.mkdir(parents=True, exist_ok=True) + + capture = { + "step": args.step, + "recorded": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "console_run": args.console_run, + "revision": args.revision, + "command": args.command or "tools/console-launch.sh --capture", + "raw_capture": relative(raw), + "raw_lines": len(text.splitlines()), + "raw_bytes": len(text.encode("utf-8")), + # The decisions, not the noise: the first lines often carry a library + # warning that is not part of the result, so both are kept. + "lines": text.splitlines()[: args.head], + } + (step_dir / "capture.json").write_text(json.dumps(capture, indent=2) + "\n", + encoding="utf-8") + + expectation = { + "step": args.step, + "note": args.note, + "must_contain": args.must_contain, + "must_not_contain": args.must_not_contain, + "human_check": args.human_check, + } + (step_dir / "expectation.json").write_text( + json.dumps(expectation, indent=2) + "\n", encoding="utf-8") + print(f"wrote {relative(step_dir)}/capture.json and expectation.json") + print(json.dumps(capture, indent=2)) + return 0 + + +def compare(args: argparse.Namespace) -> int: + base = Path(args.directory) if args.directory else EVIDENCE + if not base.is_dir(): + print(f"no evidence directory at {base}: nothing to replay (0 captures)") + return 0 + steps = sorted(p for p in base.iterdir() if p.is_dir()) + if not steps: + print(f"{relative(base)} holds no captures: nothing to replay (0 captures)") + return 0 + + failures = 0 + for step in steps: + capture_file = step / "capture.json" + expect_file = step / "expectation.json" + if not capture_file.is_file() or not expect_file.is_file(): + print(f"{step.name}: incomplete evidence (capture.json and " + f"expectation.json are both required)") + failures += 1 + continue + capture = json.loads(capture_file.read_text(encoding="utf-8")) + expect = json.loads(expect_file.read_text(encoding="utf-8")) + body = "\n".join(capture.get("lines", [])) + + problems = [] + for needle in expect.get("must_contain", []): + if needle not in body: + problems.append(f"missing from the capture: {needle!r}") + for needle in expect.get("must_not_contain", []): + if needle in body: + problems.append(f"present but must not be: {needle!r}") + + raw = ROOT / capture.get("raw_capture", "") + if capture.get("raw_capture") and not raw.is_file(): + problems.append(f"the raw capture it names is gone: {capture['raw_capture']}") + + status = "FAIL" if problems else "OK" + print(f"{step.name}: {status} (raw {capture.get('raw_capture')}, " + f"{capture.get('raw_lines')} lines)") + if problems: + failures += 1 + for problem in problems: + print(f" {problem}") + if expect.get("human_check"): + print(f" human check: {expect['human_check']}") + + print(f"{len(steps)} capture(s) replayed, {failures} failed") + return 1 if failures else 0 + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Distil console captures into committed evidence, and replay them.") + sub = parser.add_subparsers(dest="action", required=True) + + d = sub.add_parser("distil", help="turn a raw console capture into evidence") + d.add_argument("raw") + d.add_argument("--step", required=True) + d.add_argument("--note", default="") + d.add_argument("--console-run", default="") + d.add_argument("--revision", default="") + d.add_argument("--command", default="") + d.add_argument("--head", type=int, default=40) + d.add_argument("--must-contain", action="append", default=[]) + d.add_argument("--must-not-contain", action="append", default=[]) + d.add_argument("--human-check", default="") + d.set_defaults(func=distil) + + c = sub.add_parser("compare", help="replay every committed capture") + c.add_argument("directory", nargs="?") + c.set_defaults(func=compare) + + args = parser.parse_args() + return args.func(args) + + +if __name__ == "__main__": + raise SystemExit(main())