Files
b489af4f08 RomM integration: browse, download and play games from your server (#21)
* Download sources: games on the network in the Library, downloaded when wanted

A download source is a server in the network that has Switch games. Its games are
listed in the Library beside the ones on the console, with the sources that have them.
A source says what it knows of a game: its title ID, its ids at metadata providers
(several at once) and whether its name comes from such metadata. The same game on
several sources is one title, and a game already on the console is not offered again:
by title ID, then a provider id both have, then the normalised name, else the file
name. Cross downloads a game and starts it once
it is there, asking first which source when several have it; Square puts it in a
download queue, which goes on in the background and after the app was reopened. Nothing
is streamed: a game runs from its file in roms/, its updates and DLC go to updates/. A
download is written to .remote-downloads/ in the game files folder and, once all of
the game's files are complete, moved into place (a cancel never touches roms/ or
updates/); a stopped one resumes where it was, its last 4 MB fetched again. A game
a source has can be deleted from the console in its settings: the game and all of its
updates and DLC go, its save data and settings stay. Settings > Downloads shows the
sources and the queue.

The sources are set up in config/remote/sources.json. Each entry names its backend's
"type": a backend implements the Source interface (headless/remote/source.h) and is made
in backends.cpp; remote.cpp, the launcher and EdenServices only know that interface.
This commit has no backend yet. remote/http.c is HTTP for the backends that need it,
linked into the update check's libcurl object.

Settings has one category more, so the preview walks one row further to Language.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: a RomM backend

"type": "romm" in sources.json is a RomM server (https://github.com/rommapp/romm), with
its address and a client API token, or a user name and password. Its Switch games come
from the game list's pages with each ROM's files; RomM's file categories say what is the
game and what are its updates and DLC (manuals, mods and the like stay on the server);
its ids at the metadata providers RomM matched, its title ID when the server has the keys,
and whether RomM identified it tell the game apart. A
file is downloaded by its own id (/api/roms/<file id>/files/content/<name>), the endpoint
RomM's feeds use, and goes on from where it was (Range). A current RomM is assumed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: tests and preview pictures

tools/check-remote.py builds the download sources (headless/remote) with their RomM
backend on the host and runs headless/remote_check.cpp against tools/romm-mock-server.py,
a stand-in for the parts of RomM's API the backend uses: sign-in, pages of games with
their files' categories, covers, downloads by file id with Range, a server that ignores
Range, a slow download to stop and cancel. It checks which game is which (title IDs,
provider ids, normalised names, file names), one source and two (a game of another
file name on the second one is the same title, the same game queued from one of them
only, a source of an unknown type, a source taken out of sources.json), going on after a stop with a spoilt end, a cancel that leaves the
player's own updates alone, a server that pages fewer games than asked for, two copies
of a game on one source, the cleaning of .remote-downloads/ and the kept queue. make test runs it; it is skipped without
libcurl's headers.

The launcher preview's sample services have two sources, Home and Office, one game on
both, and the preview draws the new screens (80-98): the Library with games on the
sources, the download dialog, choosing a source, a failed download, Settings >
Downloads, and deleting a game from the console.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: written by the console's FTP server, covers, speed and status

Written by the app itself, a download slowed the drive down as the file grew
(65 MB/s at first, 2 MB/s after 2 GB) and stopped the menu for seconds per
frame. From a payload the same drive keeps its speed, so the app now hands
the bytes to an FTP server running on the console (ftpsrv, etaHEN's, ...)
over loopback: REST to go on at a byte, STOR to write (headless/remote/ftp.h).
Downloads need such a server; without one they fail and say so. sources.json
may name its port ("ftp_port", 2121 by default) and a sign-in. File names
with a line break are refused, so a source cannot send the server commands.
When the server stops taking a file (a full drive), the download fails with
what it said ("No space left on device"); what was written stays in
.remote-downloads/ to go on from once there is room.

- Crash: the free-space check called statfs, which the SDK's libc.a has as
  a raw system call that ends a title. The check is left out on the console.
- Library: a game downloaded from a source is that source's game by the
  file name it came as, before the title IDs: a server's title ID can be
  another than the console reads from the file (Super Mario Galaxy 2 on the
  test server), and the game was then listed twice.
- Library: it reads all of the console's games again only when a source's
  list changed or a cover came, not after every reading of an unchanged
  list (that rewrote every cover and could make the menu flicker).
- Covers: Eden builds stb_image for JPEG only, and RomM's covers are PNG
  ("the cover of ... cannot be read"). metadata_bridge.cpp has a reader of
  its own for both, private to it.
- Speed: measured by the download thread, shown in the download dialog and
  in Settings > Downloads; the time left comes from it. Game files are not
  asked for compressed, and libcurl takes them in pieces of up to 1 MB.
- Menu: a cancel no longer deletes gigabytes while holding the queue's lock
  (the download thread does it, also while a game runs), and the menu asks
  for covers without touching the drive.
- Log: download sources write only what went wrong (on this console every
  line is a write to the drive the menu may wait for).
- Settings > Downloads: says when the lists asked for are in (a message
  gives way to the buttons after 4 s: with one source no row can be moved
  to, which takes it away elsewhere), shows the FTP server's port, and
  without a source shows an example with a RomM server. German added for
  the new texts.
- README: the FTP server is named where download sources are.
- Tests: tools/ftp-mock-server.py writes as ftpsrv does; check-remote runs
  every download through it, plus a download without an FTP server, a full
  drive and a file name with a line break.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: backends by what they can do, RomM's server part on its own

Groundwork for more uses of a backend than downloads (the save data of a
profile next): a backend's type now says what it can be used for, and what
all uses of a RomM server share is a part of its own.

- backends.h: Backends() lists each type with what it makes, one column per
  use (for now: `source`, a download source; nullptr when its servers have
  no games to download). MakeSource moved here from source.h and says when a
  type cannot be a download source. A new use is a column and a Make... of
  its own.
- romm/romm_client.h: Romm::Client, made from an entry of any config file
  (its errors name that file): the address, the sign-in, a JSON GET, a file
  into memory, the platform's id and the list of games. NormalUrl and
  ParsePage moved with it.
- romm/romm_source.cpp: the download source is what is left: the cover and
  the file download, on top of Romm::Client. Nothing changes for sources.json.
- remote_check: the table, and a RomM entry read for another file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: review fixes

- A download cancelled and queued again before its transfer stopped goes
  on instead of being dropped with its files.
- A cancel that comes once the game is all in place leaves it done: the
  Library reads its list again.
- The covers' paths are taken while the lock is held: Start sets the
  paths again while a list thread may still fetch covers.
- Settings > Downloads: a row the queue no longer has (a download that
  finished meanwhile) is not drawn from past its end; trying a failed
  download again moves the rows with the queue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Download sources: a download's contents checked as they come

RomM does not hash Switch files (they are in its NON_HASHABLE_PLATFORMS),
so a download could only be told whole by its size. Now each file is
checked while its bytes stream to the console's FTP server, as Eden's
integrity check checks a game, with no keys needed and without reading
the file again afterwards.

- remote/stream_check: reads the file's layout from the stream itself
  (an NSP's PFS0, an XCI's header, root HFS0 and secure HFS0) and hashes
  each NCA with OpenSSL's SHA-256 as its bytes go by, against the half of
  the SHA-256 it is named after. The first 1 MB is kept for headers asked
  for after their bytes went by. A gap, an unknown layout, a header that
  points past any file or a file that is no NSP or XCI leaves it
  unchecked, not damaged.
- remote: each file's bytes go to a StreamCheck on their way to FTP. A
  download that goes on from where it was reads what its file has so far
  from the drive first (the queue shows it as verifying, up to where it
  goes on); a server that sends the whole file anyway starts the check
  again. A damaged file is deleted and the download fails saying so, so
  trying again downloads it again. What the FTP server writes is not
  read back.
- Launcher: "Checking the game files 45%" in the download dialog,
  "Checking 45%" in the Library and Settings > Downloads, with the
  progress bar. German added; the preview's Orbit Postman goes on from
  40% and is shot while checking (83b). README.
- remote_check: synthetic NSPs and XCIs fed in pieces, overlapping,
  restarted, with gaps, spoilt, with headers of absurd sizes and as no
  NSP; check-remote links libcrypto and is skipped without its headers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Save sync: save-sync.json, and a game's save data in step with RomM

Each player profile can keep its save data on a server, apart from the
download sources: another server, or another user of the same one. This
is the part without the menu: the config file, the store's interface,
RomM's store and the sync itself.

- save-sync.json (save_sync_config.h): the app writes an entry for every
  profile and keeps it in step with them (added, renamed, removed, in
  their order); the player fills one in over FTP with a server's "type",
  "url" and sign-in. What the player wrote stays; a file that is not JSON
  is left alone and nothing syncs. eden-save-sync-check.
- remote/save_store.h: a save store, a backend's interface of its own
  beside the download sources'; Backends() has a column for it.
- remote/save_archive.h: a game's save folder as one zip with the title's
  folder at its top, as Argosy packs Eden's, and RomM's content hash of
  it (MD5 of "<name>:<MD5>" lines by name, MD5 from OpenSSL), told from
  the folder. A folder that cannot be read whole fails instead of being
  taken as less. Unpacking takes JKSV's zips and Mac zips (without
  __MACOSX/ and .DS_Store), keeps a zip of one folder of the game's own,
  and refuses entries outside the folder. miniz is built into the app.
- remote/save_sync.h: SyncSaveData finds the game on the store and tells
  which side changed by the save data, not by the server's times: the
  console keeps the content hashes of both sides from when they were last
  in step (synced.json), bound to the server, the user and the store's
  game. One side changed: it uploads or downloads; both, or no record:
  the player chooses, so save data changed offline is never replaced
  unasked. Replaced save data goes to a backup folder first, the last
  three of a game kept.
- remote/romm/romm_saves.cpp: RomM's save sync, RomM 5.0.0 or newer
  (Romm::kMinimumVersion; an older server is refused at each sync). The
  console is registered once as a device of the user; a game's save data
  is in the slot "autosave" of the emulator "eden", as Argosy keeps it;
  the server's state is read from /api/saves (RomM 5.3's negotiate only
  for the session, as it pairs other emulators' save data with the
  console's). A refused upload (409) is a conflict.
- romm_client: requests with a JSON body, uploads as a form field and
  downloads to a file, all stopped when the sync is; a refusal (403)
  names both causes, a wrong sign-in or the token's missing scope.
- remote/http: POST and PUT with a JSON body or a file as a form field; a
  redirect sends a POST again with its body.
- remote_check: the zips, RomM's times and versions, save stores made
  from entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Save sync: synced before a game starts and after it ended

- Launcher: a dialog while a game's save data is synced before it
  starts; the game starts when it is done. Circle goes back without the
  game (the sync stops, nothing more changes). When the server cannot
  be reached: Play anyway, Try again or Back.
- After a game, once the menu is back, its save data goes up with a
  notice at the top right, and is tried again on later starts until it
  worked (pending.json). "auto": false stops those too.
- Changed on both sides: a dialog asks which one stays (Keep this
  console's, Take the server's, Change nothing), with when each was
  saved and the device of the server's.
- A server older than the save sync takes says so in a dialog to
  confirm, with both versions (before a game: play anyway or not; after
  one: Understood, or a notice when another game is about to start).
- The sync runs on a thread of its own in EdenServices; a game's title
  ID is read there, so the menu never waits for a scan of the games.
- arc() moves to widgets for the sync dialog's spinner.
- Preview A0-A7; German added; README: Save sync.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Save sync: profiles paired by a QR code

Setting a profile's save sync up needs no FTP and no typing.

- remote/pairing.h: a backend's pairing (as RFC 8628): a code and the
  address to approve it at, then polled until approved, denied or
  expired, stopped when it is cancelled. RomM's: /api/auth/device/init
  and /token, approved on the server's /pair/device signed in as the
  profile's user; RomM makes a device of the console with a client API
  token bound to it, which the save sync then is (no second device).
- save_sync_config.h: SetEntry links or unlinks a profile's entry, the
  player's own fields kept.
- Settings > Save sync: each profile with its server and user; Cross
  pairs it with one of the download sources' servers that can pair
  (asked first when several can; the source's "platform" kept), Square
  twice unlinks it. The pairing dialog shows a QR code of the approval
  page (Project Nayuki's QR Code generator library,
  third_party/qrcodegen, MIT), the code, the address, the steps and how
  long the code works.
- Preview B0-B6; German added; README.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Checks: the download sources and the save sync against RomM and ftpsrv in Docker

tools/check-romm.py [version ...] starts, for each RomM version, RomM
and MariaDB on a free port of localhost with a library of fake Switch
games (laid out as real NSPs and XCIs), and ftpsrv (the console's FTP
server, built for Linux from a pinned commit) on the host's network
(its passive mode needs it); makes an admin and a player with client
API tokens, sets a cover in RomM, scans the library over RomM's socket
(the scan task cannot be started over REST), and runs
headless/romm_live_check.cpp against it; then takes it all down. By
default it checks kMinimumVersion (5.0.0), 5.3.1, and 4.9.0, which has
to be refused. make test runs it after check-remote.py; without Docker
it is skipped, ROMM_CHECK=0 skips it too.

romm_live_check:
- Download sources: sign-ins, the list kept, a cover, what is no game, a
  game of 400 MB stopped part way and gone on with by Range (seen
  reading what it had from the drive first) and compared byte for byte,
  a spoilt end, an NSP and an XCI spoilt on the drive found damaged,
  updates and DLC, a cancel, leftovers, no FTP server, two sources on
  one server, sources removed or moved, the kept queue.
- Save sync: two consoles and a phone (Argosy's API calls): uploads, a
  save of 150 MB with 400 files, a device deleted in RomM, the versions
  kept, Argosy's and JKSV's zips, another emulator's and slot's save
  data, both sides changed in both orders, backups, a slot emptied in
  RomM, a token without scopes, a server that does not answer, an older
  RomM refused at each sync, and pairing approved, denied and cancelled.

check-remote.py keeps what needs no server and, against the stand-ins,
only what a real RomM and ftpsrv do not show on demand: capped pages, a
server that cannot resume, metadata ids and title IDs, another file
name on a second source, a full drive. romm-mock-server.py serves the
list and files only. docs/BUILDING.md says how to run it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Sample names in the checks and comments: invented ones

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: BlackBearReloaded <blackbearreloaded@protonmail.com>
2026-10-08 19:15:47 -04:00

138 lines
6.9 KiB
C++

// SPDX-License-Identifier: GPL-3.0-or-later
// Host check of save-sync.json (save_sync_config.h): made with an empty entry for each profile,
// kept in step with the profiles (added, renamed, removed, in their order) without touching what
// the player wrote, written only when something changed, and left alone when it is no JSON.
#include "save_sync_config.h"
#include <cassert>
#include <cstdlib>
#include <filesystem>
#include <fstream>
#include <sstream>
#include <unistd.h>
namespace {
std::string Contents(const std::string& file) {
std::ifstream in(file);
std::stringstream text;
text << in.rdbuf();
return text.str();
}
} // namespace
int main() {
using namespace Eden::SaveSync;
char folder[] = "/tmp/eden-save-sync-XXXXXX";
assert(mkdtemp(folder));
const std::string file = std::string(folder) + "/config/remote/save-sync.json";
const std::string one = "1F1E1D1C1B1A19181716151413121110";
const std::string two = "AFAEADACABAAA9A8A7A6A5A4A3A2A1A0";
const std::string three = "000102030405060708090A0B0C0D0E0F";
// No file: nothing is set up, and that is readable.
Config config = Read(file);
assert(config.readable && config.automatic && config.entries.empty());
// Made, with its folder, an empty entry for each profile.
assert(Reconcile({{one, "Player 1"}, {two, "Kids"}}, file));
config = Read(file);
assert(config.readable && config.automatic && config.entries.size() == 2);
assert(config.entries[0].profile == one && config.entries[0].profile_name == "Player 1" &&
config.entries[0].type.empty() && config.entries[0].settings.contains("url") &&
config.entries[0].settings.contains("token"));
assert(config.entries[1].profile == two && config.entries[1].profile_name == "Kids");
// The fields of an entry in the order a reader expects them.
const std::string made = Contents(file);
assert(made.find("\"profile\"") < made.find("\"__profile_name\"") && made.find("\"__profile_name\"") < made.find("\"type\"") &&
made.find("\"type\"") < made.find("\"url\"") && made.find("\"url\"") < made.find("\"token\""));
// The player fills one in, with fields of its own; the app's next look changes nothing.
{
Json document = Json::parse(made);
document["auto"] = false;
document["note"] = "mine";
document["profiles"][1]["type"] = "romm";
document["profiles"][1]["url"] = "http://nas:3000";
document["profiles"][1]["token"] = "";
document["profiles"][1]["username"] = "kids";
document["profiles"][1]["password"] = "secret";
std::ofstream(file) << document.dump(4);
}
const auto written = std::filesystem::last_write_time(file);
const std::string filled = Contents(file);
assert(Reconcile({{one, "Player 1"}, {two, "Kids"}}, file));
assert(Contents(file) == filled && std::filesystem::last_write_time(file) == written);
config = Read(file);
assert(!config.automatic && config.entries[1].type == "romm" &&
config.entries[1].settings["username"] == "kids" && config.entries[1].settings["password"] == "secret");
const Entry* kids = Find(config, two);
assert(kids && kids->settings["url"] == "http://nas:3000" && Find(config, three) == nullptr);
// Renamed, a profile added, the profiles in another order: names written, the new profile's
// entry empty, the entries in the profiles' order, the player's fields kept.
assert(Reconcile({{two, "Children"}, {one, "Player 1"}, {three, "Guest"}}, file));
config = Read(file);
assert(config.entries.size() == 3 && config.entries[0].profile == two && config.entries[0].profile_name == "Children" &&
config.entries[0].settings["username"] == "kids" && config.entries[1].profile == one &&
config.entries[2].profile == three && config.entries[2].type.empty());
assert(Json::parse(Contents(file))["note"] == "mine" && !config.automatic);
// A profile removed: its entry goes; a second entry of the same profile goes too, the first
// one counts.
{
Json document = Json::parse(Contents(file));
Json twin = document["profiles"][0];
twin["username"] = "someone else";
document["profiles"].push_back(twin);
std::ofstream(file) << document.dump(2);
}
config = Read(file);
assert(config.entries.size() == 3 && Find(config, two)->settings["username"] == "kids");
assert(Reconcile({{two, "Children"}, {three, "Guest"}}, file));
config = Read(file);
const Json reconciled = Json::parse(Contents(file));
assert(config.entries.size() == 2 && reconciled["profiles"].size() == 2 &&
config.entries[0].settings["username"] == "kids" && config.entries[1].profile == three);
// Entries the file cannot tell apart are not entries.
{
Json document = reconciled;
document["profiles"].push_back(Json{{"type", "romm"}});
document["profiles"].push_back("text");
std::ofstream(file) << document.dump(2);
}
assert(Read(file).entries.size() == 2);
// Linked from the menu (pairing): the sign-in replaced, the name and the player's fields kept;
// unlinked: type, url and token empty.
{
Json document = Json::parse(Contents(file));
document["profiles"][0]["note"] = "mine";
std::ofstream(file) << document.dump(2);
}
assert(SetEntry(two, Json{{"type", "romm"}, {"url", "http://nas:3000"}, {"token", "rmm_paired"}, {"__server_user", "kids"}}, file));
config = Read(file);
assert(config.entries[0].type == "romm" && config.entries[0].settings["token"] == "rmm_paired" &&
!config.entries[0].settings.contains("username") && !config.entries[0].settings.contains("password") &&
config.entries[0].settings["note"] == "mine" && config.entries[0].profile_name == "Children");
assert(SetEntry(two, Json::object(), file));
config = Read(file);
assert(config.entries[0].type.empty() && config.entries[0].settings["url"] == "" &&
!config.entries[0].settings.contains("__server_user") &&
config.entries[0].settings["token"] == "" && config.entries[0].settings["note"] == "mine");
assert(!SetEntry("00000000000000000000000000000000", Json{{"type", "romm"}}, file));
// Not JSON (a typing error over FTP): left as it is, and nothing syncs.
{ std::ofstream(file) << "{ \"profiles\": [ { \"profile\": "; }
const std::string broken = Contents(file);
config = Read(file);
assert(!config.readable && !config.error.empty() && config.entries.empty());
assert(!Reconcile({{two, "Children"}}, file) && Contents(file) == broken);
// "profiles" that is no list is not the app's to replace either.
{ std::ofstream(file) << "{ \"profiles\": {} }"; }
assert(!Read(file).readable && !Reconcile({{two, "Children"}}, file) && Contents(file) == "{ \"profiles\": {} }");
std::filesystem::remove_all(folder);
std::puts("save sync check: PASS");
return 0;
}