mirror of
https://github.com/Swordpdf/ProsperoEden.git
synced 2026-10-21 21:22:16 -05:00
* Download sources: games on the network in the Library, downloaded when wanted A download source is a server in the network that has Switch games. Its games are listed in the Library beside the ones on the console, with the sources that have them. A source says what it knows of a game: its title ID, its ids at metadata providers (several at once) and whether its name comes from such metadata. The same game on several sources is one title, and a game already on the console is not offered again: by title ID, then a provider id both have, then the normalised name, else the file name. Cross downloads a game and starts it once it is there, asking first which source when several have it; Square puts it in a download queue, which goes on in the background and after the app was reopened. Nothing is streamed: a game runs from its file in roms/, its updates and DLC go to updates/. A download is written to .remote-downloads/ in the game files folder and, once all of the game's files are complete, moved into place (a cancel never touches roms/ or updates/); a stopped one resumes where it was, its last 4 MB fetched again. A game a source has can be deleted from the console in its settings: the game and all of its updates and DLC go, its save data and settings stay. Settings > Downloads shows the sources and the queue. The sources are set up in config/remote/sources.json. Each entry names its backend's "type": a backend implements the Source interface (headless/remote/source.h) and is made in backends.cpp; remote.cpp, the launcher and EdenServices only know that interface. This commit has no backend yet. remote/http.c is HTTP for the backends that need it, linked into the update check's libcurl object. Settings has one category more, so the preview walks one row further to Language. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: a RomM backend "type": "romm" in sources.json is a RomM server (https://github.com/rommapp/romm), with its address and a client API token, or a user name and password. Its Switch games come from the game list's pages with each ROM's files; RomM's file categories say what is the game and what are its updates and DLC (manuals, mods and the like stay on the server); its ids at the metadata providers RomM matched, its title ID when the server has the keys, and whether RomM identified it tell the game apart. A file is downloaded by its own id (/api/roms/<file id>/files/content/<name>), the endpoint RomM's feeds use, and goes on from where it was (Range). A current RomM is assumed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: tests and preview pictures tools/check-remote.py builds the download sources (headless/remote) with their RomM backend on the host and runs headless/remote_check.cpp against tools/romm-mock-server.py, a stand-in for the parts of RomM's API the backend uses: sign-in, pages of games with their files' categories, covers, downloads by file id with Range, a server that ignores Range, a slow download to stop and cancel. It checks which game is which (title IDs, provider ids, normalised names, file names), one source and two (a game of another file name on the second one is the same title, the same game queued from one of them only, a source of an unknown type, a source taken out of sources.json), going on after a stop with a spoilt end, a cancel that leaves the player's own updates alone, a server that pages fewer games than asked for, two copies of a game on one source, the cleaning of .remote-downloads/ and the kept queue. make test runs it; it is skipped without libcurl's headers. The launcher preview's sample services have two sources, Home and Office, one game on both, and the preview draws the new screens (80-98): the Library with games on the sources, the download dialog, choosing a source, a failed download, Settings > Downloads, and deleting a game from the console. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: written by the console's FTP server, covers, speed and status Written by the app itself, a download slowed the drive down as the file grew (65 MB/s at first, 2 MB/s after 2 GB) and stopped the menu for seconds per frame. From a payload the same drive keeps its speed, so the app now hands the bytes to an FTP server running on the console (ftpsrv, etaHEN's, ...) over loopback: REST to go on at a byte, STOR to write (headless/remote/ftp.h). Downloads need such a server; without one they fail and say so. sources.json may name its port ("ftp_port", 2121 by default) and a sign-in. File names with a line break are refused, so a source cannot send the server commands. When the server stops taking a file (a full drive), the download fails with what it said ("No space left on device"); what was written stays in .remote-downloads/ to go on from once there is room. - Crash: the free-space check called statfs, which the SDK's libc.a has as a raw system call that ends a title. The check is left out on the console. - Library: a game downloaded from a source is that source's game by the file name it came as, before the title IDs: a server's title ID can be another than the console reads from the file (Super Mario Galaxy 2 on the test server), and the game was then listed twice. - Library: it reads all of the console's games again only when a source's list changed or a cover came, not after every reading of an unchanged list (that rewrote every cover and could make the menu flicker). - Covers: Eden builds stb_image for JPEG only, and RomM's covers are PNG ("the cover of ... cannot be read"). metadata_bridge.cpp has a reader of its own for both, private to it. - Speed: measured by the download thread, shown in the download dialog and in Settings > Downloads; the time left comes from it. Game files are not asked for compressed, and libcurl takes them in pieces of up to 1 MB. - Menu: a cancel no longer deletes gigabytes while holding the queue's lock (the download thread does it, also while a game runs), and the menu asks for covers without touching the drive. - Log: download sources write only what went wrong (on this console every line is a write to the drive the menu may wait for). - Settings > Downloads: says when the lists asked for are in (a message gives way to the buttons after 4 s: with one source no row can be moved to, which takes it away elsewhere), shows the FTP server's port, and without a source shows an example with a RomM server. German added for the new texts. - README: the FTP server is named where download sources are. - Tests: tools/ftp-mock-server.py writes as ftpsrv does; check-remote runs every download through it, plus a download without an FTP server, a full drive and a file name with a line break. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: backends by what they can do, RomM's server part on its own Groundwork for more uses of a backend than downloads (the save data of a profile next): a backend's type now says what it can be used for, and what all uses of a RomM server share is a part of its own. - backends.h: Backends() lists each type with what it makes, one column per use (for now: `source`, a download source; nullptr when its servers have no games to download). MakeSource moved here from source.h and says when a type cannot be a download source. A new use is a column and a Make... of its own. - romm/romm_client.h: Romm::Client, made from an entry of any config file (its errors name that file): the address, the sign-in, a JSON GET, a file into memory, the platform's id and the list of games. NormalUrl and ParsePage moved with it. - romm/romm_source.cpp: the download source is what is left: the cover and the file download, on top of Romm::Client. Nothing changes for sources.json. - remote_check: the table, and a RomM entry read for another file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: review fixes - A download cancelled and queued again before its transfer stopped goes on instead of being dropped with its files. - A cancel that comes once the game is all in place leaves it done: the Library reads its list again. - The covers' paths are taken while the lock is held: Start sets the paths again while a list thread may still fetch covers. - Settings > Downloads: a row the queue no longer has (a download that finished meanwhile) is not drawn from past its end; trying a failed download again moves the rows with the queue. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Download sources: a download's contents checked as they come RomM does not hash Switch files (they are in its NON_HASHABLE_PLATFORMS), so a download could only be told whole by its size. Now each file is checked while its bytes stream to the console's FTP server, as Eden's integrity check checks a game, with no keys needed and without reading the file again afterwards. - remote/stream_check: reads the file's layout from the stream itself (an NSP's PFS0, an XCI's header, root HFS0 and secure HFS0) and hashes each NCA with OpenSSL's SHA-256 as its bytes go by, against the half of the SHA-256 it is named after. The first 1 MB is kept for headers asked for after their bytes went by. A gap, an unknown layout, a header that points past any file or a file that is no NSP or XCI leaves it unchecked, not damaged. - remote: each file's bytes go to a StreamCheck on their way to FTP. A download that goes on from where it was reads what its file has so far from the drive first (the queue shows it as verifying, up to where it goes on); a server that sends the whole file anyway starts the check again. A damaged file is deleted and the download fails saying so, so trying again downloads it again. What the FTP server writes is not read back. - Launcher: "Checking the game files 45%" in the download dialog, "Checking 45%" in the Library and Settings > Downloads, with the progress bar. German added; the preview's Orbit Postman goes on from 40% and is shot while checking (83b). README. - remote_check: synthetic NSPs and XCIs fed in pieces, overlapping, restarted, with gaps, spoilt, with headers of absurd sizes and as no NSP; check-remote links libcrypto and is skipped without its headers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Save sync: save-sync.json, and a game's save data in step with RomM Each player profile can keep its save data on a server, apart from the download sources: another server, or another user of the same one. This is the part without the menu: the config file, the store's interface, RomM's store and the sync itself. - save-sync.json (save_sync_config.h): the app writes an entry for every profile and keeps it in step with them (added, renamed, removed, in their order); the player fills one in over FTP with a server's "type", "url" and sign-in. What the player wrote stays; a file that is not JSON is left alone and nothing syncs. eden-save-sync-check. - remote/save_store.h: a save store, a backend's interface of its own beside the download sources'; Backends() has a column for it. - remote/save_archive.h: a game's save folder as one zip with the title's folder at its top, as Argosy packs Eden's, and RomM's content hash of it (MD5 of "<name>:<MD5>" lines by name, MD5 from OpenSSL), told from the folder. A folder that cannot be read whole fails instead of being taken as less. Unpacking takes JKSV's zips and Mac zips (without __MACOSX/ and .DS_Store), keeps a zip of one folder of the game's own, and refuses entries outside the folder. miniz is built into the app. - remote/save_sync.h: SyncSaveData finds the game on the store and tells which side changed by the save data, not by the server's times: the console keeps the content hashes of both sides from when they were last in step (synced.json), bound to the server, the user and the store's game. One side changed: it uploads or downloads; both, or no record: the player chooses, so save data changed offline is never replaced unasked. Replaced save data goes to a backup folder first, the last three of a game kept. - remote/romm/romm_saves.cpp: RomM's save sync, RomM 5.0.0 or newer (Romm::kMinimumVersion; an older server is refused at each sync). The console is registered once as a device of the user; a game's save data is in the slot "autosave" of the emulator "eden", as Argosy keeps it; the server's state is read from /api/saves (RomM 5.3's negotiate only for the session, as it pairs other emulators' save data with the console's). A refused upload (409) is a conflict. - romm_client: requests with a JSON body, uploads as a form field and downloads to a file, all stopped when the sync is; a refusal (403) names both causes, a wrong sign-in or the token's missing scope. - remote/http: POST and PUT with a JSON body or a file as a form field; a redirect sends a POST again with its body. - remote_check: the zips, RomM's times and versions, save stores made from entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Save sync: synced before a game starts and after it ended - Launcher: a dialog while a game's save data is synced before it starts; the game starts when it is done. Circle goes back without the game (the sync stops, nothing more changes). When the server cannot be reached: Play anyway, Try again or Back. - After a game, once the menu is back, its save data goes up with a notice at the top right, and is tried again on later starts until it worked (pending.json). "auto": false stops those too. - Changed on both sides: a dialog asks which one stays (Keep this console's, Take the server's, Change nothing), with when each was saved and the device of the server's. - A server older than the save sync takes says so in a dialog to confirm, with both versions (before a game: play anyway or not; after one: Understood, or a notice when another game is about to start). - The sync runs on a thread of its own in EdenServices; a game's title ID is read there, so the menu never waits for a scan of the games. - arc() moves to widgets for the sync dialog's spinner. - Preview A0-A7; German added; README: Save sync. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Save sync: profiles paired by a QR code Setting a profile's save sync up needs no FTP and no typing. - remote/pairing.h: a backend's pairing (as RFC 8628): a code and the address to approve it at, then polled until approved, denied or expired, stopped when it is cancelled. RomM's: /api/auth/device/init and /token, approved on the server's /pair/device signed in as the profile's user; RomM makes a device of the console with a client API token bound to it, which the save sync then is (no second device). - save_sync_config.h: SetEntry links or unlinks a profile's entry, the player's own fields kept. - Settings > Save sync: each profile with its server and user; Cross pairs it with one of the download sources' servers that can pair (asked first when several can; the source's "platform" kept), Square twice unlinks it. The pairing dialog shows a QR code of the approval page (Project Nayuki's QR Code generator library, third_party/qrcodegen, MIT), the code, the address, the steps and how long the code works. - Preview B0-B6; German added; README. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Checks: the download sources and the save sync against RomM and ftpsrv in Docker tools/check-romm.py [version ...] starts, for each RomM version, RomM and MariaDB on a free port of localhost with a library of fake Switch games (laid out as real NSPs and XCIs), and ftpsrv (the console's FTP server, built for Linux from a pinned commit) on the host's network (its passive mode needs it); makes an admin and a player with client API tokens, sets a cover in RomM, scans the library over RomM's socket (the scan task cannot be started over REST), and runs headless/romm_live_check.cpp against it; then takes it all down. By default it checks kMinimumVersion (5.0.0), 5.3.1, and 4.9.0, which has to be refused. make test runs it after check-remote.py; without Docker it is skipped, ROMM_CHECK=0 skips it too. romm_live_check: - Download sources: sign-ins, the list kept, a cover, what is no game, a game of 400 MB stopped part way and gone on with by Range (seen reading what it had from the drive first) and compared byte for byte, a spoilt end, an NSP and an XCI spoilt on the drive found damaged, updates and DLC, a cancel, leftovers, no FTP server, two sources on one server, sources removed or moved, the kept queue. - Save sync: two consoles and a phone (Argosy's API calls): uploads, a save of 150 MB with 400 files, a device deleted in RomM, the versions kept, Argosy's and JKSV's zips, another emulator's and slot's save data, both sides changed in both orders, backups, a slot emptied in RomM, a token without scopes, a server that does not answer, an older RomM refused at each sync, and pairing approved, denied and cancelled. check-remote.py keeps what needs no server and, against the stand-ins, only what a real RomM and ftpsrv do not show on demand: capped pages, a server that cannot resume, metadata ids and title IDs, another file name on a second source, a full drive. romm-mock-server.py serves the list and files only. docs/BUILDING.md says how to run it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Sample names in the checks and comments: invented ones --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: BlackBearReloaded <blackbearreloaded@protonmail.com>
138 lines
6.9 KiB
C++
138 lines
6.9 KiB
C++
// SPDX-License-Identifier: GPL-3.0-or-later
|
|
// Host check of save-sync.json (save_sync_config.h): made with an empty entry for each profile,
|
|
// kept in step with the profiles (added, renamed, removed, in their order) without touching what
|
|
// the player wrote, written only when something changed, and left alone when it is no JSON.
|
|
#include "save_sync_config.h"
|
|
#include <cassert>
|
|
#include <cstdlib>
|
|
#include <filesystem>
|
|
#include <fstream>
|
|
#include <sstream>
|
|
#include <unistd.h>
|
|
|
|
namespace {
|
|
std::string Contents(const std::string& file) {
|
|
std::ifstream in(file);
|
|
std::stringstream text;
|
|
text << in.rdbuf();
|
|
return text.str();
|
|
}
|
|
} // namespace
|
|
|
|
int main() {
|
|
using namespace Eden::SaveSync;
|
|
char folder[] = "/tmp/eden-save-sync-XXXXXX";
|
|
assert(mkdtemp(folder));
|
|
const std::string file = std::string(folder) + "/config/remote/save-sync.json";
|
|
const std::string one = "1F1E1D1C1B1A19181716151413121110";
|
|
const std::string two = "AFAEADACABAAA9A8A7A6A5A4A3A2A1A0";
|
|
const std::string three = "000102030405060708090A0B0C0D0E0F";
|
|
|
|
// No file: nothing is set up, and that is readable.
|
|
Config config = Read(file);
|
|
assert(config.readable && config.automatic && config.entries.empty());
|
|
|
|
// Made, with its folder, an empty entry for each profile.
|
|
assert(Reconcile({{one, "Player 1"}, {two, "Kids"}}, file));
|
|
config = Read(file);
|
|
assert(config.readable && config.automatic && config.entries.size() == 2);
|
|
assert(config.entries[0].profile == one && config.entries[0].profile_name == "Player 1" &&
|
|
config.entries[0].type.empty() && config.entries[0].settings.contains("url") &&
|
|
config.entries[0].settings.contains("token"));
|
|
assert(config.entries[1].profile == two && config.entries[1].profile_name == "Kids");
|
|
// The fields of an entry in the order a reader expects them.
|
|
const std::string made = Contents(file);
|
|
assert(made.find("\"profile\"") < made.find("\"__profile_name\"") && made.find("\"__profile_name\"") < made.find("\"type\"") &&
|
|
made.find("\"type\"") < made.find("\"url\"") && made.find("\"url\"") < made.find("\"token\""));
|
|
|
|
// The player fills one in, with fields of its own; the app's next look changes nothing.
|
|
{
|
|
Json document = Json::parse(made);
|
|
document["auto"] = false;
|
|
document["note"] = "mine";
|
|
document["profiles"][1]["type"] = "romm";
|
|
document["profiles"][1]["url"] = "http://nas:3000";
|
|
document["profiles"][1]["token"] = "";
|
|
document["profiles"][1]["username"] = "kids";
|
|
document["profiles"][1]["password"] = "secret";
|
|
std::ofstream(file) << document.dump(4);
|
|
}
|
|
const auto written = std::filesystem::last_write_time(file);
|
|
const std::string filled = Contents(file);
|
|
assert(Reconcile({{one, "Player 1"}, {two, "Kids"}}, file));
|
|
assert(Contents(file) == filled && std::filesystem::last_write_time(file) == written);
|
|
config = Read(file);
|
|
assert(!config.automatic && config.entries[1].type == "romm" &&
|
|
config.entries[1].settings["username"] == "kids" && config.entries[1].settings["password"] == "secret");
|
|
const Entry* kids = Find(config, two);
|
|
assert(kids && kids->settings["url"] == "http://nas:3000" && Find(config, three) == nullptr);
|
|
|
|
// Renamed, a profile added, the profiles in another order: names written, the new profile's
|
|
// entry empty, the entries in the profiles' order, the player's fields kept.
|
|
assert(Reconcile({{two, "Children"}, {one, "Player 1"}, {three, "Guest"}}, file));
|
|
config = Read(file);
|
|
assert(config.entries.size() == 3 && config.entries[0].profile == two && config.entries[0].profile_name == "Children" &&
|
|
config.entries[0].settings["username"] == "kids" && config.entries[1].profile == one &&
|
|
config.entries[2].profile == three && config.entries[2].type.empty());
|
|
assert(Json::parse(Contents(file))["note"] == "mine" && !config.automatic);
|
|
|
|
// A profile removed: its entry goes; a second entry of the same profile goes too, the first
|
|
// one counts.
|
|
{
|
|
Json document = Json::parse(Contents(file));
|
|
Json twin = document["profiles"][0];
|
|
twin["username"] = "someone else";
|
|
document["profiles"].push_back(twin);
|
|
std::ofstream(file) << document.dump(2);
|
|
}
|
|
config = Read(file);
|
|
assert(config.entries.size() == 3 && Find(config, two)->settings["username"] == "kids");
|
|
assert(Reconcile({{two, "Children"}, {three, "Guest"}}, file));
|
|
config = Read(file);
|
|
const Json reconciled = Json::parse(Contents(file));
|
|
assert(config.entries.size() == 2 && reconciled["profiles"].size() == 2 &&
|
|
config.entries[0].settings["username"] == "kids" && config.entries[1].profile == three);
|
|
|
|
// Entries the file cannot tell apart are not entries.
|
|
{
|
|
Json document = reconciled;
|
|
document["profiles"].push_back(Json{{"type", "romm"}});
|
|
document["profiles"].push_back("text");
|
|
std::ofstream(file) << document.dump(2);
|
|
}
|
|
assert(Read(file).entries.size() == 2);
|
|
|
|
// Linked from the menu (pairing): the sign-in replaced, the name and the player's fields kept;
|
|
// unlinked: type, url and token empty.
|
|
{
|
|
Json document = Json::parse(Contents(file));
|
|
document["profiles"][0]["note"] = "mine";
|
|
std::ofstream(file) << document.dump(2);
|
|
}
|
|
assert(SetEntry(two, Json{{"type", "romm"}, {"url", "http://nas:3000"}, {"token", "rmm_paired"}, {"__server_user", "kids"}}, file));
|
|
config = Read(file);
|
|
assert(config.entries[0].type == "romm" && config.entries[0].settings["token"] == "rmm_paired" &&
|
|
!config.entries[0].settings.contains("username") && !config.entries[0].settings.contains("password") &&
|
|
config.entries[0].settings["note"] == "mine" && config.entries[0].profile_name == "Children");
|
|
assert(SetEntry(two, Json::object(), file));
|
|
config = Read(file);
|
|
assert(config.entries[0].type.empty() && config.entries[0].settings["url"] == "" &&
|
|
!config.entries[0].settings.contains("__server_user") &&
|
|
config.entries[0].settings["token"] == "" && config.entries[0].settings["note"] == "mine");
|
|
assert(!SetEntry("00000000000000000000000000000000", Json{{"type", "romm"}}, file));
|
|
|
|
// Not JSON (a typing error over FTP): left as it is, and nothing syncs.
|
|
{ std::ofstream(file) << "{ \"profiles\": [ { \"profile\": "; }
|
|
const std::string broken = Contents(file);
|
|
config = Read(file);
|
|
assert(!config.readable && !config.error.empty() && config.entries.empty());
|
|
assert(!Reconcile({{two, "Children"}}, file) && Contents(file) == broken);
|
|
// "profiles" that is no list is not the app's to replace either.
|
|
{ std::ofstream(file) << "{ \"profiles\": {} }"; }
|
|
assert(!Read(file).readable && !Reconcile({{two, "Children"}}, file) && Contents(file) == "{ \"profiles\": {} }");
|
|
|
|
std::filesystem::remove_all(folder);
|
|
std::puts("save sync check: PASS");
|
|
return 0;
|
|
}
|