mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
worktree-timelapse-test-gcode
47
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr ShyshatskyiandClaude Opus 5
|
a51b09884d |
klippy is exec'd directly; start-stop-daemon matched moonraker
klipper/run launched klippy with
exec start-stop-daemon -S --exec "$FF_PYTHON" -- "$KLIPPY" ...
-S starts the program unless a matching process is found, and --exec matches
on the interpreter rather than on the script. moonraker runs $FF_PYTHON too,
so start-stop-daemon found moonraker, reported the service already up, and
returned without ever exec'ing klippy; s6 saw a longrun exit and respawned it
for as long as the printer was on. One boot log carried 508 copies of
"/usr/data/anvil/bin/python3.13 is already running", with moonraker up,
klipper disconnected, ff-startup giving up with KLIPPER DID NOT FINISH
STARTING, and HelixScreen in front of a machine with no motion and no heaters.
Not intermittent: klipper waits on the mcu-bringup oneshot and moonraker
depends on nothing, so moonraker is always already running by then.
There was nothing for start-stop-daemon to do. s6-supervise is the parent --
it knows the pid, runs one copy of a servicedir at a time, and restarts what
dies -- so a duplicate check in a run script has no true positive left to find
and can only return false ones. The one case s6 does not cover, a klippy that
outlived its supervisor, was already swept by cmdline against klippy.py.
The guard is static rather than a replica test because the replica cannot
reproduce this. Printer binaries run there under qemu-mipsel-static, so a
moonraker process has cmdline[0] and /proc/PID/exe of qemu-mipsel-static, not
$FF_PYTHON; start-stop-daemon matches nothing and starts klippy correctly.
Measured: a replica baked from a package carrying the broken line passes a
behaviour test asserting klippy starts with moonraker up. A behaviour test
there would report this failure as green.
The rest of the diff is comment reduction, 119 lines to 66.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
28eb20f3da |
Package numpy, because klippy does not start without it
extras/stepper_resonance_tester.py opens with a bare `import numpy as np`; klippy.py:122 walks EVERY config section through a load_object whose importlib.import_module is not wrapped in except ImportError; printer.base.cfg includes FlashForge's printer.vibration.cfg, which declares [stepper_resonance_tester]. Since klippy moved to our own CPython 3.13 that has not been a printer missing resonance testing. It has been a klippy that never reaches ready. So this is not a scientific extra. anvil-klipper depends on anvil-python-numpy the way it depends on cffi, except that the others mean "klippy imports this" and this one means "klippy starts". It also gives back SHAPER_CALIBRATE and TEST_RESONANCES, dead since the same switch, and the VFA calibration behind STEPPER_RESONANCE_FACTORY_CALIBRATE that docs/notes/44-vfa-calibration.md traces end to end. WHAT WAS NOT OBVIOUS, in the order it bit. numpy 2.x builds with meson-python, and meson has never heard of _PYTHON_SYSCONFIGDATA_NAME -- the trick every other python-* recipe here leans on. pkg_mesoncross writes a real cross file instead, naming the $PKG_XW wrappers rather than the raw toolchain: the wrappers carry -EL -mnan=2008, and a cross file that reaches past them builds a tree that passes every gate which only reads symbols and is refused at exec() on the printer. pkg_pywheel grew a PKG_PY_PIP_ARGS branch to pass the file as --config-settings, which is the one thing PEP 517 does standardise. The cross file's `python =` is the BUILD python, and that is not a mistake: meson introspects an interpreter by running it and asking sysconfig, and pkg_pytarget has already made that sysconfig answer for mipsel. The old trick does reach meson, just not by its own mechanism. numpy's meson.build hard-errors below gcc 8.4 and ours is 7.2. The gate is the SciPy Toolchain Roadmap's support statement, not a requirement of the code: this toolchain compiles if constexpr, structured bindings, std::optional, std::string_view, and pocketfft_hdronly.h -- 3635 lines of the heaviest template C++ in the tree, and the half np.fft.rfft needs. Probed before the floor was lowered, and lowered rather than removed, behind a guard that fails loudly if upstream moves the check. 19 extension modules, every one at e_flags 0x70001405/0x70001407, 14MB shipped after pruning tests/, f2py/, stubs and static libs. meson-python is self-hosting, so pkg_buildpython runs meson and ninja just to install the backend -- hence ninja-build in the image and PYPKG_HOST_LIST in dependency order rather than alphabetical. And pip puts meson and cython in work/.py-host/bin, which nothing had ever looked in: without that on PATH the backend fails to install with `meson executable "meson" not found` while pip list shows it right there. THE GATE THAT WOULD HAVE CAUGHT ALL OF IT is qa/replica/test_klippy_extras_import.py: walk the real /usr/data/config include graph, map every section to its extra, import each on the printer's own interpreter. It needs no MCU, because config load happens before klippy:mcu_identify. Nothing asked that question before, which is how two comments in this repo came to assert the module guarded its own import. test_numpy.py asks for numeric answers rather than a module object -- linalg.lstsq specifically, because -Dallow-noblas selects the bundled lapack_lite and _r2_sinusoidal_fit calls it every calibration round. And test_install.py's include check read only ff-*.cfg, so FlashForge's four -- printer.vibration.cfg among them -- were never asked about at all. ff-toolchange.cfg wraps the two search commands in _FF_SHAPER_PREP, as SHAPER_CALIBRATE already was. STEPPER_RESONANCE_FACTORY_CALIBRATE is deliberately NOT wrapped: it is a gcode_macro, not a registered command, and same-named sections merge under RawConfigParser(strict=False) -- a wrapper would not wrap it, it would BE it, with rename_existing pointing at itself. It needs none, since its body is two calls to a command that is wrapped. No SAVE_CONFIG in any macro either; it restarts klippy and stays the operator's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EJr2x7aDaY889qwSxbdLt8 |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b96390b579 |
Eleven gates read our own scripts as text; none of them read a package
Each one grepped a file in this repo and asserted a spelling. That is green when the behaviour is broken and red when nothing is, and both halves happened here: test_the_archives_are_built_by_upstream wanted the literal "opkg-make-index" in bin/build-packages.sh. The script calls $OPKG_INDEX_BIN, which bin/common.sh points at exactly that tool, so the gate went red for a variable it approves of. test_the_klipper_service_runs_the_payload_tree_on_our_python asserted "/usr/prog/klipper/klippy" was absent from the klipper run script. The command line is $MODDIR/klipper/klippy/klippy.py and has been for some time; the match was on the header paragraph explaining that the tree USED to be read from /usr/prog. The gate read its own warning as the thing it warns about. test_klippy_imports_resolve_on_our_python goes for a different reason: it sourced anvil-env.sh without MODDIR, against that file's stated contract, so FF_PYTHON came out /bin/python3.13 and four imports failed on a missing interpreter. What it meant to ask is answered by klipper starting at all. The rest are the same technique: payload.sh, lib.sh, build-packages.sh and bin/*.sh grepped for a spelling, one gate grepped ANOTHER TEST FILE for the words nan2008/o32/mips32r2, and two linted every recipe for forbidden constructs. What survives asks the built artifact or the replica. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S64uRMdZbj9xPMns2dqMG8 |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
2e9f2214a0 |
Merge master into the comment trim
Master moved twelve commits while this branch ran: patch.sh became
payload.sh, run-pre.sh and run-append.sh became runFirmwareExe.sh, the
install manifest and anvil.conf are gone, and versions.env was trimmed
harder there than here (53 comment lines against 185).
Resolved so the merge contributes comments and nothing else. Verified
mechanically: every one of the 270 files in the tree now carries
exactly master's code, and no file this branch touched is resurrected.
- master's deletions accepted: bin/patch.sh, bin/verify.sh,
installer/run-{pre,append}.sh, anvil.conf.in
- versions.env: master's, which is the better trim
- 26 files master only renamed through: this branch's trim kept, with
patch.sh -> payload.sh applied
- 14 files master rewrote to describe the new installer: master's
text taken wholesale, to be re-trimmed separately
- stale references to the manifest and run-append.sh corrected in
pkgs/3rdparty/busybox/build.sh and pkgs/moonraker/build.sh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMMpQTajN7RDJySrhQ2cTE
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5d7754285e |
Wipe $MODDIR and extract, instead of pruning by a manifest
The installer deletes /usr/data/anvil outright and extracts into the empty directory. The installed set is the shipped set by construction, so a renamed file cannot survive beside its replacement and no list has to be shipped, read or trusted for that to hold. What that costs is everything else under $MODDIR, and the audit of what was actually there is docs/notes/86-wipe-and-extract.md. backup/stock had no restore path in any script and omitted the one file a restore would need. .prev-root-hash was already dead state. config-installed was a twelve-file snapshot serving one md5, for a compare whose stated rationale -- "there is no include-and-override seam" -- stopped being true when moonraker-custom.conf became the last include, and which inverted its own purpose by keeping the webcam block and the API lockdown off any printer that tripped it. So moonraker.conf is overwritten like every ff-*.cfg. HelixScreen's /tmp stash is untouched and is now the only preservation the installer does, so the gate asserts it with a value that could only have come from the printer. Gone with it: both delete passes, the forged-path guard, the rmdir pass, the pre-manifest branch, the unconditional init.d/anvil-service/ anvil.conf removal, the etc/s6 scandir sweep and the $MODDIR __pycache__ sweep -- the last two existed only because a manifest cannot see paths made at runtime. firmwareExe already makes the scandir itself. test_upgrade.py rewritten: the negative control is inverted rather than deleted, and the legacy fixture goes -- there is one path now. qa/static green (112). The replica lane needs a package in work/out, which needs config.env with FF_KEY and STOCK_TGZ; not run here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KXKk3zz8LAUPUDZkVEZ9fR |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
7ffb39a730 |
The last ten files still named run-pre.sh and run-append.sh
Both were deleted three commits ago and the sweep that followed missed these: the recipes, the s6-rc service scripts, printer.base.cfg's header and three qa modules all still pointed at them by name. Renamed to runFirmwareExe.sh, which is the file that actually does each thing described. Two needed more than a rename. anvil-link-prog.sh's ORDERING note explained itself in terms of stock run.sh's line numbers -- "distributes the software component at its lines 107-179, before the payload is extracted at line 180" -- and none of that is the shape of an install any more. What the note is really for is why the links cannot ship as files, which is unchanged, and it now says that instead. The paragraph about the component no longer carrying firmwareExe is replaced by the reason that matters: no component ships at all, so the version directory under /usr/prog/PROGRAM/software stays FlashForge's and app_startup.sh's watchdog has their binary to fall back on. test_moddir_paths.py's ALLOWED set excused two paths that no longer exist. anvil.conf went when the settings it held moved into the services, and .prev-root-hash is not written any more -- this installer only deletes the stale ones. Both were dead weight in a list whose whole job is to say "not shipped, and correctly so": an entry for a file nobody guards on excuses nothing and hides the next real one. Gates: 112 static tests, shellcheck clean over the edited payload scripts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b98552386b |
No FlashForge component ships; patch.sh is payload.sh
The software component existed to give the mod one exec hook. It is not needed for that any more, so a release now carries our installer and our payload and nothing else: 26MB smaller, 21MB of which was FlashForge's own firmwareExe travelling back to the partition it was already on, along with their klippy tree, their wifi module and their configs. WHAT IT GIVES BACK. /usr/prog/PROGRAM/software is no longer rewritten, so the version directory there stays FlashForge's with their genuine firmwareExe inside it -- and app_startup.sh's five-second watchdog, which copies one out of that directory when nothing by that name is running, has something to copy again. A printer whose payload fails to install now comes up on the stock UI instead of a blank screen. That recovery has been broken since we started deleting their binary out of a component we shipped; docs/how-it-works.md claimed it worked throughout. THE ROOT PASSWORD moves with it, because it used to ride in the component's shadow file. bin/pack.sh bakes ROOT_PW_HASH into the installer, single-quoted -- an unquoted $6$... would have the printer's shell expand $6 as a positional parameter and ship a truncated hash -- and the installer writes it to /usr/prog/etc/shadow. The random-password path asks one question instead of four: FlashForge ship the same root hash on every machine, so a live shadow still carrying it means nobody has set a password and it is ours to set. The .prev-root-hash recording, the shipped-vs-live comparison and the restore-it- afterwards step were all machinery for surviving a file that nothing overwrites any more, and are gone. pack.sh checks the hardcoded constant against the stock package it built from, because getting it wrong is silent and total: every printer would read as "already set" and none would ever get a password. bin/patch.sh is bin/payload.sh. It patched nothing any more -- no splice, no shadow edit, no firmwareExe deletion -- and what is left is the payload: the feed installed by the printer's own opkg, the seeded config, the manifest. Its sections are renumbered 1-5 with no gaps, and the five tombstone comments marking where removed sections used to be are gone with them. test_recipe_layout.py loses six of its nine tests. Each compared the tree against a description of the tree -- no fourth directory kind, each kind still used, 3rdparty carries no files of ours, installer/ holds exactly two named files -- and none could go red for a reason a printer would notice, while all six went red the moment the layout legitimately changed. The two that bite are kept: a payload/ path escaping $MODDIR, and a recipe name at both levels, which makes pkg_dir silently build the wrong package. SW_VER is deleted from config.env.example and both workflows: it stamped a component that is not built. Gates: 112 static tests pass. The generated installer is verified after substitution -- all four lines present, and `sh -n` over the result, because a hash with a stray character in it would otherwise reach a printer as a syntax error whose only symptom is a package that installs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
2a6331a831 |
We own runFirmwareExe.sh; the install is no longer spliced into run.sh
app_startup.sh runs whatever it finds under that name in the package it just decrypted. That is the entire contract -- plus the exit code, where 0 means "flashed, power-cycle me" and non-zero means "carry on booting" -- so owning the name is all it takes to own the install. We were not using it. We shipped FlashForge's copy verbatim and reached the printer through the one exec hook their file offers: the software component's run.sh, into which bin/patch.sh spliced two marked blocks with 40 lines of regex. installer/runFirmwareExe.sh is that file now. run-pre.sh and run-append.sh are folded into it unchanged in substance -- the manifest delete, the HelixScreen settings rescue, the s6 scandir sweep, the config merge, the password dance are all the same code -- and the two orderings that mattered are kept explicitly rather than by where a regex put them: the backups run BEFORE any component is installed, because the software component ships /usr/prog's shadow and passwd and its run.sh copies them over the live ones. The four component installers become one function. Their semantics are kept exactly, including that a component failing md5sum.list is discarded rather than half-installed, and that every previous version directory is wiped before the new one lands. What is NOT kept is the third of that file which never ran: create_key(), defined and never called, and built its paths without a slash so it would have written /usr/progprivate.pem; SSID_NAME/SSID_PASS, assigned and never read; the commented-out kernel version gate; the screwflag factory bypass of the model check, which we ship no screwflag for. bin/pack.sh stages it, rewriting MACHINE=, PID= and MOD_PW_AUTO= and then GREPPING for all three -- a package whose gate still said Creator5Pro because a sed missed would install on the wrong machine. MOD_PW_AUTO is 0 in the checkout, which is the safe default and not an arbitrary one: test_upgrade.py runs the unsubstituted file, and 1 would have it reroll the replica's root password and drop a password file on /mnt every run. The software component still ships, unchanged, and its run.sh is left exactly as FlashForge wrote it. Dropping it is the next commit; this one moves the vehicle without changing what reaches /usr/prog. Gates: qa/static/test_shell_syntax.py already covered installer/*.sh under both bash -n and the printer's ash plus shellcheck, so the new file was checked the day it landed -- 91 pass. test_upgrade.py runs the real installer over two synthetic payloads and is repointed at it. Two doc claims corrected while their sections were open, both false before this change rather than because of it: how-it-works.md said app_startup.sh's firmwareExe recovery "still works" -- it cannot, because it copies out of the version directory and ours has had firmwareExe deleted from it -- and it described a Moonraker preflight-and-rollback that went with the /usr/prog copy; moonraker-preflight.py exists nowhere in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
9b5b562a22 |
Comments: keep the contract and the trap, drop the narration
A sweep over every commented file in the tree. What is kept is what a reader needs to change the code safely: what a thing is, its signature, the invariants other files depend on, and the traps that fail silently (find's non-zero exit under set -e, s6's baked-in libexecdir, the -lm that decides whether _sqlite3 exists, opkg's compile-time prefix). What goes is the narration around it: incident histories, arguments against alternatives already rejected, restatements of the code below, and paragraphs re-explaining a rule three files already state. One paragraph in pkgs/3rdparty/opkg/build.sh was duplicated outright. Comment lines 7971 -> 6158. Every change is comment-only: no non-comment line differs anywhere in the 86 files, checked mechanically. Commented-out config directives (#fast_feed: 30000 and friends) are documentation of defaults and are kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NMMpQTajN7RDJySrhQ2cTE |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
57c776ee63 |
test/ is deleted; qa/ is the only suite
The eight files were already gone from the working tree. What this commit does
is everything that had to follow them, because nothing else had been updated
and three of the four things pointing at test/ were on a build or release path.
WHAT WAS STILL WIRED TO IT
pkgs/klipper/build.sh ran test/test-chelper.py under `|| pkg_die`. With the
file gone `make packages` would have died on every
klipper build.
bin/verify.sh ran the same script and read a non-zero exit as
"c_helper.so does not match the shipped klippy tree
(stale build)" -- a false red on every release verify
rather than an error about a missing file. Its
ship-boundary check also globbed `find bin test
docker`; that is `find bin docker` now.
pytest.ini testpaths = test, so a bare `pytest` collected
nothing at all. It is `qa` now, which is what the
comment above it was already describing.
qa/static/ PY_GLOBS listed test/*.py and test/integration/*.py,
test_shell_syntax.py and _files() makes an empty glob a collection error
on purpose -- so the whole static lane failed to
collect, taking 93 unrelated tests with it.
Also dropped: the `pytest -- the host-side tree` step in ci.yml, the `test-py`
target with its .PHONY entry and two help lines, and qa/conftest.py's opening
paragraph explaining why qa/ sat beside test/ rather than inside it.
THE CHELPER SYMBOL CHECK IS GONE, NOT MOVED
It compared the .so's dynamic symbols against what klippy cdefs, and it was the
only check for a c_helper.so built from older sources than the klippy tree
beside it -- the failure that shipped once and bricked klippy startup on
hardware, because cffi resolves lazily and a stale .so imports cleanly.
qa/replica/test_install.py covers the ELF arch and nan2008 and nothing else.
What stands in its place is structural rather than a gate: pkgs/klipper
compiles the .so from the chelper sources of the very tree it ships, so the
mismatch is not something the recipe can produce. The comments in
pkgs/klipper/build.sh and docs/building.md now say that is the whole defence.
DOCS
docs/testing.md's "what is left in test/" section is now "test/ is gone", and
it names what went uncovered rather than leaving it to be found: the tool-frame
arithmetic in ff_toolchange, the chamber macros' per-model branching, the stamp
discipline in ff-startup.py, and the verification G-code against the shipped
macros. docs/hardware-testing.md said `make test-py` keeps the two
gcode/creator5-*.gcode files honest; it now says nothing does, and to read them
by hand when a macro is renamed. Paths corrected in docs/building.md,
docs/printer-replica.md and docs/qa-migration.md's intro.
Verified: `pytest` collects 366; qa/static is 136 passed, up from 43 because
the lane could not collect before this. Its 110 errors are this checkout's
missing vendor/opkg-utils, the same cause before and after. bin/verify.sh,
pkgs/klipper/build.sh and the Makefile parse; ci.yml is valid YAML.
Not touched, and stale before this commit: tools/python/README.md and
tools/supervisor/README.md still document test/integration/printer/case-*.sh
and a printer-exec.py that earlier commits removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bLGYpVAeVzAX6NeAxcU42
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c92b3c9e86 |
anvil.conf is gone; a component runs because it is installed
The file held three unrelated things behind one name: gates deciding which services boot, switches for the splash / the startup wait / the first-boot import, and tunables for the camera and the nice values. Every one of them defaulted to on, and what each offered was a half-installed printer -- a state nobody chose on purpose and everything downstream carried a second answer for. firmwareExe no longer expands the ok-all bundle and filters it, it asks for the bundle: BOOT_SET=ok-all. A printer that should not run a component does not have it -- `opkg remove`, or a stock FlashForge package. The settings worth keeping are each stated once now, in the service that uses them: the camera's resolution in its run script, nice 10 there and nice 5 in moonraker's (the values every install was getting from the template, not the 0 the inline fallback said). ff-startup's `up` stops passing --timeout and --klipper-tries at all, because 240 and 6 are already ff-startup.py's own defaults and two statements of a default can disagree. Nothing sources the file, so nothing reads it, so it must not survive an upgrade looking editable. run-append.sh stops preserving it and removes it unconditionally beside init.d and anvil-service.sh -- unconditionally because the pre-manifest sweep removes seven DIRECTORIES and would leave this file sitting at the top of $MODDIR. The two upgrade tests that asserted the edit survived now assert the file does not. seed/ leaves anvil-core entirely, which takes with it the recipe's one exception to "every sibling directory is shipped", the seed mount in the replica plumbing, and its exemption in the .ipk allowlist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015E7armHFDcnZFZwdAbqkbt |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
4ce928f160 |
The last two artefact ABI gates go; the toolchain check stays
Follow-on to |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
9aece5b890 |
klipperDaemon was never replaced on any printer
installer/run-append.sh guards the klipperDaemon replacement with
if [ -f $MODDIR/bin/klipperDaemon ] && [ -d /usr/prog/klipper ]; then
and the file ships at $MODDIR/prog/klipperDaemon. It moved there in
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
7dda2fee0d |
ABI gate: one check, on the filesystem the kernel loads from
mips_abi_gate answered "did this recipe stage the right ABI", once per recipe, over a PKG_ROOT. Four implementations of that one rule had grown up around it -- the function in bin/common.sh, the call at the packaging boundary, two recipes that re-ran it on their own output, and a readelf|grep nan2008 in bin/verify.sh reading a single .so -- and between them they still could not see the largest binary we ship: bin/patch.sh unpacks HelixScreen into /usr/prog and no gate ever read that tree. pkgs/helixscreen/pkg.conf had carried a note saying exactly that. qa/replica/test_abi.py replaces all of it. It walks the installed filesystem on the printer's own CPython -- one qemu process, ~24000 files, 8s -- and reads the ELF header of everything it finds. That is the only place where what the recipes staged, what opkg unpacked, what patch.sh copied and what FlashForge shipped are all visible at once, and it is the thing the question is actually about: the kernel does not read a PKG_ROOT. MEASURED, not assumed: 603 ELF objects on a real install. 569 the loader handles and 568 of those conform. The exemptions are the other two, both narrow. ET_REL objects (33 stock .ko files, plus a stray python.o in FlashForge's CPython 3.8 tree) are not asked about the FP ABI, because insmod and the linker read them rather than the ELF loader and kernel code has no FP ABI to agree about -- by e_type, so the python.o needs no case of its own. And /usr/prog/PROGRAM/ffstartup-arm is an ARM binary FlashForge ships that nothing in the stock tree names; exempt by path, so an ARM binary anywhere else still fails. The architecture is settled before anything is classified, and that ordering came from planting a counterexample rather than from reasoning: a big-endian object decodes its own e_type big-endian, so ET_DYN reads back as 768, lands in neither bucket, and an earlier draft looked straight at it and passed. Every gate here was driven red on purpose before it was kept -- an x86-64 .so in $MODDIR, a real mipsel object with EF_MIPS_NAN2008 cleared, an ARM binary outside the exempt path, a big-endian object, an x86-64 object among the .ko files, and a truncated sweep. test_install.py's two chelper header tests go with it: they read one file out of the several hundred on the same fixture, a strict subset of what now runs. Whether the package ships a chelper at all stays test_install.py's question and is not asked twice. test_ipk.py's assertion that the gate exists becomes an assertion that a second one has not grown back beside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NpH87TJJEjDyRhQPu18pgp |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
965cf67d93 |
Compile the s6-rc database where the packages are installed
The replica already installs the feed with the printer's own opkg, and anvil-s6-rc already ships s6-rc-compile -- pkgs/3rdparty/s6-rc/build.sh keeps it deliberately, "the only way to recover a printer whose database is wrong without rebuilding a package on a laptop". So the compiler was sitting in the container the whole time, three lines after the install that puts the source tree there. bin/patch.sh section 5b-2 built a second, NATIVE skalibs+execline+s6+s6-rc instead -- four ./configure && make passes cached in work/.s6-native -- purely to get an s6-rc-compile that runs on x86. 110 lines, and its entire hazard was self-inflicted: s6-rc-compile bakes the #! of the execline the COMPILER was linked against into the oneshot runner, so a native stack configured with a different --prefix ships a database whose every oneshot ENOENTs on the printer. Compiling with the compiler we ship cannot get that wrong. The output is the same object. docs/notes/80-s6-migration.md records the measurement from 2026-08-28: one source tree, the host compiler and the target one under qemu, byte-identical db, n and resolve.cdb and identical servicedirs. That measurement was made to justify compiling on the host; it works just as well in the other direction. The rejected alternative in the old comment -- ship s6-rc-compile and compile at INSTALL time on the printer -- is a different thing and stays rejected. qa/static/test_s6rc_source.py is dropped. It was the only consumer of the native compiler, and it hard-failed on any checkout that had not run `make build`: 14 of the static suite's errors were it saying so. What goes with it is the timeout and edge assertions, which are now made nowhere; qa/replica/ test_s6rc.py still boots the real thing. bin/verify.sh keeps the half a host can still check, moved out of patch.sh: the oneshot runner's shebang, and that compiled/current resolves -- read as text straight out of the shipped payload, so it needs no mipsel binary. bin/patch.sh: 278 -> 207 lines. Static suite: 122 passed, errors 124 -> 110. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c185ef3285 |
The replica is a build tool; test/ is 28 unit tests
Two changes that turned out to be one.
MOVED, NOT DELETED. `test/ffsim/` and `test/integration/printer/` are
`tools/replica/`. They were never tests: bin/patch.sh calls
bin/build-payload.py, which starts a replica and assembles the payload by
running the PRINTER'S OWN opkg against / inside it -- so the "test harness"
was on the build path, and deleting it as asked would have broken `make
build`. qa/lib/replica.py drives the same directory unmodified. No content
changed; what changed is qa/lib/replica.py's build directory, three launcher
sys.path lines, the Makefile targets and the CI shellcheck glob.
tools/replica/README.md says who drives it and why it is not under test/.
CUT TO 28. The five surviving modules were 136 tests; they are 28, chosen on
one question: would this failure be SILENT? What is kept is the mistakes that
run perfectly and are wrong --
test_startup 8 no stamp unless the values are verifiably saved (a
stamp written early is a printer that has lost its
factory calibration for good), and the handover order
test_tool_transform 7 every one is a move to the wrong place: a sign, a
frame left applied while probing, a stale gcode_move
cache, a config a calibrated printer cannot parse
test_ffscreen 5 the four ways it can do harm rather than look wrong
test_chamber 4 the chamber mistakes that cost a print before anyone
sees them, plus the parse gate they rest on
test_gcode 4 a renamed macro, and the safe file being cold
What went: the panel narration, the geometry probe and parse_geometry, the
z_adjust and per-job-Z bookkeeping, the config layout checks, the G-code
bounds checks. Each is either a wrong number shown to a person, or a failure
the printer reports loudly the first time -- and qa/replica/test_boot_screen.py
already renders the real frames on the real machine.
The rewrite is not a delete: assertions were folded so a kept test asks its
question in every direction it can be got wrong (both models, both rotations,
all four ways the migration can fail), and each file's header now says what it
dropped and why, where the next person will open it.
Also gone: conftest's `rootfs` fixture, orphaned two commits ago. Nothing under
test/ needs the firmware, docker, or a network any more.
28 host-side tests pass in 0.15s. qa/static unchanged at 218. `make rootfs`
re-verified end to end from its new path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UzLebCrvc8ssWR5JuEsha4
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
465c9902f1 |
test/: drop the seven modules whose subject was the fake
Eight files, 46 tests and make-stock-fixture.sh. Each of the seven test modules built a stand-in -- a klippy stub, an argparse Namespace, a config tree -- elaborate enough that what they pinned down was the stand-in's behaviour rather than the printer's. What is left under test/ is the Klipper config gate and the three modules whose subject is our own Python: ff-startup, ffscreen, and the tool transform. make-stock-fixture.sh goes with them. docs/qa-migration.md argued for keeping it as the input a future build-path test would need, but it fed a gate that no longer exists, and a fixture generator with no consumer rots exactly as silently as a test that asserts nothing. What it had learnt is recorded in the doc so whoever ports the build path does not have to find it twice. The coverage this gives up is written down rather than glossed: a new section in docs/qa-migration.md names every dropped file, what it asserted, and what now asks the question instead -- for four of them, nothing. The one worth watching is the shebang-executable check, which qa/ makes only of qa/replica/actions/*.sh. Falling out of it: the rootfs fixture in test/conftest.py had no users left, and nothing under test/ needs the proprietary firmware any more. Comments in qa/static/test_probes.py, qa/replica/test_mcu_bringup.py, bin/patch.sh and ci.yml that pointed at the deleted files say what is true now; the tree listing in docs/building.md was stale from the run-tests.py removal and is rewritten. 136 host-side tests pass. qa/static is unchanged at 218. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UzLebCrvc8ssWR5JuEsha4 |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6aa622d2ff |
6,280 lines of case scripts reported 13 bits. They are 118 tests now.
test/run-tests.py is gone, and with it the bespoke harness that wrapped
pytest, re-parsed its JUnit XML, and drove thirteen replica case scripts to
get one boolean back from each. Every gate it ran is either a module under
qa/replica/ or deliberately retired.
PORTED. case-install.sh (579 lines, 1 bit) is test_install.py, 22 tests, asked
of the machine the printer's own app_startup.sh built rather than one the
harness assembled. case-upgrade.sh is 21, case-supervisor.sh 31,
case-boot-screen.sh 17, case-mcu-bringup.sh 8, and case-nginx.sh plus
case-moonraker313-s6.sh are test_web.py's 7.
RETIRED, subject deleted by the s6-rc switch: case-services.sh, case-nginx.sh,
case-moonraker.sh, case-moonraker313-s6.sh, case-camera.sh and
case-priority.sh drove $MODDIR/init.d/S* or sourced anvil-service.sh, neither
of which exists. Decided against real output -- `make test-services` failed
with "the payload ships no anvil-service.sh". case-pyext.sh and
case-moonraker313.sh went too: no gate, no target, no CI step, never run once.
RETIRED as a coverage decision: case-python.sh (test_web.py runs real Moonraker
on that interpreter, which is harder), case-libpath.sh, and case-recovery.sh.
The last is the brick-safety rollback and has no replacement; it is recorded in
docs/qa-migration.md rather than left to be discovered.
THREE ASSERTIONS HAD STOPPED MEANING ANYTHING, and each had been green for
months:
* case-mcu-bringup.sh's negative control had INVERTED. It asserted the
bring-up fails without LD_LIBRARY_PATH; CPython is --disable-shared now and
needs no library path, so the case's own skip branch was the one firing.
* case-upgrade.sh's headline assertion was VACUOUS. run-append.sh:150 does
`rm -rf $MODDIR/init.d` unconditionally, outside the manifest branch, so
"the update removed the stale S60web" passed regardless of the manifest
logic it was written to test.
* case-boot-screen.sh CONFIRMED ITS OWN ARITHMETIC, computing expected pixel
packing by calling ffscreen._pixel inside the replica -- a swapped R/B
matched itself.
case-install.sh's boot-log grep is not reproduced for the same reason: the log
line names every service in the boot set including the ones that then fail, so
grepping it cannot go red. The compiled database is asked instead.
TWO REAL BUGS FOUND ON THE WAY.
assemble.sh attached /stick.img with `losetup` and wrote the device to
/stick.loop, which NOTHING HAS EVER READ. Loop devices are the host's, not the
container's, so every USB_STICK=1 run leaked one permanently and a WSL2 kernel
has thirteen. Once they were gone every replica run failed with "failed to
setup loop device", which looks like a docker fault and is not. Detached now in
entrypoint.sh on the way out, and in qa's stop() for the held-open containers
that never take that path.
test_startup.py was 43 tests red: it built its argparse.Namespace by hand and
had never been taught --only-bringup / --no-bringup, and its FakeKlipper still
modelled start.sh plus a separate stop when klipper is restarted with one
`s6-svc -wr -t`.
CI AND RELEASE. The replica job was getting its package as a side effect of
run-tests.py running unpack/patch/pack; that is an explicit step now, as the
comment there predicted it would have to be. release.yml tested the exact files
it ships through sim-install.py, whose exit status the loop was discarding --
it is `REAL_PKG=<pkg> pytest ./qa/replica` per model now, 22 named results each.
WHAT IS NOT TESTED ANY MORE: bin/unpack.sh, patch.sh, pack.sh and verify.sh.
The packaging build on a synthetic stock package went with the runner and has
no counterpart in qa/. That is the path producing the .tgz a user flashes.
make-stock-fixture.sh is kept as the input a replacement needs.
Measured: qa/static 246 passed, qa/replica 110 passed and 8 blocked on hardware
the replica has not got (no /dev/ttyS*, no /dev/video*, so ok-all is
unreachable), test/ pytest 185 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BTAXgjtF8GdhqSxvdXCFW
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
abd4a01d73 |
Adopt the s6-rc boot graph -- DOES NOT BOOT YET, see below
Merge of qa-suite-bridgehead into the packaging line. The seven init.d/S*
scripts and the hand-written etc/s6 servicedirs are gone; anvil-core ships
etc/s6-rc/source/ and bin/patch.sh compiles it into etc/s6-rc/compiled/ after
the payload is installed. klipperDaemon joins firmwareExe and start.sh in
payload/prog/, all three linked into place by anvil-link-prog.sh.
WHAT IS KNOWN BROKEN. qa/replica/test_s6rc.py fails: the tree never comes up,
because s6-rc-init dies with
unable to supervise service directories in /run/s6-rc/servicedirs:
Value too large for defined data type
That is EOVERFLOW out of the readdir() loop in s6rc_servicedir_manage.c, which
checks errno after every entry. Measured, and these are ruled out:
* not the live directory's filesystem -- fails identically with -l on
/usr/data as with the default /run, and the inodes there are 3 and
1358496, nowhere near 32 bits
* not the database -- s6-rc-db lists all eleven services out of it, and a
database recompiled inside the replica by the TARGET s6-rc-compile fails
exactly like the one compiled natively on x86
* not a missing -D_FILE_OFFSET_BITS=64 -- both s6-svscan and s6-rc-init
reference readdir64 and never plain readdir
What is left is the toolchain. qa-suite-bridgehead built s6, s6-rc, execline
and skalibs with the Bootlin musl toolchain, where readdir has no EOVERFLOW
path at all; the packaging line builds them as recipes against the Ingenic
glibc, where it does. s6-svscan looked healthy only because it was started on
an EMPTY scandir, which never reaches the loop.
Everything else is green: 262 static tests, and bin/verify.sh passes on the
73MB package, including the graph's own "compiled s6-rc database" check.
Resolution notes: the five MOD_* switches stay deleted -- firmwareExe defaults
every gate to 1, so nothing changes -- but MOD_CAM is now worth having by hand
on a cameraless printer, because the camera is a longrun that waits out
timeout-up. anvil-service.sh went with init.d; nothing sourced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
39b5cb0d0b |
ipk-install was a stand-in for an opkg the printer now has
The script says so itself: it existed so phase 1 could ship without waiting for a cross-built opkg, and it would hand its database over when the real client arrived. pkgs/3rdparty/opkg builds that client, and anvil-opkg is in bin/patch.sh's payload roots, so the printer has opkg and the stand-in has nothing left to do. Nothing called it. bin/build-payload.py assembles the payload with the printer's own opkg inside the replica, and the test that existed only to keep ipk-install off the build path was a guard against a file that is now gone. Its five round-trip tests went with it. The shellcheck lane loses one file rather than a rule: _files() refuses a glob that matches nothing, so the two entries had to go with the script. Cherry-picked from fc98b04 on worktree-drop-ipk-install, so the two branches carry the same patch. Resolved against this branch's payload build: the host opkg that commit still described is deleted here, and the lane globs are payload/prog/ rather than prog/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5f36cd14e3 |
Say what the code does, not what it used to
Comment cleanup over the files this branch touched, plus the dead code the
last two commits left:
* pkgs/lib.sh's pkg_buildopkg, 89 lines building the x86-64 opkg nothing
calls any more, and the qa test that checked its configure flags
* patch.sh's "5d. what is no longer here" section, and its mkdir of the
prefix directories -- build-payload.py replaces $PAYLOAD_DIR wholesale, so
they were created and then thrown away
* the two emptied creator5*-config recipe directories
The rest is prose. Comments that narrated a previous design -- what the host
opkg did, which package used to own the chamber config, that prog/ was once a
sibling of payload/ -- are gone rather than shortened; the ones explaining
constraints that still hold are trimmed to the constraint.
docs/building.md and docs/toolchange.md describe the packages as they are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ea1364ebca |
The payload is installed by the printer, not by a host opkg
bin/build-payload.py starts the replica, lets the machine's own opkg install the feed onto the machine's own filesystem under qemu-mipsel, and tars /usr/data/anvil back out. bin/patch.sh's section 0 is that call now. WHAT THE OLD WAY COULD NOT DO. A host opkg under --offline-root leaves every package Status: unpacked, because it cannot run maintainer scripts against a staging tree -- so patch.sh passed --force-postinstall to make the database claim work that had not happened. The payload shipped a postinst nothing had ever executed. It runs now, on the target userland, and its output is in the build log: anvil-core's postinst links the boot path and says so. THE REPLICA NEEDED A WAY TO HAND SOMETHING BACK. Every mount was :ro, which is right for a test harness and wrong for a build. run_case takes out_dir and mounts one writable /out -- but not at /printer/out, because assemble.sh binds /printer-src over /printer and a bind does not carry submounts, so the mount would be silently buried. entrypoint-out.sh makes the seam the stock entrypoint lacks: run it with a case that does nothing, then mount, then run the real case. It chowns /out back to the caller afterwards, because a privileged container writing into the repo leaves root-owned files the build user cannot delete. RUNBUILD IS A THIRD LANE for the same reason. The build has to reach the docker socket now AND still run as you, or work/ fills with root-owned files -- the two things the Makefile's existing split kept apart. --group-add with the socket's own gid, read rather than assumed, answers both. Left behind: pkgs/lib.sh's pkg_buildopkg is now defined and never called. It builds the host opkg this replaces. Deleting it is a separate change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
26e3388895 |
The printer picks its own chamber config
anvil-klipper-config carries printer.base.cfg and BOTH chamber configs, under config/chamber/<Machine>.cfg, and anvil-link-prog.sh symlinks the right one into /usr/data/config. The model comes from FlashForge's own app_startup.sh, which carries MACHINE= at its top -- so the machine answers the question about itself instead of a build being told the answer and having to be right. THAT REMOVES A PACKAGE PAIR THAT COULD NOT COEXIST. anvil-klipper-creator5-config and -creator5pro-config each owned config/printer.chamber.cfg, so they Conflicted, opkg refused the pair with exit 255, and bin/patch.sh had to pick one from TARGET_MACHINE. The payload was model-specific for the sake of one file; it is not any more. TARGET_MACHINE still names the output, because runFirmwareExe.sh gates on it. THE SOFTWARE COMPONENT'S klipper/config IS NOW BYTE-IDENTICAL TO STOCK -- diffed, not assumed. printer.base.cfg was the last thing patch.sh wrote there. The seven configs we do NOT modify stay stock and unpackaged, and that works because Klipper resolves [include] against the directory of the file doing the including, by the path as opened rather than the resolved target (configfile.py: dirname = os.path.dirname(source_filename)). So a symlinked printer.base.cfg still finds printer.filament.cfg and friends beside it in /usr/data/config. run-append.sh no longer copies the files the linker links -- copying would only put a real file in the way of the symlink about to replace it. moonraker.conf and moonraker-custom.conf keep their branches: those are user-editable, and a symlink would lose the .mod-new preservation a tuned trusted_clients block depends on. anvil-klipper now Depends on anvil-klipper-config, which is where that dependency always belonged -- it used to arrive only because the per-model packages pulled it in, and naming it in MOD_ROOTS would answer a missing dependency in the one place a printer's own `opkg install` never reads. prog/ is gone as a recipe subtree. Its own test said an empty one was the good outcome; it emptied, so the rule went with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
722804364a |
The boot path follows the package, not the last .tgz
firmwareExe and start.sh have to sit on /usr/prog because FlashForge's own scripts look for them by absolute path. They are package files now (anvil-core, under payload/prog/) and anvil-link-prog.sh replaces the copies on /usr/prog with symlinks into $MODDIR, so what the printer executes is what the package owns. WHAT THAT BUYS: `opkg upgrade anvil-core` on a running printer used to update $MODDIR and change nothing the boot path reads -- the new firmwareExe sat in the payload while the old copy kept being run, and only a full .tgz could land it. The postinst calls the same script, so an upgrade is now enough. THE LINKS ARE MADE AFTER EXTRACTION, and that is the whole ordering. Stock run.sh distributes the component at its lines 107-179 and run-append.sh does not extract anvil.tar.xz until line 180, so the component cannot carry the links itself: on a first install they would dangle, and on an upgrade they would resolve to the payload being replaced and silently install the previous release's files. The component still carries real copies. That is what keeps a printer bootable when the payload is not there, and patch.sh now takes them from the installed payload rather than from a directory beside the recipe -- the same move it already makes for the klippy fork. A postinst needed the recipe system to have maintainer scripts at all, so build-packages.sh copies pkgs/<name>/control/* into CONTROL/. Note what that exposes: patch.sh installs with a host opkg under --offline-root --force-postinstall, and that combination DOES run maintainer scripts, with IPKG_INSTROOT empty, so their absolute paths are taken as written. Inside the build image that is harmless -- there is no /usr/prog there -- and the script's guard is what makes it a quiet no-op rather than an error. pkg_payload_hash covers control/ too, or an edited postinst would read as "already current". It takes two find calls because `find a b` with b missing exits non-zero, which under the recipes' set -euo pipefail killed every build whose recipe has no control/ -- silently, the existing 2>/dev/null swallowing the one line that said so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
09e1e6521f |
patch.sh was asking the payload questions its recipes had already answered
Four sections went. Three of them were checks: _sqlite3 by name, which pkgs/3rdparty/python/build.sh already made; libsodium's symlink, which pkgs/3rdparty/libsodium/build.sh already made; and an ABI walk of the whole payload, which bin/build-packages.sh makes over every PKG_ROOT on the way into every .ipk -- from the packaging loop rather than a build.sh, so cached trees are re-read too. The fourth was not a check. config.env's BUSYBOX_BIN was copied into the payload here, which made it the one file no package owned: an allowance in the "every file is owned" test, an ABI gate of its own because the payload-wide one ran before the copy, and a file no `opkg remove` could take away. It is pkgs/3rdparty/busybox now, PKG_WHEN-gated on BUSYBOX_BIN, and it needed a third source verb -- pkg_prebuilt, beside pkg_unpack and pkg_intree -- because a recipe names its source exactly once and this one's source is a path someone typed. Once it was a package, nothing in the payload came from outside the feed, which is what made the payload-wide ABI gate redundant rather than merely duplicated. The "at least twelve extension modules in site-packages" count was the one check with nowhere to go and no duplicate. It is pkg_pynative in the seven python-* recipes that ship a .so: an aggregate could say that a native package had fallen back to a pure-python wheel but never which one, and each of those recipes can say it about itself. What that costs is written into the section it left: a gate inside build.sh does not run on a cache hit, and no python recipe sets PKG_STAMP_EXTRA. That is pkg_stamp's problem, not an argument for a second copy of every check here. Measured: `make packages` builds anvil-busybox_1.31.1-1_mipsel_xburst2.ipk, `make build` assembles a 32-package payload whose bin/busybox is listed in anvil-busybox.list, and qa-static is 265 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gKRUCqj18TVF1Vz1oMpmC |
||
|
Oleksandr Shyshatskyi
|
21cf19f2e2 |
BUILD_KLIPPER=stock kept a Klipper nothing else was written for
The flag chose between the Klipper4FlashForge fork and FlashForge's own v0.12 tree. Every other thing the mod ships had already stopped being neutral about that choice: the ff_*.py extras are the fork's package's own files, the ff-*.cfg declare sections only those extras implement, and c_helper.so is compiled from the tree it ships beside. A stock build was a printer with the mod's configuration and none of the code behind it -- and the previous commit turned that into a hard error, which is what showed the flag had nothing left to select. So it is gone rather than guarded. pkgs/klipper loses its PKG_WHEN, patch.sh section 1 loses the case and is straight-line, verify.sh section 7 stops being conditional -- a package with no klippy tree is now unrepresentable rather than merely gated -- and fetch-assets.sh pulls the fork tarball on every build, the way it already does s6 and CPython. Two fixture-lane repairs come with it. test/run-tests.py said BUILD_KLIPPER=stock to keep the lane off the network. That reason had already expired: since the payload became the feed installed, the lane runs build-packages first, and that needs the ~203MB Ingenic toolchain from vendor/ for CPython alone. The Klipper tarball is 27MB beside it, cached on versions.env like everything else. The lane now builds the fork, so verify.sh's ABI and symbol checks on c_helper.so run there for the first time. And the synthetic Moonraker tarball unpacked to moonraker-fixture/, while pkgs/moonraker/build.sh looks for moonraker-$MOONRAKER_VERSION/ -- the shape GitHub generates. That lane could therefore never build anvil-moonraker at all. It passed only while work/pkg/moonraker held a tree from a REAL build, because pkg_stamp keys on the version and not on the tarball; the first run against a cold cache died with "no moonraker/moonraker.py", a true statement about a tarball nobody could have made work. CI caches work/pkg on versions.env, so the first run after any pin bump would have hit it. Confirmed pre-existing by running the parent commit against the same cold cache: it fails identically. make qa-static 261 passed; make test 9 gates, 0 failed, with the fixture lane building Klipper and Moonraker from its own assets for the first time; CI's shellcheck gate clean. |
||
|
Oleksandr Shyshatskyi
|
53a2bc010b |
Five extras were on the firmware partition because of a flag
pkgs/klipper/prog/klippy/extras/ff_*.py sat on /usr/prog for one reason, and it was not a technical one: anvil-klipper answers to BUILD_KLIPPER and the extras answered to BUILD_TOOLCHANGE, and a package cannot hold files gated differently from itself. So they stayed outside every package, copied to the firmware partition by bin/patch.sh. They are anvil-klipper's now, at payload/klipper/klippy/extras/, and BUILD_TOOLCHANGE no longer gates them. An extra is inert until a config section instantiates it, and those sections are anvil-klipper-config's, where the flag still applies -- so the gate bought nothing and cost the mismatch. Inside the tree is the only place they can go: klippy resolves an extra as dirname(klippy.py)/extras/<name>.py (klippy.py:90-103), with no search path and no environment variable. bin/patch.sh still copies them into the software component -- klippy runs from /usr/prog until the s6 service lands -- but reads them out of the payload now, the way section 1 already reads the klippy tree, so the recipe directory is not a second source for a file a package holds. It names the five rather than globbing ff_*.py: the fork ships an ff_eddy.py of its own and the glob could not tell them apart. Verified the component still gets exactly the same five files. The flags are coupled by this, and patch.sh says so rather than shipping a broken printer: BUILD_TOOLCHANGE=1 with BUILD_KLIPPER=stock now fails with the fix in the message. It has to -- the extras are written against the fork's v0.13 klippy, and dropping them onto FlashForge's v0.12 is the half-overwritten mixture that shipped as v20260824. The fixture lane in test/run-tests.py used exactly that pair (BUILD_KLIPPER=stock is a no-network convenience there) and now sets BUILD_TOOLCHANGE=0. pkgs/klipper/pkg.conf gains PKG_STAMP_EXTRA=$(pkg_payload_hash): KLIPPER_VERSION says nothing about payload/, so without it an edit to ff_toolchange.py would read as "already current". make qa-static 261 passed; make test 9 gates, 0 failed. The .ipk carries all five under klipper/klippy/extras/. |
||
|
Oleksandr Shyshatskyi
|
dce1f62146 |
Naming the release and letting Depends work out the rest
bin/patch.sh enumerated the payload by hand: every recipe from pkg_recipes, minus the -dev packages, minus the chamber config for the other model. Forty lines to compute a closure opkg already computes, and computing it here meant the Depends fields were never read by anything. A printer running `opkg install anvil-moonraker` would have been the first to find out whether they were right. The feed is indexed, so point opkg at it -- `src anvil file:$PKG_FEED` and an `opkg update` -- and name the twelve packages the release IS. opkg pulls the other nineteen and marks them Auto-Installed. `file:` is answered before curl is reached at all (libopkg/opkg_download.c:134), which is why this works with the --disable-curl host opkg. The PKG_WHEN gates still reach here, by absence: a gated-off recipe leaves no .ipk in the feed, so BUILD_KLIPPER=stock has nothing to install and no BUILD_* flag is restated outside the pkg.conf that owns it. The cost is that a misspelled root drops out the same silent way, which test_the_payload_roots_name_real_packages now catches, and test_the_payload_roots_stay_a_short_list keeps a missing file from being answered by naming the package here instead of in the depending recipe's PKG_DEPENDS. Four of the twelve are Recommends, a field opkg does not have: pillow and preprocess-cancellation are Moonraker's thumbnail path, which pkgs/moonraker argues out of Depends; greenlet and cffi are klippy's, which pkgs/klipper cannot declare while klippy still runs under FlashForge's interpreter. anvil-core is the one version that cannot be left to opkg -- its PKG_VERSION is MOD_VER, today's date -- so its expected filename is still checked before installing. Payload verified against a build from the parent commit: 3298 paths, modes and symlinks identical, every file's sha256 identical except var/lib/opkg/status, which improves. Dependencies now say "Auto-Installed: yes" and the flag field is "ok" rather than "user" on packages nobody asked for by name. 31 packages, 154 ELF objects, as before. make qa-static 261 passed; make test 9 gates, 0 failed. |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8f54d87513 |
Comments that restated the code, and a few that had stopped being true
A pass over what this branch added, on the same rule as the branch merged before it: a comment earns its place by saying WHY, and the ones here had grown into essays. bin/patch.sh 781 -> 555 lines, its comments 523 -> 297. The install section keeps the three things a reader cannot get from the code -- the set comes from pkg_recipes so the BUILD_ flags are read once, -dev is filtered by package name because nine recipes ARE the dev package, --force-postinstall is what stops the database saying "unpacked" about files that are in place -- and loses the narration around them. Sections 3 and 4 were mostly an account of the sections that used to be there; they are four lines now. The CPython and libsodium headers keep the facts that took measuring (FF_PYTHON's callers, the missing _sqlite3 that pins Moonraker, libnacl's __file__ arithmetic and its dependence on "lib" appearing in the prefix) and drop the rest. Same for pkg_buildopkg, the payload-path block in bin/common.sh, and the four test docstrings. Also fixed, found while reading rather than looked for: bin/build-packages.sh still called itself "the proof-of-concept half" and said bin/patch.sh "still stages libsodium into the payload exactly as it did and `make build` is untouched". None of that survived the migration. The -maxdepth 1 comment had been wedged between `# shellcheck disable` and the command it applies to. Eleven "bin/patch.sh stages ..." references across pkgs/, test/ and qa/ described a script that stages nothing now. Payload bytes unchanged except S40s6, whose comment was one of the eleven. qa-static 259; make test 9 gates, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
51f524b5cf |
Merge worktree-comment-cleanup
The branch tightens comments across the tree -- removing the ones that
explained the code's past rather than the code. It branched before both the
pkg/ -> pkgs/ rename and the migration of bin/patch.sh onto the feed, so most
of its conflicts were with text describing code that has since moved or gone.
Resolved by taking its brevity and correcting what it could not have known:
bin/patch.sh ours, entirely. Its only non-comment change there deletes a
trailing comment from a line this branch rewrote, and every
comment it edits annotates one of the eight staging sections
that no longer exist.
bin/fetch-assets.sh ours. Theirs reinstates a KLIPPER_FORK condition; that
knob was deleted with the move to pkgs/ -- a recipe names its
source exactly once.
pkgs/lib.sh theirs, except the two hunks where our side carries code it
never saw (pkg_payload_hash and pkg_dir; the PKG_CC_SHARED
branch of pkg_build). Its prose, our functions.
bin/common.sh theirs throughout, with pkg/ -> pkgs/ and $MOD_PAYLOAD ->
$PAYLOAD_DIR, and minus its claim that the toolchain is used
"by bin/patch.sh for Klipper's c_helper.so" -- patch.sh
compiles nothing now.
pkg/anvil-core/build.sh deleted. The file is pkgs/anvil-core/build.sh and
was rewritten as a loop; its comments are superseded.
Two of its deletions were right about things this branch had wrong:
bzip2 named a pkg_make verb that was retired into pkg_build.
zlib claimed a ./configure line of its own as "the one sanctioned
exception". It has not had one since PKG_CONFIGURE_AUTO=0 and CHOST; the
rule in qa/static/test_ipk.py is absolute, and its docstring says so. I had
kept that paragraph in an earlier pass of this merge and was wrong to.
Three stale hints in the Makefile went with them: work/pkg is filled by
`make packages`, not by bin/patch.sh, in the two gate targets and one error
message.
Verified: the payload rebuilds to the same 31 packages and 154 ELF objects,
and the six files whose bytes changed are five payload scripts this branch
comment-edited (zero non-comment lines between them) plus the opkg status
file, which records their sizes. qa-static 259; make test 9 gates, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8b50ccd930 |
Every file in the payload should name the package that put it there
Four tests for the properties the migration rests on, two of them asked of the
assembled payload rather than of the scripts that assemble it.
test_every_payload_file_is_owned_by_a_package walks work/modpayload-root and
checks each path against opkg's own record of what it installed. The pass is
not the interesting part -- the allowlist is. It is the complete set of files
that are in the payload and in no package:
.install-manifest generated after everything else, because it
describes the payload
anvil.conf templated from config.env, preserved across
updates -- a package member is overwritten on
every upgrade by definition
config/moonraker-custom.conf created once, never overwritten
bin/busybox optional, from config.env, gated on its own
var, var/lib, var/run opkg's own scaffolding
Six entries and three of them are opkg's. That list is the remaining to-do
for phase 2 of docs/notes/85-packaging.md; it should shrink and must not grow
by accident. Proven by dropping a file into $MODDIR/bin and watching it fail
by name.
test_the_payload_database_has_no_clock_in_it asks the status file whether
every Installed-Time is the same value. opkg takes it from time() and ignores
SOURCE_DATE_EPOCH, so before bin/patch.sh normalised it two builds of one
commit differed by one line per package. It asks the database rather than the
tarball because anvil.tar.xz is not reproducible yet for unrelated reasons --
bin/pack.sh's tar neither sorts nor clamps -- and this is the part that
regressed and is fixed.
test_the_host_opkg_is_built_for_the_prefix guards the two flags that fail
silently: --prefix=$MODDIR (opkg bakes its state directory in at compile time,
so any other prefix writes a database nothing reads) and --disable-shared (the
prefix is in libopkg too). pkg_buildopkg checks its own output for both; this
checks it still asks, since that check is only reachable if the build gets
that far.
test_ipk_install_is_not_on_the_build_path keeps the printer's hand-repair
installer off it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e3d1bdcdf7 |
The payload is the feed installed, not the build trees copied
bin/patch.sh ran every recipe and then cp -a'd its build tree into
work/modpayload. Eight sections of it, each a second description of what the
.ipk that recipe produces already contains -- the same duplication the recipes
themselves removed one level down, one level further up.
It now installs the feed. pkg_buildopkg's x86-64 opkg gets one invocation with
the whole selected set and an --offline-root, and what lands there is the
payload. "What does this release install?" is answerable with
`opkg list-installed` against the payload itself, and the tarball and the
package cannot describe different bytes because the tarball is made of the
package.
THE SET COMES FROM pkg_recipes, which is already PKG_WHEN-gated -- so
BUILD_HELIX, BUILD_MAINSAIL, BUILD_MOONRAKER, BUILD_TOOLCHANGE and
BUILD_KLIPPER are read in the pkg.conf that owns each and nowhere else,
instead of being restated by an `if` here that could drift from it.
-dev IS FILTERED BY PACKAGE NAME, and the first attempt got this wrong in a
way worth recording. execline, s6 and CPython ship both halves and pkg_ipk
names the runtime one; zlib, openssl, sqlite, expat, libffi, xz, bzip2,
libarchive and skalibs ship ONLY a dev package and say so by setting
PKG_NAME=anvil-<x>-dev outright. Asking "does this recipe have a runtime half"
installed all nine: 40 packages and 2165 ELF objects where there should be 31
and 154.
--force-postinstall, because opkg skips configuration under an offline root
and would leave every package Status: unpacked. Nothing runs opkg on the
printer after run-append.sh extracts the tarball -- the extraction IS the
install -- so a database saying otherwise would make the first real `opkg
upgrade` argue with the filesystem. It runs nothing today: no recipe defines a
maintainer script, and opkg 0.7.0 would not run one here anyway (it sets no
IPKG_INSTROOT; see libopkg/pkg.c:1339).
Installed-Time is normalised afterwards. opkg takes it from the clock and
honours SOURCE_DATE_EPOCH only for a man-page date, so two builds of one
commit differed by one line per package -- measured, then fixed, then measured
again.
Gone with the staging: the hand-written PKG_DEV_FILES prune (the dev half is
not installed, so bin/verify.sh's "no headers" check now passes because they
were never there), three partial ABI gates in favour of one over the finished
payload, and the last pkg_out call -- section 1 reads klippy out of the
installed anvil-klipper.
The payload gains anvil-klipper's tree at $MODDIR/klipper and bin/opkg. The
first is inert until phase 7 of docs/notes/80-s6-migration.md and is 2.4MB of
duplicate until then; the second is route 1 of the bootstrap in
docs/notes/85-packaging.md -- the first payload that can install its own
successor.
VERIFIED by building the payload from the parent commit and from this one, on
the synthetic stock fixture with the real vendored assets:
files removed 0
content changes among the 2778 shared files 1 (.install-manifest,
which lists the payload)
mode changes 0
symlink changes 0
added klipper/**, var/lib/opkg/** (63), bin/opkg, and the
www/mainsail/.version the previous commit stopped deleting
two builds in a row byte-identical
bin/verify.sh passes every check including the ship boundary; qa-static 255;
make test 9 gates, 0 failed; test-py 181.
test/run-tests.py builds the feed as the first step of the fixture lane, into
a PKG_FEED of its own -- sharing work/packages let the MOD_VER=ci build rename
every dated package and the prune delete the rest on the way past, measured as
44 becoming 43.
qa/static/test_ipk.py's share-one-build test is inverted rather than deleted:
the property was "one build, two vehicles" and is now "one vehicle", so what
it asserts is that nobody reintroduces the second path -- no recipe run from
patch.sh, no pkg_out, no work/pkg, and opkg rather than pkgs/ipk-install,
which stays the printer's hand-repair tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
dff6fe3198 |
anvil-core was holding four other components' configuration
It held all of it for one reason: it was the first recipe written, so
everything of ours started there. With the files visible per recipe the
misfilings read straight off the tree.
moonraker.conf -> pkgs/moonraker, with the server it configures
ff-*.cfg, printer.base.cfg -> pkgs/klipper-config (new)
printer.chamber.cfg -> pkgs/klipper-creator5-config and
pkgs/klipper-creator5pro-config (new)
PRINTER.BASE.CFG WAS IN pkgs/klipper AND THAT WAS WRONG TWICE. It is Klipper
CONFIG rather than any part of the Klipper program -- it is the file carrying
[include ff-toolchange.cfg] and [include printer.chamber.cfg], so it is the
entry point to everything in the config package -- and pkgs/klipper is gated
on BUILD_KLIPPER while this file is installed under BUILD_TOOLCHANGE. A
BUILD_KLIPPER=stock build has no pkgs/klipper recipe at all and was still
reaching into its prog/ directory for a file.
THE ff-*.cfg ARE NOT IN pkgs/klipper EITHER, for the same gate: ask for
BUILD_KLIPPER=stock and that recipe does not exist, and the toolchanger would
have become a silent casualty of a flag about which klippy is shipped.
FlashForge's own klippy loads ff_*.py and reads these .cfg perfectly well.
They travel with the feature they configure and the feature's flag is
BUILD_TOOLCHANGE.
THE MODEL IS A PACKAGE NAME NOW, not a build flag. printer.chamber.cfg.creator5
and .creator5pro used to be picked by patch.sh from TARGET_MACHINE and copied
under their real name, which made the model a property of the BUILD: two
builds of one commit produced different bytes under one version number, and
nothing in the feed or on the printer recorded which machine a package was
for. Both are built on every build; patch.sh stages the one this tarball is
for. This is the feed's first use of Provides/Conflicts, both on a virtual
anvil-klipper-chamber-config -- Provides so anvil-klipper-config can depend on
"a chamber config" and either satisfies it, Conflicts so a printer cannot
install both and have two packages own one path with opkg letting whichever
unpacked last win. Neither field alone expresses "exactly one", and
printer.base.cfg's [include printer.chamber.cfg] is unconditional, so "none"
is a printer whose klippy does not start.
printer.chamber.cfg also CHANGED WHERE IT LANDS: $MODDIR/config, like the
ff-*.cfg, instead of riding the software component to /usr/prog and reaching
/usr/data/config only because the stock run.sh copies that whole directory.
run-append.sh overwrites it on the same mod-owned terms, and by this repo's
own rule -- files the update overwrites must say so -- it now carries the
DO NOT EDIT banner the ff-*.cfg have always had. test_config_ownership.py is
what demanded that, which is what it is for.
TWO FLAGS BECAME HONEST, and both are behaviour changes:
* BUILD_TOOLCHANGE=0 shipped the toolchanger's .cfg anyway, because they
rode in anvil-core while everything else the flag controls was gated.
It no longer does.
* BUILD_MOONRAKER=0 overwrote the printer's moonraker.conf with one written
for a server it was not going to install -- [webcam anvil] and all. It no
longer does, which is what docs/building.md already said the flag meant.
WHAT DELIBERATELY DID NOT MOVE. init.d/ and etc/s6/ look like they belong to
the services they start, and moving them would brick a printer: those services
are gated at RUNTIME by flags in anvil.conf, while the packages that would own
the scripts are gated at BUILD time. A BUILD_KLIPPER=stock tarball has no
anvil-klipper package, so it would carry no S70klipper and nothing would start
Klipper at all. The script that reads a flag must not sit behind a second,
different flag. nginx.conf stays for a plainer reason: nginx is the rootfs's
binary and there is no package here to give it to.
Measured:
* 44 packages (41 + three config packages), feed builds clean.
* anvil-moonraker carries config/moonraker.conf; anvil-klipper-config the
seven ff-*.cfg; each model package one printer.chamber.cfg. Checked by
listing the built .ipk, which is how the first attempt was caught staging
moonraker.conf and never shipping it -- pkg_ship takes an explicit list.
* make test: 8 passed, 0 failed, including unpack/patch/pack/verify.
* bin/verify.sh gained two gates over the payload -- a chamber config and
the ff-*.cfg -- and BOTH WERE PROVEN BY BREAKING THE BUILD: deleting the
staging line fails patch.sh's own check, and bypassing that too fails
verify with "mod payload has no config/printer.chamber.cfg". Which is
also the proof that a normal build carries them.
* make qa-static: 256 passed. make test-py: 181 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f67e7a269f |
ls pkg/ was thirty-eight entries and four of them mattered
pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and
carry no files of this repo move down a level into pkgs/3rdparty/. What is
left at the top is the four components a person actually edits: anvil-core,
klipper, moonraker, helixscreen.
THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory
rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day
somebody patches zlib nobody would agree which side it belongs on, and the
answer would be a rename touching CI and docs decided by taste. "Does it carry
files of ours" is a fact about the tree: it is exactly whether the recipe has
a payload/, prog/ or seed/, checkable in one assertion, and
test_recipe_layout.py now demands the move in both directions -- a 3rdparty
recipe that grows one of the three must come up, and a top-level recipe with
none of them must go down, or the top level slowly refills and the split stops
meaning anything.
ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id
to its directory and nothing else spells either path -- not a recipe, not
bin/build-packages.sh, not a test. Names stay unique across both levels
because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare
name; a duplicate would resolve to whichever level is searched first and
build the wrong thing silently, so that is a test too.
Recipes now source `. ./bin/common.sh` instead of counting `..` from $0.
Depth-counting would have needed two spellings for two levels, and it was
never buying anything: `. pkg/lib.sh` on the next line already required the
working directory to be the repo root.
A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files()
expanded a list of globs and returned whatever matched, so "pkgs/*/build.sh"
quietly stopped covering thirty-four recipes the moment they moved -- the lane
stayed green, and the only visible sign was a test count dropping by 34 in a
run nobody was counting. _files() now refuses a pattern that matches nothing.
That is the same failure the ash-compatibility check had a guard for and the
syntax check did not, which is why the guard reads the way it does.
AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of
.py helpers wholesale, so any test that imported one left a __pycache__ beside
it -- gitignored, invisible in a diff, copied by cp -a like anything else. The
package's contents depended on whether pytest had been run on the machine that
built it: two builds of one commit, different sha256, cause nowhere in the
tree. What it shipped was worthless as well as unstable, compiled by the build
image's python 3.11 for a printer running the 3.13 this repo cross-builds. The
sweep is in pkg_ship now, once, and the two hand-written copies in
pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe
that stages a tree of .py files, and the two recipes that had hit it had each
solved it locally while the third, which had not, was the one shipping.
Measured:
* the rename changes no package bytes. Same cached build trees, feed built
with the diff stashed and unstashed: all 41 .ipk match.
* with the bytecode sweep, anvil-core returns to exactly the bytes the
previous commit produced.
* no package in the feed contains a .pyc or a __pycache__.
* make qa-static: 247 passed (236 before this work + 11 layout tests).
* make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the
synthetic stock package. make test-py: 181 passed.
Not explained: anvil-python-dev produced two different sha256 from an
UNCHANGED work/pkg/python at two points today, under code that was identical
both times -- so it is not reproducible from a cached tree, and it is not this
commit's doing (the stashed and unstashed builds above agree on it). Worth a
look; the dev split in bin/build-packages.sh is where to start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
54a1e72cc4 |
A file that ships belongs to a package, and the path should say which
payload/ was organised by DESTINATION -- init.d/, bin/, etc/s6/,
klipper/config/ -- plus an assets/ directory of three .conf files. That was
right when there was one owner. With 38 recipes it held three different kinds
of file behind identical-looking paths:
* files a recipe stages into its .ipk,
* files that go to /usr/prog and CANNOT be in a package of ours, because
every path in one lands under $MODDIR,
* files that never ship as files at all -- run-pre.sh and run-append.sh are
spliced into FlashForge's own run.sh at build time.
The only thing recording which package owned any of it was a thirty-line
comment at the top of pkg/anvil-core/build.sh. A comment cannot be checked.
Files now live with the recipe that owns them, and the subtree says how they
ship: payload/ is staged into the .ipk laid out exactly as it lands under
$MODDIR, prog/ is placed on /usr/prog by patch.sh, seed/ is templated or
seeded user state. installer/ sits outside pkg/ and holds the two files that
are never files on a printer. qa/static/test_recipe_layout.py fails if a
fourth kind of directory appears, which is the failure this is actually for:
a misfiled file is caught by a build, an unruled directory is caught by
nothing.
anvil-core's recipe is a loop over a directory now. Six pkg_stage lines with
hand-written destinations plus a glob became one loop, because payload/ IS the
overlay -- a file added to that tree ships without an edit to build.sh.
A REAL BUG FELL OUT. anvil-core's PKG_STAMP_EXTRA was an eight-path list
naming payload/ and assets/ subtrees by hand, and one of those paths was
payload/helixscreen/ -- HelixScreen's printer-database entry, hashed into the
cache key of the package that does NOT contain it. Editing that json rebuilt
anvil-core and left pkg/helixscreen sitting in the cache. pkg_payload_hash is
now the one way a recipe keys its own files and it can only hash its own.
WHAT ELSE MOVED, and why each is not where it looks like it should be:
* ff-*.cfg stay with anvil-core rather than pkg/klipper. They are the
package that has always shipped them; ownership moves in the next commit,
with the two model-config packages. This one is a move, not a repackaging.
* printer.base.cfg is pkg/klipper's prog/, not payload/: it is FlashForge's
file with one block replaced and it lands beside their Klipper tree.
* moonraker.conf is anvil-core's for now; moonraker-custom.conf is
pkg/moonraker's seed/, because it is the user's and a package member is
overwritten on every upgrade by definition.
The cfgdir fixture is a merged copy of both config directories rather than a
path into the checkout. The split is a fact about which package ships what --
on the printer there is no split, the stock run.sh force-copies
printer.base.cfg into /usr/data/config/ beside the ff-*.cfg, which is why its
includes are bare and relative. A test about include resolution has to model
where files END UP or it is testing the repository's filing system.
qa/lib/replica.py's one payload mount is three, and entrypoint.sh reassembles
/tmp/payload out of them byte-for-byte as it was. Not re-pointed: every
case-*.sh copies out of /tmp/payload with 2>/dev/null, so a path that stops
resolving does not fail -- it copies nothing and the case runs green against
an empty $MODDIR.
Measured:
* all 41 .ipk byte-identical. anvil-core and helixscreen were rebuilt from
HEAD and from this tree and compared directly; the other 39 match the
feed built before the move.
* that comparison caught a mistake in this commit: the HelixScreen json was
mirrored one directory too high (helixscreen/printer_database.d/ instead
of helixscreen/config/printer_database.d/). A move that changes no bytes
is only a claim until it is diffed.
* anvil-core: two cold builds byte-identical.
* make test: 8 passed, 0 failed -- including unpack/patch/pack/verify on the
synthetic stock package, which is the gate that covers patch.sh end to end.
* make qa-static: 244 passed (236 + 8 new). make test-py: 181 passed.
Known gap, pre-existing and left alone: pkg/*/payload/bin/*.sh is in no ash
glob, so wifi-action.sh's dialect is unchecked. The old glob was payload/*.sh,
one level above it, so this is not a regression -- it belongs with the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
09a9f868e3 |
klippy compiles c_helper.so at first run, and this printer has no compiler
So the "build system" for that .so genuinely is one gcc line -- the argument
list in klippy/chelper/__init__.py -- and pkg_build could not express it,
because it always ran make. That is why Klipper was the last cross-build
still living in bin/patch.sh, which unpacked the Ingenic toolchain into
work/.mips-toolchain, wrote its own gcc invocation, cached on an mtime
comparison and gated the result itself: every one of the things pkg/lib.sh
exists to do once.
PKG_CC_SHARED is that line, appended to `$CC -shared -fPIC`. A knob and not
a verb, for the reason pkg_build's own header gives about the three
mechanisms with one user each: the recipe says WHAT to link, pkg_build owns
HOW, and -EL -mnan=2008 stay spelled once in pkg_toolchain where the wrapper
is gated before anything is built on it.
pkg/klipper -> anvil-klipper_0.13.0+gitb722a4c8-1_mipsel_xburst2.ipk
Both gates moved into the recipe, next to the ship list they check -- the
same move the s6 presence checks made. They now run on `make packages` too,
so a c_helper.so that is not nan2008, or one missing a symbol the klippy
beside it cdefs, fails the build that produced the .ipk rather than only a
full firmware build. The symbol gate is the one that matters: cffi resolves
lazily, so a stale .so imports cleanly and dies at connect on the printer.
KLIPPER_FORK is gone. It pointed the build at a local checkout instead of
the pin, which is a second source for a recipe allowed exactly one -- and it
is the same variable that let v20260824-nova-kakhovka ship a klippy tree
nobody had built, set to "" in CI. To build your own fork, put its tarball
in vendor/ and re-pin KLIPPER_VERSION and KLIPPER_SHA256; the sha is then
the guarantee every other source in this repo carries. bin/fetch-assets.sh
loses two of its conditions with it, and test/test-chelper.py loses the
config.env fallback that printed "SKIP: no KLIPPER_FORK configured" -- a
green line for a check that had not run.
bin/patch.sh section 1 is now staging and nothing else: a cp -a into the
SOFTWARE component, where the stock klipperDaemon looks, plus the
chelper.tar its run.sh extracts. That tar stays out of the .ipk, where it
would be a second copy of a file the package already contains. The package
installs the tree under $MODDIR/klipper, inert until phase 7 of
docs/notes/80-s6-migration.md makes the installed copy the live one -- at
which point these lines are deleted rather than moved.
Measured:
* two cold builds of pkg/klipper byte-identical, work/pkg deleted between
* c_helper.so is 0x70001407 -- nan2008, o32, mips32r2
* all 66 functions klippy declares are exported, checked after pkg_ship
strips rather than before
* the whole feed still builds: 38 recipes, 41 packages
* 236 static tests, including the staged-recipe list that now names klipper
* BUILD_KLIPPER=stock removes the recipe from pkg_recipes entirely
Not verified here: bin/patch.sh end to end. It needs the stock FlashForge
package, which the packaging lane deliberately does without; bin/verify.sh's
klippy-tree and chelper.tar assertions are what cover the staging rewrite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
de3e369f07 |
Comments that explained the code's past, not the code
Across 46 files the comments had accumulated a second job: narrating what
each mechanism replaced. Some of that narration outlived what it described,
which is worse than verbose -- it is wrong.
Removed, because the thing referenced is gone:
- pkg/lib.sh's example recipe called pkg_autotools, and pkg/bzip2 pointed
at pkg_make for the shape it uses. qa/static/test_ipk.py asserts both
names stay deleted.
- bin/common.sh explained S6_STAMP, PY_BUILD and PY_STAMP at length. None
of the three is defined anywhere; mips_abi_gate still said it walked
$PY_BUILD, and that section 5b cross-builds with a musl toolchain that
left with the second libc.
- pkg/lib.sh and bin/common.sh had section 5c carrying its own toolchain
setup and CPython cross-build. It runs recipes and stages the output.
- pkg/zlib's header claimed a ./configure carve-out enforced by name in
test_ipk.py; the body sets PKG_CONFIGURE_AUTO=0 and the test permits no
exceptions.
- case-supervisor.sh was fed out of work/.s6, which is now three recipe
outputs merged by the Makefile.
Shortened everywhere else to the logic that is actually hard: the -mnan=2008
driver baking, exporting CC because OpenSSL reads it where autoconf reads
--host, the ar/ranlib determinism split and why the cross ranlib cannot do
it, the sysconfig cross trick, busybox start-stop-daemon returning before the
process is dead, and the manifest property an update has to keep. The
measurements stay; the archaeology goes.
No code changed. qa/static: 111 passed, unchanged from before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8cc92e8266 |
A wheel could only be built while three scratch trees happened to coexist
That is why bin/patch.sh section 5c was 825 lines and why a Pillow bump
rebuilt CPython. Cross-building an extension module needs the untrimmed
staging tree (INCLUDEPY points into it), the private sysroot of static C
libraries (pillow links zlib, cffi links libffi) and an x86-64 build-python
of the same version -- and all three existed only mid-section, so the wheels
had to be built there too, on a cache with two stamps that invalidated each
other.
Each of those three is now a durable thing: anvil-python-dev carries the
headers, pkg_deps fills each recipe's own sysroot from the feed, and
pkg_buildpython caches one build-python for all nineteen recipes. So the
coupling goes away with them -- `make packages PKG=python-pillow` builds
Pillow.
pkg/python -> anvil-python + anvil-python-dev
pkg/python-<name> x18 -> anvil-python-distro, anvil-python-pillow, ...
Three verbs in pkg/lib.sh, shared by all nineteen rather than one per awkward
project: pkg_buildpython provides the build-python, pkg_pytarget points its
sysconfig at the target headers in the sysroot, pkg_pywheel builds one wheel
and unpacks it into the staging site-packages. greenlet's gcc 7.2 designated
initializers, lmdb's LMDB_FORCE_CPYTHON and pillow's build_ext options are
recipe lines and a PKG_PY_SETUP_ARGS knob, not three more verbs.
Eighteen versions that were invisible are now in the feed index, anvil-moonraker
declares the thirteen packages it imports, and a printer that does not want
gcode thumbnails can leave Pillow uninstalled.
PY_STAMP, PYPKG_STAMP and PY_BUILD are gone: pkg_stamp derives the key from
PKG_BUILD_DEPENDS, and bin/fetch-assets.sh asks pkg_needs instead of
re-deriving a stamp bin/patch.sh also derived. Two of its four toolchain
conditions were that second spelling.
Measured:
* 40 packages; 959 site-packages files, none owned by two packages
* 12 mipsel extension modules -- the count the old gate demanded
* every Depends resolves inside the feed (24 declare one, 0 unsatisfiable)
* two cold builds byte-identical, with work/pkg, work/.py-host and the
unpacked toolchain deleted in between
* 233 static tests, 4 new: the PYPKG_LIST/recipe agreement both ways, no
version pinned twice, patch.sh builds every package, and no Depends
naming a package no recipe produces
Not verified here: the replica suite. It needs the stock FlashForge package,
which the packaging lane deliberately does without.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
94a03800f7 |
Perl's || is not //, so SOURCE_DATE_EPOCH=0 meant "use the clock"
Six more recipes -- openssl, sqlite, libffi, xz, bzip2, expat -- and one
build environment instead of two. Twenty packages.
ONE VERB, NOT FOUR. There was pkg_autotools with eleven users, a
hand-written ./configure in pkg/zlib permitted by name in the test that
forbids it everywhere else, a pkg_make for bzip2, and a pkg_build for
OpenSSL. Three mechanisms with one user each is three exceptions wearing
function names. Projects do not differ in four ways, they differ in four
SETTINGS of the same three steps, so pkg_build takes knobs -- which
configure, whether it takes --host, what to make, what to install -- and
every recipe goes through it, zlib included. The test is now absolute: no
recipe runs a configure script, no exceptions, and it knows about
./Configure as well as ./configure.
THE TOOLCHAIN REACHED PROJECTS TWO WAYS AND THAT WAS THE REAL BUG. Only
PATH was set, so autoconf found the cross compiler through --host and
anything that reads $CC did not. OpenSSL takes a target name rather than
--host: it used the build machine's gcc and produced 936 x86-64 objects
inside the container. mips_abi_gate caught it at the package boundary.
pkg_toolchain now exports CC/CXX/AR/RANLIB/STRIP/NM/OBJCOPY/OBJDUMP/LD as
well, so every project gets the toolchain the same way.
The same failure in miniature: pkg_build dropped its trailing arguments
when a project had no configure step, so bzip2's CC= went nowhere and
libbz2.a came out x86. Arguments now go to whichever step consumes them.
THE BUILD ENVIRONMENT IS THE CONTAINER, and it was not. Eleven recipes had
been built on the host and the twelfth needed perl's IPC::Cmd, which Debian
bookworm has and this host does not -- so the container was about to become
"the environment for the awkward ones". Two builds, two results: the
Moonraker and HelixScreen packages differed between host and container
because GNU tar restores a directory's exact mode as root and applies the
umask otherwise. Packages now normalise modes at the boundary (0755 dirs,
0755 executables, 0644 the rest), the way ownership and timestamps already
were, so what a package contains no longer depends on who built it.
And the build lane runs as the invoking user, not root. Root-owned files in
work/ are what make `rm -rf work/pkg` fail and running the build outside the
container tempting -- which is how a project ends up with two environments
in the first place.
OpenSSL was then the one package that still would not reproduce: two builds
differed in libcrypto-lib-cversion.o, which carries a "built on:" string.
It does honour SOURCE_DATE_EPOCH -- util/mkbuildinf.pl reads it -- but as
`$ENV{'SOURCE_DATE_EPOCH'} || time()`, and Perl's || treats 0 as false. The
one value we passed was the single value that silently falls back to the
clock. The default is 1 now.
Twenty packages, byte-identical across two cold container builds. 172
static tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
46d2973165 |
A printer with no compiler was carrying 248K of headers
PKG_DEV_FILES splits one build into two archives: <name> with what a printer runs, <name>-dev with the headers and the static library only a build machine opens. Installing anvil-execline + anvil-s6 + anvil-s6-rc now leaves bin/ and libexec/ and nothing else -- include/ and lib/ are gone. The three libraries that are ONLY ever linked against -- zlib, libarchive, skalibs -- have no runtime half at all, so they are renamed rather than split: anvil-zlib-dev, anvil-libarchive-dev, anvil-skalibs-dev. The name now says what the section already did. THE RULE CHANGED, deliberately. "One recipe builds one package" becomes "one recipe builds one SOURCE, and may emit several packages from that one build". The rule was written to stop one script compiling several different upstream projects -- zlib inside the opkg recipe -- and a dev split is not that: one source, one configure, one version, sorted afterwards. The invariant that actually catches the thing it was for is countable and unchanged: exactly one source verb per recipe. The split MOVES files, never copies. A path in both archives is a path two packages own, opkg lets whichever installed last win, and removing either deletes a file the other still lists -- damage that surfaces long after the install that caused it. Two tests hold the line, both negative-tested by making a runtime package carry a dev file. Renaming also exposed a feed bug worth its own fix: the old anvil-zlib, anvil-libarchive and anvil-skalibs archives stayed in work/packages and opkg-make-index went on advertising three packages no recipe produced any more. A full build now prunes what no recipe claims. A single-recipe build does not, because it was not asked about the rest. Fourteen packages, byte-identical across two cold builds. 154 static tests. docs/notes/85-packaging.md phase 2 gains the target this is all for: bin/pack.sh should be handed a tree produced by installing the feed, not one patch.sh assembled, at which point every staging copy in patch.sh -- each of them a second description of what a package already contains -- goes away. CPython is the blocker and is next. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6e97f946b0 |
execline was 2.1MB unstripped against musl; it is 1008KB and mandatory
Eight recipes: skalibs, execline, s6, s6-rc, mainsail, moonraker,
helixscreen, anvil-core. bin/patch.sh loses 183 lines and stops being a
build system for any of them.
The supervision stack moves to the Ingenic glibc toolchain and links libc
dynamically, so the musl toolchain -- the only second libc in this tree --
is deleted along with MUSL_TOOLCHAIN_*, S6_STAMP and S6_BUILD. The
measurement that chose musl compared two STATIC builds; dynamic glibc was
never in it. Measured now: s6's shipped tree is 696K against ~930K for the
static musl one, and the whole stack including execline and s6-rc is 2.0M.
Three things that were not guesses:
* s6 and s6-rc do not link against glibc 2.29 without -lpthread. skalibs'
pthread_mutex_tailock reaches pthread_mutex_timedlock, which glibc kept
in libpthread until 2.34. Upstream never sees it and musl never had the
split. PKG_MAKE_ARGS is the lever, because s6's PTHREAD_LIB is wired
only to --enable-nsss and its Makefile appends LDLIBS last.
* s6 ships 21 binaries, not 13. s6-rc's generated scripts exec s6-sudo,
s6-sudoc, the s6-ipcserver chain, s6-sudod and s6-fdholder-daemon --
found by running a real up/down cycle with a restricted PATH, not by
reading the documentation.
* --enable-static-libc has never been a skalibs option. Its configure
warns on unknown flags and patch.sh sent that to /dev/null, so it was a
no-op for as long as section 5b existed.
execline ships whole. A subset picked from today's services fails the day
somebody adds a pipeline, at boot, naming a program nobody chose to leave
out. tools/supervisor/README.md and 80-s6-migration.md said it was not
needed; both now say why that stopped being true.
PY_STAMP was spelled in three places and happened to agree -- the same
shape as the s6 stamp bug, caught before it inverted. It is in common.sh
now, and the test that guarded S6_STAMP by name is rewritten around the
rule so it covers both and dies with neither.
HelixScreen's binaries reach mips_abi_gate for the first time: patch.sh
never gated them. They pass, 3 objects, nan2008/o32/mips32r2.
anvil-core carries a content hash in PKG_STAMP_EXTRA because its version is
a release date: pinned, an edit to payload/ would read as current; unpinned,
it rebuilds daily. The version says which release, the stamp says what is
in it. anvil.conf stays out -- it is user state that run-append.sh
preserves, and a package member is overwritten on every upgrade.
Twelve packages, byte-identical across two cold builds. 152 static tests.
The replica suite has NOT been run: docker is unavailable here, and the s6
change ships without its behavioural gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c8e33a5604 |
zlib was compiled twice because it belonged to whoever needed it first
pkg/opkg/build.sh unpacked zlib, built it into a private sysroot, unpacked
libarchive, built that against it, and then built the one binary it shipped.
Section 5c did the same with its own zlib, from the same pinned tarball with
the same flags. Neither copy could see the other, because a library built
inside somebody else's recipe has no version, no package, and no way to be
reused.
One build.sh now produces exactly one package. A recipe names what it builds
against in PKG_BUILD_DEPENDS and pkg_deps unpacks those out of the feed with
opkg-unbuild -- upstream's inverse of opkg-build, from the checkout that
already provides both. That dissolves the bootstrap: nothing needs a working
opkg in order to build packages, so opkg is an ordinary recipe that waits on
nothing except its own dependencies, and no qemu enters the build path.
Building against the package rather than the build tree is the part worth
paying for. A package that forgets a header fails the next recipe's configure,
on the build that produced it.
One libc. skalibs, s6 and opkg were musl because a STATIC glibc s6 tree
measured 73MB -- a static-to-static comparison that never considered dynamic.
Two libcs also meant two of every library both worlds wanted, which a feed
cannot express without lying about one of them. opkg moves to the Ingenic
toolchain and links libc.so.6 dynamically with zlib and libarchive inside it;
it got smaller (212K against 332K). Section 5b stays musl until it becomes a
recipe too, and is now the only place left where one script builds two
libraries.
Found on the way, each of which had been true for a while:
* ar archives are not reproducible. Every member header carries an mtime and
the builder's uid/gid, and the symbol index carries another timestamp --
none of it reachable by SOURCE_DATE_EPOCH or opkg-build's -o 0 -g 0,
because it is inside a file the tarball merely contains. objcopy -D fixes
the members; the cross binutils is 2.27 and its ranlib -D writes a fresh
current timestamp into the index instead of zeroing it, so the index is
rewritten with the build image's ranlib.
* The ABI gate accepted exactly 0x70001405 and 0x70001407, both measured from
linked binaries where crt objects set NOREORDER. Object files need not:
libarchive's xxhash.o is 0x70001406, identical ABI, and the gate called it
wrong. It also read one Flags: line, while readelf -h on an archive prints
one per member -- 122 for libarchive.a. It now masks the three non-ABI bits
and checks every header.
* bin/fetch-assets.sh compared work/.s6/.version against two fields where
patch.sh wrote three, so the test could never be false and 71MB was
re-hashed every run. The comment above it described a fast path that had
never once been taken.
Gate: 127 static tests (29 packaging), and `make packages` from a cold cache
producing four byte-reproducible .ipk files plus the index. The end-to-end
install under qemu has NOT been re-run since opkg went dynamic -- a dynamic
binary needs qemu -L <sysroot>, and the command is written down in
docs/notes/85-packaging.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
1191f87411 |
The graph is asserted by compiling it, not by reading it.
Phase 8 made the boot a compiled s6-rc database. This is the gate for it, plus
the honest accounting of what the change did to the suites that tested the
boot it replaced.
qa/static/test_s6rc_source.py -- 14 tests, no docker, ~1s. It COMPILES
payload/etc/s6-rc/source/ with the native s6-rc-compile that bin/patch.sh
section 5b-2 builds, then reads every answer back out with s6-rc-db. Never out
of the source tree: the compiler is free to reject, rewrite or silently drop
what it was given, and `down` is the one that proved it -- accepted and
discarded, producing a byte-identical servicedir. A test that reads the source
would have called that a pass.
It asserts the bundle is the boot set; klipper waits for mcu-bringup,
ff-startup for moonraker and klipper, ui for ff-startup; mcu-bringup is
reachable but NOT in the bundle, which is the only way s6-rc can express "start
it, but not because the boot asked"; nginx does not gate moonraker, because as
an edge that preference would make `s6-rc -d change nginx` stop the API with
the UI; and the timeouts survive, without which a camera that never appears
holds the transition open for ever and the UI never starts.
Two of them earn their place beyond the graph:
* The generated oneshot-runner shebang must be $MODDIR/bin/execlineb. It
comes from the execline the COMPILER was linked against, so a native stack
configured with the wrong --prefix ships a database where every oneshot
dies with ENOENT and every longrun keeps working. Nothing else in the build
can see that.
* The oneshots' up/down are execline command lines carrying shell inside a
/bin/sh -c argument. bash -n and shellcheck parse the wrapper and pass, so
this extracts the body s6-rc-db actually stored and runs sh -n over it.
The NUL split is also the proof the shell arrived as ONE argument --
execline lexing it into several would be a script that runs its own first
line.
Both were checked by breaking them: dropping ui/dependencies.d/ff-startup
fails one named test with 13 still reporting, and a dangling `then` inside
wifi's up fails the parse test. A first attempt at that second control did not
fail, correctly -- `[ 1 = 1 ; then` is a missing `]`, a runtime error, not a
syntax one.
qa/replica/test_s6rc.py -- written, NEVER RUN, and it says so at the top. There
is no stock FlashForge package on this machine, so `make build` is impossible
and the replica image holds a pre-s6-rc payload. Its `box` fixture checks for
s6-rc-init once and fails with a single message naming `make build`, so the
lane reports twelve errors with one cause rather than forty assertion failures
sharing it. It covers what neither suite ever has: a kill -9'd daemon coming
back, which is the entire reason for shipping 14 binaries.
RETIRED: qa/replica/test_services.py, and the Service class it was the only
caller of. It was the port of case-services.sh and its subject was the wrapper
contract -- seven S* scripts loading a shared library, answering status, naming
themselves, rejecting an unknown verb. None of that exists. Three assertions
survived into test_s6rc.py: the scanner is the cross-built ELF we shipped, its
log is empty, every run in the scandir is executable.
THE OLD SUITE IS RED AND I HAVE NOT TOUCHED IT. case-services.sh,
case-nginx.sh and case-moonraker.sh assert the wrapper contract and the
MOD_S6=0 fallback; case-priority.sh calls svc_start_daemon; test_startup.py
asserts FF_SKIP_MCU_BRINGUP. All deleted. They also cannot be RUN here, and
editing shell I cannot execute to make it assert something new is the thing
this repo's own rule forbids -- never decide whether a script works by reading
it. docs/qa-migration.md now carries the table: per case, what it asserted and
where that lives now, including the two entries that honestly say "nowhere".
Retiring them is a coverage decision and it belongs to whoever has a stock
package and a real run.
bin/patch.sh also ships s6-rc-db beside the native compiler now; the static
lane needs it and it was the only piece missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5025aac171 |
opkg-utils already builds ipks, and opkg is the second package that proves lib.sh
Two corrections to the packaging PoC, both from review.
FIRST: THERE WAS ALREADY A TOOL. bin/mkipk.sh is deleted. opkg-utils ships
opkg-build and opkg-make-index -- 423 lines of POSIX sh and a python indexer,
upstream, maintained -- and this repo now drives them instead of re-deriving
the format. What is left of build-packages.sh is the part that is genuinely
ours: laying out the tree opkg-build packages. The hand-rolled version worked,
and it was still us reimplementing member ORDER, control validation and the tar
flags that make a build reproducible, all of which fail as "a package that
inspects fine and installs nowhere".
opkg-utils is PINNED BY COMMIT, uniquely in this repo, and not by choice: it
publishes no release tarball anywhere. Upstream's cgit has snapshots disabled,
downloads.yoctoproject.org carries opkg but not opkg-utils, and the GitHub
mirror last saw a commit in 2012 with no tags. A git sha is a hash of the whole
tree -- the same guarantee every sha256 here gives -- and fetch-assets.sh
verifies HEAD against it and refuses a dirty checkout. Both were tested.
SECOND: A SECOND PACKAGE, AND IT IS opkg ITSELF. pkg/lib.sh now holds the part
of a cross-build that is the same every time -- the two toolchains, the gcc
wrappers that bake -EL -mnan=2008 into the DRIVER because autotools link lines
do not all forward CFLAGS, the ABI self-test of those wrappers, autotools,
the build cache, and pkg_ship. Both recipes fit on a screen and share exactly
that file.
opkg is the honest test of it: different toolchain (Bootlin musl, not Ingenic
glibc), different libc, different link mode (static, not shared), and a
dependency chain three builds deep (zlib -> libarchive -> opkg) against
libsodium's none. It is static musl for the same reason s6 is -- a standalone
binary that must match nothing -- so no libarchive.so or libz.so ships beside
it. qa/static/test_ipk.py fails any recipe that sources its own toolchain,
spells its own -mnan=2008, or calls ./configure directly; the gate was checked
by breaking a recipe on purpose. zlib is the one named exception, because its
configure has never accepted --host.
THE LOOP IS CLOSED ON THE TARGET. The opkg this repo builds -- 775KB, static,
nan2008/o32/mips32r2 -- runs under qemu-mipsel and installs both our packages,
including itself, then lists their files and removes them leaving nothing
behind. The opkg it installed then runs and reports its version. Its status
database landed at /usr/data/anvil/var/lib/opkg, which is where pkg/ipk-install
writes, because both agree rather than by coincidence.
Three findings paid for by building it, all in the note:
* opkg BAKES ITS STATE DIRECTORY IN AT COMPILE TIME -- --offline-root does
not move it. Built with any other prefix it believes nothing is installed
and reinstalls the world. Third instance of this trap here after s6's
libexecdir and execline's shebangdir.
* opkg 0.6.3 does not build against musl: two files call basename() without
<libgen.h> and get away with it only under glibc. 0.7.0 fixed both, which
is the difference between a pin bump and carrying a patch.
* libtool SWALLOWS -static. A program configured LDFLAGS=-static comes out
dynamic and the only symptom is a NEEDED entry nobody reads. -all-static,
at make time, is the flag that means it -- hence the explicit check.
Both packages are byte-identical across a full cold rebuild. The musl toolchain
fetch now also triggers on a stale work/.opkg, and its cache key lives in
common.sh so the fetcher and the recipe cannot spell it differently -- the
hazard already written above pypkg_stamp.
Also fixed: two `# shellcheck disable=... -- prose` directives were malformed,
which made shellcheck abandon those files rather than check them. A gate that
examined nothing, in the file arguing against those.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
057a3a11d7 |
The boot was seven scripts in filename order. It is one compiled graph now.
firmwareExe ran $MODDIR/init.d/S* one at a time in the foreground, and that
loop WAS the boot order. Three of the seven were already thin wrappers over a
supervisor that was already running, so what the loop still carried was not
supervision, it was sequencing -- expressed as filenames, in a directory
nothing but the loop read.
This ships s6-rc and moves the sequencing into a database compiled at build
time. firmwareExe now starts the scanner as its own child, initialises s6-rc,
asks for one machine state, and blocks on `wait "$SCANNER_PID"`.
WHAT THE GRAPH ACTUALLY SAYS
mcu-bringup -> klipper -> ff-startup -> ui
^
moonraker
ok-all = wifi nginx moonraker camera klipper ff-startup ui
Those are the first real edges the plan has ever had; it predicted they would
not arrive until phase 7. bin/ff-startup.py became two services, which is what
made them possible: the MCU bring-up has to precede klipper and the readiness
wait has to follow it, and both facts now live where a program can read them.
Deliberately still absent: nginx before moonraker. S60nginx's header wants it,
but dependencies.d/nginx under moonraker means `s6-rc -d change nginx` stops
moonraker too -- the exact failure splitting S60web into two scripts prevented.
The cost of not writing it is a second where a reloading browser gets a refused
connection instead of a 502.
AND WHAT THEY DO NOT SAY. klipper has no notification-fd, so s6-rc calls it up
when it is forked, not when klippy is usable. ff-startup keeps its own poll of
moonraker's API. moonraker and camera DO carry one, so their readiness is real.
The asymmetry is written down at each site rather than glossed.
init.d/ IS DELETED, REVERSING PHASE 4
Phase 4 promised the S* scripts would survive as thin wrappers, because
`S60nginx restart` is in people's hands. They did not survive, and the reason
is the argument nginx/run already made: a service that only works when a shell
script happened to run first is not supervised, it is lucky. Every prerequisite
a wrapper did belongs in the run script anyway, because s6 restarts a service
without going near a shell. Once they moved, the wrappers were verb translation
in a directory whose S<NN> numbers meant nothing. s6-rc is the interface now.
anvil-service.sh goes with them -- 625 lines whose last caller was firmwareExe,
and a shared library for one consumer is the redundancy the wrappers were.
So do the fallbacks. S60nginx and S62moonraker each carried a second,
unsupervised implementation reachable only via MOD_S6=0; that switch is gone,
and with it the argument for a code path exercised only on broken printers. The
cost is stated plainly in anvil.conf: a printer whose s6 fails now has nothing.
klipperDaemon is replaced with a no-op `start`. It is FlashForge's script and
off-repo callers remain -- stock start.sh, the UI's restart path, habits over
ssh -- and each of them would fork a second klippy beside the supervised one.
WHAT SUPERVISES THE SUPERVISOR, ANSWERED
Phase 3 left this open and guessed a dead scanner was a dead printer either
way. It is not. firmwareExe's tail used to be `while true; do sleep 3600; done`
purely to satisfy app_startup.sh's watchdog, which greps ps for our name 5s in.
It waits on the scanner instead, so exiting IS the recovery: app_startup.sh --
FlashForge's own init, which we never patch -- re-execs us and the tree comes
back. Not a respawn loop in ash, which phase 3 rightly refused. It restarts
everything rather than the scanner alone, and that is written down.
MEASURED, NOT READ OFF THE MANUAL
Host- and target-compiled databases are byte-identical (db md5
80f5261437191fa591744ada6087c6f7 both ways), so the database is built here and
shipped. /run is tmpfs and writable, so the live directory needs no fallback.
Both were BLOCKING UNKNOWNs in the note and are now settled sections.
Three traps the replica found that reading could not:
* s6-svscan resolves s6-supervise through PATH. Without $MODDIR/bin on it the
scanner answers s6-svscanctl -a perfectly and then fails every spawn with
ENOENT for ever, and s6-rc-init HANGS rather than failing.
* s6-rc-init creates one symlink per service in the scandir and dies with
"File exists" if the name is taken. So the cutover is not additive, and
run-append.sh sweeps.
* s6-rc-compile does not reject a `down` file. It accepts it and SILENTLY
DROPS it -- the servicedir is byte-identical. "Down by default" can only
mean absent from the bundle, which is why mcu-bringup is the one service
outside ok-all and arrives as klipper's dependency.
execline turned out not to be optional: s6-rc GENERATES two servicedirs whose
run scripts are execline, and every oneshot's `up` goes through one of them.
The binary list was read off a real compile, not guessed -- seven execline
binaries of 53, plus six s6 ones we built and did not ship.
Two build bugs fell out. fetch-assets.sh compared against a stamp patch.sh
never wrote, so every build pulled the ~100MB toolchain the skip exists to
avoid; S6_STAMP is one string in common.sh now. And `chmod +x .../bin/s6-*`
would have missed all seven execline binaries.
qa: 83 static. The suite gained the s6-rc run scripts and klipperDaemon, which
no glob had ever covered, and lost a dead payload/init.d/S* glob -- exactly the
silently-retired check that file's own docstring warns about. The oneshots'
up/down are NOT in it: they are execline carrying shell inside a /bin/sh -c
argument, so both checkers would parse the wrapper and pass.
Not verified here: no config.env and no stock package on this machine, so the
mipsel cross-build of execline/s6-rc and the ABI gate over 33 binaries have
never run. The native compile and the database are proven; that half is not.
The replica gate is the next commit, and it still owes an answer on whether
helix-screen runs under s6-supervise at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
1c89a9cd80 |
Neither Alpine nor OpenWrt has a binary we can install. That picks opkg.
The evaluation asked for apk vs opkg. The comparison turns on one fact that makes most of the usual argument irrelevant here: Alpine has no MIPS port at all, and OpenWrt's mipsel_24kc is musl against this printer's glibc 2.29. So no upstream repository is reachable either way, every package is cross-built here regardless, and a package manager buys packaging and installing -- not software. That reduces the choice to which format is cheapest to PRODUCE on every build, and .ipk wins by a distance: ar + two tarballs + a three-byte version file, no keypair, no Alpine host, no abuild. apk would need the signed three-concatenated-gzip layout implemented by hand or abuild imported into a Debian build image, paid per recipe forever, for a format with an in-progress v2->v3 transition and no users on our architecture. The full table, the two honest points for apk, and the phased plan are in docs/notes/85-packaging.md. The proof of concept is libsodium, end to end and measured rather than described: make packages -> libsodium_1.0.20-1_mipsel_xburst2.ipk + a feed index Verified against real opkg 0.6.3 built from source -- offline-root install, both symlinks preserved as symlinks, list-installed, files, remove leaving nothing behind. dpkg-deb reads it too; the container format is Debian's. A full recompile from a cold cache produces a byte-identical package, so a pin bump is reviewable as a hash diff. libsodium is now compiled ONCE. patch.sh section 5d's build block moved to pkg/libsodium/build.sh and patch.sh runs it, so the tarball's copy and the packaged copy cannot become different libraries wearing one version number -- asserted, not trusted. MODDIR, PY_HOST, PY_TOOLCHAIN_DIR and mips_abi_gate moved to common.sh for the same reason: a package is a shipping vehicle and gets ABI-gated like one, on a machine that may never run patch.sh. Nothing ships. make build is unchanged and still produces one anvil.tar.xz; make packages needs no stock FlashForge package, which is the point -- a packaging gate that only runs where the proprietary firmware is stops being a gate. Two findings worth more than the code. opkg bakes its state directory in at COMPILE time, so a phase-2 opkg must be configured --prefix=/usr/data/anvil or it comes up believing nothing is installed -- the s6 libexecdir trap one layer up. And opkg 0.6.3 will not configure without libarchive, which is what phase 2 actually costs; the opkg binary itself is 33KB. So pkg/ipk-install walks the ar headers with tail/head instead: this busybox is 1.31.1 built small, with no timeout, nc or ionice applet, and betting an installer on an ar nobody has checked for is how an update fails at the last step. qa/static/test_ipk.py is 18 tests that build their own package from a synthetic tree -- no toolchain, no firmware -- so the lane keeps the property qa/conftest.py insists on and can never be skipped into looking green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
16d00be525 |
case-services.sh is 360 lines and one exit code. That contract is 44 tests now.
test/ has two test frameworks. run-tests.py + ffsim is 1,657 lines reimplementing what pytest already does -- skip accounting, gating, reporting, then parsing pytest's own JUnit XML back out -- and it calls pytest as gate #5 of about twenty. Underneath it, 6,280 lines of case-*.sh execute inside the qemu chroot and report 13 booleans between them; case-moonraker.sh is 1,059 lines that collapse to one pass/fail. Nothing runs in parallel and nothing can be selected: each replica gate is its own `docker run --privileged`, re-registering binfmt and re-assembling the mount layout, thirteen of them in sequence. All three follow from one decision -- THE REPLICA CONTAINER IS TREATED AS A TEST RUNNER INSTEAD OF AS A FIXTURE. That is what pushes assertions into ash, which yields one bit per container, which then needs a bespoke framework above pytest to aggregate the bits. qa/ is the replacement, built beside test/ rather than inside it, with two lanes and one framework. Both suites run in CI; test/ is still the release gate; nothing has been deleted. docs/qa-migration.md has the order the rest moves in, and the rule: a case script is deleted only once its replacement here has been green, so there is never a window where coverage drops. THE MACHINE IS UNCHANGED. qa/ reuses test/integration/printer/'s Dockerfile, entrypoint.sh, assemble.sh, binfmt.sh and seed-prog.sh as they are, and its `docker run` argv is deliberately ffsim's, minus --rm and plus -d. While both suites run they have to be testing the same machine or neither proves anything about the other. The only new trick is how the container is held open: entrypoint.sh ends by running the case script it was given, and the one it is given here touches a marker and sleeps, so setup completes exactly as it always has and the container simply does not exit. The marker is also the readiness signal, because setup ranges from under a second to over a minute and a constant would either waste the fast case or fail the slow one. TESTS RUN AGAINST A MACHINE THE REAL INSTALLER PRODUCED. The `printer` fixture is a replica with the package installed by the printer's own app_startup.sh, off a genuine FAT filesystem on /dev/sda1. The first version of this tree did what case-services.sh does -- copy payload/*.sh and payload/init.d/S* into $MODDIR by hand -- and two things were wrong with it. It is a second implementation of the install, so everything downstream asserted against a layout the harness built rather than one the installer produced, and the real installer could have broken without a single test noticing. And it can only place what is in payload/, so everything the BUILD produces was missing: s6 is a build output that bin/patch.sh stages into the package, so a hand-placed payload has no supervisor at all, which is why the tests had to carry a stand-in scanner and gate one assertion on whether it was in use. Installing for real fixes both and deletes the stand-in. Hand-placed, /usr/data/anvil held anvil-env.sh, anvil-service.sh, anvil.conf, init.d and etc/s6. Really installed it holds VERSION, backup, bin, config, config-installed, helixscreen, lib, libexec, moonraker, nginx and www as well -- bin/lib/libexec being the cross-built s6 and CPython 3.13. The file is SHORTER than the case script it replaces while reporting 44 results instead of one. case-install.sh's header records this same lesson from the first time somebody learned it: "An earlier version of this file replayed app_startup.sh by hand -- which meant a bug in our reading of it could never be caught." The install is the machine's own shell under qemu, so it is baked into an image once per package and cached under a tag derived from the package's md5 -- build-printer-image.sh's trick for the stock baseline. Rebuild the package and the tag changes, so the next run bakes again instead of testing yesterday's build; deliberately not a hand-written staleness check, because those are the ones that eventually get it wrong. A MISSING TOOL OR RESOURCE FAILS, and there is no flag. This tree briefly had --strict-skip=<lane|all>, which was ALLOW_SKIP inverted -- and ALLOW_SKIP's defect was never the direction it named things in, it was that the decision lived in the CI invocation. An accept-list has to be edited whenever a gate moves; a strict-list has to be remembered whenever a job is added; both degrade silently when the command line is wrong, which is the same disease as the printer-sim job that was gated on an unset secret and never ran once in its existence. So the verdict lives with the knowledge: no shellcheck, no pyflakes, no docker, no daemon, no replica image, no built package -- each fails where it is needed, with the fix in the message. There are no skips in this suite. MEASURED, on monstrofil/creator5-printer:latest, both suites agreeing on the same machine: 131 tests in 40s, all green; qa/static alone 1.3s with no docker; one assertion selected by node id 2.9s, against the whole 360-line case being the smallest unit test/ can run. Granularity checked by injecting the regression case-services.sh exists to catch -- deleting SVC_EXTRA_VERBS="|force-start" from S70klipper -- which failed one named test, printed the usage line that came back, left the other 41 reporting, and took 4s to re-run alone. Two bugs the port found in its own harness, both only visible by running it. bake.sh shipped without its executable bit, so the lane died at container init with 43 errors all saying "could not install the package" -- a file mode wearing an installer failure's clothes; test_action_scripts_are_executable now guards it, the same check the payload's own s6 run scripts already have. And an ELF magic assertion used `od -An`, which the printer's busybox od does not have -- the replica catching host-vs-printer drift in the test code itself, which is the entire reason it exists. Also here: qa/static/test_probes.py, which has no counterpart in test/. Printer.ps() and Printer.listening() are parsers, and a parser is where a harness goes quietly wrong -- a listening() that never matches makes every readiness assertion pass by the absence of contradiction. Neither failure looks like a failure; they look like a green suite. So they are tested against captured output, in the static lane, with no docker involved. Two things found while surveying, neither fixed here. case-pyext.sh is 567 lines with no gate in gates.py, no Makefile target and no CI step: it has never run. And case-mcu-bringup.sh:4 says the handshake is "tested elsewhere against a pty" -- there is no pty anywhere in the repo, and every documented past bug in ff_mcu_bringup.py lived in the code that comment excuses from testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |