mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
worktree-timelapse-test-gcode
64
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr ShyshatskyiandClaude Opus 5
|
c3a64e0394 |
The stock firmware is a pinned download like everything else
config.env.example opened with DESKTOP="$HOME/Desktop", and four settings hung off it. Nothing in bin/, pkgs/, qa/ or installer/ ever read it: three of the four were commented-out overrides for files `make vendor` already fetches, and the fourth, BUSYBOX_BIN, names a binary with no source at all. What DESKTOP really did was carry the stock FlashForge .tgz -- the one input a person had to go and find before they could build anything. It did not have to be. .github/workflows/ci.yml has been fetching that exact file for as long as printer-sim has run, from ghzserg/FF, pinned by sha256, with its own get(), its own cache and its own secrets -- a second spelling of "download the firmware and check the hash" that no human path shared. The pins here are those pins, verified byte-for-byte against a fresh download of both assets. So versions.env pins the stock package the way it pins everything else, bin/fetch-assets.sh downloads it into vendor/, and bin/common.sh defaults STOCK_TGZ_* there. A cold checkout is now `cp config.env.example config.env && make build`, with nothing to edit. --stock is a separate flag from --all, not folded into it. The packaging lane wants no FlashForge package -- that is most of the point of it being runnable in CI on a bare checkout -- so `--all` stays "every payload piece, no firmware", `make build` asks for the one model it is building, and `make vendor` (--all --stock) takes both. The one download that will not overwrite an override. HELIX_TGZ and friends name a public release anyone can re-fetch in a minute; STOCK_TGZ_* names 93MB of proprietary firmware someone downloaded by hand, and clobbering it on a hash mismatch would destroy the only copy on the machine. It is skipped instead, and unchecked -- their file, their problem. CI loses its duplicate fetcher, its stock/ cache and the STOCK_URL_* secrets. A fork testing its own firmware re-pins versions.env, which works locally too; the secrets only ever worked in CI. qa/lib/config.py reads versions.env for the pinned filenames so stock_for() looks where the download actually lands. An explicit STOCK_TGZ_* still wins. An existing config.env keeps working untouched: its paths are now overrides, so the build skips the download and uses the file it always used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DPU6RR4UJH2Qf9DR5hb2gg |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
83a4600df8 |
Stop naming things that are not there any more
A sweep for comments and messages that describe machinery this tree no longer
has. Each one read as current and was not.
* bin/verify.sh, retired two commits before this branch, was still named as
the thing `make release` and docs/building.md were explaining the absence
of. The absence needs no explaining -- `make qa-replica` is the gate.
* `make test-recovery` does not exist. docs/hardware-testing.md told owners
it "used to" prove the stock rollback; the useful half of that -- nothing
tests the rollback today, treat it as unverified -- is kept and the dead
target name is not. runFirmwareExe.sh pointed at it too.
* run-append.sh in bin/build-payload.py, case-install.sh in
docs/printer-replica.md: both files are gone.
* `make test`/`make test-py` over test/, and the note that both suites run
in CI "until a case-*.sh has a green replacement" -- test/ was deleted and
there are no case scripts left to replace.
* docs/building.md still said a package carries the software component.
CI's shellcheck globs are the one real defect this turned up. Two of the four
-- pkgs/*/payload/init.d/S* and pkgs/*/prog/*.sh -- match nothing: the recipes
stopped using both paths, and payload/prog/ is where those scripts live now.
shellcheck given a pattern that expands to nothing writes usage to stderr and
exits 1, so that step has been failing for its globs rather than for the code.
Fixed to the paths that exist, with a pointer to
qa/static/test_shell_syntax.py, which addresses the same files and refuses a
glob that matches nothing -- the guard this step did not have.
Verified: the corrected invocation runs clean over the 21 files it now
matches, and 112 static tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b98552386b |
No FlashForge component ships; patch.sh is payload.sh
The software component existed to give the mod one exec hook. It is not needed for that any more, so a release now carries our installer and our payload and nothing else: 26MB smaller, 21MB of which was FlashForge's own firmwareExe travelling back to the partition it was already on, along with their klippy tree, their wifi module and their configs. WHAT IT GIVES BACK. /usr/prog/PROGRAM/software is no longer rewritten, so the version directory there stays FlashForge's with their genuine firmwareExe inside it -- and app_startup.sh's five-second watchdog, which copies one out of that directory when nothing by that name is running, has something to copy again. A printer whose payload fails to install now comes up on the stock UI instead of a blank screen. That recovery has been broken since we started deleting their binary out of a component we shipped; docs/how-it-works.md claimed it worked throughout. THE ROOT PASSWORD moves with it, because it used to ride in the component's shadow file. bin/pack.sh bakes ROOT_PW_HASH into the installer, single-quoted -- an unquoted $6$... would have the printer's shell expand $6 as a positional parameter and ship a truncated hash -- and the installer writes it to /usr/prog/etc/shadow. The random-password path asks one question instead of four: FlashForge ship the same root hash on every machine, so a live shadow still carrying it means nobody has set a password and it is ours to set. The .prev-root-hash recording, the shipped-vs-live comparison and the restore-it- afterwards step were all machinery for surviving a file that nothing overwrites any more, and are gone. pack.sh checks the hardcoded constant against the stock package it built from, because getting it wrong is silent and total: every printer would read as "already set" and none would ever get a password. bin/patch.sh is bin/payload.sh. It patched nothing any more -- no splice, no shadow edit, no firmwareExe deletion -- and what is left is the payload: the feed installed by the printer's own opkg, the seeded config, the manifest. Its sections are renumbered 1-5 with no gaps, and the five tombstone comments marking where removed sections used to be are gone with them. test_recipe_layout.py loses six of its nine tests. Each compared the tree against a description of the tree -- no fourth directory kind, each kind still used, 3rdparty carries no files of ours, installer/ holds exactly two named files -- and none could go red for a reason a printer would notice, while all six went red the moment the layout legitimately changed. The two that bite are kept: a payload/ path escaping $MODDIR, and a recipe name at both levels, which makes pkg_dir silently build the wrong package. SW_VER is deleted from config.env.example and both workflows: it stamped a component that is not built. Gates: 112 static tests pass. The generated installer is verified after substitution -- all four lines present, and `sh -n` over the result, because a hash with a stray character in it would otherwise reach a printer as a syntax error whose only symptom is a package that installs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
56b5b6c334 |
The printer image is the only replica; make rootfs was a second way nothing took
`make rootfs` unsquashed rootfs.squashfs out of the stock package into
work/rootfs, and a thin container bind-mounted it with PROG_DUMP supplying
/usr/prog from the factory image. Every caller set PRINTER_IMAGE anyway --
both workflows wrote it into test.env BEFORE running `make rootfs`, so CI
unpacked a 93MB package, unsquashed a rootfs and downloaded a 182MB factory
image on every push and read none of it. build-printer-image.sh never used
work/rootfs either: Dockerfile.full does its own extraction from the same
public firmware and needs no stock package at all.
So the fallback went, and PROG_DUMP with it -- it was only ever read when the
replica was built locally, and both implementations already passed it empty
otherwise. Gone: the target, extract-rootfs.py, gates.extract_rootfs and its
two tar helpers, the thin printer/Dockerfile, the `prebuilt` flag threaded
through both Replica implementations, and Replica.start's want_output, whose
only message was that the local build was about to be slow.
PRINTER_IMAGE is pinned to 1.9.7-1.2.9-20260810 rather than :latest. It is
now the entire substrate the brick-safety gates run on and the only input
nothing checksums, so a floating tag would let it change under a green suite
-- the drift the sha256 pins in both workflows exist to prevent.
THREE THINGS THAT COULD NOT HAVE PASSED, found while doing it:
* Neither workflow built the feed. bin/patch.sh has refused to start
without one since
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
fbb3182e68 |
bin/verify.sh is retired; four of its forty checks move to qa/replica
verify.sh described itself as "simulate every check the printer performs,
against a built package". The word carrying that sentence is SIMULATE, and it
is why thirty-six of the forty are deleted rather than moved.
THE REPLICA LANE DOES NOT SIMULATE THE INSTALL. install-package.sh puts the
package on a genuine FAT filesystem at /dev/sda1 and runs the machine's own
/usr/prog/app_startup.sh over it, verbatim, under qemu. So these are
PRECONDITIONS of the lane now rather than assertions in it -- break any and
installed_image() raises with the install log and the whole lane goes red:
* the decrypt with the firmware key (the printer's unTar does it)
* runFirmwareExe.sh present, components plain tar not xz (bare `tar -xvf`)
* md5sum.list verifying (the installer aborts and deletes the payload)
* the MACHINE= gate (runFirmwareExe.sh refuses a foreign package)
* the /mnt/<Model>-*.tgz glob (app_startup.sh finds the file by it)
A host-side re-reading of those rules is a second implementation, and a second
implementation can agree with itself while disagreeing with the printer. That
is the defect qa/replica/conftest.py records this repo learning twice about
hand-placed payloads.
A second block was asking what the suite already answers, of an installed
filesystem rather than of a tar listing: the klippy tree and c_helper.so
(test_install, test_abi), the s6-rc database and the oneshot runner's execline
(test_s6rc), s6 and s6-ftrigrd (test_supervisor), the .install-manifest
(test_upgrade), libsodium's bare .so symlink and the chamber configs
(static/test_ipk), and every shipped script's syntax
(static/test_shell_syntax).
THE FOUR THAT MOVED are the ones nothing else asks, and each is stronger for
the move -- a `tar -t` listing cannot tell you that a shared object loads, or
what a name resolves to on a PATH:
* the interpreter IMPORTS sqlite3, lmdb and _cffi_backend, rather than a
grep finding three .so files. This is what catches a cross build that
resolved to an x86_64 manylinux wheel: present on disk, raises on import.
* nothing in $MODDIR/bin shadows FlashForge's python3 -- asked with
`command -v` after sourcing anvil-env.sh, which is the file that PREPENDS
$MODDIR/bin to PATH and so creates the hazard in the first place.
* CPython's dev half (headers, pkgconfig, config-3.13-*) did not ship.
* the ship boundary: no file on the installed printer is byte-identical to
one under bin/ or docker/. Asked of the machine rather than of the .tgz,
so it also sees whatever a maintainer script wrote as root at install
time. The sweep filters on exact size before hashing, which is what keeps
a walk of /usr/prog through qemu cheap.
WHAT GENUINELY LEFT, recorded in docs/qa-migration.md rather than only here:
there is no longer a check you can run on the build host, in seconds and
without docker, against a package you are about to flash. Nothing warns you
before you copy the file to a stick that it was built from the wrong model's
stock package. The printer still refuses it and `make qa-replica` still
catches it, but the fast host-side warning the hardware checklist used is
gone. `make verify` is removed and `make release` no longer calls it.
VERIFIED. All four ran against a real replica (the cached installed image,
PRINTER_IMAGE=monstrofil/creator5-printer:latest): 4 passed in 17s. Then each
was DRIVEN RED on purpose with a throwaway module, which is this repo's own
standard from
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
57c776ee63 |
test/ is deleted; qa/ is the only suite
The eight files were already gone from the working tree. What this commit does
is everything that had to follow them, because nothing else had been updated
and three of the four things pointing at test/ were on a build or release path.
WHAT WAS STILL WIRED TO IT
pkgs/klipper/build.sh ran test/test-chelper.py under `|| pkg_die`. With the
file gone `make packages` would have died on every
klipper build.
bin/verify.sh ran the same script and read a non-zero exit as
"c_helper.so does not match the shipped klippy tree
(stale build)" -- a false red on every release verify
rather than an error about a missing file. Its
ship-boundary check also globbed `find bin test
docker`; that is `find bin docker` now.
pytest.ini testpaths = test, so a bare `pytest` collected
nothing at all. It is `qa` now, which is what the
comment above it was already describing.
qa/static/ PY_GLOBS listed test/*.py and test/integration/*.py,
test_shell_syntax.py and _files() makes an empty glob a collection error
on purpose -- so the whole static lane failed to
collect, taking 93 unrelated tests with it.
Also dropped: the `pytest -- the host-side tree` step in ci.yml, the `test-py`
target with its .PHONY entry and two help lines, and qa/conftest.py's opening
paragraph explaining why qa/ sat beside test/ rather than inside it.
THE CHELPER SYMBOL CHECK IS GONE, NOT MOVED
It compared the .so's dynamic symbols against what klippy cdefs, and it was the
only check for a c_helper.so built from older sources than the klippy tree
beside it -- the failure that shipped once and bricked klippy startup on
hardware, because cffi resolves lazily and a stale .so imports cleanly.
qa/replica/test_install.py covers the ELF arch and nan2008 and nothing else.
What stands in its place is structural rather than a gate: pkgs/klipper
compiles the .so from the chelper sources of the very tree it ships, so the
mismatch is not something the recipe can produce. The comments in
pkgs/klipper/build.sh and docs/building.md now say that is the whole defence.
DOCS
docs/testing.md's "what is left in test/" section is now "test/ is gone", and
it names what went uncovered rather than leaving it to be found: the tool-frame
arithmetic in ff_toolchange, the chamber macros' per-model branching, the stamp
discipline in ff-startup.py, and the verification G-code against the shipped
macros. docs/hardware-testing.md said `make test-py` keeps the two
gcode/creator5-*.gcode files honest; it now says nothing does, and to read them
by hand when a macro is renamed. Paths corrected in docs/building.md,
docs/printer-replica.md and docs/qa-migration.md's intro.
Verified: `pytest` collects 366; qa/static is 136 passed, up from 43 because
the lane could not collect before this. Its 110 errors are this checkout's
missing vendor/opkg-utils, the same cause before and after. bin/verify.sh,
pkgs/klipper/build.sh and the Makefile parse; ci.yml is valid YAML.
Not touched, and stale before this commit: tools/python/README.md and
tools/supervisor/README.md still document test/integration/printer/case-*.sh
and a printer-exec.py that earlier commits removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bLGYpVAeVzAX6NeAxcU42
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c185ef3285 |
The replica is a build tool; test/ is 28 unit tests
Two changes that turned out to be one.
MOVED, NOT DELETED. `test/ffsim/` and `test/integration/printer/` are
`tools/replica/`. They were never tests: bin/patch.sh calls
bin/build-payload.py, which starts a replica and assembles the payload by
running the PRINTER'S OWN opkg against / inside it -- so the "test harness"
was on the build path, and deleting it as asked would have broken `make
build`. qa/lib/replica.py drives the same directory unmodified. No content
changed; what changed is qa/lib/replica.py's build directory, three launcher
sys.path lines, the Makefile targets and the CI shellcheck glob.
tools/replica/README.md says who drives it and why it is not under test/.
CUT TO 28. The five surviving modules were 136 tests; they are 28, chosen on
one question: would this failure be SILENT? What is kept is the mistakes that
run perfectly and are wrong --
test_startup 8 no stamp unless the values are verifiably saved (a
stamp written early is a printer that has lost its
factory calibration for good), and the handover order
test_tool_transform 7 every one is a move to the wrong place: a sign, a
frame left applied while probing, a stale gcode_move
cache, a config a calibrated printer cannot parse
test_ffscreen 5 the four ways it can do harm rather than look wrong
test_chamber 4 the chamber mistakes that cost a print before anyone
sees them, plus the parse gate they rest on
test_gcode 4 a renamed macro, and the safe file being cold
What went: the panel narration, the geometry probe and parse_geometry, the
z_adjust and per-job-Z bookkeeping, the config layout checks, the G-code
bounds checks. Each is either a wrong number shown to a person, or a failure
the printer reports loudly the first time -- and qa/replica/test_boot_screen.py
already renders the real frames on the real machine.
The rewrite is not a delete: assertions were folded so a kept test asks its
question in every direction it can be got wrong (both models, both rotations,
all four ways the migration can fail), and each file's header now says what it
dropped and why, where the next person will open it.
Also gone: conftest's `rootfs` fixture, orphaned two commits ago. Nothing under
test/ needs the firmware, docker, or a network any more.
28 host-side tests pass in 0.15s. qa/static unchanged at 218. `make rootfs`
re-verified end to end from its new path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UzLebCrvc8ssWR5JuEsha4
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6aa622d2ff |
6,280 lines of case scripts reported 13 bits. They are 118 tests now.
test/run-tests.py is gone, and with it the bespoke harness that wrapped
pytest, re-parsed its JUnit XML, and drove thirteen replica case scripts to
get one boolean back from each. Every gate it ran is either a module under
qa/replica/ or deliberately retired.
PORTED. case-install.sh (579 lines, 1 bit) is test_install.py, 22 tests, asked
of the machine the printer's own app_startup.sh built rather than one the
harness assembled. case-upgrade.sh is 21, case-supervisor.sh 31,
case-boot-screen.sh 17, case-mcu-bringup.sh 8, and case-nginx.sh plus
case-moonraker313-s6.sh are test_web.py's 7.
RETIRED, subject deleted by the s6-rc switch: case-services.sh, case-nginx.sh,
case-moonraker.sh, case-moonraker313-s6.sh, case-camera.sh and
case-priority.sh drove $MODDIR/init.d/S* or sourced anvil-service.sh, neither
of which exists. Decided against real output -- `make test-services` failed
with "the payload ships no anvil-service.sh". case-pyext.sh and
case-moonraker313.sh went too: no gate, no target, no CI step, never run once.
RETIRED as a coverage decision: case-python.sh (test_web.py runs real Moonraker
on that interpreter, which is harder), case-libpath.sh, and case-recovery.sh.
The last is the brick-safety rollback and has no replacement; it is recorded in
docs/qa-migration.md rather than left to be discovered.
THREE ASSERTIONS HAD STOPPED MEANING ANYTHING, and each had been green for
months:
* case-mcu-bringup.sh's negative control had INVERTED. It asserted the
bring-up fails without LD_LIBRARY_PATH; CPython is --disable-shared now and
needs no library path, so the case's own skip branch was the one firing.
* case-upgrade.sh's headline assertion was VACUOUS. run-append.sh:150 does
`rm -rf $MODDIR/init.d` unconditionally, outside the manifest branch, so
"the update removed the stale S60web" passed regardless of the manifest
logic it was written to test.
* case-boot-screen.sh CONFIRMED ITS OWN ARITHMETIC, computing expected pixel
packing by calling ffscreen._pixel inside the replica -- a swapped R/B
matched itself.
case-install.sh's boot-log grep is not reproduced for the same reason: the log
line names every service in the boot set including the ones that then fail, so
grepping it cannot go red. The compiled database is asked instead.
TWO REAL BUGS FOUND ON THE WAY.
assemble.sh attached /stick.img with `losetup` and wrote the device to
/stick.loop, which NOTHING HAS EVER READ. Loop devices are the host's, not the
container's, so every USB_STICK=1 run leaked one permanently and a WSL2 kernel
has thirteen. Once they were gone every replica run failed with "failed to
setup loop device", which looks like a docker fault and is not. Detached now in
entrypoint.sh on the way out, and in qa's stop() for the held-open containers
that never take that path.
test_startup.py was 43 tests red: it built its argparse.Namespace by hand and
had never been taught --only-bringup / --no-bringup, and its FakeKlipper still
modelled start.sh plus a separate stop when klipper is restarted with one
`s6-svc -wr -t`.
CI AND RELEASE. The replica job was getting its package as a side effect of
run-tests.py running unpack/patch/pack; that is an explicit step now, as the
comment there predicted it would have to be. release.yml tested the exact files
it ships through sim-install.py, whose exit status the loop was discarding --
it is `REAL_PKG=<pkg> pytest ./qa/replica` per model now, 22 named results each.
WHAT IS NOT TESTED ANY MORE: bin/unpack.sh, patch.sh, pack.sh and verify.sh.
The packaging build on a synthetic stock package went with the runner and has
no counterpart in qa/. That is the path producing the .tgz a user flashes.
make-stock-fixture.sh is kept as the input a replacement needs.
Measured: qa/static 246 passed, qa/replica 110 passed and 8 blocked on hardware
the replica has not got (no /dev/ttyS*, no /dev/video*, so ok-all is
unreachable), test/ pytest 185 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BTAXgjtF8GdhqSxvdXCFW
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5f36cd14e3 |
Say what the code does, not what it used to
Comment cleanup over the files this branch touched, plus the dead code the
last two commits left:
* pkgs/lib.sh's pkg_buildopkg, 89 lines building the x86-64 opkg nothing
calls any more, and the qa test that checked its configure flags
* patch.sh's "5d. what is no longer here" section, and its mkdir of the
prefix directories -- build-payload.py replaces $PAYLOAD_DIR wholesale, so
they were created and then thrown away
* the two emptied creator5*-config recipe directories
The rest is prose. Comments that narrated a previous design -- what the host
opkg did, which package used to own the chamber config, that prog/ was once a
sibling of payload/ -- are gone rather than shortened; the ones explaining
constraints that still hold are trimmed to the constraint.
docs/building.md and docs/toolchange.md describe the packages as they are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ea1364ebca |
The payload is installed by the printer, not by a host opkg
bin/build-payload.py starts the replica, lets the machine's own opkg install the feed onto the machine's own filesystem under qemu-mipsel, and tars /usr/data/anvil back out. bin/patch.sh's section 0 is that call now. WHAT THE OLD WAY COULD NOT DO. A host opkg under --offline-root leaves every package Status: unpacked, because it cannot run maintainer scripts against a staging tree -- so patch.sh passed --force-postinstall to make the database claim work that had not happened. The payload shipped a postinst nothing had ever executed. It runs now, on the target userland, and its output is in the build log: anvil-core's postinst links the boot path and says so. THE REPLICA NEEDED A WAY TO HAND SOMETHING BACK. Every mount was :ro, which is right for a test harness and wrong for a build. run_case takes out_dir and mounts one writable /out -- but not at /printer/out, because assemble.sh binds /printer-src over /printer and a bind does not carry submounts, so the mount would be silently buried. entrypoint-out.sh makes the seam the stock entrypoint lacks: run it with a case that does nothing, then mount, then run the real case. It chowns /out back to the caller afterwards, because a privileged container writing into the repo leaves root-owned files the build user cannot delete. RUNBUILD IS A THIRD LANE for the same reason. The build has to reach the docker socket now AND still run as you, or work/ fills with root-owned files -- the two things the Makefile's existing split kept apart. --group-add with the socket's own gid, read rather than assumed, answers both. Left behind: pkgs/lib.sh's pkg_buildopkg is now defined and never called. It builds the host opkg this replaces. Deleting it is a separate change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8f54d87513 |
Comments that restated the code, and a few that had stopped being true
A pass over what this branch added, on the same rule as the branch merged before it: a comment earns its place by saying WHY, and the ones here had grown into essays. bin/patch.sh 781 -> 555 lines, its comments 523 -> 297. The install section keeps the three things a reader cannot get from the code -- the set comes from pkg_recipes so the BUILD_ flags are read once, -dev is filtered by package name because nine recipes ARE the dev package, --force-postinstall is what stops the database saying "unpacked" about files that are in place -- and loses the narration around them. Sections 3 and 4 were mostly an account of the sections that used to be there; they are four lines now. The CPython and libsodium headers keep the facts that took measuring (FF_PYTHON's callers, the missing _sqlite3 that pins Moonraker, libnacl's __file__ arithmetic and its dependence on "lib" appearing in the prefix) and drop the rest. Same for pkg_buildopkg, the payload-path block in bin/common.sh, and the four test docstrings. Also fixed, found while reading rather than looked for: bin/build-packages.sh still called itself "the proof-of-concept half" and said bin/patch.sh "still stages libsodium into the payload exactly as it did and `make build` is untouched". None of that survived the migration. The -maxdepth 1 comment had been wedged between `# shellcheck disable` and the command it applies to. Eleven "bin/patch.sh stages ..." references across pkgs/, test/ and qa/ described a script that stages nothing now. Payload bytes unchanged except S40s6, whose comment was one of the eleven. qa-static 259; make test 9 gates, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
51f524b5cf |
Merge worktree-comment-cleanup
The branch tightens comments across the tree -- removing the ones that
explained the code's past rather than the code. It branched before both the
pkg/ -> pkgs/ rename and the migration of bin/patch.sh onto the feed, so most
of its conflicts were with text describing code that has since moved or gone.
Resolved by taking its brevity and correcting what it could not have known:
bin/patch.sh ours, entirely. Its only non-comment change there deletes a
trailing comment from a line this branch rewrote, and every
comment it edits annotates one of the eight staging sections
that no longer exist.
bin/fetch-assets.sh ours. Theirs reinstates a KLIPPER_FORK condition; that
knob was deleted with the move to pkgs/ -- a recipe names its
source exactly once.
pkgs/lib.sh theirs, except the two hunks where our side carries code it
never saw (pkg_payload_hash and pkg_dir; the PKG_CC_SHARED
branch of pkg_build). Its prose, our functions.
bin/common.sh theirs throughout, with pkg/ -> pkgs/ and $MOD_PAYLOAD ->
$PAYLOAD_DIR, and minus its claim that the toolchain is used
"by bin/patch.sh for Klipper's c_helper.so" -- patch.sh
compiles nothing now.
pkg/anvil-core/build.sh deleted. The file is pkgs/anvil-core/build.sh and
was rewritten as a loop; its comments are superseded.
Two of its deletions were right about things this branch had wrong:
bzip2 named a pkg_make verb that was retired into pkg_build.
zlib claimed a ./configure line of its own as "the one sanctioned
exception". It has not had one since PKG_CONFIGURE_AUTO=0 and CHOST; the
rule in qa/static/test_ipk.py is absolute, and its docstring says so. I had
kept that paragraph in an earlier pass of this merge and was wrong to.
Three stale hints in the Makefile went with them: work/pkg is filled by
`make packages`, not by bin/patch.sh, in the two gate targets and one error
message.
Verified: the payload rebuilds to the same 31 packages and 154 ELF objects,
and the six files whose bytes changed are five payload scripts this branch
comment-edited (zero non-comment lines between them) plus the opkg status
file, which records their sizes. qa-static 259; make test 9 gates, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e3d1bdcdf7 |
The payload is the feed installed, not the build trees copied
bin/patch.sh ran every recipe and then cp -a'd its build tree into
work/modpayload. Eight sections of it, each a second description of what the
.ipk that recipe produces already contains -- the same duplication the recipes
themselves removed one level down, one level further up.
It now installs the feed. pkg_buildopkg's x86-64 opkg gets one invocation with
the whole selected set and an --offline-root, and what lands there is the
payload. "What does this release install?" is answerable with
`opkg list-installed` against the payload itself, and the tarball and the
package cannot describe different bytes because the tarball is made of the
package.
THE SET COMES FROM pkg_recipes, which is already PKG_WHEN-gated -- so
BUILD_HELIX, BUILD_MAINSAIL, BUILD_MOONRAKER, BUILD_TOOLCHANGE and
BUILD_KLIPPER are read in the pkg.conf that owns each and nowhere else,
instead of being restated by an `if` here that could drift from it.
-dev IS FILTERED BY PACKAGE NAME, and the first attempt got this wrong in a
way worth recording. execline, s6 and CPython ship both halves and pkg_ipk
names the runtime one; zlib, openssl, sqlite, expat, libffi, xz, bzip2,
libarchive and skalibs ship ONLY a dev package and say so by setting
PKG_NAME=anvil-<x>-dev outright. Asking "does this recipe have a runtime half"
installed all nine: 40 packages and 2165 ELF objects where there should be 31
and 154.
--force-postinstall, because opkg skips configuration under an offline root
and would leave every package Status: unpacked. Nothing runs opkg on the
printer after run-append.sh extracts the tarball -- the extraction IS the
install -- so a database saying otherwise would make the first real `opkg
upgrade` argue with the filesystem. It runs nothing today: no recipe defines a
maintainer script, and opkg 0.7.0 would not run one here anyway (it sets no
IPKG_INSTROOT; see libopkg/pkg.c:1339).
Installed-Time is normalised afterwards. opkg takes it from the clock and
honours SOURCE_DATE_EPOCH only for a man-page date, so two builds of one
commit differed by one line per package -- measured, then fixed, then measured
again.
Gone with the staging: the hand-written PKG_DEV_FILES prune (the dev half is
not installed, so bin/verify.sh's "no headers" check now passes because they
were never there), three partial ABI gates in favour of one over the finished
payload, and the last pkg_out call -- section 1 reads klippy out of the
installed anvil-klipper.
The payload gains anvil-klipper's tree at $MODDIR/klipper and bin/opkg. The
first is inert until phase 7 of docs/notes/80-s6-migration.md and is 2.4MB of
duplicate until then; the second is route 1 of the bootstrap in
docs/notes/85-packaging.md -- the first payload that can install its own
successor.
VERIFIED by building the payload from the parent commit and from this one, on
the synthetic stock fixture with the real vendored assets:
files removed 0
content changes among the 2778 shared files 1 (.install-manifest,
which lists the payload)
mode changes 0
symlink changes 0
added klipper/**, var/lib/opkg/** (63), bin/opkg, and the
www/mainsail/.version the previous commit stopped deleting
two builds in a row byte-identical
bin/verify.sh passes every check including the ship boundary; qa-static 255;
make test 9 gates, 0 failed; test-py 181.
test/run-tests.py builds the feed as the first step of the fixture lane, into
a PKG_FEED of its own -- sharing work/packages let the MOD_VER=ci build rename
every dated package and the prune delete the rest on the way past, measured as
44 becoming 43.
qa/static/test_ipk.py's share-one-build test is inverted rather than deleted:
the property was "one build, two vehicles" and is now "one vehicle", so what
it asserts is that nobody reintroduces the second path -- no recipe run from
patch.sh, no pkg_out, no work/pkg, and opkg rather than pkgs/ipk-install,
which stays the printer's hand-repair tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
a1b4ea6d60 |
The Makefile said the release path did not need the feed; python's seven build deps disagreed
`make packages` carried the comment "NOTHING ON THE RELEASE PATH DEPENDS ON
THIS", and it was false when it was written. pkgs/3rdparty/python declares
PKG_BUILD_DEPENDS="openssl sqlite zlib libffi xz bzip2 expat"; pkg_deps
resolves those by unpacking their .ipk out of work/packages and dies when they
are absent; bin/patch.sh section 5c runs that recipe and builds none of the
seven itself. A cold checkout has never been able to run `make build`.
What it did instead was fail two hundred lines in, inside a recipe, with a
message naming a .ipk path -- so the answer ("run make packages") had to be
worked out rather than read. patch.sh now checks the feed before it touches
anything and names the command.
Counting .ipk rather than testing for the directory, because an existing but
empty work/packages is the state a cleaned tree is in, and that is the one
that produced the confusing error.
No payload change: the tree is diff -r identical either side of this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f67e7a269f |
ls pkg/ was thirty-eight entries and four of them mattered
pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and
carry no files of this repo move down a level into pkgs/3rdparty/. What is
left at the top is the four components a person actually edits: anvil-core,
klipper, moonraker, helixscreen.
THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory
rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day
somebody patches zlib nobody would agree which side it belongs on, and the
answer would be a rename touching CI and docs decided by taste. "Does it carry
files of ours" is a fact about the tree: it is exactly whether the recipe has
a payload/, prog/ or seed/, checkable in one assertion, and
test_recipe_layout.py now demands the move in both directions -- a 3rdparty
recipe that grows one of the three must come up, and a top-level recipe with
none of them must go down, or the top level slowly refills and the split stops
meaning anything.
ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id
to its directory and nothing else spells either path -- not a recipe, not
bin/build-packages.sh, not a test. Names stay unique across both levels
because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare
name; a duplicate would resolve to whichever level is searched first and
build the wrong thing silently, so that is a test too.
Recipes now source `. ./bin/common.sh` instead of counting `..` from $0.
Depth-counting would have needed two spellings for two levels, and it was
never buying anything: `. pkg/lib.sh` on the next line already required the
working directory to be the repo root.
A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files()
expanded a list of globs and returned whatever matched, so "pkgs/*/build.sh"
quietly stopped covering thirty-four recipes the moment they moved -- the lane
stayed green, and the only visible sign was a test count dropping by 34 in a
run nobody was counting. _files() now refuses a pattern that matches nothing.
That is the same failure the ash-compatibility check had a guard for and the
syntax check did not, which is why the guard reads the way it does.
AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of
.py helpers wholesale, so any test that imported one left a __pycache__ beside
it -- gitignored, invisible in a diff, copied by cp -a like anything else. The
package's contents depended on whether pytest had been run on the machine that
built it: two builds of one commit, different sha256, cause nowhere in the
tree. What it shipped was worthless as well as unstable, compiled by the build
image's python 3.11 for a printer running the 3.13 this repo cross-builds. The
sweep is in pkg_ship now, once, and the two hand-written copies in
pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe
that stages a tree of .py files, and the two recipes that had hit it had each
solved it locally while the third, which had not, was the one shipping.
Measured:
* the rename changes no package bytes. Same cached build trees, feed built
with the diff stashed and unstashed: all 41 .ipk match.
* with the bytecode sweep, anvil-core returns to exactly the bytes the
previous commit produced.
* no package in the feed contains a .pyc or a __pycache__.
* make qa-static: 247 passed (236 before this work + 11 layout tests).
* make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the
synthetic stock package. make test-py: 181 passed.
Not explained: anvil-python-dev produced two different sha256 from an
UNCHANGED work/pkg/python at two points today, under code that was identical
both times -- so it is not reproducible from a cached tree, and it is not this
commit's doing (the stashed and unstashed builds above agree on it). Worth a
look; the dev split in bin/build-packages.sh is where to start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
de3e369f07 |
Comments that explained the code's past, not the code
Across 46 files the comments had accumulated a second job: narrating what
each mechanism replaced. Some of that narration outlived what it described,
which is worse than verbose -- it is wrong.
Removed, because the thing referenced is gone:
- pkg/lib.sh's example recipe called pkg_autotools, and pkg/bzip2 pointed
at pkg_make for the shape it uses. qa/static/test_ipk.py asserts both
names stay deleted.
- bin/common.sh explained S6_STAMP, PY_BUILD and PY_STAMP at length. None
of the three is defined anywhere; mips_abi_gate still said it walked
$PY_BUILD, and that section 5b cross-builds with a musl toolchain that
left with the second libc.
- pkg/lib.sh and bin/common.sh had section 5c carrying its own toolchain
setup and CPython cross-build. It runs recipes and stages the output.
- pkg/zlib's header claimed a ./configure carve-out enforced by name in
test_ipk.py; the body sets PKG_CONFIGURE_AUTO=0 and the test permits no
exceptions.
- case-supervisor.sh was fed out of work/.s6, which is now three recipe
outputs merged by the Makefile.
Shortened everywhere else to the logic that is actually hard: the -mnan=2008
driver baking, exporting CC because OpenSSL reads it where autoconf reads
--host, the ar/ranlib determinism split and why the cross ranlib cannot do
it, the sysconfig cross trick, busybox start-stop-daemon returning before the
process is dead, and the manifest property an update has to keep. The
measurements stay; the archaeology goes.
No code changed. qa/static: 111 passed, unchanged from before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8cc92e8266 |
A wheel could only be built while three scratch trees happened to coexist
That is why bin/patch.sh section 5c was 825 lines and why a Pillow bump
rebuilt CPython. Cross-building an extension module needs the untrimmed
staging tree (INCLUDEPY points into it), the private sysroot of static C
libraries (pillow links zlib, cffi links libffi) and an x86-64 build-python
of the same version -- and all three existed only mid-section, so the wheels
had to be built there too, on a cache with two stamps that invalidated each
other.
Each of those three is now a durable thing: anvil-python-dev carries the
headers, pkg_deps fills each recipe's own sysroot from the feed, and
pkg_buildpython caches one build-python for all nineteen recipes. So the
coupling goes away with them -- `make packages PKG=python-pillow` builds
Pillow.
pkg/python -> anvil-python + anvil-python-dev
pkg/python-<name> x18 -> anvil-python-distro, anvil-python-pillow, ...
Three verbs in pkg/lib.sh, shared by all nineteen rather than one per awkward
project: pkg_buildpython provides the build-python, pkg_pytarget points its
sysconfig at the target headers in the sysroot, pkg_pywheel builds one wheel
and unpacks it into the staging site-packages. greenlet's gcc 7.2 designated
initializers, lmdb's LMDB_FORCE_CPYTHON and pillow's build_ext options are
recipe lines and a PKG_PY_SETUP_ARGS knob, not three more verbs.
Eighteen versions that were invisible are now in the feed index, anvil-moonraker
declares the thirteen packages it imports, and a printer that does not want
gcode thumbnails can leave Pillow uninstalled.
PY_STAMP, PYPKG_STAMP and PY_BUILD are gone: pkg_stamp derives the key from
PKG_BUILD_DEPENDS, and bin/fetch-assets.sh asks pkg_needs instead of
re-deriving a stamp bin/patch.sh also derived. Two of its four toolchain
conditions were that second spelling.
Measured:
* 40 packages; 959 site-packages files, none owned by two packages
* 12 mipsel extension modules -- the count the old gate demanded
* every Depends resolves inside the feed (24 declare one, 0 unsatisfiable)
* two cold builds byte-identical, with work/pkg, work/.py-host and the
unpacked toolchain deleted in between
* 233 static tests, 4 new: the PYPKG_LIST/recipe agreement both ways, no
version pinned twice, patch.sh builds every package, and no Depends
naming a package no recipe produces
Not verified here: the replica suite. It needs the stock FlashForge package,
which the packaging lane deliberately does without.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
94a03800f7 |
Perl's || is not //, so SOURCE_DATE_EPOCH=0 meant "use the clock"
Six more recipes -- openssl, sqlite, libffi, xz, bzip2, expat -- and one
build environment instead of two. Twenty packages.
ONE VERB, NOT FOUR. There was pkg_autotools with eleven users, a
hand-written ./configure in pkg/zlib permitted by name in the test that
forbids it everywhere else, a pkg_make for bzip2, and a pkg_build for
OpenSSL. Three mechanisms with one user each is three exceptions wearing
function names. Projects do not differ in four ways, they differ in four
SETTINGS of the same three steps, so pkg_build takes knobs -- which
configure, whether it takes --host, what to make, what to install -- and
every recipe goes through it, zlib included. The test is now absolute: no
recipe runs a configure script, no exceptions, and it knows about
./Configure as well as ./configure.
THE TOOLCHAIN REACHED PROJECTS TWO WAYS AND THAT WAS THE REAL BUG. Only
PATH was set, so autoconf found the cross compiler through --host and
anything that reads $CC did not. OpenSSL takes a target name rather than
--host: it used the build machine's gcc and produced 936 x86-64 objects
inside the container. mips_abi_gate caught it at the package boundary.
pkg_toolchain now exports CC/CXX/AR/RANLIB/STRIP/NM/OBJCOPY/OBJDUMP/LD as
well, so every project gets the toolchain the same way.
The same failure in miniature: pkg_build dropped its trailing arguments
when a project had no configure step, so bzip2's CC= went nowhere and
libbz2.a came out x86. Arguments now go to whichever step consumes them.
THE BUILD ENVIRONMENT IS THE CONTAINER, and it was not. Eleven recipes had
been built on the host and the twelfth needed perl's IPC::Cmd, which Debian
bookworm has and this host does not -- so the container was about to become
"the environment for the awkward ones". Two builds, two results: the
Moonraker and HelixScreen packages differed between host and container
because GNU tar restores a directory's exact mode as root and applies the
umask otherwise. Packages now normalise modes at the boundary (0755 dirs,
0755 executables, 0644 the rest), the way ownership and timestamps already
were, so what a package contains no longer depends on who built it.
And the build lane runs as the invoking user, not root. Root-owned files in
work/ are what make `rm -rf work/pkg` fail and running the build outside the
container tempting -- which is how a project ends up with two environments
in the first place.
OpenSSL was then the one package that still would not reproduce: two builds
differed in libcrypto-lib-cversion.o, which carries a "built on:" string.
It does honour SOURCE_DATE_EPOCH -- util/mkbuildinf.pl reads it -- but as
`$ENV{'SOURCE_DATE_EPOCH'} || time()`, and Perl's || treats 0 as false. The
one value we passed was the single value that silently falls back to the
clock. The default is 1 now.
Twenty packages, byte-identical across two cold container builds. 172
static tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6e97f946b0 |
execline was 2.1MB unstripped against musl; it is 1008KB and mandatory
Eight recipes: skalibs, execline, s6, s6-rc, mainsail, moonraker,
helixscreen, anvil-core. bin/patch.sh loses 183 lines and stops being a
build system for any of them.
The supervision stack moves to the Ingenic glibc toolchain and links libc
dynamically, so the musl toolchain -- the only second libc in this tree --
is deleted along with MUSL_TOOLCHAIN_*, S6_STAMP and S6_BUILD. The
measurement that chose musl compared two STATIC builds; dynamic glibc was
never in it. Measured now: s6's shipped tree is 696K against ~930K for the
static musl one, and the whole stack including execline and s6-rc is 2.0M.
Three things that were not guesses:
* s6 and s6-rc do not link against glibc 2.29 without -lpthread. skalibs'
pthread_mutex_tailock reaches pthread_mutex_timedlock, which glibc kept
in libpthread until 2.34. Upstream never sees it and musl never had the
split. PKG_MAKE_ARGS is the lever, because s6's PTHREAD_LIB is wired
only to --enable-nsss and its Makefile appends LDLIBS last.
* s6 ships 21 binaries, not 13. s6-rc's generated scripts exec s6-sudo,
s6-sudoc, the s6-ipcserver chain, s6-sudod and s6-fdholder-daemon --
found by running a real up/down cycle with a restricted PATH, not by
reading the documentation.
* --enable-static-libc has never been a skalibs option. Its configure
warns on unknown flags and patch.sh sent that to /dev/null, so it was a
no-op for as long as section 5b existed.
execline ships whole. A subset picked from today's services fails the day
somebody adds a pipeline, at boot, naming a program nobody chose to leave
out. tools/supervisor/README.md and 80-s6-migration.md said it was not
needed; both now say why that stopped being true.
PY_STAMP was spelled in three places and happened to agree -- the same
shape as the s6 stamp bug, caught before it inverted. It is in common.sh
now, and the test that guarded S6_STAMP by name is rewritten around the
rule so it covers both and dies with neither.
HelixScreen's binaries reach mips_abi_gate for the first time: patch.sh
never gated them. They pass, 3 objects, nan2008/o32/mips32r2.
anvil-core carries a content hash in PKG_STAMP_EXTRA because its version is
a release date: pinned, an edit to payload/ would read as current; unpinned,
it rebuilds daily. The version says which release, the stamp says what is
in it. anvil.conf stays out -- it is user state that run-append.sh
preserves, and a package member is overwritten on every upgrade.
Twelve packages, byte-identical across two cold builds. 152 static tests.
The replica suite has NOT been run: docker is unavailable here, and the s6
change ships without its behavioural gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c8e33a5604 |
zlib was compiled twice because it belonged to whoever needed it first
pkg/opkg/build.sh unpacked zlib, built it into a private sysroot, unpacked
libarchive, built that against it, and then built the one binary it shipped.
Section 5c did the same with its own zlib, from the same pinned tarball with
the same flags. Neither copy could see the other, because a library built
inside somebody else's recipe has no version, no package, and no way to be
reused.
One build.sh now produces exactly one package. A recipe names what it builds
against in PKG_BUILD_DEPENDS and pkg_deps unpacks those out of the feed with
opkg-unbuild -- upstream's inverse of opkg-build, from the checkout that
already provides both. That dissolves the bootstrap: nothing needs a working
opkg in order to build packages, so opkg is an ordinary recipe that waits on
nothing except its own dependencies, and no qemu enters the build path.
Building against the package rather than the build tree is the part worth
paying for. A package that forgets a header fails the next recipe's configure,
on the build that produced it.
One libc. skalibs, s6 and opkg were musl because a STATIC glibc s6 tree
measured 73MB -- a static-to-static comparison that never considered dynamic.
Two libcs also meant two of every library both worlds wanted, which a feed
cannot express without lying about one of them. opkg moves to the Ingenic
toolchain and links libc.so.6 dynamically with zlib and libarchive inside it;
it got smaller (212K against 332K). Section 5b stays musl until it becomes a
recipe too, and is now the only place left where one script builds two
libraries.
Found on the way, each of which had been true for a while:
* ar archives are not reproducible. Every member header carries an mtime and
the builder's uid/gid, and the symbol index carries another timestamp --
none of it reachable by SOURCE_DATE_EPOCH or opkg-build's -o 0 -g 0,
because it is inside a file the tarball merely contains. objcopy -D fixes
the members; the cross binutils is 2.27 and its ranlib -D writes a fresh
current timestamp into the index instead of zeroing it, so the index is
rewritten with the build image's ranlib.
* The ABI gate accepted exactly 0x70001405 and 0x70001407, both measured from
linked binaries where crt objects set NOREORDER. Object files need not:
libarchive's xxhash.o is 0x70001406, identical ABI, and the gate called it
wrong. It also read one Flags: line, while readelf -h on an archive prints
one per member -- 122 for libarchive.a. It now masks the three non-ABI bits
and checks every header.
* bin/fetch-assets.sh compared work/.s6/.version against two fields where
patch.sh wrote three, so the test could never be false and 71MB was
re-hashed every run. The comment above it described a fast path that had
never once been taken.
Gate: 127 static tests (29 packaging), and `make packages` from a cold cache
producing four byte-reproducible .ipk files plus the index. The end-to-end
install under qemu has NOT been re-run since opkg went dynamic -- a dynamic
binary needs qemu -L <sysroot>, and the command is written down in
docs/notes/85-packaging.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5025aac171 |
opkg-utils already builds ipks, and opkg is the second package that proves lib.sh
Two corrections to the packaging PoC, both from review.
FIRST: THERE WAS ALREADY A TOOL. bin/mkipk.sh is deleted. opkg-utils ships
opkg-build and opkg-make-index -- 423 lines of POSIX sh and a python indexer,
upstream, maintained -- and this repo now drives them instead of re-deriving
the format. What is left of build-packages.sh is the part that is genuinely
ours: laying out the tree opkg-build packages. The hand-rolled version worked,
and it was still us reimplementing member ORDER, control validation and the tar
flags that make a build reproducible, all of which fail as "a package that
inspects fine and installs nowhere".
opkg-utils is PINNED BY COMMIT, uniquely in this repo, and not by choice: it
publishes no release tarball anywhere. Upstream's cgit has snapshots disabled,
downloads.yoctoproject.org carries opkg but not opkg-utils, and the GitHub
mirror last saw a commit in 2012 with no tags. A git sha is a hash of the whole
tree -- the same guarantee every sha256 here gives -- and fetch-assets.sh
verifies HEAD against it and refuses a dirty checkout. Both were tested.
SECOND: A SECOND PACKAGE, AND IT IS opkg ITSELF. pkg/lib.sh now holds the part
of a cross-build that is the same every time -- the two toolchains, the gcc
wrappers that bake -EL -mnan=2008 into the DRIVER because autotools link lines
do not all forward CFLAGS, the ABI self-test of those wrappers, autotools,
the build cache, and pkg_ship. Both recipes fit on a screen and share exactly
that file.
opkg is the honest test of it: different toolchain (Bootlin musl, not Ingenic
glibc), different libc, different link mode (static, not shared), and a
dependency chain three builds deep (zlib -> libarchive -> opkg) against
libsodium's none. It is static musl for the same reason s6 is -- a standalone
binary that must match nothing -- so no libarchive.so or libz.so ships beside
it. qa/static/test_ipk.py fails any recipe that sources its own toolchain,
spells its own -mnan=2008, or calls ./configure directly; the gate was checked
by breaking a recipe on purpose. zlib is the one named exception, because its
configure has never accepted --host.
THE LOOP IS CLOSED ON THE TARGET. The opkg this repo builds -- 775KB, static,
nan2008/o32/mips32r2 -- runs under qemu-mipsel and installs both our packages,
including itself, then lists their files and removes them leaving nothing
behind. The opkg it installed then runs and reports its version. Its status
database landed at /usr/data/anvil/var/lib/opkg, which is where pkg/ipk-install
writes, because both agree rather than by coincidence.
Three findings paid for by building it, all in the note:
* opkg BAKES ITS STATE DIRECTORY IN AT COMPILE TIME -- --offline-root does
not move it. Built with any other prefix it believes nothing is installed
and reinstalls the world. Third instance of this trap here after s6's
libexecdir and execline's shebangdir.
* opkg 0.6.3 does not build against musl: two files call basename() without
<libgen.h> and get away with it only under glibc. 0.7.0 fixed both, which
is the difference between a pin bump and carrying a patch.
* libtool SWALLOWS -static. A program configured LDFLAGS=-static comes out
dynamic and the only symptom is a NEEDED entry nobody reads. -all-static,
at make time, is the flag that means it -- hence the explicit check.
Both packages are byte-identical across a full cold rebuild. The musl toolchain
fetch now also triggers on a stale work/.opkg, and its cache key lives in
common.sh so the fetcher and the recipe cannot spell it differently -- the
hazard already written above pypkg_stamp.
Also fixed: two `# shellcheck disable=... -- prose` directives were malformed,
which made shellcheck abandon those files rather than check them. A gate that
examined nothing, in the file arguing against those.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
1c89a9cd80 |
Neither Alpine nor OpenWrt has a binary we can install. That picks opkg.
The evaluation asked for apk vs opkg. The comparison turns on one fact that makes most of the usual argument irrelevant here: Alpine has no MIPS port at all, and OpenWrt's mipsel_24kc is musl against this printer's glibc 2.29. So no upstream repository is reachable either way, every package is cross-built here regardless, and a package manager buys packaging and installing -- not software. That reduces the choice to which format is cheapest to PRODUCE on every build, and .ipk wins by a distance: ar + two tarballs + a three-byte version file, no keypair, no Alpine host, no abuild. apk would need the signed three-concatenated-gzip layout implemented by hand or abuild imported into a Debian build image, paid per recipe forever, for a format with an in-progress v2->v3 transition and no users on our architecture. The full table, the two honest points for apk, and the phased plan are in docs/notes/85-packaging.md. The proof of concept is libsodium, end to end and measured rather than described: make packages -> libsodium_1.0.20-1_mipsel_xburst2.ipk + a feed index Verified against real opkg 0.6.3 built from source -- offline-root install, both symlinks preserved as symlinks, list-installed, files, remove leaving nothing behind. dpkg-deb reads it too; the container format is Debian's. A full recompile from a cold cache produces a byte-identical package, so a pin bump is reviewable as a hash diff. libsodium is now compiled ONCE. patch.sh section 5d's build block moved to pkg/libsodium/build.sh and patch.sh runs it, so the tarball's copy and the packaged copy cannot become different libraries wearing one version number -- asserted, not trusted. MODDIR, PY_HOST, PY_TOOLCHAIN_DIR and mips_abi_gate moved to common.sh for the same reason: a package is a shipping vehicle and gets ABI-gated like one, on a machine that may never run patch.sh. Nothing ships. make build is unchanged and still produces one anvil.tar.xz; make packages needs no stock FlashForge package, which is the point -- a packaging gate that only runs where the proprietary firmware is stops being a gate. Two findings worth more than the code. opkg bakes its state directory in at COMPILE time, so a phase-2 opkg must be configured --prefix=/usr/data/anvil or it comes up believing nothing is installed -- the s6 libexecdir trap one layer up. And opkg 0.6.3 will not configure without libarchive, which is what phase 2 actually costs; the opkg binary itself is 33KB. So pkg/ipk-install walks the ar headers with tail/head instead: this busybox is 1.31.1 built small, with no timeout, nc or ionice applet, and betting an installer on an ar nobody has checked for is how an update fails at the last step. qa/static/test_ipk.py is 18 tests that build their own package from a synthetic tree -- no toolchain, no firmware -- so the lane keeps the property qa/conftest.py insists on and can never be skipped into looking green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
16d00be525 |
case-services.sh is 360 lines and one exit code. That contract is 44 tests now.
test/ has two test frameworks. run-tests.py + ffsim is 1,657 lines reimplementing what pytest already does -- skip accounting, gating, reporting, then parsing pytest's own JUnit XML back out -- and it calls pytest as gate #5 of about twenty. Underneath it, 6,280 lines of case-*.sh execute inside the qemu chroot and report 13 booleans between them; case-moonraker.sh is 1,059 lines that collapse to one pass/fail. Nothing runs in parallel and nothing can be selected: each replica gate is its own `docker run --privileged`, re-registering binfmt and re-assembling the mount layout, thirteen of them in sequence. All three follow from one decision -- THE REPLICA CONTAINER IS TREATED AS A TEST RUNNER INSTEAD OF AS A FIXTURE. That is what pushes assertions into ash, which yields one bit per container, which then needs a bespoke framework above pytest to aggregate the bits. qa/ is the replacement, built beside test/ rather than inside it, with two lanes and one framework. Both suites run in CI; test/ is still the release gate; nothing has been deleted. docs/qa-migration.md has the order the rest moves in, and the rule: a case script is deleted only once its replacement here has been green, so there is never a window where coverage drops. THE MACHINE IS UNCHANGED. qa/ reuses test/integration/printer/'s Dockerfile, entrypoint.sh, assemble.sh, binfmt.sh and seed-prog.sh as they are, and its `docker run` argv is deliberately ffsim's, minus --rm and plus -d. While both suites run they have to be testing the same machine or neither proves anything about the other. The only new trick is how the container is held open: entrypoint.sh ends by running the case script it was given, and the one it is given here touches a marker and sleeps, so setup completes exactly as it always has and the container simply does not exit. The marker is also the readiness signal, because setup ranges from under a second to over a minute and a constant would either waste the fast case or fail the slow one. TESTS RUN AGAINST A MACHINE THE REAL INSTALLER PRODUCED. The `printer` fixture is a replica with the package installed by the printer's own app_startup.sh, off a genuine FAT filesystem on /dev/sda1. The first version of this tree did what case-services.sh does -- copy payload/*.sh and payload/init.d/S* into $MODDIR by hand -- and two things were wrong with it. It is a second implementation of the install, so everything downstream asserted against a layout the harness built rather than one the installer produced, and the real installer could have broken without a single test noticing. And it can only place what is in payload/, so everything the BUILD produces was missing: s6 is a build output that bin/patch.sh stages into the package, so a hand-placed payload has no supervisor at all, which is why the tests had to carry a stand-in scanner and gate one assertion on whether it was in use. Installing for real fixes both and deletes the stand-in. Hand-placed, /usr/data/anvil held anvil-env.sh, anvil-service.sh, anvil.conf, init.d and etc/s6. Really installed it holds VERSION, backup, bin, config, config-installed, helixscreen, lib, libexec, moonraker, nginx and www as well -- bin/lib/libexec being the cross-built s6 and CPython 3.13. The file is SHORTER than the case script it replaces while reporting 44 results instead of one. case-install.sh's header records this same lesson from the first time somebody learned it: "An earlier version of this file replayed app_startup.sh by hand -- which meant a bug in our reading of it could never be caught." The install is the machine's own shell under qemu, so it is baked into an image once per package and cached under a tag derived from the package's md5 -- build-printer-image.sh's trick for the stock baseline. Rebuild the package and the tag changes, so the next run bakes again instead of testing yesterday's build; deliberately not a hand-written staleness check, because those are the ones that eventually get it wrong. A MISSING TOOL OR RESOURCE FAILS, and there is no flag. This tree briefly had --strict-skip=<lane|all>, which was ALLOW_SKIP inverted -- and ALLOW_SKIP's defect was never the direction it named things in, it was that the decision lived in the CI invocation. An accept-list has to be edited whenever a gate moves; a strict-list has to be remembered whenever a job is added; both degrade silently when the command line is wrong, which is the same disease as the printer-sim job that was gated on an unset secret and never ran once in its existence. So the verdict lives with the knowledge: no shellcheck, no pyflakes, no docker, no daemon, no replica image, no built package -- each fails where it is needed, with the fix in the message. There are no skips in this suite. MEASURED, on monstrofil/creator5-printer:latest, both suites agreeing on the same machine: 131 tests in 40s, all green; qa/static alone 1.3s with no docker; one assertion selected by node id 2.9s, against the whole 360-line case being the smallest unit test/ can run. Granularity checked by injecting the regression case-services.sh exists to catch -- deleting SVC_EXTRA_VERBS="|force-start" from S70klipper -- which failed one named test, printed the usage line that came back, left the other 41 reporting, and took 4s to re-run alone. Two bugs the port found in its own harness, both only visible by running it. bake.sh shipped without its executable bit, so the lane died at container init with 43 errors all saying "could not install the package" -- a file mode wearing an installer failure's clothes; test_action_scripts_are_executable now guards it, the same check the payload's own s6 run scripts already have. And an ELF magic assertion used `od -An`, which the printer's busybox od does not have -- the replica catching host-vs-printer drift in the test code itself, which is the entire reason it exists. Also here: qa/static/test_probes.py, which has no counterpart in test/. Printer.ps() and Printer.listening() are parsers, and a parser is where a harness goes quietly wrong -- a listening() that never matches makes every readiness assertion pass by the absence of contradiction. Neither failure looks like a failure; they look like a green suite. So they are tested against captured output, in the static lane, with no docker involved. Two things found while surveying, neither fixed here. case-pyext.sh is 567 lines with no gate in gates.py, no Makefile target and no CI step: it has never run. And case-mcu-bringup.sh:4 says the handshake is "tested elsewhere against a pty" -- there is no pty anywhere in the repo, and every documented past bug in ff_mcu_bringup.py lived in the code that comment excuses from testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
441dd80d56 |
Merge svc-nice-20260825: moonraker and the camera yield to klippy
NICE_MOONRAKER (5) and NICE_CAM (10) in anvil.conf, baked in from
config.env like the MOD_* switches. Stock ran neither moonraker, nginx
nor the streamer -- both lines are commented out in FlashForge's own
start.sh -- so klippy had this two-core SoC to itself; the mod runs all
three, and a host that misses Klipper's step-generation deadlines does
not print slowly, it stops (MCU "Timer too close"). svc_start_daemon
passes the value through as start-stop-daemon -N; case-priority.sh
proves both that mechanism and the `nice` applet actually work on this
printer's busybox (1.31.1, no ionice), reading the value back out of
/proc/<pid>/stat rather than grepping the scripts for "-N".
Two conflicts, both from the same cause: this branch's s6 migration
landed after svc-nice-20260825 branched, so its patch only reaches the
code that predates supervision.
* payload/init.d/S62moonraker and S65camera: svc-nice's changes target
the OLD unsupervised start paths (a direct svc_start_daemon call for
moonraker, a hand-rolled `while true` respawn loop for the camera).
Moonraker still has that path (start_unsupervised(), the MOD_S6=0 /
no-scanner fallback) and keeps the -N there. The camera's equivalent
loop is gone entirely under s6 -- start() now hard-fails without a
scanner -- so that half of the patch is dead code under this
architecture and was dropped rather than merged in.
* Extended past what the patch reached, for the reason above: s6
supervision is the PRIMARY path for both services now, not a
fallback, so leaving the nice values reachable only from the code
path most printers never take would ship the feature dead. Added
`nice -n "${NICE_MOONRAKER:-0}"` / `"${NICE_CAM:-0}"` in front of the
exec in payload/etc/s6/moonraker/run and payload/etc/s6/camera/run --
`nice` execs its target in its own place (verified: this is still
one process, and $FF_PYTHON / ./mjpg_streamer is still what s6 ends
up holding the pid of, exactly as the `exec` comments already in
both files require), re-applied on every s6 respawn since
s6-supervise re-execs the whole run script each time.
Everything else -- anvil-service.sh's svc_start_daemon, the Makefile and
config.env.example plumbing, case-priority.sh itself -- merged cleanly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ee665d08ba |
Moonraker on 3.13 under s6, through the real boot path, for real
Every earlier gate proved one link in the chain: components import ( |
||
|
Oleksandr Shyshatskyi
|
05b24f7c56 |
Moonraker and the camera yield to klippy
Stock ran neither moonraker nor nginx nor the streamer -- both lines are
commented out in FlashForge's own start.sh -- so klippy had this two-core
SoC to itself. The mod runs all of them. A host that misses Klipper's
step-generation deadlines does not print slowly, it stops: the MCU reports
"Timer too close" and shuts down mid-job.
So the services around klippy now start at a nice value instead of
competing with it on equal terms. NICE_MOONRAKER (5) and NICE_CAM (10) in
anvil.conf, baked in from config.env like the MOD_* switches and editable
on the printer afterwards; 0 restores the old behaviour exactly.
The two take different routes because they start differently. Moonraker
goes through svc_start_daemon, which now passes start-stop-daemon's -N when
SVC_NICE is set -- not a `nice` wrapper, because --exec is what
start-stop-daemon matches and records, and wrapping would put the nice
binary there instead of the program. The streamer is started by its own
respawn supervisor, so it takes the nice applet, re-applied per respawn.
Modest values, and deliberately so: this is insurance, not a fix. klippy is
already ahead on both counts -- klipper_pri.sh puts its main thread on
SCHED_FIFO 50, which outranks every nice value there is, and its serial
reader thread calls os.nice(-20) itself in serialhdl.py's _bg_thread
(FlashForge's own change, not upstream). These numbers only matter for
whatever klippy thread is neither of those, and for the case where the chrt
or the nice did not take -- which the log will say:
grep -i nice /usr/data/logs/printer.log
Nothing here addresses memory or eMMC contention. This printer's busybox
has no ionice applet at all, so there is no knob for the second one.
case-priority.sh is the gate, and it rides along with the services one
because it is the same subject from the other side. It is not a grep for
"-N": whether the option is honoured is a property of THIS busybox (1.31.1,
and the missing ionice is why that is worth measuring), so every assertion
reads the nice value back out of /proc/<pid>/stat on the printer's own
shell. Verified there: -N 7 lands as 7, unset and 0 both land as 0.
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
813973620c |
The payload carries a CPython 3.13 with a working sqlite3
Nothing runs on it. FF_PYTHON still points at FlashForge's 3.8.2, and the
comment beside it now says at length why changing that line is a one-line way
to break the whole userspace: klippy, Moonraker and ff-startup.py import
third-party C extensions -- tornado, lmdb, cffi, greenlet, pillow, libnacl --
that exist on this printer only as mipsel objects built against 3.8. None has
been cross-built for 3.13 yet. The interpreter ships ahead of the switch for
the same reason s6 did: it puts the artefact on real printers where it can be
measured, and it leaves the switch a one-file change rather than a build
change and a boot change at once.
Why it exists at all: the Moonraker pin is frozen at commit 9d0d09d, December
2023, for exactly one reason -- this firmware's Python has no _sqlite3 and
there is no libsqlite3 anywhere on the box, while Moonraker v0.9.0 moved its
database off lmdb onto sqlite. That is a cost paid at every release. The gate
proves the premise rather than assuming it: the stock 3.8.2, given the library
path it needs to start, imports json and zlib fine and cannot import sqlite3.
Built by bin/patch.sh from eight pinned sources with the Ingenic toolchain
that already builds c_helper.so -- gcc 7.2, glibc 2.29, cross to
mipsel-linux-gnu. Not musl, and this is not a preference: a musl interpreter
cannot dlopen a glibc c_helper.so, which is exactly how klippy loads it.
Three things about the build are load-bearing and all three were found the
hard way:
* -EL -mnan=2008 must reach the LINK lines, not only the compile lines.
They are baked into PATH wrapper scripts around the gcc driver rather than
passed as CFLAGS, because trusting eight separate build systems to forward
flags to their linkers is how you get legacy-NaN objects that the printer
refuses to load.
* _sqlite3 disappears silently without LIBS="-latomic -lm". A static
libsqlite3.a carries no DT_NEEDED on libm, so configure's
sqlite3_bind_double probe fails to link and CPython records the module as
"missing" -- a probe failure, so nothing in the build output says why. The
build now hard-fails when _sqlite3 is absent rather than shipping an
interpreter whose whole purpose is gone.
* OpenSSL's linux-mips32 target hardcodes -mips2, which collides with this
toolchain's -mfp64. -mips32r2 as a user cflag lands after the target's and
wins.
Built --disable-shared, unlike FlashForge's own 3.8.2. Static libpython and
static dependencies mean the shipped tree contains no shared object of ours at
all: nothing to version-skew, nothing that can resolve to a /usr/prog copy by
accident, and no new mandatory LD_LIBRARY_PATH entry -- which matters here
specifically, because removing /usr/prog version strings from anvil-env.sh is
the point. The usual objection to static, that C extensions cannot find the
interpreter's symbols, was tested rather than assumed: an out-of-tree
extension cross-built against the staged include/ imports and runs on the
printer with no libpython mapped.
It installs into $MODDIR itself, --prefix=/usr/data/anvil, the same prefix
root s6 uses: bin/python3.13 beside bin/s6-svscan, stdlib in lib/python3.13
beside lib/. One install, not two trees.
That creates a hazard worth naming, because anvil-env.sh PREPENDS $MODDIR/bin
to PATH so s6-svscan can find s6-supervise. `make install` also writes
bin/python3, and a shipped bin/python3 would then sit in front of
FlashForge's for every process that sources anvil-env.sh -- silently handing
klippy and Moonraker an interpreter with none of their extensions. So the
build deletes it and asserts bin/ holds exactly python3.13, verify.sh fails a
package carrying bin/python3, and the gate asks the shell what `python3`
resolves to on anvil-env.sh's own PATH. It answers
/usr/prog/Python-3.8.2/bin/python3, while python3.13 answers ours.
About 28MB and 600-odd files, which takes the install manifest from 70 entries
to 728 -- case-upgrade.sh still passes, which is the check that the manifest
deletion loop scales.
The build image grows a host gcc, which looks redundant beside two
cross-toolchains and is not: cross-building CPython needs a build-python of
exactly the same version to freeze modules and byte-compile the stdlib, so
patch.sh builds an x86-64 3.13 from the same tarball first.
tools/python/ keeps the measurement build and what it cost, as
tools/supervisor/ does for s6. Two gaps recorded there rather than lost: the
printer has NO CA certificates -- /etc/ssl/certs exists and is empty, so TLS
negotiates and verifies nothing, which is true of the stock interpreter too --
and libatomic.so.1 is a new runtime dependency, present on the rootfs,
measured, and removable by linking it statically if that is ever wanted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
bcd38fae28 |
Six of the ten library paths were never loaded by anything
ANVIL_LIBS handed every mod process ten /usr/prog library directories. Four of
them are load-bearing and six have no consumer at all, which was established
by measurement rather than argument: /proc/PID/maps of the processes the mod
actually runs, plus a DT_NEEDED walk over every ELF in
/usr/prog/{PROGRAM,bin,nginx,mjpg-streamer,klipper,module,modules,wifi},
/usr/bin, /usr/sbin, /bin and /sbin.
That walk found exactly ONE consumer of ffmpeg, x264, opencv, nim and libzip
between them: /usr/prog/PROGRAM/software/firmwareExe -- FlashForge's Qt
binary, which this mod replaces with a shell script. curl is the interesting
one, because it did have a Python consumer: pycurl, in site-packages, which
does not import on this firmware at all. "undefined symbol:
curl_global_sslset", a symbol that arrived in curl 7.56 while /usr/prog ships
7.55.1, and the shipped pycurl links libssl.so.1.1 where this curl links
1.0.0.
The comment that stood here is replaced rather than deleted. It argued the
list should be the union, because which subset a script needs is a property of
the component set in moonraker.conf and the user edits that file. The
measurement empties that defence: no moonraker.conf edit can reach a library
through a module that cannot be imported in the first place. What replaces it
says what was measured, and what would have to be true to put an entry back --
some process the mod runs has to map it, which is a measurement and not an
argument.
Each of the four that stay now names the failure it prevents, and openssl's is
worth having written down: it is NOT for `import ssl`. _ssl and zlib are
BUILTIN on this interpreter, statically linked into libpython, so
libpython3.8.so.1.0 itself carries DT_NEEDED on libssl.so.1.0.0 -- take the
directory away and the interpreter does not start, before it has read a line
of anyone's python. The rootfs has 1.1 only, a different soname that cannot
stand in.
case-libpath.sh is the measurement, kept as a gate and running in both
directions. Each of the four is removed one at a time and required to produce
a real failure -- two of them the interpreter refusing to start, one an
ImportError from ctypes, one an OSError from libnacl, which is the shape that
once made a missing library read as a component crash. And a probe is run
twice: on the four shipped entries, and again on the full old ten-entry path.
The second run is the one that carries the weight. With every directory
available it still maps none of the six, while it does map libffi and
libsodium -- so it was loading libraries, it just never wanted those.
Two of the case's checks SKIP in the replica, honestly and for the same reason
case-moonraker.sh skips the identical one: only the payload directory is
mounted, so FlashForge's 2022 Moonraker tree stands in and has no
moonraker.server. The component-import proof needs a checkout where patch.sh
has staged the mod's own tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
26ed883a39 |
nginx and the camera are supervised, and the camera says when it is ready
Two services move into the scandir S40s6 has been scanning empty. Each ships a directory under etc/s6 holding a `run` that execs the daemon in the FOREGROUND -- s6 supervises the process it spawns, so a daemonising one is supervised as a process that exits immediately and respawned for ever. The nginx gate's negative control is exactly that: without `daemon off;` the pids churn, 677, 689, 701, while s6-svstat says up. The init scripts stay and keep their verbs. `S60nginx restart` over ssh is in the docs and in people's hands; changing the supervisor is not a reason to change the interface. Both are now thin wrappers over the svc_s6_* helpers. $MODDIR/bin joins PATH in anvil-env.sh, and this is the phase's real find. s6 bakes its --prefix in at compile time, so it was reasonable to assume the binaries find each other wherever they are. They do not, and the two rules differ: s6-ftrigrd comes from the compiled-in libexecdir, but s6-svscan execs S6-SUPERVISE BY NAME OFF PATH, and s6-svc -w execs s6-svlisten the same way. Without it the scanner starts, stays up, answers its control socket and supervises nothing, saying so once in a log nobody reads. It could not have been caught one phase earlier: the scandir was empty, so no supervisor was ever spawned, so nothing ever needed the thing it could not find. Both service agents hit it independently. The camera is the reason s6 was chosen over runit, and it is the one that changed shape. Its hand-rolled `while true; do mjpg_streamer; sleep 3; done` is gone -- s6 does that. Its 30-second poll of /dev/video0 moved INTO the run script, which turns "give up" into "retry": the script exits non-zero, s6 runs it again, and a camera plugged in ten minutes after boot now comes up on its own, which the old poll could never do. Readiness is declared when the HTTP port is listening, not when the device node appears and not when the process forks, because the port is what nginx's /webcam/ proxy actually needs. The gate proves the distinction rather than assuming it: s6-svwait -U is asserted to be STILL BLOCKING while s6-svstat already says up, and only then to return when :8080 binds. Asserted together, so it cannot pass vacuously against a service that never started. Both services ship a `down` file, permanently. The MOD_WEB and MOD_CAM gates are read from anvil.conf at runtime while the payload is built once in CI, so the scandir cannot hold that decision; without `down`, MOD_CAM=0 would mean "the camera runs for a moment on every boot and is then shot". The init script's start is the runtime half. nginx needed one thing nobody predicted: SIGKILL orphans its worker, and the orphan keeps :80. The respawned master then cannot bind, retries five times and exits, and s6 loops for ever while an unsupervised leftover serves the UI. The run script sweeps those before exec'ing, reading /proc/PID/cmdline -- busybox ps truncates to terminal width and cut the path in half, and busybox kill refuses a negative pid, so process-group signalling is not available on this box. Both were tried. With no scanner the two degrade differently, on purpose. nginx falls back to starting directly and says so, because a printer that loses Mainsail because a SUPERVISOR did not build has lost it for an unrelated reason and ssh is the only door left. The camera refuses and explains, because a second respawn implementation exercised only on broken printers is how the services drifted apart in the first place. case-services.sh now installs the service directories too -- init.d without them is a layout no printer has -- and checks the half where the quiet mistakes live: a run script that arrived without its executable bit, which s6 reports only in its own log, and a service shipped without `down`. Its old "nothing is supervised yet" assertion was true for exactly one phase. bin/patch.sh stages etc/s6 with cp -a, because `down` and `notification-fd` are not scripts and a run without +x is a service s6 can never start. case-nginx.sh and case-camera.sh are registered as gates rather than left to be run by hand, and both Skip rather than pass when there is no built s6: what they test is s6 supervising our definitions, so a stand-in supervisor would be testing the wrong half. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ae99b8bff5 |
The supervisor is running, and it is supervising nothing
S40s6 starts s6-svscan on an empty $MODDIR/etc/s6. Nothing has moved into it; that is the next phase. What this buys is that the scanner is alive on every boot before any service could want it, so moving services later is one small change per service rather than one big change to the boot. The number is load-bearing the same way S60nginx's is: firmwareExe runs init.d/S* in filename order, so 40 puts the scanner ahead of every service that will later be supervised by it. Below 40 is left free for something that has to run before a supervisor. The sharp edge is that s6-svscan never returns -- that is its whole job -- and these scripts run in the FOREGROUND. Inline it does not slow the boot, it ends it: no wifi, no ssh, no Klipper, no screen, for ever, with no way in to find out why. So it is backgrounded through svc_detach, with exec inside the detached function so the pid in the pidfile is the scanner's own and not that of a shell that happened to start one. start() then waits up to 5s for the scanner to answer before claiming it is scanning, which is where the EOVERFLOW build trap surfaces by name instead of silently, three phases from now, as a service that never comes up. WHAT RESTARTS A DEAD SCANNER: nothing, and that is a decision. A watchdog would be one more hand-rolled respawn loop in ash, which is what s6 is being bought to delete. The realistic failure is a bad build rather than a crash, and a respawn loop would hide that for ever. If the scanner does die its s6-supervise children keep running and keep respawning their services, so it is a printer that stopped noticing scandir changes, not one that stopped printing. firmwareExe re-checks once after the init loop and restarts it there, in the same shape as the Klipper last resort and for the same reason: the check is cheap, it runs when something has already gone wrong, and the log line is what you would otherwise have had to guess. The reasoning is written down in both files so the next person does not have to rediscover it. anvil-service.sh grows svc_s6_running / svc_s6_svc / svc_s6_up / svc_s6_down / svc_s6_stat / svc_s6_ready. Every command line in them was run against the real mipsel binaries on the replica before being written down. Liveness is `s6-svscanctl -a` rather than a process match: it travels down the control FIFO, so it can only succeed if something is reading the other end -- the distinction a `ps | grep` cannot make, and the one that made a stale pidfile lie about moonraker. The existing svc_* functions are untouched; all six services still use them. etc/s6 ships in the payload rather than being created only at runtime, so it is named in the install manifest and a later update can account for it. S40s6 mkdir -p's it anyway: the manifest pass rmdir's empty listed directories before the new tarball extracts, and hand-made installs exist. The gate's most important check is the negative control: an init script that runs the scanner in the FOREGROUND must make the bounded replica of firmwareExe's loop time out. It does, at 15s, while the real sequence returns in 2. Without that check this phase's one catastrophic failure mode is invisible. Two gate-wiring fixes on top. gates.services() now hands the case the real cross-built s6 whenever work/.s6 holds one -- it was only ever running against the stand-in scanner, which would pass on a printer that cannot exec the supervisor it ships -- and case-supervisor.sh is registered as a gate at last, so the suite runs it instead of waiting to be asked by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
54a9b93ed3 |
An update removes what it installed, not the directories it found
run-append.sh began every update with
rm -rf $MODDIR/bin $MODDIR/www $MODDIR/nginx $MODDIR/helixscreen \
$MODDIR/config $MODDIR/moonraker $MODDIR/init.d
which is correct exactly as long as every file under those seven directories
is ours, and stops being correct the moment anything else lives there.
$MODDIR/bin is where a supervisor and later a Python are going to live, and
it is the obvious place for someone to drop a script of their own. All of it
was destroyed on every update, silently, by the installer.
The property that line was written for still has to hold: the installed set
must end up exactly the shipped set, or a RENAMED init script leaves a stale
twin and firmwareExe runs both -- which is what happened when S60web became
S60nginx and S62moonraker. A manifest gives that for free. bin/patch.sh now
reads the staged payload and ships the list of every path it installs, and
the next update deletes what the LAST manifest names: files first, then the
directories, deepest first, and only the ones rmdir finds empty. A directory
still holding something nobody shipped keeps it, and keeps itself.
The manifest is generated, never hand-maintained -- a hand-written list is
wrong one release after somebody adds a directory, and wrong here means
either a file that never goes away or a file that should have stayed. It
names itself, because the next update must be free to replace it. It is
written to a temp file and moved into place, or find would walk the tree and
list the half-written manifest as one more payload path.
Printers running any earlier package have no manifest and there is no way to
work out after the fact what that package installed, so the old sweep stays
as the fallback, byte for byte, marked for deletion once no pre-manifest
install can still be upgraded. It is now the only piece of this installer
that can still destroy a file nobody asked it to.
The deletion loop pastes $MODDIR onto the front of whatever the manifest
says, as root, so an entry that is absolute or contains .. is refused and
reported rather than followed. Nothing reaches rm -rf; the loop only ever
runs rm -f on one path and rmdir on one directory.
test/integration/printer/case-upgrade.sh runs the real installer twice over
two hand-built payloads and asserts on what survived: a file version 1
shipped and version 2 does not is gone, a file nobody shipped -- dropped by
hand into $MODDIR/bin, beside one that had to go, so only reading the
manifest can tell them apart -- is still there, anvil.conf kept the user's
edit, config-installed survived, and the pre-manifest fallback still clears a
stale init script. bin/verify.sh now fails a build whose payload has no
manifest: such a package installs fine and leaves every printer that takes it
falling back to the sweep forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8994b461b0 |
Every service starts the same way, and Moonraker stays on /usr/data
Moonraker is no longer copied to /usr/prog. The payload already extracts
that exact tree to /usr/data/anvil/moonraker, so the installer was making a
second identical copy onto the small firmware partition -- the one install
step that could fail on space, whose failure was "this printer has no
working web UI". Worse, /usr/prog is what a stock flash overwrites while
/usr/data survives, so a firmware update silently reverted Moonraker to
FlashForge's 2022 build and left the rest of the mod in place: exactly the
"which Moonraker is this printer running?" ambiguity the previous commit set
out to end. Neither of the two things thought to pin it there holds -- the
moonraker-env virtualenv is not on sys.path (imports resolve from
Python-3.8.2/lib/python3.8/site-packages, checked on the real image) and
moonrakerDaemon is never invoked. FlashForge's tree is left alone, unused.
S60web became S60nginx and S62moonraker. nginx and moonraker fail for
different reasons and are now stopped, started and diagnosed separately;
MOD_WEB still governs both.
payload/anvil-service.sh is the shape all six services now share. They had
been written one at a time and had arrived at four different answers to "is
it running" and two to "restart", none of them decisions. Two real bugs were
living in that drift, and both are fixed here:
* busybox start-stop-daemon -K returns before the process is dead and
never removes the pidfile, so restart raced its own start -- stop
returned, start found the pidfile still held, and the service ended up
stopped rather than restarted. The `sleep 2` in front of restart was
standing in for that wait; svc_stop_daemon does it properly and the
sleep is gone.
* klippy does not unlink /tmp/uds on exit. supervise() knew that; stop()
did not, so status reported a dead klippy as running and restart
declined to do anything at all.
run-append.sh now clears $MODDIR/init.d before extracting. Without it the
rename above would leave a stale S60web beside the two new scripts on any
printer that already has the mod, and firmwareExe runs every executable
S* in filename order -- starting nginx and moonraker twice, from scripts
disagreeing about where moonraker lives.
bin/patch.sh stages anvil-service.sh. It is sourced, so no chmod +x.
test/integration/printer/case-services.sh is a new replica gate: it runs
every service on the printer's own busybox and checks the contract they
share rather than grepping them for how they are spelled. case-moonraker.sh
now asserts moonraker runs from /usr/data, that nothing under /usr/prog is
executed, that restart yields a genuinely new pid, and that stopping
moonraker leaves nginx up -- the point of the split.
The .modold rollback recovery in docs/how-it-works.md described a code path
removed in the previous commit and is rewritten for what happens now.
Verified in the replica (real rootfs.squashfs, qemu-mipsel): case-moonraker,
case-services, case-mcu-bringup and case-boot-screen all pass; pytest 181
passed, 5 skipped. case-install.sh and case-recovery.sh are syntax-checked
only -- they need a stock FlashForge package, which is not in this checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
bcdd059008 |
S60web sets moonraker's whole library path instead of inheriting most of it
A printer came up with no web UI at all: no Mainsail, no API, just the screen. S60web named three /usr/prog/*/lib directories and inherited the other nine from app_startup.sh, which exports twelve before it runs the wrapper. That inheritance holds on a normal boot and nowhere else -- re-run the script over ssh, or perturb the boot chain, and the environment is bare. The one that matters is libsodium. moonraker's `authorization` component is linked against it, so its absence is not a missing-module error on something obscure: moonraker exits during component load, which from the outside is indistinguishable from "moonraker did not start". moonraker-preflight.py exists because that same library caught us once already, and run-append.sh sets the full list explicitly for exactly this reason -- S60web was the last place still trusting somebody else's exports. The replica proves both halves on the printer's own interpreter: with the list S60web really sets, read out of the shipped script rather than retyped, `authorization` imports; with libsodium removed from that same list it dies on "Could not locate nacl lib". A path entry that is not load-bearing would fail that second check, so the list cannot quietly become cargo. Not taken from the reported workaround: running `python3.8 moonraker.py &` directly. moonrakerDaemon owns the pidfile that `stop` needs, and it sets TMPDIR=/usr/data/tmp -- without which moonraker writes to /tmp, a ramdisk here. Those are two regressions for no gain once the library path is right. Also: a background check now says in the boot log when moonraker is not running, with the tail of its own log. It exits quietly, minutes before anyone opens a browser, and until now the only symptom was a user reporting a blank page days later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c7a821bb9d |
ff_mcu_bringup.py is importable by name
The filename was the only reason for the importlib shim: a module called ff-mcu-bringup cannot be written as an import, so ff-startup.py loaded it from a spec and hand-rolled the failure handling. Renaming it removes all of that and leaves a plain guarded import beside the one ffscreen already had. start.sh still runs it as a script, by path, on the fallback route -- a rename costs that nothing. What the shim was hiding is worth keeping visible, though: these imports resolve only because python puts a SCRIPT's own directory on sys.path, which is true when the wrapper runs the file by path and false the moment anything loads it another way. So there is now a --selftest that reports what it can actually reach, and the replica gate calls that instead of loading the module itself -- which it was doing before, and which would have passed while the real path was broken. It is worth having on a printer over ssh too. The unit tests were making the same mistake in the other direction: loading ff-startup.py through importlib with payload/bin absent from sys.path, so its siblings were missing and the selftest correctly said so. The loader now puts that directory on the path, which is what the runtime looks like, and a separate test runs the script from another directory entirely to prove the premise rather than assume it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
56cf907447 |
The boot screen can be looked at, rendered by the printer's own python
"It works" was resting on assertions about byte counts and pixel colours, which is a poor way to review something whose entire job is to look right. So there are now two renders of the same eight frames. make boot-screen draws them on the host in about a second, no docker. make boot-screen-sim draws them inside the replica -- FlashForge's own python3, MIPS under qemu, the printer's real rootfs -- and pipes them out as base64 PNGs, because every mount the replica gets is read-only and stdout is the only way anything leaves it. The two agree byte for byte across all eight, which is the part that matters: it means the fast host preview is a faithful stand-in for the panel, and reviewing a layout change does not need docker at all. The replica render is there for when that claim needs re-establishing. case-boot-screen.sh, the gate, ran green against the same replica: the importer starts under the printer's interpreter, ffscreen imports beside it, the frame is exactly one screen at stride 4096, background, title and bar all land, clear() blacks the panel, and a missing framebuffer stays a no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ac4f72a9d7 |
The first boot says what it is doing instead of showing a black screen
The migration runs before HelixScreen and can take minutes: it waits for klipper to find its MCUs, which on this machine routinely needs a restart or two, then makes a SAVE_CONFIG that restarts klippy again. All of that happened with nothing on the panel. A black screen through the longest wait of the install reads as a brick, and the thing people reach for is the power switch -- which is the one action that can leave the config half written. bin/ffscreen.py paints straight onto /dev/fb0: a 5x7 font compiled into the file, geometry read from sysfs rather than assumed, no toolkit and no dependencies, because this runs before anything that could provide one. The importer drives it through each phase with a progress bar, and hands the panel back black so HelixScreen starts on its own splash. None of it is allowed to matter. Panel.say swallows everything and retires the screen on first failure, an unfamiliar pixel format draws nothing rather than guessing, and a test asserts the migration completes identically against a screen that throws on every call -- which is how that boundary got found. Geometry can also be passed explicitly. The replica mounts the HOST's /sys read-only and makes /dev/fb0 a plain file, so probing there would describe a developer's monitor; case-boot-screen.sh passes the panel size instead and checks the frame is exactly one screen, has its background, title and bar, and that clear() leaves it black -- under FlashForge's own python3 on qemu, which is the half no amount of reading can settle. Wired up as `make test-boot-screen` and into `make test`. Also `make boot-screen`: renders every phase to PNGs with nothing but the standard library, so a layout change can be looked at without a printer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Fable 5
|
f3d12ac84a |
An update keeps the root password the printer already has
A random password on every install meant a trip back to the USB stick on every update. Now the pre-block records the live root hash before the stock installer replaces the shadow file, and the post-block puts it back when it differs from what the package shipped -- so the random password is chosen once, on the first install, and an update preserves it, including one set by hand with `passwd`. Flashing stock in between resets the cycle, as it should. The end-to-end install test asserts all of it across its two installs: same hash after the re-install, one generation line in the log, the stick file untouched. With it, `make passwd`: the ROOT_PW_HASH generator now runs inside the pinned build image, because the previous advice -- host `openssl passwd -6` -- fails outright on LibreSSL builds that have no -6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ac558f6686 |
Document the Moonraker swap honestly, and finish its doc coverage
The reversibility claim was the strongest promise in the README and
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
14672ffe7a |
Section H: delete the dead code the comments implied was live
* ff_print.py TAIL_BYTES -- nothing reads the tail; the module says outright that the slicer's config block is not a source. The rationale comment claiming a measured ~24 KB offset for it went with it. * ff_tool_offset.py Z_TARGET_APP and APP_ACCEL_RESTORE -- unreferenced, but sitting in a block of constants that ARE wired to config defaults. The numbers are kept as comments on the live constants beside them, since they record what the app did; they no longer look like code. * bin/patch.sh MARK_BEGIN / MARK_END -- the real markers are hardcoded inside the inline Python, in four different literals. Editing these did nothing. * bin/patch.sh -- the /etc bind-mount comment was written twice, the second a truncated copy of the first. * bin/common.sh WORK -- nothing reads it; bin/ hardcodes work/. * Makefile clean -- work/uninst and work/uninst-sw are never created. * payload/run-append.sh -- the ssh host-key block is guarded on bin/dropbearkey, which patch.sh never installs. We ship no dropbear at all: stock S50dropbear is already listening. * config.env.example STOCK_SOFTWARE -- referenced nowhere; the base comes from unpack.sh extracting software-*.tar.xz out of the stock package. * test/ffcfg.py -- sections() bound `name` and never used it. Checked: 48 passed / 5 skipped, bash -n over bin/, sh -n over the payload and the test shell, py_compile over every module, the HelixScreen JSON parses, and the Makefile still parses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
d05565284d |
Sections E, F, G: wrong values, stale claims, overclaims
E -- values and counts. The START_PRINT prologue has no M400 and an idle timeout of 864000, not 1800000. The nozzle-clean temperature comes from the material table FIRST, so NOZZLE=240 with no per-tool temp cleans at 220 -- documented backwards before. TOOLS is the paired 0:220,2:240 form. The wipe Z is the raw eddy frame, not the homed one. The runout M117 strings are "T<n> clog" / "T<n> out of filament", not E-codes. Grab-verify raises E0051+tool, not E0143. Docks are at X~297; 250/280 are staging waypoints. The grab sequence is documented in the order _grab actually emits it. ff_print exposes seven status fields, not nine, and never emits TOOLS=. Package sizes reconciled against the real artifacts (62MB built, ~93MB stock, ~370MB of CI assets, ~60s not 90s). testing.md's sample output block matched no header the code prints, so it is replaced with the real ones; its pytest row listed two of six suites. F -- stale claims. _FF_PREFLIGHT is not "missing". SDCARD_SET_CHANNEL is not still issued ( |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e5bd171cf8 |
Correct the docs the code had drifted away from
Applies the audit's sections B, C and D, plus two decisions. Behaviour (A1): fm_ex1..3 carried stock's `extruder: extruder`, so a clog on T1..T3 was measured against T0's filament movement. Each motion sensor now watches its own extruder. Gate count (A6): "four gates" was the number of make test-* targets, not of gates run -- run-tests.py emits twelve. The number is dropped rather than restated, in docs/testing.md and the Makefile help. B, install instructions that failed as written: STOCK_TGZ_CREATOR5 missing its .tgz; a pointer to a ./build.sh that does not exist; klippy-extras/ paths that moved to payload/klipper/extras/ in 0ef9d6b; a calibration recipe both commands refuse without PLATE_REMOVED=1; a storage table that put dock_x/dock_y in an include, which SAVE_CONFIG refuses; four files telling the user to add includes printer.base.cfg already ships; ./pack.sh style paths that do not resolve from where the scripts leave you. The MAINSAIL_ZIP / HELIX_TGZ override was documented in three places as "used as-is, never checksummed". It is checksummed, and fetch-assets.sh overwrites the user's file with the pinned release -- SKIP re-downloads too. Corrected, with the way that actually works (match the hash in versions.env). C, files that contradicted themselves: the -5/-3 ESTOP target, the t<n>_offset_* storage names, the chamber heater's model split, the LED premise, the include order comment in test_includes.py, gear_stepper as a "feed hub" when it is the tool lock, an object list ff-toolchange.cfg says does not exist, the eddy probe vs the under-bed cylinder, "installs them without clobbering" against a gate that enforces the clobber, "keeps two versions" when the installer keeps one, "two changes from stock" when there are three, and ff_legacy's "nothing here runs during normal operation" when auto_import runs on every ready. D, dangling references: OKF/* and firmwareExe-decompiled/* repointed at docs/notes/*, plus case-boot.sh, sim-image.sh, printer-exec.sh, the test/printer build context, wait_for_banner, ff_toolchanger_compat and a nonexistent orca snippet. Not touched, pending a decision: sections E-H, and the printer.base.cfg "verbatim" claim (A7), which depends on how the unpack drift check is fixed. 48 passed, 5 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f70369155b |
Run the brick-safety gates in CI, and stop skipping quietly
The replica gates -- the only ones that decide whether a package bricks a
printer -- had never run in CI. They were gated on a STOCK_URL_CREATOR5PRO
secret that was never set, so printer-sim rendered as a grey dash beside
two green ticks and read as "CI passed".
The packages it needs are already public. ghzserg/FF publishes both stock
packages and the factory image, and this repo already points at that
release in docs/printer-replica.md. All three are now pinned in ci.yml by
sha256 and verified byte-for-byte against the copies used locally:
Creator5Pro-1.9.7-1.2.9-20260810.tgz 1f3252fe... 93MB
Creator5-1.9.7-1.2.9-20260810.tgz 72009be1... 93MB
Creator5Pro-factory.tar.xz 45657924... 190MB
Nothing proprietary is redistributed from here; the workflow fetches what
is published elsewhere, cached on their checksums so an unchanged pin
costs that archive nothing. Secrets still override for a fork with its own
firmware -- but a URL override without a matching checksum now refuses to
run rather than downloading something nobody verified. have-firmware is
gone with the `if:` it fed.
Rehearsing that job locally found the reason it would have failed anyway.
The replica gates call ./bin/patch.sh directly, and patch.sh expects
vendor/ already populated -- bin/build.sh is what fetches Mainsail and
HelixScreen, and the gates do not go through build.sh. No local run ever
noticed, because vendor/ was always already there. Both workflows now
fetch assets before the suite; in release.yml that step has to precede the
gates, since `make release` would otherwise fetch them one step too late.
Second half: a skip must be legible.
ALLOW_SKIP=1 silenced the explanation as well as the exit code -- the
summary printed "2 skipped" and then nothing about what or why. That is
the disease this harness exists to treat, reintroduced by the flag meant
to manage it. Now:
- Every skip is listed again in the summary with its reason and whether
it was accepted, whatever ALLOW_SKIP says, and becomes a GitHub
annotation. Nobody reads a green job's log.
- ALLOW_SKIP takes names: ALLOW_SKIP="pytest,the printer replica". `1`
still accepts everything, but it is a standing promise never to notice
a skip again, so CI uses the list and a third skip fails there.
- The pytest gate names the tests that skipped instead of counting them.
That immediately showed the exec-bit test skipping wherever .git is
absent -- a bare "6 skipped" would not have.
VERIFIED:
- printer-sim rehearsed end to end on the actual published bytes, real
git checkout, REQUIRE_PRINTER_SIM=1 and no ALLOW_SKIP: 12 passed,
0 failed, 0 skipped, exit 0, in about four minutes. Before the vendor
step it failed at patch.sh, which is what CI would have done.
- The four ALLOW_SKIP cases, in the build image with no firmware: none
set -> exit 1 naming both gaps; "pytest" only -> exit 1, replica still
NOT ACCEPTED; both named -> exit 0, both still printed; and under
GITHUB_ACTIONS the ::warning:: annotations appear.
- pytest 30 passed. Both workflows parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
30b21262ed |
Ship the two verification files, not the thing that wrote one
The generator was a build step for an artifact that is written once and then read by a person standing at a printer. Keeping it meant the file you send to the machine did not exist in the repo, and the header of the file you did get told you to re-run a script instead of telling you what to watch. gcode/ now holds the G-code itself and bin/gen-test-gcode.py is gone. Added alongside it: creator5-safe-moves.gcode, the feature print with the heat and the filament taken out. Once homed, nothing in it goes below Z50, nothing extrudes, no heater is ever given a target. It homes, laps the bed slow then fast, runs the diagonals a failing corexy motor turns into an L, changes tools, sweeps the fans and checks the chamber gate -- so that a dock that does not latch or a Z offset that is wrong is found 50 mm up rather than in the plate. It names its tools with SELECT_TOOL rather than a bare `Tn`, and carries no M104 or M140. ff_print scans a file's head for exactly those to decide what to prepare, so this one leaves it nothing to find and is safe whether FF_BEFORE_PRINT_START.prepare is 0 or 1. The feature print still wants it at 0. test_gcode.py reads the committed files now instead of running a generator, and covers both. The safe file gets the two checks that are its whole reason to exist -- never below Z50, and never a heater target or an E word -- because those are exactly the properties a later edit would break while leaving a file that still runs and still looks safe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
959fbd356a |
Generate the print that proves the macros, not just the machine
The on-hardware checklist ended at "a single-tool test print" and "a multi-tool test print" without saying which print. Any cube proves the motion system; none of them prove START_PRINT's preflight gate, the per-tool purge, the print Z offset, the M106 fan map or the M141 model gate, which are the parts of this firmware that have never run on a printer. bin/gen-test-gcode.py writes one file that drives all of them in the order a real print drives them, and lays down nested squares -- one ring per tool, a fixed gap apart -- so the tool offsets can be measured with calipers afterwards instead of eyeballed. It calibrates nothing: TOOL_OFFSET_CALIBRATE and STATION_CALIBRATE need the build plate off, which a print does not have, so the file only reads the geometry into the log. PAUSE is left out for the same kind of reason -- a resume that fails should not be buried inside the print the offsets are being measured from, so the header tells the operator to press Pause in Mainsail instead. Four things the first draft got wrong on the machine and not on paper: T<n> ships with restore_axis unset, so a change restores no position at all and every move after one has to re-state itself; descending to layer height before the cross-bed travel dragged the nozzle back over the work from the dock; START_PRINT's clean leaves each tool retracted by purge_retract and the start block primes only the tool it starts on, so a tool's first ring was the one recovering those 5 mm; and parked tools sat at full temperature oozing into their docks for the whole print. test_gcode.py is the gate that keeps the file honest: every command in it must be one the shipped configs or extras define, TOOLS= must list every tool the file uses, and every coordinate must be inside the configured axis limits. Renaming a macro now breaks the suite rather than the print. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e0dad3513c |
Rewrite the host half of the harness in Python, so a broken gate fails
The shell suite decided "this gate did not run" by grepping the child's
stdout:
if "$@" >"$out" 2>&1; then
if grep -qE '^[[:space:]]*SKIP:|[0-9]+ skipped' "$out"
then skip "$name"; else pass "$name"; fi
Text on stdout cannot tell you what happened inside a process. A launcher that
had ALREADY failed reached that same line by another road: the file that
sources config.env failed to load, so STOCK_TGZ_* was never set, so the
launcher concluded there was nothing to test, printed those five characters
and exited 0 -- on a machine with docker and the firmware sitting right there.
That is how five broken launchers shipped and stayed broken.
So the host half is Python and a skip is an exception. Output cannot imitate
one, and a gate that fails cannot claim it was skipped.
test/ffsim/ config loading, docker plumbing, gate reporting
test/run-tests.py the orchestrator
test/integration/ sim-install.py, sim-roundtrip.py, printer-exec.py,
extract-rootfs.py
test/integration/printer/*.sh and the whole payload stay shell. They are
executed by the printer's own busybox ash under qemu, and surviving that is a
large part of what the suite proves; rewriting them would test a Python the
printer does not have. build-printer-image.sh and make-stock-fixture.sh stay
too: both are `set -euo pipefail`, neither is a gate, and one is a release
tool run by hand.
Four things that used to report as skips are now failures -- a config.env that
does not parse, a missing case script, a missing package, a missing baseline.
A genuinely absent precondition is still a skip, because that is still true.
Three more bug classes are gone by construction rather than by being tested
for. The repo root is found by searching upward for bin/common.sh instead of
counting `..` from $0, so a launcher can move to any depth. Sourcing a config
with a syntax error is checked -- `.` returns non-zero and does NOT stop the
shell, so the old code carried on with half a file applied, which is exactly
the broken-looks-absent confusion again. And pytest counts come from the JUnit
XML rather than from grepping "N skipped".
test_script_roots.py becomes test_harness.py: the `..` check now has two
remaining shell targets rather than seven, and it gains a byte-compile of the
whole harness. That last one matters -- sim-install.py and friends are only
executed by `make test-install`, which needs docker and the firmware, so
without it a typo would wait quietly until release day. It also pins the
property that replaced the `..` counting, from four different depths.
VERIFIED, and the verification is the point, because this machine has no
docker daemon and these are exactly the scripts that cannot run here:
- Stubbed the docker binary and diffed the recorded `docker run` argv,
shell against Python, across five paths: install with a prebuilt image,
install with a local build, local build plus PROG_DUMP, roundtrip, and
printer-exec. IDENTICAL in all five.
- Ran the old orchestrator and the new one in the same tree: both give
6 passed, 1 failed, 2 skipped, exit 1, with the same gates in the same
order. (The failure is shellcheck, which lives in the build image.)
- pytest: 24 passed / 5 skipped bare, 29 passed against a real extracted
rootfs.
- Compared failure modes directly. The four cases above go from "exit 0,
reported skipped" to "exit 1, named". Case 2 is the sharpest: with a
missing case script the shell launcher exited 0 in silence, having cp'd
nothing and mounted a directory over /case.sh.
Still unproven here: the gates that need a live daemon. `make test-install`,
`test-mcu` and `test-recovery` want one run on a docker-capable machine. The
argv diff says they will be handed the same command as before; it cannot say
what docker does with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
43baf4054e |
Drop the unit lane; one suite, in test/integration
The split existed so a plain pull request had something to run. This repo has one maintainer who always has the firmware to hand, so it was a boundary kept in sync for a contributor who never arrives. One directory now. The tests moved rather than went away. test_chamber.py is the only check in the suite that can catch a printer heating something it has no element for, and the replica cannot cover it: case-install.sh touches .cfg files at exactly two lines, both a grep for a sentinel proving the user config survived, and it asserts klippy.py exists and c_helper.so has the right ABI without ever starting klippy. Declaring [heater_generic chamber_heater] on the plain Creator 5 fails four of these tests, including M191 waiting on a target the machine can never reach; that same package installs, boots, and passes every replica gate. make-stock-fixture.sh came too, since the packaging pipeline is built on it. Ordering changed, and that is the part to look at. With two lanes, pytest ran twice -- once per lane, each with its own -m expression or path. With one it runs once, so it has to run at the point where it can see the most: the stock package is now resolved and the rootfs extracted BEFORE pytest rather than after it. On a configured machine that is 28 tests in one pass; on a bare tree the five that read the rootfs skip. config.env is read in a subshell while doing that. It is the BUILD config, and it is now consulted upstream of the fixture build, where the entire point is that the fixture config wins -- sourcing it into this shell would put MOD_NAME and TARGET_MACHINE where that build could see them. Verified by pointing a config.env at MOD_NAME=LEAKED and confirming the fixture still builds Creator5Pro-anvil-ci.tgz. One thing gets worse, and it is inherent to having a single lane: on a tree with no rootfs, pytest reports 5 skips, so sub() marks the whole gate SKIP rather than ok, and the summary reads 6 passed instead of 7. Those 23 passes are real and the line above the SKIP says so. This is the existing policy working as designed -- a gate that did not fully run must not read as success -- but the summary is coarser than it was when a lane existed that never skipped. Verified: 28 passed with a real extracted rootfs, 23 passed / 5 skipped without; the full suite gives 7 passed / 1 failed / 1 skipped with a rootfs present and 6 / 1 / 2 without, the failure being shellcheck, which lives in the build image; no test/unit reference is left outside the passages that explain why it is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
2ae9736e97 |
Name the two lanes unit and integration, and fix what the move broke
test/repo -> test/unit, test/replica -> test/integration. The old names said
where the input comes from; these say what the tests are, which is the thing
you already have a word for.
The rename is also how the bug below surfaced, because "integration" is one
character-count away from nothing and running the launchers by hand is not.
Every launcher that moved out of test/ into a subdirectory opened with
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
and one `..` was correct at test/ and wrong a directory deeper. $ROOT became
test/, so the first thing each of them did -- source $ROOT/test/integration/
sim-image.sh -- went looking for test/test/integration/. Five scripts:
printer-exec.sh, sim-install.sh, sim-roundtrip.sh, build-printer-image.sh and
extract-rootfs.sh, which reached bin/common.sh the same way.
The previous commit shipped this. The path-existence sweep that was supposed
to catch it only checked paths written as literals; these are assembled at
runtime from $0. Nothing else caught it either, and that is the part worth
naming: the integration half cannot run without docker and the stock package,
so on any machine missing either -- this one, and a fork's CI without the
secrets -- five broken scripts still reported a clean SKIP.
So test/unit/test_script_roots.py now checks it statically: for every .sh
under test/, count the `..` after `dirname "$0"` and compare it against how
deep the file actually sits. It needs no docker, no firmware and no shell, so
it runs in the lane that always runs, which is the whole point -- the failure
it pins only appears in the lane that often does not.
Verified: the guard fails with "climbs 1, is 2 deep" against a reintroduced
bug and passes once it is reverted; all four launchers now reach their own
preconditions (docker daemon, config.env) instead of dying on a missing file;
21 in test/unit and 7 in test/integration against a real extracted rootfs;
run-tests.sh reports 7 passed / 1 failed / 1 skipped, the same as before the
rename (shellcheck lives in the build image, and no stock package is
configured in this worktree); no test/repo or test/replica reference is left
anywhere in the tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8337359a68 |
Split test/ into the two lanes it already had
The suite has run in two halves for a long time, and the rest of the repo is
built around that: RUN vs RUNSIM in the Makefile, `test` vs `printer-sim` in
CI, two halves in run-tests.sh. Only test/ itself was flat, so which half a
file belonged to was something you learned by reading its header.
test/repo/ needs nothing but this checkout
test/replica/ needs the printer's real rootfs
test/ run-tests.sh, and the fixtures both lanes share
The axis is the precondition, not the runner. test/replica is everything that
cannot run without the proprietary package -- the qemu'd sim launchers and the
replica image, but also test_paths.py, which only reads the extracted rootfs
as files and needs no docker at all. Grouping by "does it use docker" would
have put that one in the wrong place; grouping by what has to exist first puts
every file where the question "can this run on a public checkout?" is answered
by the directory name.
The `rootfs` pytest marker goes with it. It existed to let run-tests.sh pass
`-m 'not rootfs'` in the PR lane and `-m rootfs` in the replica lane; the
directories say the same thing without anyone having to remember to tag a new
test, and a file in the wrong lane now shows up in `git status`. The two
lanes are `pytest ./test/repo` and `pytest ./test/replica`.
pytest.ini is new and is load-bearing: without a pinned rootdir, naming a lane
on the command line makes pytest take that lane as the root and never load
test/conftest.py, so the fixtures both lanes share would vanish depending on
which one you asked for.
Renames only, plus the path references: test/fixtures/make-stock-fixture.sh ->
test/repo/, the sim launchers and test/printer/ -> test/replica/, and
test_applets.py -> test/replica/test_paths.py, which is what it has actually
checked since the command-word half was dropped.
Verified: 19 in test/repo and 7 in test/replica against a real extracted
rootfs, the same 26 as before the split; run-tests.sh reports 7 passed /
1 failed / 1 skipped, identical to the pre-split run on this host (shellcheck
is absent here and lives in the build image, and no stock package is
configured in this worktree); every referenced path in the repo resolves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
91604c2dfb |
Cut the suite to four gates, and rewrite the Python half as pytest
Twelve gates, most of them the weaker copy of a gate that survives. What is
left: pytest for the Klipper config, and three replica gates -- the end-to-end
install, MCU bring-up, and stock recovery.
Fix first, because the cut depends on it: a skip counted as a pass. Every
launcher exits 0 when it decides it cannot run, and sub() reported that as
`ok`. That is how test-abi.sh sat in the suite printing green while checking
nothing -- run-tests.sh deleted work/modpayload immediately before it ran and
CI set KLIPPER_FORK="", so it had no targets on any run, ever. Skips are now
counted apart, the summary says so, and the suite exits non-zero unless
ALLOW_SKIP=1 says the gap is deliberate. With four gates instead of twelve, a
gate that silently does not run is the failure mode that matters most.
Dropped, each because something else already does the job properly:
test-abi.sh bin/patch.sh refuses to build a non-nan2008
c_helper.so, bin/verify.sh re-checks the packaged
one, and case-install.sh reads the ELF header of
the file that actually landed on the machine.
test-ash-conformance.sh case-install.sh runs `sh -n` over every installed
script with the same qemu'd busybox and then greps
the boot log for "Syntax error".
test-model-gate.sh verify.sh SS8b/SS9 checks the same invariant and
pack.sh exits on MODEL MISMATCH before a package
exists. Its header claimed it proved the two models
ship different files; no such check was in it.
test-macros.py It used jinja2.Environment() -- default '{{ }}'.
The configs have no '{{' at all, only single-brace
expressions, so it checked '{% %}' block structure
and never looked inside an expression. Folded into
the chamber tests, which had the delimiters right.
test-base-cfg.py It read work/software/klipper/config/printer.base.cfg
as its stock reference -- the file bin/patch.sh
overwrites with ours. It was diffing our file
against itself: green on a cold tree, red on a
second run, blind to real drift either way. The
comparison moves to bin/unpack.sh, where a pristine
tree exists, as a warning rather than a gate.
lint-danger.sh Its structural greps were satisfiable by comments:
firmwareExe:44 and S80ui:16 *mention* helix and
SAFE-MODE, so they passed with the logic deleted.
case-ui.sh Goes with the helix-only decision.
applets.allow Ten entries, none of which the extractor had asked
about in a long time.
The command-word half of test-applets went too. It extracted only the first
word of a simple unprefixed command, so `if timeout 5 foo; then` yielded
['if','echo','fi'] -- the one failure its docstring named was invisible to it.
The absolute-path half survives: S50wifi's binaries sit behind a
[ -d /sys/class/net/wlan0 ] guard no simulation ever reaches, so nothing else
checks they exist.
The Python half is pytest now. Real assertions instead of hand-rolled
PASS/FAIL counters, underscored filenames so the modules import normally
(ffcfg keeps only the parser; the dash in test-macros.py was why one test used
to reach the other's parser through importlib), and a `rootfs` marker that
splits the tests needing the proprietary package from those that do not --
deselected in the PR lane, run in the replica half, so neither lane reports
skips it did not expect. test_default_delimiters_are_blind pins the jinja2 bug
so it cannot come back.
The config gate also moves out of the replica half into the half that runs on
a plain pull request. It needs python3, jinja2 and files in the repo, and it
is the only safety-relevant check in the suite -- a chamber heater declared on
a Creator 5 that has no element for it -- so it had no business being gated
behind the proprietary firmware.
Verified: pytest 21 passed / 5 rootfs-marked; the fixture pipeline
(unpack, patch, pack, verify) passes end to end; injecting a broken
single-brace expression fails test_every_macro_parses with the exact line, and
parses clean under the old defaults; the skip path reports SKIP and exits 1,
and ALLOW_SKIP=1 accepts it.
Also drops the CI `abi` job, which ran on a bare checkout with no binaries and
so was permanently green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
3a28fdf65b |
Drop test-chelper and its build gate
test-chelper.py compared c_helper.so's exported symbols against the cdef blocks in the klippy tree shipped beside it, and bin/patch.sh ran it as a hard build gate with ALLOW_STALE_CHELPER=1 as the escape hatch. Both are gone, along with the Makefile target, the suite entry and the ALLOW_STALE_CHELPER passthrough that had nothing left to pass through. patch.sh still refuses a c_helper.so that is not MIPS32r2/nan2008/o32, so the ABI failure mode is still caught. The symbol failure mode is not: cffi resolves lazily, so a .so older than its klippy imports cleanly, passes the ABI check, installs, boots, and dies at connect with a traceback that names cffi rather than the stale build. That is a real thing that shipped once, so it is written down in docs/building.md and docs/notes/20-klipper-fork.md rather than left for someone to rediscover -- the fork path this guarded is winding down anyway, per the same note. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |