mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
worktree-timelapse-test-gcode
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr ShyshatskyiandClaude Opus 5
|
5d7754285e |
Wipe $MODDIR and extract, instead of pruning by a manifest
The installer deletes /usr/data/anvil outright and extracts into the empty directory. The installed set is the shipped set by construction, so a renamed file cannot survive beside its replacement and no list has to be shipped, read or trusted for that to hold. What that costs is everything else under $MODDIR, and the audit of what was actually there is docs/notes/86-wipe-and-extract.md. backup/stock had no restore path in any script and omitted the one file a restore would need. .prev-root-hash was already dead state. config-installed was a twelve-file snapshot serving one md5, for a compare whose stated rationale -- "there is no include-and-override seam" -- stopped being true when moonraker-custom.conf became the last include, and which inverted its own purpose by keeping the webcam block and the API lockdown off any printer that tripped it. So moonraker.conf is overwritten like every ff-*.cfg. HelixScreen's /tmp stash is untouched and is now the only preservation the installer does, so the gate asserts it with a value that could only have come from the printer. Gone with it: both delete passes, the forged-path guard, the rmdir pass, the pre-manifest branch, the unconditional init.d/anvil-service/ anvil.conf removal, the etc/s6 scandir sweep and the $MODDIR __pycache__ sweep -- the last two existed only because a manifest cannot see paths made at runtime. firmwareExe already makes the scandir itself. test_upgrade.py rewritten: the negative control is inverted rather than deleted, and the legacy fixture goes -- there is one path now. qa/static green (112). The replica lane needs a package in work/out, which needs config.env with FF_KEY and STOCK_TGZ; not run here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KXKk3zz8LAUPUDZkVEZ9fR |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b98552386b |
No FlashForge component ships; patch.sh is payload.sh
The software component existed to give the mod one exec hook. It is not needed for that any more, so a release now carries our installer and our payload and nothing else: 26MB smaller, 21MB of which was FlashForge's own firmwareExe travelling back to the partition it was already on, along with their klippy tree, their wifi module and their configs. WHAT IT GIVES BACK. /usr/prog/PROGRAM/software is no longer rewritten, so the version directory there stays FlashForge's with their genuine firmwareExe inside it -- and app_startup.sh's five-second watchdog, which copies one out of that directory when nothing by that name is running, has something to copy again. A printer whose payload fails to install now comes up on the stock UI instead of a blank screen. That recovery has been broken since we started deleting their binary out of a component we shipped; docs/how-it-works.md claimed it worked throughout. THE ROOT PASSWORD moves with it, because it used to ride in the component's shadow file. bin/pack.sh bakes ROOT_PW_HASH into the installer, single-quoted -- an unquoted $6$... would have the printer's shell expand $6 as a positional parameter and ship a truncated hash -- and the installer writes it to /usr/prog/etc/shadow. The random-password path asks one question instead of four: FlashForge ship the same root hash on every machine, so a live shadow still carrying it means nobody has set a password and it is ours to set. The .prev-root-hash recording, the shipped-vs-live comparison and the restore-it- afterwards step were all machinery for surviving a file that nothing overwrites any more, and are gone. pack.sh checks the hardcoded constant against the stock package it built from, because getting it wrong is silent and total: every printer would read as "already set" and none would ever get a password. bin/patch.sh is bin/payload.sh. It patched nothing any more -- no splice, no shadow edit, no firmwareExe deletion -- and what is left is the payload: the feed installed by the printer's own opkg, the seeded config, the manifest. Its sections are renumbered 1-5 with no gaps, and the five tombstone comments marking where removed sections used to be are gone with them. test_recipe_layout.py loses six of its nine tests. Each compared the tree against a description of the tree -- no fourth directory kind, each kind still used, 3rdparty carries no files of ours, installer/ holds exactly two named files -- and none could go red for a reason a printer would notice, while all six went red the moment the layout legitimately changed. The two that bite are kept: a payload/ path escaping $MODDIR, and a recipe name at both levels, which makes pkg_dir silently build the wrong package. SW_VER is deleted from config.env.example and both workflows: it stamped a component that is not built. Gates: 112 static tests pass. The generated installer is verified after substitution -- all four lines present, and `sh -n` over the result, because a hash with a stray character in it would otherwise reach a printer as a syntax error whose only symptom is a package that installs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
56b5b6c334 |
The printer image is the only replica; make rootfs was a second way nothing took
`make rootfs` unsquashed rootfs.squashfs out of the stock package into
work/rootfs, and a thin container bind-mounted it with PROG_DUMP supplying
/usr/prog from the factory image. Every caller set PRINTER_IMAGE anyway --
both workflows wrote it into test.env BEFORE running `make rootfs`, so CI
unpacked a 93MB package, unsquashed a rootfs and downloaded a 182MB factory
image on every push and read none of it. build-printer-image.sh never used
work/rootfs either: Dockerfile.full does its own extraction from the same
public firmware and needs no stock package at all.
So the fallback went, and PROG_DUMP with it -- it was only ever read when the
replica was built locally, and both implementations already passed it empty
otherwise. Gone: the target, extract-rootfs.py, gates.extract_rootfs and its
two tar helpers, the thin printer/Dockerfile, the `prebuilt` flag threaded
through both Replica implementations, and Replica.start's want_output, whose
only message was that the local build was about to be slow.
PRINTER_IMAGE is pinned to 1.9.7-1.2.9-20260810 rather than :latest. It is
now the entire substrate the brick-safety gates run on and the only input
nothing checksums, so a floating tag would let it change under a green suite
-- the drift the sha256 pins in both workflows exist to prevent.
THREE THINGS THAT COULD NOT HAVE PASSED, found while doing it:
* Neither workflow built the feed. bin/patch.sh has refused to start
without one since
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
fbb3182e68 |
bin/verify.sh is retired; four of its forty checks move to qa/replica
verify.sh described itself as "simulate every check the printer performs,
against a built package". The word carrying that sentence is SIMULATE, and it
is why thirty-six of the forty are deleted rather than moved.
THE REPLICA LANE DOES NOT SIMULATE THE INSTALL. install-package.sh puts the
package on a genuine FAT filesystem at /dev/sda1 and runs the machine's own
/usr/prog/app_startup.sh over it, verbatim, under qemu. So these are
PRECONDITIONS of the lane now rather than assertions in it -- break any and
installed_image() raises with the install log and the whole lane goes red:
* the decrypt with the firmware key (the printer's unTar does it)
* runFirmwareExe.sh present, components plain tar not xz (bare `tar -xvf`)
* md5sum.list verifying (the installer aborts and deletes the payload)
* the MACHINE= gate (runFirmwareExe.sh refuses a foreign package)
* the /mnt/<Model>-*.tgz glob (app_startup.sh finds the file by it)
A host-side re-reading of those rules is a second implementation, and a second
implementation can agree with itself while disagreeing with the printer. That
is the defect qa/replica/conftest.py records this repo learning twice about
hand-placed payloads.
A second block was asking what the suite already answers, of an installed
filesystem rather than of a tar listing: the klippy tree and c_helper.so
(test_install, test_abi), the s6-rc database and the oneshot runner's execline
(test_s6rc), s6 and s6-ftrigrd (test_supervisor), the .install-manifest
(test_upgrade), libsodium's bare .so symlink and the chamber configs
(static/test_ipk), and every shipped script's syntax
(static/test_shell_syntax).
THE FOUR THAT MOVED are the ones nothing else asks, and each is stronger for
the move -- a `tar -t` listing cannot tell you that a shared object loads, or
what a name resolves to on a PATH:
* the interpreter IMPORTS sqlite3, lmdb and _cffi_backend, rather than a
grep finding three .so files. This is what catches a cross build that
resolved to an x86_64 manylinux wheel: present on disk, raises on import.
* nothing in $MODDIR/bin shadows FlashForge's python3 -- asked with
`command -v` after sourcing anvil-env.sh, which is the file that PREPENDS
$MODDIR/bin to PATH and so creates the hazard in the first place.
* CPython's dev half (headers, pkgconfig, config-3.13-*) did not ship.
* the ship boundary: no file on the installed printer is byte-identical to
one under bin/ or docker/. Asked of the machine rather than of the .tgz,
so it also sees whatever a maintainer script wrote as root at install
time. The sweep filters on exact size before hashing, which is what keeps
a walk of /usr/prog through qemu cheap.
WHAT GENUINELY LEFT, recorded in docs/qa-migration.md rather than only here:
there is no longer a check you can run on the build host, in seconds and
without docker, against a package you are about to flash. Nothing warns you
before you copy the file to a stick that it was built from the wrong model's
stock package. The printer still refuses it and `make qa-replica` still
catches it, but the fast host-side warning the hardware checklist used is
gone. `make verify` is removed and `make release` no longer calls it.
VERIFIED. All four ran against a real replica (the cached installed image,
PRINTER_IMAGE=monstrofil/creator5-printer:latest): 4 passed in 17s. Then each
was DRIVEN RED on purpose with a throwaway module, which is this repo's own
standard from
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
4850be432c |
Docs: say what is there now, not what the last three commits removed
Deleting test/ and anvil.conf left claims behind in nine files. Every one of
these was present tense about something that no longer exists.
THE CHELPER SYMBOL GATE, in three places that each described it as live:
notes/20-klipper-fork.md said pkgs/klipper gates the .so on "both the ELF
flags and test-chelper.py's symbol check". It
gates on neither now: the ABI is asked once of the
installed filesystem in qa/replica/test_abi.py,
and the single-source rule -- the .so is compiled
from the chelper sources of the tree it ships -- is
the whole defence. The page says that instead.
notes/85-packaging.md described the KLIPPER_FORK fallback the script no
longer had; the script itself is gone too.
qa-migration.md "What survives in test/" listed both the 28 unit
tests and test-chelper.py as surviving. It is
"What survived in test/, and then did not", and it
names what went uncovered rather than leaving that
to be discovered.
ANVIL.CONF, in the two notes that still read it at runtime:
notes/80-s6-migration.md is a dated plan, so it keeps its phases and gains
a "Where this ended up" note: anvil.conf and every
MOD_* switch were removed, so the phases that
source it describe an intermediate state, and the
case-*.sh gates they name are qa/replica/ modules.
notes/85-packaging.md predicted seed/ emptying via a postinst that seeds
anvil.conf.default. It emptied by anvil.conf being
deleted instead. Its payload allowlist said six
entries including anvil.conf and bin/busybox; the
real one in test_ipk.py has five and neither.
THE TWO STANDALONE TREES under tools/, stale before any of this and now fixed:
tools/python/README.md and tools/supervisor/README.md both told the reader to
run their gate with `test/integration/printer-exec.py` over a case-*.sh, and
one offered a `make test-python` target. None of the three has existed for
several commits. Both now point at qa/replica/, which installs the real package
and asks more than hand-feeding a tarball to a case script could. The four
places tools/python/README.md narrates what case-python.sh proves are past
tense, since that is now a record of how the measurements were taken.
versions.env's sqlite3 note pointed at the same dead target.
NAV: every notes/ page was listed except 85-packaging.md, and qa-migration.md
was reachable only by following a link out of Testing, Building or How it
works. Both are in the nav now, which also clears the two "not included in the
nav" lines from every build. .venv/ is gitignored -- docs/requirements.txt
tells you to make one right there in the tree.
Verified: `mkdocs build --strict` is clean and no longer reports pages outside
the nav. qa/static is 123 passed, its 110 errors the same missing
vendor/opkg-utils as before. No .md file is read by any lane, so the check that
matters here is the build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bLGYpVAeVzAX6NeAxcU42
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
57c776ee63 |
test/ is deleted; qa/ is the only suite
The eight files were already gone from the working tree. What this commit does
is everything that had to follow them, because nothing else had been updated
and three of the four things pointing at test/ were on a build or release path.
WHAT WAS STILL WIRED TO IT
pkgs/klipper/build.sh ran test/test-chelper.py under `|| pkg_die`. With the
file gone `make packages` would have died on every
klipper build.
bin/verify.sh ran the same script and read a non-zero exit as
"c_helper.so does not match the shipped klippy tree
(stale build)" -- a false red on every release verify
rather than an error about a missing file. Its
ship-boundary check also globbed `find bin test
docker`; that is `find bin docker` now.
pytest.ini testpaths = test, so a bare `pytest` collected
nothing at all. It is `qa` now, which is what the
comment above it was already describing.
qa/static/ PY_GLOBS listed test/*.py and test/integration/*.py,
test_shell_syntax.py and _files() makes an empty glob a collection error
on purpose -- so the whole static lane failed to
collect, taking 93 unrelated tests with it.
Also dropped: the `pytest -- the host-side tree` step in ci.yml, the `test-py`
target with its .PHONY entry and two help lines, and qa/conftest.py's opening
paragraph explaining why qa/ sat beside test/ rather than inside it.
THE CHELPER SYMBOL CHECK IS GONE, NOT MOVED
It compared the .so's dynamic symbols against what klippy cdefs, and it was the
only check for a c_helper.so built from older sources than the klippy tree
beside it -- the failure that shipped once and bricked klippy startup on
hardware, because cffi resolves lazily and a stale .so imports cleanly.
qa/replica/test_install.py covers the ELF arch and nan2008 and nothing else.
What stands in its place is structural rather than a gate: pkgs/klipper
compiles the .so from the chelper sources of the very tree it ships, so the
mismatch is not something the recipe can produce. The comments in
pkgs/klipper/build.sh and docs/building.md now say that is the whole defence.
DOCS
docs/testing.md's "what is left in test/" section is now "test/ is gone", and
it names what went uncovered rather than leaving it to be found: the tool-frame
arithmetic in ff_toolchange, the chamber macros' per-model branching, the stamp
discipline in ff-startup.py, and the verification G-code against the shipped
macros. docs/hardware-testing.md said `make test-py` keeps the two
gcode/creator5-*.gcode files honest; it now says nothing does, and to read them
by hand when a macro is renamed. Paths corrected in docs/building.md,
docs/printer-replica.md and docs/qa-migration.md's intro.
Verified: `pytest` collects 366; qa/static is 136 passed, up from 43 because
the lane could not collect before this. Its 110 errors are this checkout's
missing vendor/opkg-utils, the same cause before and after. bin/verify.sh,
pkgs/klipper/build.sh and the Makefile parse; ci.yml is valid YAML.
Not touched, and stale before this commit: tools/python/README.md and
tools/supervisor/README.md still document test/integration/printer/case-*.sh
and a printer-exec.py that earlier commits removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016bLGYpVAeVzAX6NeAxcU42
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
965cf67d93 |
Compile the s6-rc database where the packages are installed
The replica already installs the feed with the printer's own opkg, and anvil-s6-rc already ships s6-rc-compile -- pkgs/3rdparty/s6-rc/build.sh keeps it deliberately, "the only way to recover a printer whose database is wrong without rebuilding a package on a laptop". So the compiler was sitting in the container the whole time, three lines after the install that puts the source tree there. bin/patch.sh section 5b-2 built a second, NATIVE skalibs+execline+s6+s6-rc instead -- four ./configure && make passes cached in work/.s6-native -- purely to get an s6-rc-compile that runs on x86. 110 lines, and its entire hazard was self-inflicted: s6-rc-compile bakes the #! of the execline the COMPILER was linked against into the oneshot runner, so a native stack configured with a different --prefix ships a database whose every oneshot ENOENTs on the printer. Compiling with the compiler we ship cannot get that wrong. The output is the same object. docs/notes/80-s6-migration.md records the measurement from 2026-08-28: one source tree, the host compiler and the target one under qemu, byte-identical db, n and resolve.cdb and identical servicedirs. That measurement was made to justify compiling on the host; it works just as well in the other direction. The rejected alternative in the old comment -- ship s6-rc-compile and compile at INSTALL time on the printer -- is a different thing and stays rejected. qa/static/test_s6rc_source.py is dropped. It was the only consumer of the native compiler, and it hard-failed on any checkout that had not run `make build`: 14 of the static suite's errors were it saying so. What goes with it is the timeout and edge assertions, which are now made nowhere; qa/replica/ test_s6rc.py still boots the real thing. bin/verify.sh keeps the half a host can still check, moved out of patch.sh: the oneshot runner's shebang, and that compiled/current resolves -- read as text straight out of the shipped payload, so it needs no mipsel binary. bin/patch.sh: 278 -> 207 lines. Static suite: 122 passed, errors 124 -> 110. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
214a4e58b9 |
anvil-core owns firmwareExe and start.sh; kill the 3.8.2 myth
Three things, all downstream of klippy having moved to $MODDIR and our own
CPython 3.13.
SECTIONS 8 AND 9 ARE GONE. Both files install at $MODDIR/prog/ and
anvil-link-prog.sh points the stock paths at them -- from run-append.sh on a
flash and from anvil-core's postinst on `opkg upgrade anvil-core`. Nothing
execs either between run.sh distributing the component and link-prog running,
so the component copy bought no ordering. It cost, though: our firmwareExe
with no payload behind it logs "no anvil-env.sh", finds no s6-svscan and then
`while true; do sleep 3600; done` -- a printer that boots to a blank screen for
ever and never reaches app_startup.sh's own recovery. Absent, that recovery
restores a firmwareExe from a version directory. So patch.sh now REMOVES both
from the component: the stock component carries them, and deleting the two cp
lines alone would have shipped FlashForge's originals.
SECTION 5c IS GONE. It did no work -- two du prints and a comment whose first
line was the falsehood below.
THE FALSEHOOD. "klippy runs on the stock 3.8.2, started by
/usr/prog/klipper/start.sh" had spread well past patch.sh, and it had reached
things people follow:
- docs/toolchange.md told owners to scp the ff_*.py extras to
/usr/prog/klipper/klippy/extras/ -- the wrong partition -- and warned that
an OTA would delete them there. Same wrong path in three ff-*.cfg headers
and in ff_toolchange.py.
- versions.env labelled our CPython 3.13 and its eighteen packages
"SHIPPED BUT NOT USED".
- docs/how-it-works.md said Moonraker runs on python3.8, contradicting line
126 of the same file.
All corrected. Historical notes keep their history and get a [Superseded: ...]
marker instead -- including 80-s6-migration.md's "s6-rc-compile is not
shipped", which pkgs/3rdparty/s6-rc/build.sh contradicts deliberately.
Found while reviewing: every docs/ reference to the ff-*.cfg pointed at
pkgs/anvil-core/payload/config/, which does not exist -- they are
anvil-klipper-config's. 28 references across 9 files, several of them broken
markdown links on the published site. All 24 relative links under docs/ now
resolve.
Gates: verify.sh syntax-checks prog/start.sh and prog/firmwareExe out of the
payload (firmwareExe was never syntax-checked before) and fails if the
component carries either. A new replica test asserts both stock paths are
symlinks into $MODDIR/prog and do not dangle. One existing replica test stubbed
firmwareExe by redirecting into $FE, which now writes THROUGH the symlink into
the packaged wrapper; it moves the link aside instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c185ef3285 |
The replica is a build tool; test/ is 28 unit tests
Two changes that turned out to be one.
MOVED, NOT DELETED. `test/ffsim/` and `test/integration/printer/` are
`tools/replica/`. They were never tests: bin/patch.sh calls
bin/build-payload.py, which starts a replica and assembles the payload by
running the PRINTER'S OWN opkg against / inside it -- so the "test harness"
was on the build path, and deleting it as asked would have broken `make
build`. qa/lib/replica.py drives the same directory unmodified. No content
changed; what changed is qa/lib/replica.py's build directory, three launcher
sys.path lines, the Makefile targets and the CI shellcheck glob.
tools/replica/README.md says who drives it and why it is not under test/.
CUT TO 28. The five surviving modules were 136 tests; they are 28, chosen on
one question: would this failure be SILENT? What is kept is the mistakes that
run perfectly and are wrong --
test_startup 8 no stamp unless the values are verifiably saved (a
stamp written early is a printer that has lost its
factory calibration for good), and the handover order
test_tool_transform 7 every one is a move to the wrong place: a sign, a
frame left applied while probing, a stale gcode_move
cache, a config a calibrated printer cannot parse
test_ffscreen 5 the four ways it can do harm rather than look wrong
test_chamber 4 the chamber mistakes that cost a print before anyone
sees them, plus the parse gate they rest on
test_gcode 4 a renamed macro, and the safe file being cold
What went: the panel narration, the geometry probe and parse_geometry, the
z_adjust and per-job-Z bookkeeping, the config layout checks, the G-code
bounds checks. Each is either a wrong number shown to a person, or a failure
the printer reports loudly the first time -- and qa/replica/test_boot_screen.py
already renders the real frames on the real machine.
The rewrite is not a delete: assertions were folded so a kept test asks its
question in every direction it can be got wrong (both models, both rotations,
all four ways the migration can fail), and each file's header now says what it
dropped and why, where the next person will open it.
Also gone: conftest's `rootfs` fixture, orphaned two commits ago. Nothing under
test/ needs the firmware, docker, or a network any more.
28 host-side tests pass in 0.15s. qa/static unchanged at 218. `make rootfs`
re-verified end to end from its new path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UzLebCrvc8ssWR5JuEsha4
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
465c9902f1 |
test/: drop the seven modules whose subject was the fake
Eight files, 46 tests and make-stock-fixture.sh. Each of the seven test modules built a stand-in -- a klippy stub, an argparse Namespace, a config tree -- elaborate enough that what they pinned down was the stand-in's behaviour rather than the printer's. What is left under test/ is the Klipper config gate and the three modules whose subject is our own Python: ff-startup, ffscreen, and the tool transform. make-stock-fixture.sh goes with them. docs/qa-migration.md argued for keeping it as the input a future build-path test would need, but it fed a gate that no longer exists, and a fixture generator with no consumer rots exactly as silently as a test that asserts nothing. What it had learnt is recorded in the doc so whoever ports the build path does not have to find it twice. The coverage this gives up is written down rather than glossed: a new section in docs/qa-migration.md names every dropped file, what it asserted, and what now asks the question instead -- for four of them, nothing. The one worth watching is the shebang-executable check, which qa/ makes only of qa/replica/actions/*.sh. Falling out of it: the rootfs fixture in test/conftest.py had no users left, and nothing under test/ needs the proprietary firmware any more. Comments in qa/static/test_probes.py, qa/replica/test_mcu_bringup.py, bin/patch.sh and ci.yml that pointed at the deleted files say what is true now; the tree listing in docs/building.md was stale from the run-tests.py removal and is rewritten. 136 host-side tests pass. qa/static is unchanged at 218. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UzLebCrvc8ssWR5JuEsha4 |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6aa622d2ff |
6,280 lines of case scripts reported 13 bits. They are 118 tests now.
test/run-tests.py is gone, and with it the bespoke harness that wrapped
pytest, re-parsed its JUnit XML, and drove thirteen replica case scripts to
get one boolean back from each. Every gate it ran is either a module under
qa/replica/ or deliberately retired.
PORTED. case-install.sh (579 lines, 1 bit) is test_install.py, 22 tests, asked
of the machine the printer's own app_startup.sh built rather than one the
harness assembled. case-upgrade.sh is 21, case-supervisor.sh 31,
case-boot-screen.sh 17, case-mcu-bringup.sh 8, and case-nginx.sh plus
case-moonraker313-s6.sh are test_web.py's 7.
RETIRED, subject deleted by the s6-rc switch: case-services.sh, case-nginx.sh,
case-moonraker.sh, case-moonraker313-s6.sh, case-camera.sh and
case-priority.sh drove $MODDIR/init.d/S* or sourced anvil-service.sh, neither
of which exists. Decided against real output -- `make test-services` failed
with "the payload ships no anvil-service.sh". case-pyext.sh and
case-moonraker313.sh went too: no gate, no target, no CI step, never run once.
RETIRED as a coverage decision: case-python.sh (test_web.py runs real Moonraker
on that interpreter, which is harder), case-libpath.sh, and case-recovery.sh.
The last is the brick-safety rollback and has no replacement; it is recorded in
docs/qa-migration.md rather than left to be discovered.
THREE ASSERTIONS HAD STOPPED MEANING ANYTHING, and each had been green for
months:
* case-mcu-bringup.sh's negative control had INVERTED. It asserted the
bring-up fails without LD_LIBRARY_PATH; CPython is --disable-shared now and
needs no library path, so the case's own skip branch was the one firing.
* case-upgrade.sh's headline assertion was VACUOUS. run-append.sh:150 does
`rm -rf $MODDIR/init.d` unconditionally, outside the manifest branch, so
"the update removed the stale S60web" passed regardless of the manifest
logic it was written to test.
* case-boot-screen.sh CONFIRMED ITS OWN ARITHMETIC, computing expected pixel
packing by calling ffscreen._pixel inside the replica -- a swapped R/B
matched itself.
case-install.sh's boot-log grep is not reproduced for the same reason: the log
line names every service in the boot set including the ones that then fail, so
grepping it cannot go red. The compiled database is asked instead.
TWO REAL BUGS FOUND ON THE WAY.
assemble.sh attached /stick.img with `losetup` and wrote the device to
/stick.loop, which NOTHING HAS EVER READ. Loop devices are the host's, not the
container's, so every USB_STICK=1 run leaked one permanently and a WSL2 kernel
has thirteen. Once they were gone every replica run failed with "failed to
setup loop device", which looks like a docker fault and is not. Detached now in
entrypoint.sh on the way out, and in qa's stop() for the held-open containers
that never take that path.
test_startup.py was 43 tests red: it built its argparse.Namespace by hand and
had never been taught --only-bringup / --no-bringup, and its FakeKlipper still
modelled start.sh plus a separate stop when klipper is restarted with one
`s6-svc -wr -t`.
CI AND RELEASE. The replica job was getting its package as a side effect of
run-tests.py running unpack/patch/pack; that is an explicit step now, as the
comment there predicted it would have to be. release.yml tested the exact files
it ships through sim-install.py, whose exit status the loop was discarding --
it is `REAL_PKG=<pkg> pytest ./qa/replica` per model now, 22 named results each.
WHAT IS NOT TESTED ANY MORE: bin/unpack.sh, patch.sh, pack.sh and verify.sh.
The packaging build on a synthetic stock package went with the runner and has
no counterpart in qa/. That is the path producing the .tgz a user flashes.
make-stock-fixture.sh is kept as the input a replacement needs.
Measured: qa/static 246 passed, qa/replica 110 passed and 8 blocked on hardware
the replica has not got (no /dev/ttyS*, no /dev/video*, so ok-all is
unreachable), test/ pytest 185 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BTAXgjtF8GdhqSxvdXCFW
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
abd4a01d73 |
Adopt the s6-rc boot graph -- DOES NOT BOOT YET, see below
Merge of qa-suite-bridgehead into the packaging line. The seven init.d/S*
scripts and the hand-written etc/s6 servicedirs are gone; anvil-core ships
etc/s6-rc/source/ and bin/patch.sh compiles it into etc/s6-rc/compiled/ after
the payload is installed. klipperDaemon joins firmwareExe and start.sh in
payload/prog/, all three linked into place by anvil-link-prog.sh.
WHAT IS KNOWN BROKEN. qa/replica/test_s6rc.py fails: the tree never comes up,
because s6-rc-init dies with
unable to supervise service directories in /run/s6-rc/servicedirs:
Value too large for defined data type
That is EOVERFLOW out of the readdir() loop in s6rc_servicedir_manage.c, which
checks errno after every entry. Measured, and these are ruled out:
* not the live directory's filesystem -- fails identically with -l on
/usr/data as with the default /run, and the inodes there are 3 and
1358496, nowhere near 32 bits
* not the database -- s6-rc-db lists all eleven services out of it, and a
database recompiled inside the replica by the TARGET s6-rc-compile fails
exactly like the one compiled natively on x86
* not a missing -D_FILE_OFFSET_BITS=64 -- both s6-svscan and s6-rc-init
reference readdir64 and never plain readdir
What is left is the toolchain. qa-suite-bridgehead built s6, s6-rc, execline
and skalibs with the Bootlin musl toolchain, where readdir has no EOVERFLOW
path at all; the packaging line builds them as recipes against the Ingenic
glibc, where it does. s6-svscan looked healthy only because it was started on
an EMPTY scandir, which never reaches the loop.
Everything else is green: 262 static tests, and bin/verify.sh passes on the
73MB package, including the graph's own "compiled s6-rc database" check.
Resolution notes: the five MOD_* switches stay deleted -- firmwareExe defaults
every gate to 1, so nothing changes -- but MOD_CAM is now worth having by hand
on a cameraless printer, because the camera is a longrun that waits out
timeout-up. anvil-service.sh went with init.d; nothing sourced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f67e7a269f |
ls pkg/ was thirty-eight entries and four of them mattered
pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and
carry no files of this repo move down a level into pkgs/3rdparty/. What is
left at the top is the four components a person actually edits: anvil-core,
klipper, moonraker, helixscreen.
THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory
rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day
somebody patches zlib nobody would agree which side it belongs on, and the
answer would be a rename touching CI and docs decided by taste. "Does it carry
files of ours" is a fact about the tree: it is exactly whether the recipe has
a payload/, prog/ or seed/, checkable in one assertion, and
test_recipe_layout.py now demands the move in both directions -- a 3rdparty
recipe that grows one of the three must come up, and a top-level recipe with
none of them must go down, or the top level slowly refills and the split stops
meaning anything.
ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id
to its directory and nothing else spells either path -- not a recipe, not
bin/build-packages.sh, not a test. Names stay unique across both levels
because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare
name; a duplicate would resolve to whichever level is searched first and
build the wrong thing silently, so that is a test too.
Recipes now source `. ./bin/common.sh` instead of counting `..` from $0.
Depth-counting would have needed two spellings for two levels, and it was
never buying anything: `. pkg/lib.sh` on the next line already required the
working directory to be the repo root.
A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files()
expanded a list of globs and returned whatever matched, so "pkgs/*/build.sh"
quietly stopped covering thirty-four recipes the moment they moved -- the lane
stayed green, and the only visible sign was a test count dropping by 34 in a
run nobody was counting. _files() now refuses a pattern that matches nothing.
That is the same failure the ash-compatibility check had a guard for and the
syntax check did not, which is why the guard reads the way it does.
AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of
.py helpers wholesale, so any test that imported one left a __pycache__ beside
it -- gitignored, invisible in a diff, copied by cp -a like anything else. The
package's contents depended on whether pytest had been run on the machine that
built it: two builds of one commit, different sha256, cause nowhere in the
tree. What it shipped was worthless as well as unstable, compiled by the build
image's python 3.11 for a printer running the 3.13 this repo cross-builds. The
sweep is in pkg_ship now, once, and the two hand-written copies in
pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe
that stages a tree of .py files, and the two recipes that had hit it had each
solved it locally while the third, which had not, was the one shipping.
Measured:
* the rename changes no package bytes. Same cached build trees, feed built
with the diff stashed and unstashed: all 41 .ipk match.
* with the bytecode sweep, anvil-core returns to exactly the bytes the
previous commit produced.
* no package in the feed contains a .pyc or a __pycache__.
* make qa-static: 247 passed (236 before this work + 11 layout tests).
* make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the
synthetic stock package. make test-py: 181 passed.
Not explained: anvil-python-dev produced two different sha256 from an
UNCHANGED work/pkg/python at two points today, under code that was identical
both times -- so it is not reproducible from a cached tree, and it is not this
commit's doing (the stashed and unstashed builds above agree on it). Worth a
look; the dev split in bin/build-packages.sh is where to start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
54a1e72cc4 |
A file that ships belongs to a package, and the path should say which
payload/ was organised by DESTINATION -- init.d/, bin/, etc/s6/,
klipper/config/ -- plus an assets/ directory of three .conf files. That was
right when there was one owner. With 38 recipes it held three different kinds
of file behind identical-looking paths:
* files a recipe stages into its .ipk,
* files that go to /usr/prog and CANNOT be in a package of ours, because
every path in one lands under $MODDIR,
* files that never ship as files at all -- run-pre.sh and run-append.sh are
spliced into FlashForge's own run.sh at build time.
The only thing recording which package owned any of it was a thirty-line
comment at the top of pkg/anvil-core/build.sh. A comment cannot be checked.
Files now live with the recipe that owns them, and the subtree says how they
ship: payload/ is staged into the .ipk laid out exactly as it lands under
$MODDIR, prog/ is placed on /usr/prog by patch.sh, seed/ is templated or
seeded user state. installer/ sits outside pkg/ and holds the two files that
are never files on a printer. qa/static/test_recipe_layout.py fails if a
fourth kind of directory appears, which is the failure this is actually for:
a misfiled file is caught by a build, an unruled directory is caught by
nothing.
anvil-core's recipe is a loop over a directory now. Six pkg_stage lines with
hand-written destinations plus a glob became one loop, because payload/ IS the
overlay -- a file added to that tree ships without an edit to build.sh.
A REAL BUG FELL OUT. anvil-core's PKG_STAMP_EXTRA was an eight-path list
naming payload/ and assets/ subtrees by hand, and one of those paths was
payload/helixscreen/ -- HelixScreen's printer-database entry, hashed into the
cache key of the package that does NOT contain it. Editing that json rebuilt
anvil-core and left pkg/helixscreen sitting in the cache. pkg_payload_hash is
now the one way a recipe keys its own files and it can only hash its own.
WHAT ELSE MOVED, and why each is not where it looks like it should be:
* ff-*.cfg stay with anvil-core rather than pkg/klipper. They are the
package that has always shipped them; ownership moves in the next commit,
with the two model-config packages. This one is a move, not a repackaging.
* printer.base.cfg is pkg/klipper's prog/, not payload/: it is FlashForge's
file with one block replaced and it lands beside their Klipper tree.
* moonraker.conf is anvil-core's for now; moonraker-custom.conf is
pkg/moonraker's seed/, because it is the user's and a package member is
overwritten on every upgrade by definition.
The cfgdir fixture is a merged copy of both config directories rather than a
path into the checkout. The split is a fact about which package ships what --
on the printer there is no split, the stock run.sh force-copies
printer.base.cfg into /usr/data/config/ beside the ff-*.cfg, which is why its
includes are bare and relative. A test about include resolution has to model
where files END UP or it is testing the repository's filing system.
qa/lib/replica.py's one payload mount is three, and entrypoint.sh reassembles
/tmp/payload out of them byte-for-byte as it was. Not re-pointed: every
case-*.sh copies out of /tmp/payload with 2>/dev/null, so a path that stops
resolving does not fail -- it copies nothing and the case runs green against
an empty $MODDIR.
Measured:
* all 41 .ipk byte-identical. anvil-core and helixscreen were rebuilt from
HEAD and from this tree and compared directly; the other 39 match the
feed built before the move.
* that comparison caught a mistake in this commit: the HelixScreen json was
mirrored one directory too high (helixscreen/printer_database.d/ instead
of helixscreen/config/printer_database.d/). A move that changes no bytes
is only a claim until it is diffed.
* anvil-core: two cold builds byte-identical.
* make test: 8 passed, 0 failed -- including unpack/patch/pack/verify on the
synthetic stock package, which is the gate that covers patch.sh end to end.
* make qa-static: 244 passed (236 + 8 new). make test-py: 181 passed.
Known gap, pre-existing and left alone: pkg/*/payload/bin/*.sh is in no ash
glob, so wifi-action.sh's dialect is unchecked. The old glob was payload/*.sh,
one level above it, so this is not a regression -- it belongs with the fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
1191f87411 |
The graph is asserted by compiling it, not by reading it.
Phase 8 made the boot a compiled s6-rc database. This is the gate for it, plus
the honest accounting of what the change did to the suites that tested the
boot it replaced.
qa/static/test_s6rc_source.py -- 14 tests, no docker, ~1s. It COMPILES
payload/etc/s6-rc/source/ with the native s6-rc-compile that bin/patch.sh
section 5b-2 builds, then reads every answer back out with s6-rc-db. Never out
of the source tree: the compiler is free to reject, rewrite or silently drop
what it was given, and `down` is the one that proved it -- accepted and
discarded, producing a byte-identical servicedir. A test that reads the source
would have called that a pass.
It asserts the bundle is the boot set; klipper waits for mcu-bringup,
ff-startup for moonraker and klipper, ui for ff-startup; mcu-bringup is
reachable but NOT in the bundle, which is the only way s6-rc can express "start
it, but not because the boot asked"; nginx does not gate moonraker, because as
an edge that preference would make `s6-rc -d change nginx` stop the API with
the UI; and the timeouts survive, without which a camera that never appears
holds the transition open for ever and the UI never starts.
Two of them earn their place beyond the graph:
* The generated oneshot-runner shebang must be $MODDIR/bin/execlineb. It
comes from the execline the COMPILER was linked against, so a native stack
configured with the wrong --prefix ships a database where every oneshot
dies with ENOENT and every longrun keeps working. Nothing else in the build
can see that.
* The oneshots' up/down are execline command lines carrying shell inside a
/bin/sh -c argument. bash -n and shellcheck parse the wrapper and pass, so
this extracts the body s6-rc-db actually stored and runs sh -n over it.
The NUL split is also the proof the shell arrived as ONE argument --
execline lexing it into several would be a script that runs its own first
line.
Both were checked by breaking them: dropping ui/dependencies.d/ff-startup
fails one named test with 13 still reporting, and a dangling `then` inside
wifi's up fails the parse test. A first attempt at that second control did not
fail, correctly -- `[ 1 = 1 ; then` is a missing `]`, a runtime error, not a
syntax one.
qa/replica/test_s6rc.py -- written, NEVER RUN, and it says so at the top. There
is no stock FlashForge package on this machine, so `make build` is impossible
and the replica image holds a pre-s6-rc payload. Its `box` fixture checks for
s6-rc-init once and fails with a single message naming `make build`, so the
lane reports twelve errors with one cause rather than forty assertion failures
sharing it. It covers what neither suite ever has: a kill -9'd daemon coming
back, which is the entire reason for shipping 14 binaries.
RETIRED: qa/replica/test_services.py, and the Service class it was the only
caller of. It was the port of case-services.sh and its subject was the wrapper
contract -- seven S* scripts loading a shared library, answering status, naming
themselves, rejecting an unknown verb. None of that exists. Three assertions
survived into test_s6rc.py: the scanner is the cross-built ELF we shipped, its
log is empty, every run in the scandir is executable.
THE OLD SUITE IS RED AND I HAVE NOT TOUCHED IT. case-services.sh,
case-nginx.sh and case-moonraker.sh assert the wrapper contract and the
MOD_S6=0 fallback; case-priority.sh calls svc_start_daemon; test_startup.py
asserts FF_SKIP_MCU_BRINGUP. All deleted. They also cannot be RUN here, and
editing shell I cannot execute to make it assert something new is the thing
this repo's own rule forbids -- never decide whether a script works by reading
it. docs/qa-migration.md now carries the table: per case, what it asserted and
where that lives now, including the two entries that honestly say "nowhere".
Retiring them is a coverage decision and it belongs to whoever has a stock
package and a real run.
bin/patch.sh also ships s6-rc-db beside the native compiler now; the static
lane needs it and it was the only piece missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
16d00be525 |
case-services.sh is 360 lines and one exit code. That contract is 44 tests now.
test/ has two test frameworks. run-tests.py + ffsim is 1,657 lines reimplementing what pytest already does -- skip accounting, gating, reporting, then parsing pytest's own JUnit XML back out -- and it calls pytest as gate #5 of about twenty. Underneath it, 6,280 lines of case-*.sh execute inside the qemu chroot and report 13 booleans between them; case-moonraker.sh is 1,059 lines that collapse to one pass/fail. Nothing runs in parallel and nothing can be selected: each replica gate is its own `docker run --privileged`, re-registering binfmt and re-assembling the mount layout, thirteen of them in sequence. All three follow from one decision -- THE REPLICA CONTAINER IS TREATED AS A TEST RUNNER INSTEAD OF AS A FIXTURE. That is what pushes assertions into ash, which yields one bit per container, which then needs a bespoke framework above pytest to aggregate the bits. qa/ is the replacement, built beside test/ rather than inside it, with two lanes and one framework. Both suites run in CI; test/ is still the release gate; nothing has been deleted. docs/qa-migration.md has the order the rest moves in, and the rule: a case script is deleted only once its replacement here has been green, so there is never a window where coverage drops. THE MACHINE IS UNCHANGED. qa/ reuses test/integration/printer/'s Dockerfile, entrypoint.sh, assemble.sh, binfmt.sh and seed-prog.sh as they are, and its `docker run` argv is deliberately ffsim's, minus --rm and plus -d. While both suites run they have to be testing the same machine or neither proves anything about the other. The only new trick is how the container is held open: entrypoint.sh ends by running the case script it was given, and the one it is given here touches a marker and sleeps, so setup completes exactly as it always has and the container simply does not exit. The marker is also the readiness signal, because setup ranges from under a second to over a minute and a constant would either waste the fast case or fail the slow one. TESTS RUN AGAINST A MACHINE THE REAL INSTALLER PRODUCED. The `printer` fixture is a replica with the package installed by the printer's own app_startup.sh, off a genuine FAT filesystem on /dev/sda1. The first version of this tree did what case-services.sh does -- copy payload/*.sh and payload/init.d/S* into $MODDIR by hand -- and two things were wrong with it. It is a second implementation of the install, so everything downstream asserted against a layout the harness built rather than one the installer produced, and the real installer could have broken without a single test noticing. And it can only place what is in payload/, so everything the BUILD produces was missing: s6 is a build output that bin/patch.sh stages into the package, so a hand-placed payload has no supervisor at all, which is why the tests had to carry a stand-in scanner and gate one assertion on whether it was in use. Installing for real fixes both and deletes the stand-in. Hand-placed, /usr/data/anvil held anvil-env.sh, anvil-service.sh, anvil.conf, init.d and etc/s6. Really installed it holds VERSION, backup, bin, config, config-installed, helixscreen, lib, libexec, moonraker, nginx and www as well -- bin/lib/libexec being the cross-built s6 and CPython 3.13. The file is SHORTER than the case script it replaces while reporting 44 results instead of one. case-install.sh's header records this same lesson from the first time somebody learned it: "An earlier version of this file replayed app_startup.sh by hand -- which meant a bug in our reading of it could never be caught." The install is the machine's own shell under qemu, so it is baked into an image once per package and cached under a tag derived from the package's md5 -- build-printer-image.sh's trick for the stock baseline. Rebuild the package and the tag changes, so the next run bakes again instead of testing yesterday's build; deliberately not a hand-written staleness check, because those are the ones that eventually get it wrong. A MISSING TOOL OR RESOURCE FAILS, and there is no flag. This tree briefly had --strict-skip=<lane|all>, which was ALLOW_SKIP inverted -- and ALLOW_SKIP's defect was never the direction it named things in, it was that the decision lived in the CI invocation. An accept-list has to be edited whenever a gate moves; a strict-list has to be remembered whenever a job is added; both degrade silently when the command line is wrong, which is the same disease as the printer-sim job that was gated on an unset secret and never ran once in its existence. So the verdict lives with the knowledge: no shellcheck, no pyflakes, no docker, no daemon, no replica image, no built package -- each fails where it is needed, with the fix in the message. There are no skips in this suite. MEASURED, on monstrofil/creator5-printer:latest, both suites agreeing on the same machine: 131 tests in 40s, all green; qa/static alone 1.3s with no docker; one assertion selected by node id 2.9s, against the whole 360-line case being the smallest unit test/ can run. Granularity checked by injecting the regression case-services.sh exists to catch -- deleting SVC_EXTRA_VERBS="|force-start" from S70klipper -- which failed one named test, printed the usage line that came back, left the other 41 reporting, and took 4s to re-run alone. Two bugs the port found in its own harness, both only visible by running it. bake.sh shipped without its executable bit, so the lane died at container init with 43 errors all saying "could not install the package" -- a file mode wearing an installer failure's clothes; test_action_scripts_are_executable now guards it, the same check the payload's own s6 run scripts already have. And an ELF magic assertion used `od -An`, which the printer's busybox od does not have -- the replica catching host-vs-printer drift in the test code itself, which is the entire reason it exists. Also here: qa/static/test_probes.py, which has no counterpart in test/. Printer.ps() and Printer.listening() are parsers, and a parser is where a harness goes quietly wrong -- a listening() that never matches makes every readiness assertion pass by the absence of contradiction. Neither failure looks like a failure; they look like a green suite. So they are tested against captured output, in the static lane, with no docker involved. Two things found while surveying, neither fixed here. case-pyext.sh is 567 lines with no gate in gates.py, no Makefile target and no CI step: it has never run. And case-mcu-bringup.sh:4 says the handshake is "tested elsewhere against a pty" -- there is no pty anywhere in the repo, and every documented past bug in ff_mcu_bringup.py lived in the code that comment excuses from testing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |