Oleksandr ShyshatskyiandClaude Opus 5 16d00be525 case-services.sh is 360 lines and one exit code. That contract is 44 tests now. ...

test/ has two test frameworks. run-tests.py + ffsim is 1,657 lines
reimplementing what pytest already does -- skip accounting, gating,
reporting, then parsing pytest's own JUnit XML back out -- and it calls
pytest as gate #5 of about twenty. Underneath it, 6,280 lines of
case-*.sh execute inside the qemu chroot and report 13 booleans between
them; case-moonraker.sh is 1,059 lines that collapse to one pass/fail.
Nothing runs in parallel and nothing can be selected: each replica gate
is its own `docker run --privileged`, re-registering binfmt and
re-assembling the mount layout, thirteen of them in sequence.

All three follow from one decision -- THE REPLICA CONTAINER IS TREATED
AS A TEST RUNNER INSTEAD OF AS A FIXTURE. That is what pushes assertions
into ash, which yields one bit per container, which then needs a bespoke
framework above pytest to aggregate the bits.

qa/ is the replacement, built beside test/ rather than inside it, with
two lanes and one framework. Both suites run in CI; test/ is still the
release gate; nothing has been deleted. docs/qa-migration.md has the
order the rest moves in, and the rule: a case script is deleted only
once its replacement here has been green, so there is never a window
where coverage drops.

THE MACHINE IS UNCHANGED. qa/ reuses test/integration/printer/'s
Dockerfile, entrypoint.sh, assemble.sh, binfmt.sh and seed-prog.sh as
they are, and its `docker run` argv is deliberately ffsim's, minus --rm
and plus -d. While both suites run they have to be testing the same
machine or neither proves anything about the other. The only new trick
is how the container is held open: entrypoint.sh ends by running the
case script it was given, and the one it is given here touches a marker
and sleeps, so setup completes exactly as it always has and the
container simply does not exit. The marker is also the readiness
signal, because setup ranges from under a second to over a minute and a
constant would either waste the fast case or fail the slow one.

TESTS RUN AGAINST A MACHINE THE REAL INSTALLER PRODUCED. The `printer`
fixture is a replica with the package installed by the printer's own
app_startup.sh, off a genuine FAT filesystem on /dev/sda1. The first
version of this tree did what case-services.sh does -- copy payload/*.sh
and payload/init.d/S* into $MODDIR by hand -- and two things were wrong
with it. It is a second implementation of the install, so everything
downstream asserted against a layout the harness built rather than one
the installer produced, and the real installer could have broken without
a single test noticing. And it can only place what is in payload/, so
everything the BUILD produces was missing: s6 is a build output that
bin/patch.sh stages into the package, so a hand-placed payload has no
supervisor at all, which is why the tests had to carry a stand-in
scanner and gate one assertion on whether it was in use.

Installing for real fixes both and deletes the stand-in. Hand-placed,
/usr/data/anvil held anvil-env.sh, anvil-service.sh, anvil.conf, init.d
and etc/s6. Really installed it holds VERSION, backup, bin, config,
config-installed, helixscreen, lib, libexec, moonraker, nginx and www as
well -- bin/lib/libexec being the cross-built s6 and CPython 3.13. The
file is SHORTER than the case script it replaces while reporting 44
results instead of one. case-install.sh's header records this same
lesson from the first time somebody learned it: "An earlier version of
this file replayed app_startup.sh by hand -- which meant a bug in our
reading of it could never be caught."

The install is the machine's own shell under qemu, so it is baked into
an image once per package and cached under a tag derived from the
package's md5 -- build-printer-image.sh's trick for the stock baseline.
Rebuild the package and the tag changes, so the next run bakes again
instead of testing yesterday's build; deliberately not a hand-written
staleness check, because those are the ones that eventually get it
wrong.

A MISSING TOOL OR RESOURCE FAILS, and there is no flag. This tree
briefly had --strict-skip=<lane|all>, which was ALLOW_SKIP inverted --
and ALLOW_SKIP's defect was never the direction it named things in, it
was that the decision lived in the CI invocation. An accept-list has to
be edited whenever a gate moves; a strict-list has to be remembered
whenever a job is added; both degrade silently when the command line is
wrong, which is the same disease as the printer-sim job that was gated
on an unset secret and never ran once in its existence. So the verdict
lives with the knowledge: no shellcheck, no pyflakes, no docker, no
daemon, no replica image, no built package -- each fails where it is
needed, with the fix in the message. There are no skips in this suite.

MEASURED, on monstrofil/creator5-printer:latest, both suites agreeing on
the same machine: 131 tests in 40s, all green; qa/static alone 1.3s with
no docker; one assertion selected by node id 2.9s, against the whole
360-line case being the smallest unit test/ can run. Granularity checked
by injecting the regression case-services.sh exists to catch -- deleting
SVC_EXTRA_VERBS="|force-start" from S70klipper -- which failed one named
test, printed the usage line that came back, left the other 41
reporting, and took 4s to re-run alone.

Two bugs the port found in its own harness, both only visible by
running it. bake.sh shipped without its executable bit, so the lane died
at container init with 43 errors all saying "could not install the
package" -- a file mode wearing an installer failure's clothes;
test_action_scripts_are_executable now guards it, the same check the
payload's own s6 run scripts already have. And an ELF magic assertion
used `od -An`, which the printer's busybox od does not have -- the
replica catching host-vs-printer drift in the test code itself, which is
the entire reason it exists.

Also here: qa/static/test_probes.py, which has no counterpart in test/.
Printer.ps() and Printer.listening() are parsers, and a parser is where
a harness goes quietly wrong -- a listening() that never matches makes
every readiness assertion pass by the absence of contradiction. Neither
failure looks like a failure; they look like a green suite. So they are
tested against captured output, in the static lane, with no docker
involved.

Two things found while surveying, neither fixed here.
case-pyext.sh is 567 lines with no gate in gates.py, no Makefile target
and no CI step: it has never run. And case-mcu-bringup.sh:4 says the
handshake is "tested elsewhere against a pty" -- there is no pty
anywhere in the repo, and every documented past bug in
ff_mcu_bringup.py lived in the code that comment excuses from testing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-28 09:35:14 +03:00

..

Dockerfile.build

case-services.sh is 360 lines and one exit code. That contract is 44 tests now.

2026-08-28 09:35:14 +03:00