firmware/docs/testing.md

Testing: how we know it does not brick

qa/ is the suite. One framework (pytest), two lanes, and a result per assertion rather than per case script.

make qa            # both lanes
make qa-static     # seconds, any machine, any clone -- no docker
make qa-replica    # the gates that decide whether a package bricks a printer

Selection is pytest's: -k nginx, -m static, or a single test id. qa-migration.md is the design and the history.

The lanes

Lane	Needs	What it asks
static	nothing but the checkout	every shipped script parses and is free of bashisms, every name resolves, the recipe layout holds, the .ipks are what we mean to ship, and the boot graph says the right thing
replica	docker + qemu + the firmware	what the printer does -- on a machine the real app_startup.sh installed the real package onto

The replica lane needs a built package in work/out/*.tgz (make build) and a base image: put PRINTER_IMAGE in test.env, or let it build one from work/rootfs after make rootfs. The install is baked into an image once per package, keyed on the package's md5, so rebuilding gets you a fresh bake and not yesterday's.

The replica lane, module by module

module	asks
test_install.py	what the machine's own installer produced: the stock boot chain is untouched and still parses, the wrapper is installed and starts the supervisor, every installed script is sh -n-clean under the printer's own busybox, c_helper.so is still nan2008 MIPS, klipper and the UI are services in the compiled database, klipper depends on the bring-up, the config include set is wired up, the user's printer.cfg survived, a pristine backup/stock was kept, and a boot with no stick does not go looking for an update
test_upgrade.py	an update deletes what the last package shipped and only that -- the user's edits, their moonraker-custom.conf, their HelixScreen settings and anything nobody shipped all survive, over both the manifest path and the pre-manifest sweep
test_supervisor.py	the s6 we cross-compiled: all 13 binaries load on the printer's kernel, and s6-svwait -U really waits for readiness rather than returning on the fork
test_boot_screen.py	the first-boot screen renders on our own CPython 3.13, one screen's worth of correctly packed bytes, and degrades to "no screen" rather than raising when there is no panel
test_web.py	nginx and moonraker come up under s6, nginx comes back from a kill and stays down after a stop, and moonraker serves on :7125 on our interpreter
test_s6rc.py	the boot itself -- the scanner answers, s6-rc-init lays the scandir down, a killed daemon comes back
test_mcu_bringup.py	ff_mcu_bringup.py runs on FF_PYTHON in the shipped environment and names every port it could not open

Eight of test_s6rc.py's twelve cannot pass on a replica, and that is hardware, not a bug: there are no /dev/ttyS4,5,7 and no /dev/video*, so klippy never connects, camera times out, and the ok-all bundle is unreachable. See qa-migration.md. Everything else is green.

What is left in test/

test/run-tests.py is gone, and make test with it. What remains is not a suite:

make test-py   # the host-side pytest tree

• test/integration/test_*.py -- 185 host-side tests over the Klipper configs, the gcode, ff-startup.py, ffscreen.py, the toolchange transform and the offset sampling. Already pytest, already well-seamed; they were never part of the migration and can be adopted by pointing qa/ at them.
• test/ffsim/ -- 678 lines and no longer a framework. extract_rootfs is what make rootfs runs; Replica is what make boot-screen-sim uses.
• test/integration/printer/ -- the Dockerfile, entrypoint.sh, assemble.sh, binfmt.sh and seed-prog.sh that BUILD the replica. qa/ uses them unmodified; they were never the problem.
• test/test-chelper.py -- in neither suite: bin/patch.sh and bin/verify.sh call it directly at build time.

The build path is not tested. The packaging build on a synthetic stock package -- unpack/patch/pack/verify -- went with run-tests.py and has no counterpart in qa/. That is the path producing the .tgz a user flashes. See qa-migration.md.

Why the harness is Python

The host half of the harness — the orchestrator and the replica launchers — was shell until August 2026. It moved to Python for one specific reason, and it is worth stating because the rule it enforces is the one above.

In the shell suite, "this gate did not run" travelled between processes as the string SKIP: on stdout, and the runner decided by grepping for it:

if "$@" >"$out" 2>&1; then
    if grep -qE '^[[:space:]]*SKIP:|[0-9]+ skipped' "$out"
    then skip "$name"; else pass "$name"; fi

Text on stdout cannot tell you what happened inside a process. A launcher that had already failed would reach the same line by a different road: the file that sources config.env failed to load, so STOCK_TGZ_* was never set, so the launcher concluded there was nothing to test, printed those five characters and exited 0 — on a machine with docker and the firmware sitting right there. That is not a hypothetical; it is how five broken launchers shipped and stayed broken.

What changed:

• A skip is an exception (ffsim.Skip), not a string. Output cannot imitate it, and a gate that fails cannot accidentally claim it was skipped.
• The repo root is found by searching upward for bin/common.sh, not by counting .. from $0. Moving a launcher to a different depth used to break it silently; now it cannot.
• A broken config is not an absent one. Sourcing a file with a syntax error returns non-zero and does not stop the shell, so the old code carried on with half a config. ffsim.config checks each source and names the file that failed.
• pytest results are read from JUnit XML, not from grepping N skipped.
• The suite calls the gates as functions. There is no subprocess between a gate and the thing counting results, so there is no format to agree on.

The four things that used to be reported as skips and are now failures: a config.env that does not parse, a missing case script, a missing package, and a missing baseline. A genuinely absent precondition — no docker, no stock package — is still a skip, because that is still the truth.

Speed

Almost all of a replica run used to be setup, repeated per test case:

	per run
unpack the 182MB factory image into /usr/prog	22s
install the stock package into it, under qemu	37s
the test itself (three boots, install, re-install)	~30s

make printer-image does both of those once, at build time, and publishes the result. test/integration/printer/bake.sh is the part that cannot be a docker build step — the stock install needs binfmt_misc and chroot, so it runs in a privileged container and the result is committed. The md5 of the package that was installed is recorded in /usr/prog/.BASELINE; entrypoint.sh reinstalls only if a run asks for a different one.

With PRINTER_IMAGE set in test.env, a replica starts in 0.7s and the whole end-to-end update test takes ~70s. Without it, the same test spends a minute on setup before it begins. That image contains proprietary FlashForge firmware.

run-tests.py stamps every header with elapsed seconds, so a run says for itself where the time went. The headers, in the order they are printed:

== shell syntax ==
== no bashisms in the on-printer payload ==
== extracting the printer rootfs ==            (skipped if already extracted)
== python checks ==
== packaging, on a synthetic stock package ==
== build on the fixture ==
== printer replica ==                          (or: MCU bring-up runs on the
                                                printer's own Python)
== end-to-end update on the printer replica ==
== recovery: a stock package reverts the mod ==

What takes the time is real work: two full package builds (the payload is 55MB through xz) and two replica runs. If it needs to get faster again, that is where to look — not in the harness.

What was dropped, and why

Tests that cannot fail are worse than no tests, because they read as coverage:

• test-abi's execution half. It was qemu "$f" -h || [ $? -lt 126 ], which accepts nearly every exit status — it could only fail if qemu itself was missing. Running the binaries for real is what the replica does.

• test-ash's md5sum -s probe. It printed "inconclusive" on every path and could not fail.

• The per-file sh -n "POSIX" loop in run-tests.sh. sh is bash on the build image, so it proved nothing that test-ash does not prove properly with the printer's own busybox. Its bashism grep also flagged local, which busybox ash supports.

• test-install's hand-written replay of app_startup.sh. It re-derived the glob, MACHINE and PID with sed and then called the installer itself — so a mistake in our reading of the boot script could never be caught. The replica now runs the boot script itself, and qa/replica/test_install.py asserts against what it produced.

• The release workflow's final sim-install loop ignored the exit status of every run, so the last gate before publishing could not fail. It stops on the first failure now, and it is REAL_PKG=<pkg> pytest ./qa/replica per model -- 22 named results for each package that ships, where the loop gave one exit code it was throwing away.

• test-printer-db, all of it. It began as a re-implementation of PrinterDetector's scoring formula, mirroring printer_detector.cpp line for line, so a change to the HelixScreen fork's formula left the test passing while reality moved. Narrowing it to scoring-independent invariants kept it honest but not useful enough to earn its place, and the whole check is gone.

• The hand-rolled bashism grep in run-tests.sh. It knew five constructs; shellcheck -s dash knows the whole SC3xxx family and now stands in its place. The same pass gave every replica launcher one shared home for the docker plumbing — now test/ffsim/replica.py — which also fixed two make test-ash bugs: it ran without the docker socket (so it always silently skipped), and it never read test.env, so it ignored PRINTER_IMAGE and rebuilt the local sim image every run.

• Eight checks, in one pass, once each had been read properly. The suite had grown to twelve gates and most of them were the weaker copy of a gate that survives:

  • test-abi had never asserted anything in CI. run-tests.sh deleted work/modpayload immediately before it ran and CI set KLIPPER_FORK="", so it had no targets on any run, skipped, exited 0 and was printed as ok. bin/patch.sh refuses to build a non-nan2008 c_helper.so anyway, and qa/replica/test_install.py reads the ELF header of the file that actually landed.
  • test-macros used jinja2.Environment() — the default {{ }} syntax. The configs contain no {{ at all, only single-brace expressions, so it validated {% %} block structure and never once looked inside an expression. test_default_delimiters_are_blind now pins that.
  • test-ash-conformance parsed the payload with the printer's busybox; qa/replica/test_install.py runs sh -n over every installed script with the same qemu'd busybox.
  • test-model-gate checked what verify.sh §8b/§9 checks and what pack.sh already refuses to build. Its header claimed it proved the two models ship different files; no such check existed in it.
  • test-base-cfg compared our printer.base.cfg against work/software/.../printer.base.cfg — which bin/patch.sh overwrites with our own file before the test reads it. It had been diffing our file against itself: green on a cold tree, red on a second build, and blind to real drift either way. The comparison moved into bin/unpack.sh, where a pristine stock tree actually exists.
  • test-applets' command-word scan only extracted the first word of a simple, unprefixed command: if timeout 5 foo; then yielded ['if','echo','fi'], so the one failure its docstring named was invisible to it. Its allowlist had drifted to nine entries that excused nothing. The absolute-path half survives, because S50wifi's binaries sit on a branch no simulation reaches.
  • lint-danger's structural greps were satisfiable by comments — the comments in firmwareExe and S80ui mention helix and (at the time) SAFE-MODE, so the checks passed with the logic deleted.
  • case-ui went with the helix-only decision.

Bugs these caught

Not theoretical. Writing them caught real bugs before any hardware was involved:

• backups were taken after the stock run.sh had already overwritten the files, so a restore would have restored the modified versions
• rm -rf $MODDIR/bin with an unset MODDIR expands to rm -rf /bin
• a crashed UI is a zombie, and kill -0 succeeds on a zombie — the crash detector called a dead UI healthy
• the packer emitted both models' filenames from one build, which would have handed a Creator 5 the Pro's firmware
• three checks in the brick lint had been quietly dead for months: two were guarded on a payload/boot.sh that no longer existed, and the file glob never matched firmwareExe or the init.d/S* scripts at all
• a skip counted as a pass for the whole life of the suite, which is how test-abi sat in it for months checking nothing while printing green