firmware/docs/testing.md

Testing: how we know it does not brick

There is a second suite. qa/ is the replacement for everything below, built beside it rather than inside it, and both run in CI. It is one framework (pytest), and its results are per-assertion rather than per-case. Run make qa / make qa-static / make qa-replica.

Several gates below are now RED and cannot be fixed by editing them. Phase 8 of notes/80-s6-migration.md deleted payload/init.d/ and payload/anvil-service.sh, so the cases that assert the S* wrapper contract or the MOD_S6=0 fallback are asserting something that no longer exists. qa-migration.md lists which, what each one asserted, and where that assertion lives now. Retiring them needs a stock package and a real run -- not a reading.

The rest of this page describes the suite being migrated away from.

Everything lives in test/integration, and test/run-tests.py runs it. There is still a real seam inside it, though — what needs the proprietary firmware and what does not — and it is worth knowing which side of that seam a failure is on.

Without firmware. A synthetic fixture (test/integration/make-stock-fixture.sh) reproduces the package structure, so shell syntax, the bashism pass, the pytest config gate and the whole packaging pipeline run in CI on a clean machine. This is the half that catches a chamber heater declared on a machine that has no element for it — the replica cannot, because it never starts klippy.

These used to live in a directory of their own, test/unit. The split was there so a plain pull request had something to run; this repo has one maintainer who always has the firmware to hand, so it was a boundary being maintained for a contributor who never arrived. The tests moved rather than went away.

With firmware — the printer replica. The real rootfs.squashfs, extracted from the stock package, chrooted under qemu-mipsel, with /usr/prog installed by FlashForge's own updater. The installer under test runs on the printer's busybox, tar, md5sum and unTar — not on Debian stand-ins — with a read-only root and writable prog/data partitions, exactly like the machine, and the package reaches it the way it reaches a real printer: on a FAT filesystem that the machine's own boot script finds and mounts. This is the half that can catch a brick, it needs the stock package, and .github/workflows/release.yml refuses to publish without it.

With PROG_DUMP (in test.env) pointed at a factory image, the replica has essentially nothing invented left: the klipper daemons, nginx, python3, moonraker and the printer's own OpenSSL 1.0.2d are all genuine, so package decryption is verified against the real implementation. What remains substituted is insmod/reboot/cmd_mcu, neutered because they would act on the host kernel or real hardware, and the partition sizes.

See printer-replica.md for what is authentic, what is stubbed, and what it still cannot tell you.

The tests

make test            # the gates below, and more besides

make test does not invoke these targets: test/run-tests.py re-implements the work inline, and adds three things the table has no row for — a shell syntax parse, a shellcheck -s dash bashism pass over the on-printer payload, and a full unpack/patch/pack/verify build on a synthetic stock fixture.

Gate	What it does	Replica
test-py	pytest, the whole test/ tree — 186 tests in twelve files (five of them skip until make rootfs has run). The Klipper config gate (test_chamber.py): every ff-*.cfg gcode body parses in Klipper's Jinja dialect, and the chamber macros are rendered per model to prove a chamber target is refused on a Creator 5 that has no heating element. Plus test_paths.py (every absolute path the payload names exists on the printer, once a rootfs has been extracted), test_includes.py (the [include ff-*.cfg] block is exactly the expected set, in order), test_config_ownership.py (every mod-owned config carries its DO-NOT-EDIT banner and moonraker.conf does not), test_gcode.py (every command in gcode/*.gcode is one our configs define) and test_harness.py (the harness's own self-checks)	partly
test-install	End-to-end. The package sits on a real FAT filesystem exposed as /dev/sda1, and the machine's own app_startup.sh runs verbatim through three boots: stick in -> it installs; stick still in -> it installs again (idempotence); stick pulled -> the machine boots with the mod running and the stock ps-watchdog satisfied. Asserts along the way: UI present and executable, boot scripts unmodified and still parsing, every installed script sh -n-clean under the printer's own busybox, Klipper owned by a service, c_helper.so still nan2008 MIPS, user printer.cfg preserved, the wrapper unchanged by a re-install	yes
test-moonraker	Installs the payload the way an update does and then runs the shipped tools, on our own CPython 3.13 (FF_PYTHON). anvil-env.sh must produce a working interpreter; moonraker's authorization component must import on it. Then init.d/S62moonraker must actually start moonraker from /usr/data/anvil/moonraker/moonraker.py (checked in /proc/<pid>/cmdline, along with TMPDIR off the ramdisk), answer on :7125, report itself in status, clear its pidfile on stop, come back on a new pid after restart — and stopping it must leave S60nginx running, which is what splitting S60web in two was for	yes
test-mcu	Runs ff_mcu_bringup.py on our own CPython 3.13 (FF_PYTHON) in the exact environment start.sh sets — the gate that pins the LD_LIBRARY_PATH regression that shipped broken once, and now also proves the bring-up script survives the interpreter switch	yes
test-recovery	Installs the mod, then flashes the stock package, and asserts firmwareExe, app_startup.sh and klipper/start.sh are back byte-for-byte and the leftover payload is inert. It does not check shadow, the klippy tree or Moonraker — and Moonraker genuinely is not restored, see how-it-works	yes

Few entry points, deliberately. test-install boots the machine, so it independently covers what verify.sh checks, parses every installed script with the same busybox that test-ash used, and reads c_helper.so's ELF header the way test-abi did — the separate checks were the weaker copies.

The replica gates need make rootfs first, which extracts the printer's genuine root filesystem from the stock package's kernel-*.tar.xz. It is never committed — it is FlashForge's firmware. make test skips the replica half with a loud message when no stock package is configured; REQUIRE_PRINTER_SIM=1 turns that skip into a failure.

A skip is not a pass. run-tests.py counts skips separately and exits non-zero if any gate did not run. This matters more the fewer gates there are — and it is not hypothetical, see test-abi below.

Accepting a gap is deliberate and has two forms:

ALLOW_SKIP=1                              accept any gate that did not run
ALLOW_SKIP="pytest,the printer replica"   accept exactly these

Use the named form anywhere the setting outlives the moment — CI does. 1 is a standing promise never to notice a skip again: put it in a workflow and a replica gate that starts skipping on a machine that has the firmware is accepted in silence, forever. The named form fails on anything else.

Either way every skip is listed again in the summary, with its reason and whether it was accepted, and under GitHub Actions each becomes an annotation. Nobody reads a green job's log, so a gap that exists only in stdout is a gap accepted invisibly — which is the failure this whole harness is built around. The pytest gate names the individual tests that skipped rather than reporting a count.

Why the harness is Python

The host half of the harness — the orchestrator and the replica launchers — was shell until August 2026. It moved to Python for one specific reason, and it is worth stating because the rule it enforces is the one above.

In the shell suite, "this gate did not run" travelled between processes as the string SKIP: on stdout, and the runner decided by grepping for it:

if "$@" >"$out" 2>&1; then
    if grep -qE '^[[:space:]]*SKIP:|[0-9]+ skipped' "$out"
    then skip "$name"; else pass "$name"; fi

Text on stdout cannot tell you what happened inside a process. A launcher that had already failed would reach the same line by a different road: the file that sources config.env failed to load, so STOCK_TGZ_* was never set, so the launcher concluded there was nothing to test, printed those five characters and exited 0 — on a machine with docker and the firmware sitting right there. That is not a hypothetical; it is how five broken launchers shipped and stayed broken.

What changed:

• A skip is an exception (ffsim.Skip), not a string. Output cannot imitate it, and a gate that fails cannot accidentally claim it was skipped.
• The repo root is found by searching upward for bin/common.sh, not by counting .. from $0. Moving a launcher to a different depth used to break it silently; now it cannot.
• A broken config is not an absent one. Sourcing a file with a syntax error returns non-zero and does not stop the shell, so the old code carried on with half a config. ffsim.config checks each source and names the file that failed.
• pytest results are read from JUnit XML, not from grepping N skipped.
• The suite calls the gates as functions. There is no subprocess between a gate and the thing counting results, so there is no format to agree on.

The four things that used to be reported as skips and are now failures: a config.env that does not parse, a missing case script, a missing package, and a missing baseline. A genuinely absent precondition — no docker, no stock package — is still a skip, because that is still the truth.

Speed

Almost all of a replica run used to be setup, repeated per test case:

	per run
unpack the 182MB factory image into /usr/prog	22s
install the stock package into it, under qemu	37s
the test itself (three boots, install, re-install)	~30s

make printer-image does both of those once, at build time, and publishes the result. test/integration/printer/bake.sh is the part that cannot be a docker build step — the stock install needs binfmt_misc and chroot, so it runs in a privileged container and the result is committed. The md5 of the package that was installed is recorded in /usr/prog/.BASELINE; entrypoint.sh reinstalls only if a run asks for a different one.

With PRINTER_IMAGE set in test.env, a replica starts in 0.7s and the whole end-to-end update test takes ~70s. Without it, the same test spends a minute on setup before it begins. That image contains proprietary FlashForge firmware.

run-tests.py stamps every header with elapsed seconds, so a run says for itself where the time went. The headers, in the order they are printed:

== shell syntax ==
== no bashisms in the on-printer payload ==
== extracting the printer rootfs ==            (skipped if already extracted)
== python checks ==
== packaging, on a synthetic stock package ==
== build on the fixture ==
== printer replica ==                          (or: MCU bring-up runs on the
                                                printer's own Python)
== end-to-end update on the printer replica ==
== recovery: a stock package reverts the mod ==

What takes the time is real work: two full package builds (the payload is 55MB through xz) and two replica runs. If it needs to get faster again, that is where to look — not in the harness.

What was dropped, and why

Tests that cannot fail are worse than no tests, because they read as coverage:

• test-abi's execution half. It was qemu "$f" -h || [ $? -lt 126 ], which accepts nearly every exit status — it could only fail if qemu itself was missing. Running the binaries for real is what the replica does.

• test-ash's md5sum -s probe. It printed "inconclusive" on every path and could not fail.

• The per-file sh -n "POSIX" loop in run-tests.sh. sh is bash on the build image, so it proved nothing that test-ash does not prove properly with the printer's own busybox. Its bashism grep also flagged local, which busybox ash supports.

• test-install's hand-written replay of app_startup.sh. It re-derived the glob, MACHINE and PID with sed and then called the installer itself — so a mistake in our reading of the boot script could never be caught. It now runs the boot script.

• The release workflow's final sim-install loop ignored the exit status of every run, so the last gate before publishing could not fail. It now stops on the first failure and runs with REQUIRE_PRINTER_SIM=1, which turns a skip into a failure too.

• test-printer-db, all of it. It began as a re-implementation of PrinterDetector's scoring formula, mirroring printer_detector.cpp line for line, so a change to the HelixScreen fork's formula left the test passing while reality moved. Narrowing it to scoring-independent invariants kept it honest but not useful enough to earn its place, and the whole check is gone.

• The hand-rolled bashism grep in run-tests.sh. It knew five constructs; shellcheck -s dash knows the whole SC3xxx family and now stands in its place. The same pass gave every replica launcher one shared home for the docker plumbing — now test/ffsim/replica.py — which also fixed two make test-ash bugs: it ran without the docker socket (so it always silently skipped), and it never read test.env, so it ignored PRINTER_IMAGE and rebuilt the local sim image every run.

• Eight checks, in one pass, once each had been read properly. The suite had grown to twelve gates and most of them were the weaker copy of a gate that survives:

  • test-abi had never asserted anything in CI. run-tests.sh deleted work/modpayload immediately before it ran and CI set KLIPPER_FORK="", so it had no targets on any run, skipped, exited 0 and was printed as ok. bin/patch.sh refuses to build a non-nan2008 c_helper.so anyway, and test-install reads the ELF header of the file that actually landed.
  • test-macros used jinja2.Environment() — the default {{ }} syntax. The configs contain no {{ at all, only single-brace expressions, so it validated {% %} block structure and never once looked inside an expression. test_default_delimiters_are_blind now pins that.
  • test-ash-conformance parsed the payload with the printer's busybox; test-install runs sh -n over every installed script with the same qemu'd busybox and then greps the boot log for Syntax error.
  • test-model-gate checked what verify.sh §8b/§9 checks and what pack.sh already refuses to build. Its header claimed it proved the two models ship different files; no such check existed in it.
  • test-base-cfg compared our printer.base.cfg against work/software/.../printer.base.cfg — which bin/patch.sh overwrites with our own file before the test reads it. It had been diffing our file against itself: green on a cold tree, red on a second make test, and blind to real drift either way. The comparison moved into bin/unpack.sh, where a pristine stock tree actually exists.
  • test-applets' command-word scan only extracted the first word of a simple, unprefixed command: if timeout 5 foo; then yielded ['if','echo','fi'], so the one failure its docstring named was invisible to it. Its allowlist had drifted to nine entries that excused nothing. The absolute-path half survives, because S50wifi's binaries sit on a branch no simulation reaches.
  • lint-danger's structural greps were satisfiable by comments — the comments in firmwareExe and S80ui mention helix and (at the time) SAFE-MODE, so the checks passed with the logic deleted.
  • case-ui went with the helix-only decision.

Bugs these caught

Not theoretical. Writing them caught real bugs before any hardware was involved:

• backups were taken after the stock run.sh had already overwritten the files, so a restore would have restored the modified versions
• rm -rf $MODDIR/bin with an unset MODDIR expands to rm -rf /bin
• a crashed UI is a zombie, and kill -0 succeeds on a zombie — the crash detector called a dead UI healthy
• the packer emitted both models' filenames from one build, which would have handed a Creator 5 the Pro's firmware
• three checks in the brick lint had been quietly dead for months: two were guarded on a payload/boot.sh that no longer existed, and the file glob never matched firmwareExe or the init.d/S* scripts at all
• a skip counted as a pass for the whole life of the suite, which is how test-abi sat in it for months checking nothing while printing green