firmware/docs/testing.md

Testing: how we know it does not brick

The suite has two halves, and only one of them can answer the question. They are two directories — test/unit and test/integration — so which half a file belongs to is a fact about where it lives, not something you learn by reading its header or by remembering to tag it. test/run-tests.sh is the only thing in test/ that spans both.

Unit — test/unit, static and packaging. Needs no printer and no proprietary firmware: a synthetic fixture (test/unit/make-stock-fixture.sh) reproduces the package structure, so shell syntax, the bashism pass, the pytest config gate and the whole packaging pipeline run in CI on a clean machine.

Integration — test/integration, the printer replica. The real rootfs.squashfs, extracted from the stock package, chrooted under qemu-mipsel, with /usr/prog installed by FlashForge's own updater. The installer under test runs on the printer's busybox, tar, md5sum and unTar — not on Debian stand-ins — with a read-only root and writable prog/data partitions, exactly like the machine, and the package reaches it the way it reaches a real printer: on a FAT filesystem that the machine's own boot script finds and mounts. This is the half that can catch a brick, it needs the stock package, and .github/workflows/release.yml refuses to publish without it.

With PROG_DUMP (in test.env) pointed at a factory image, the replica has essentially nothing invented left: the klipper daemons, nginx, python3, moonraker and the printer's own OpenSSL 1.0.2d are all genuine, so package decryption is verified against the real implementation. What remains substituted is insmod/reboot/cmd_mcu, neutered because they would act on the host kernel or real hardware, and the partition sizes.

See printer-replica.md for what is authentic, what is stubbed, and what it still cannot tell you.

The tests

make test            # everything below

Gate	What it does	Replica
test-py	pytest. The Klipper config gate: every ff-*.cfg gcode body parses in Klipper's Jinja dialect, and the chamber macros are rendered per model to prove a chamber target is refused on a Creator 5 that has no heating element (test/unit) — plus, when a rootfs has been extracted, that every absolute path the payload names exists on the printer (test/integration)	partly
test-install	End-to-end. The package sits on a real FAT filesystem exposed as /dev/sda1, and the machine's own app_startup.sh runs verbatim through three boots: stick in -> it installs; stick still in -> it installs again (idempotence); stick pulled -> the machine boots with the mod running and the stock ps-watchdog satisfied. Asserts along the way: UI present and executable, boot scripts unmodified and still parsing, every installed script sh -n-clean under the printer's own busybox, Klipper owned by a service, c_helper.so still nan2008 MIPS, user printer.cfg preserved, the wrapper unchanged by a re-install	yes
test-mcu	Runs ff-mcu-bringup.py on the printer's own Python 3.8.2 in the exact environment start.sh sets — the only gate that executes our Python on the real interpreter, and the one that pins the LD_LIBRARY_PATH regression that shipped broken once	yes
test-recovery	Installs the mod, then flashes the stock package, and asserts the machine is genuinely back to stock byte-for-byte and the leftover payload is inert	yes

Four gates, deliberately. test-install boots the machine, so it independently covers what verify.sh checks, parses every installed script with the same busybox that test-ash used, and reads c_helper.so's ELF header the way test-abi did — the separate checks were the weaker copies.

The replica gates need make rootfs first, which extracts the printer's genuine root filesystem from the stock package's kernel-*.tar.xz. It is never committed — it is FlashForge's firmware. make test skips the replica half with a loud message when no stock package is configured; REQUIRE_PRINTER_SIM=1 turns that skip into a failure.

A skip is not a pass. run-tests.sh counts skips separately and exits non-zero if any gate did not run; ALLOW_SKIP=1 accepts the gap deliberately. This matters more with four gates than it did with twelve — and it is not hypothetical, see test-abi below.

Speed

Almost all of a replica run used to be setup, repeated per test case:

	per run
unpack the 182MB factory image into /usr/prog	22s
install the stock package into it, under qemu	37s
the test itself (three boots, install, re-install)	~30s

make printer-image does both of those once, at build time, and publishes the result. test/integration/printer/bake.sh is the part that cannot be a docker build step — the stock install needs binfmt_misc and chroot, so it runs in a privileged container and the result is committed. The md5 of the package that was installed is recorded in /usr/prog/.BASELINE; entrypoint.sh reinstalls only if a run asks for a different one.

With PRINTER_IMAGE set in test.env, a replica starts in 0.7s and the whole end-to-end update test takes ~70s. Without it, the same test spends a minute on setup before it begins. That image contains proprietary FlashForge firmware.

make test end to end, measured, with the image: 5m26s, and the sections say where it goes — run-tests.sh stamps every header with elapsed seconds:

== python checks (klipper config) ==                      [0s]
== build on the fixture ==                               [1s]
== extracting the printer rootfs ==                     [16s]
== python checks (rootfs) / MCU bring-up ==             [18s]
== end-to-end update on the replica ==                 [111s]
== recovery: a stock package reverts the mod ==        [250s]
                                                       [326s]

What is left is real work: two full package builds (the payload is 55MB through xz) and two replica runs. If it needs to get faster again, that is where to look — not in the harness.

What was dropped, and why

Tests that cannot fail are worse than no tests, because they read as coverage:

• test-abi's execution half. It was qemu "$f" -h || [ $? -lt 126 ], which accepts nearly every exit status — it could only fail if qemu itself was missing. Running the binaries for real is what the replica does.

• test-ash's md5sum -s probe. It printed "inconclusive" on every path and could not fail.

• The per-file sh -n "POSIX" loop in run-tests.sh. sh is bash on the build image, so it proved nothing that test-ash does not prove properly with the printer's own busybox. Its bashism grep also flagged local, which busybox ash supports.

• test-install's hand-written replay of app_startup.sh. It re-derived the glob, MACHINE and PID with sed and then called the installer itself — so a mistake in our reading of the boot script could never be caught. It now runs the boot script.

• The release workflow's final sim-install loop ignored the exit status of every run, so the last gate before publishing could not fail. It now stops on the first failure and runs with REQUIRE_PRINTER_SIM=1, which turns a skip into a failure too.

• test-printer-db, all of it. It began as a re-implementation of PrinterDetector's scoring formula, mirroring printer_detector.cpp line for line, so a change to the HelixScreen fork's formula left the test passing while reality moved. Narrowing it to scoring-independent invariants kept it honest but not useful enough to earn its place, and the whole check is gone.

• The hand-rolled bashism grep in run-tests.sh. It knew five constructs; shellcheck -s dash knows the whole SC3xxx family and now stands in its place. The same pass gave every replica launcher one shared test/integration/sim-image.sh for the docker plumbing, which also fixed two make test-ash bugs: it ran without the docker socket (so it always silently skipped), and it never read test.env, so it ignored PRINTER_IMAGE and rebuilt the local sim image every run.

• Eight checks, in one pass, once each had been read properly. The suite had grown to twelve gates and most of them were the weaker copy of a gate that survives:

  • test-abi had never asserted anything in CI. run-tests.sh deleted work/modpayload immediately before it ran and CI set KLIPPER_FORK="", so it had no targets on any run, skipped, exited 0 and was printed as ok. bin/patch.sh refuses to build a non-nan2008 c_helper.so anyway, and test-install reads the ELF header of the file that actually landed.
  • test-macros used jinja2.Environment() — the default {{ }} syntax. The configs contain no {{ at all, only single-brace expressions, so it validated {% %} block structure and never once looked inside an expression. test_default_delimiters_are_blind now pins that.
  • test-ash-conformance parsed the payload with the printer's busybox; test-install runs sh -n over every installed script with the same qemu'd busybox and then greps the boot log for Syntax error.
  • test-model-gate checked what verify.sh §8b/§9 checks and what pack.sh already refuses to build. Its header claimed it proved the two models ship different files; no such check existed in it.
  • test-base-cfg compared our printer.base.cfg against work/software/.../printer.base.cfg — which bin/patch.sh overwrites with our own file before the test reads it. It had been diffing our file against itself: green on a cold tree, red on a second make test, and blind to real drift either way. The comparison moved into bin/unpack.sh, where a pristine stock tree actually exists.
  • test-applets' command-word scan only extracted the first word of a simple, unprefixed command: if timeout 5 foo; then yielded ['if','echo','fi'], so the one failure its docstring named was invisible to it. Its allowlist had drifted to nine entries that excused nothing. The absolute-path half survives, because S50wifi's binaries sit on a branch no simulation reaches.
  • lint-danger's structural greps were satisfiable by comments — firmwareExe:44 and S80ui:16 mention helix and SAFE-MODE, so the checks passed with the logic deleted.
  • case-ui went with the helix-only decision.

Bugs these caught

Not theoretical. Writing them caught real bugs before any hardware was involved:

• backups were taken after the stock run.sh had already overwritten the files, so a restore would have restored the modified versions
• rm -rf $MODDIR/bin with an unset MODDIR expands to rm -rf /bin
• a crashed UI is a zombie, and kill -0 succeeds on a zombie — the crash detector called a dead UI healthy
• the packer emitted both models' filenames from one build, which would have handed a Creator 5 the Pro's firmware
• three checks in the brick lint had been quietly dead for months: two were guarded on a payload/boot.sh that no longer existed, and the file glob never matched firmwareExe or the init.d/S* scripts at all
• a skip counted as a pass for the whole life of the suite, which is how test-abi sat in it for months checking nothing while printing green