mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-22 05:12:16 -05:00
Oleksandr Shyshatskyi
12bbd1e7dc
The probe profile was a bring-up aid. It set BUILD_APPLY=0, reinstalled the stock component byte-for-byte and wrote a report to the stick, so you could prove the update chain worked on your own machine -- the filename glob, the DES3 key, the md5 gate, run.sh execution -- before flashing anything that changed a file. That rehearsal has served its purpose. With it gone there was exactly one profile left, so the whole PROFILE indirection goes too: profiles/ is deleted and its six BUILD_* flags are plain defaults in bin/common.sh, which also means config.env now overrides them instead of being silently overridden. make default folds into make build; all-profiles and RECOVERY_PROFILE are gone with it. What is NOT gone is the undo button: flashing the stock FlashForge package still restores every file the mod touches, and case-recovery.sh still proves it. What is gone is the dry run. Note the loss: PROG_MB/DATA_MB used to come out of the probe report. The docs now point at df -h over ssh, which you only have after a successful flash.
130 lines
7.3 KiB
Markdown
130 lines
7.3 KiB
Markdown
# Testing: how we know it does not brick
|
|
|
|
The suite has two halves, and only one of them can answer the question.
|
|
|
|
**Half 1 — static and packaging.** Needs no printer and no proprietary
|
|
firmware: a synthetic fixture (`test/fixtures/make-stock-fixture.sh`)
|
|
reproduces the package *structure*, so shell syntax, bashism and brick-pattern
|
|
lint, the packaging pipeline and the model gates all run in CI on a clean
|
|
machine.
|
|
|
|
**Half 2 — the printer replica.** The real `rootfs.squashfs`, extracted from
|
|
the stock package, chrooted under `qemu-mipsel`, with `/usr/prog` installed by
|
|
FlashForge's own updater. The installer under test runs on the printer's
|
|
busybox, tar, md5sum and `unTar` — not on Debian stand-ins — with a read-only
|
|
root and writable prog/data partitions, exactly like the machine, and the
|
|
package reaches it the way it reaches a real printer: on a FAT filesystem that
|
|
the machine's own boot script finds and mounts. This is the
|
|
half that can catch a brick, it needs the stock package, and
|
|
`.github/workflows/release.yml` refuses to publish without it.
|
|
|
|
With `PROG_DUMP` (in `test.env`) pointed at a factory image, the replica has
|
|
essentially nothing invented left: the klipper daemons, `nginx`, `python3`,
|
|
`moonraker` and the printer's **own OpenSSL 1.0.2d** are all genuine, so
|
|
package decryption is verified against the real implementation. What remains
|
|
substituted is `insmod`/`reboot`/`cmd_mcu`, neutered because they would act on
|
|
the host kernel or real hardware, and the partition sizes.
|
|
|
|
See **[printer-replica.md](printer-replica.md)** for what is authentic, what
|
|
is stubbed, and what it still cannot tell you.
|
|
|
|
## The tests
|
|
|
|
```sh
|
|
make test # everything below
|
|
```
|
|
|
|
| Test | What it does | Replica |
|
|
|---|---|:-:|
|
|
| `test-lint` | Scans on-printer scripts for brick patterns: raw block-device writes, `rm -rf` of top-level paths, unguarded `rm -rf $VAR/`, missing backups, a boot chain that could end with no UI | |
|
|
| `test-model` | Each package's filename prefix matches the gate inside, the PID matches the model, and the two models ship **different** `firmwareExe` binaries | |
|
|
| `test-install` | **End-to-end.** The package sits on a real FAT filesystem exposed as `/dev/sda1`, and the machine's own `app_startup.sh` runs verbatim through three boots: stick in → it installs; stick still in → it installs again (idempotence); stick pulled → the machine boots with the mod running and the stock `ps`-watchdog satisfied. Asserts along the way: UI present and executable, boot scripts unmodified and still parsing, Klipper owned by a service, `c_helper.so` still nan2008 MIPS, user `printer.cfg` preserved, the wrapper unchanged by a re-install | ✓ |
|
|
| `test-recovery` | Installs the mod, then flashes the **stock** package, and asserts the machine is genuinely back to stock byte-for-byte and the leftover payload is inert | ✓ |
|
|
| `test-ui` | Drives the UI decision logic on the printer's shell: HelixScreen missing, crash-loop, SAFE-MODE latch and release, and that SAFE-MODE means **headless, not a respawn loop** | ✓ |
|
|
| `test-applets` | Every command the payload runs exists on the printer. Its busybox has no `timeout`, no `bash`, no `systemctl`; reaching for one is a blank screen at boot | ✓ |
|
|
| `test-ash` | Parses every on-printer script with the printer's own busybox 1.31.1 ash | ✓ |
|
|
| `test-abi` | Every shipped MIPS binary is `nan2008`/`mips32r2`/`o32` | |
|
|
|
|
The replica tests need `make rootfs` first, which extracts the printer's
|
|
genuine root filesystem from the stock package's `kernel-*.tar.xz`. It is
|
|
never committed — it is FlashForge's firmware. `make test` skips the replica
|
|
half with a loud message when no stock package is configured;
|
|
`REQUIRE_PRINTER_SIM=1` turns that skip into a failure.
|
|
|
|
## Speed
|
|
|
|
Almost all of a replica run used to be setup, repeated per test case:
|
|
|
|
| | per run |
|
|
|---|---|
|
|
| unpack the 182MB factory image into `/usr/prog` | 22s |
|
|
| install the stock package into it, under qemu | 37s |
|
|
| **the test itself** (three boots, install, re-install) | ~30s |
|
|
|
|
`make printer-image` does both of those once, at build time, and publishes the
|
|
result. `test/printer/bake.sh` is the part that cannot be a `docker build`
|
|
step — the stock install needs `binfmt_misc` and `chroot`, so it runs in a
|
|
privileged container and the result is committed. The md5 of the package that
|
|
was installed is recorded in `/usr/prog/.BASELINE`; `entrypoint.sh` reinstalls
|
|
only if a run asks for a different one.
|
|
|
|
With `PRINTER_IMAGE` set in `test.env`, a replica starts in **0.7s** and the
|
|
whole end-to-end update test takes **~70s**. Without it, the same test spends
|
|
a minute on setup before it begins. That image contains proprietary FlashForge
|
|
firmware.
|
|
|
|
`make test` end to end, measured, with the image: **5m26s**, and the sections
|
|
say where it goes — `run-tests.sh` stamps every header with elapsed seconds:
|
|
|
|
```
|
|
== brick-risk lint == [0s]
|
|
== build on the fixture == [1s]
|
|
== extracting the printer rootfs == [16s]
|
|
== applets / ash / ABI / UI safety == [18s]
|
|
== end-to-end update on the replica == [111s]
|
|
== recovery: a stock package reverts the mod == [250s]
|
|
[326s]
|
|
```
|
|
|
|
What is left is real work: two full package builds (the payload is 55MB
|
|
through `xz`) and two replica runs. If it needs to get faster again,
|
|
that is where to look — not in the harness.
|
|
|
|
## What was dropped, and why
|
|
|
|
Tests that cannot fail are worse than no tests, because they read as coverage:
|
|
|
|
- **`test-abi`'s execution half.** It was `qemu "$f" -h || [ $? -lt 126 ]`,
|
|
which accepts nearly every exit status — it could only fail if qemu itself
|
|
was missing. Running the binaries for real is what the replica does.
|
|
- **`test-ash`'s `md5sum -s` probe.** It printed "inconclusive" on every path
|
|
and could not fail.
|
|
- **The per-file `sh -n` "POSIX" loop in `run-tests.sh`.** `sh` is bash on the
|
|
build image, so it proved nothing that `test-ash` does not prove properly
|
|
with the printer's own busybox. Its bashism grep also flagged `local`, which
|
|
busybox ash supports.
|
|
- **`test-install`'s hand-written replay of `app_startup.sh`.** It re-derived
|
|
the glob, `MACHINE` and `PID` with `sed` and then called the installer
|
|
itself — so a mistake in our reading of the boot script could never be
|
|
caught. It now runs the boot script.
|
|
- **The release workflow's final `sim-install` loop** ignored the exit status
|
|
of every run, so the last gate before publishing could not fail. It now
|
|
stops on the first failure and runs with `REQUIRE_PRINTER_SIM=1`, which
|
|
turns a skip into a failure too.
|
|
|
|
## Bugs these caught
|
|
|
|
Not theoretical. Writing them caught real bugs before any hardware was
|
|
involved:
|
|
|
|
- backups were taken *after* the stock `run.sh` had already overwritten the
|
|
files, so a restore would have restored the modified versions
|
|
- `rm -rf $MODDIR/bin` with an unset `MODDIR` expands to `rm -rf /bin`
|
|
- a crashed UI is a *zombie*, and `kill -0` succeeds on a zombie — the crash
|
|
detector called a dead UI healthy
|
|
- the packer emitted both models' filenames from one build, which would have
|
|
handed a Creator 5 the Pro's firmware
|
|
- three checks in the brick lint had been quietly dead for months: two were
|
|
guarded on a `payload/boot.sh` that no longer existed, and the file glob
|
|
never matched `firmwareExe` or the `init.d/S*` scripts at all
|