mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-23 13:52:16 -05:00
Oleksandr ShyshatskyiandClaude Opus 5
e0dad3513c
The shell suite decided "this gate did not run" by grepping the child's
stdout:
if "$@" >"$out" 2>&1; then
if grep -qE '^[[:space:]]*SKIP:|[0-9]+ skipped' "$out"
then skip "$name"; else pass "$name"; fi
Text on stdout cannot tell you what happened inside a process. A launcher that
had ALREADY failed reached that same line by another road: the file that
sources config.env failed to load, so STOCK_TGZ_* was never set, so the
launcher concluded there was nothing to test, printed those five characters
and exited 0 -- on a machine with docker and the firmware sitting right there.
That is how five broken launchers shipped and stayed broken.
So the host half is Python and a skip is an exception. Output cannot imitate
one, and a gate that fails cannot claim it was skipped.
test/ffsim/ config loading, docker plumbing, gate reporting
test/run-tests.py the orchestrator
test/integration/ sim-install.py, sim-roundtrip.py, printer-exec.py,
extract-rootfs.py
test/integration/printer/*.sh and the whole payload stay shell. They are
executed by the printer's own busybox ash under qemu, and surviving that is a
large part of what the suite proves; rewriting them would test a Python the
printer does not have. build-printer-image.sh and make-stock-fixture.sh stay
too: both are `set -euo pipefail`, neither is a gate, and one is a release
tool run by hand.
Four things that used to report as skips are now failures -- a config.env that
does not parse, a missing case script, a missing package, a missing baseline.
A genuinely absent precondition is still a skip, because that is still true.
Three more bug classes are gone by construction rather than by being tested
for. The repo root is found by searching upward for bin/common.sh instead of
counting `..` from $0, so a launcher can move to any depth. Sourcing a config
with a syntax error is checked -- `.` returns non-zero and does NOT stop the
shell, so the old code carried on with half a file applied, which is exactly
the broken-looks-absent confusion again. And pytest counts come from the JUnit
XML rather than from grepping "N skipped".
test_script_roots.py becomes test_harness.py: the `..` check now has two
remaining shell targets rather than seven, and it gains a byte-compile of the
whole harness. That last one matters -- sim-install.py and friends are only
executed by `make test-install`, which needs docker and the firmware, so
without it a typo would wait quietly until release day. It also pins the
property that replaced the `..` counting, from four different depths.
VERIFIED, and the verification is the point, because this machine has no
docker daemon and these are exactly the scripts that cannot run here:
- Stubbed the docker binary and diffed the recorded `docker run` argv,
shell against Python, across five paths: install with a prebuilt image,
install with a local build, local build plus PROG_DUMP, roundtrip, and
printer-exec. IDENTICAL in all five.
- Ran the old orchestrator and the new one in the same tree: both give
6 passed, 1 failed, 2 skipped, exit 1, with the same gates in the same
order. (The failure is shellcheck, which lives in the build image.)
- pytest: 24 passed / 5 skipped bare, 29 passed against a real extracted
rootfs.
- Compared failure modes directly. The four cases above go from "exit 0,
reported skipped" to "exit 1, named". Case 2 is the sharpest: with a
missing case script the shell launcher exited 0 in silence, having cp'd
nothing and mounted a directory over /case.sh.
Still unproven here: the gates that need a live daemon. `make test-install`,
`test-mcu` and `test-recovery` want one run on a docker-capable machine. The
argv diff says they will be handed the same command as before; it cannot say
what docker does with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
48 lines
1.8 KiB
Python
48 lines
1.8 KiB
Python
"""Running one gate on its own, from the command line.
|
|
|
|
`make test-install` and friends land here. The suite does not: run-tests.py
|
|
imports the gate functions and calls them, so nothing between a gate and the
|
|
thing counting results has to agree on a text format.
|
|
|
|
Which is why the "SKIP:" printed below is now only a message to a person. It
|
|
used to be a protocol -- run-tests.sh decided a gate had been skipped by
|
|
grepping its child's stdout for that string, so a launcher that printed it
|
|
after already failing was counted as cleanly skipped. Nothing parses it any
|
|
more. If you find yourself writing a grep for it, that is the bug coming back.
|
|
"""
|
|
import os
|
|
import sys
|
|
import traceback
|
|
|
|
from . import Fail, Skip
|
|
|
|
|
|
def main(fn):
|
|
"""Run fn(), map its outcome to an exit code, and say what happened.
|
|
|
|
A skip exits 0: not being able to run a gate on this machine is a normal
|
|
state for a laptop without docker or without the proprietary firmware.
|
|
REQUIRE_PRINTER_SIM=1 makes it an error instead, which is what a release
|
|
wants -- a package that shipped without the replica gates running has not
|
|
been tested for the thing that matters.
|
|
"""
|
|
try:
|
|
fn()
|
|
except Skip as exc:
|
|
if os.environ.get("REQUIRE_PRINTER_SIM") == "1":
|
|
sys.stderr.write(" FAIL: %s (REQUIRE_PRINTER_SIM=1)\n" % exc)
|
|
return 1
|
|
sys.stdout.write(" SKIP: %s\n" % exc)
|
|
return 0
|
|
except Fail as exc:
|
|
sys.stderr.write(" FAIL: %s\n" % exc)
|
|
return 1
|
|
except Exception:
|
|
# A broken harness, not a failed test. The traceback is the point:
|
|
# guessing from a one-line message is how a working launcher gets
|
|
# rewritten by mistake.
|
|
sys.stderr.write(" FAIL: the harness itself broke\n")
|
|
traceback.print_exc()
|
|
return 1
|
|
return 0
|