mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-23 05:42:17 -05:00
Oleksandr ShyshatskyiandClaude Opus 5
e0dad3513c
The shell suite decided "this gate did not run" by grepping the child's
stdout:
if "$@" >"$out" 2>&1; then
if grep -qE '^[[:space:]]*SKIP:|[0-9]+ skipped' "$out"
then skip "$name"; else pass "$name"; fi
Text on stdout cannot tell you what happened inside a process. A launcher that
had ALREADY failed reached that same line by another road: the file that
sources config.env failed to load, so STOCK_TGZ_* was never set, so the
launcher concluded there was nothing to test, printed those five characters
and exited 0 -- on a machine with docker and the firmware sitting right there.
That is how five broken launchers shipped and stayed broken.
So the host half is Python and a skip is an exception. Output cannot imitate
one, and a gate that fails cannot claim it was skipped.
test/ffsim/ config loading, docker plumbing, gate reporting
test/run-tests.py the orchestrator
test/integration/ sim-install.py, sim-roundtrip.py, printer-exec.py,
extract-rootfs.py
test/integration/printer/*.sh and the whole payload stay shell. They are
executed by the printer's own busybox ash under qemu, and surviving that is a
large part of what the suite proves; rewriting them would test a Python the
printer does not have. build-printer-image.sh and make-stock-fixture.sh stay
too: both are `set -euo pipefail`, neither is a gate, and one is a release
tool run by hand.
Four things that used to report as skips are now failures -- a config.env that
does not parse, a missing case script, a missing package, a missing baseline.
A genuinely absent precondition is still a skip, because that is still true.
Three more bug classes are gone by construction rather than by being tested
for. The repo root is found by searching upward for bin/common.sh instead of
counting `..` from $0, so a launcher can move to any depth. Sourcing a config
with a syntax error is checked -- `.` returns non-zero and does NOT stop the
shell, so the old code carried on with half a file applied, which is exactly
the broken-looks-absent confusion again. And pytest counts come from the JUnit
XML rather than from grepping "N skipped".
test_script_roots.py becomes test_harness.py: the `..` check now has two
remaining shell targets rather than seven, and it gains a byte-compile of the
whole harness. That last one matters -- sim-install.py and friends are only
executed by `make test-install`, which needs docker and the firmware, so
without it a typo would wait quietly until release day. It also pins the
property that replaced the `..` counting, from four different depths.
VERIFIED, and the verification is the point, because this machine has no
docker daemon and these are exactly the scripts that cannot run here:
- Stubbed the docker binary and diffed the recorded `docker run` argv,
shell against Python, across five paths: install with a prebuilt image,
install with a local build, local build plus PROG_DUMP, roundtrip, and
printer-exec. IDENTICAL in all five.
- Ran the old orchestrator and the new one in the same tree: both give
6 passed, 1 failed, 2 skipped, exit 1, with the same gates in the same
order. (The failure is shellcheck, which lives in the build image.)
- pytest: 24 passed / 5 skipped bare, 29 passed against a real extracted
rootfs.
- Compared failure modes directly. The four cases above go from "exit 0,
reported skipped" to "exit 1, named". Case 2 is the sharpest: with a
missing case script the shell launcher exited 0 in silence, having cp'd
nothing and mounted a directory over /case.sh.
Still unproven here: the gates that need a live daemon. `make test-install`,
`test-mcu` and `test-recovery` want one run on a docker-capable machine. The
argv diff says they will be handed the same command as before; it cannot say
what docker does with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
2.9 KiB
Python
70 lines
2.9 KiB
Python
"""Host side of the test harness.
|
|
|
|
Everything in here runs on YOUR machine. The scripts that run on the printer
|
|
-- test/integration/printer/*.sh and the whole payload -- stay shell on
|
|
purpose: they are executed by the printer's own busybox ash under qemu, and
|
|
the fact that they survive that is a large part of what the suite proves.
|
|
Rewriting those in Python would test a Python the printer does not have.
|
|
|
|
Two things here exist to fix specific accidents.
|
|
|
|
`repo_root` finds the checkout by looking for a file that marks it, walking
|
|
upward until it does. The shell version computed it by counting `..` from
|
|
$0, so moving a launcher one directory deeper silently made $ROOT point at
|
|
test/ instead of the repo -- which happened, to five scripts at once, and
|
|
shipped. A search does not care where the file lives.
|
|
|
|
`Skip` is an exception, and that is the whole point of it. In the shell suite
|
|
"this gate did not run" was reported by printing the string "SKIP:" and
|
|
exiting 0, and the runner decided by grepping stdout for it. So a launcher
|
|
that had ALREADY failed -- one whose config never loaded because the path to
|
|
it was wrong -- concluded it had nothing to work on, printed those five
|
|
characters, exited 0, and was counted as a clean skip on a machine that could
|
|
have run it perfectly well. Text on stdout cannot tell you what happened
|
|
inside a process. An exception can, and nothing else can forge it.
|
|
"""
|
|
from pathlib import Path
|
|
|
|
__all__ = ["Skip", "Fail", "repo_root"]
|
|
|
|
|
|
class Skip(Exception):
|
|
"""This gate cannot run here, and that is a legitimate answer.
|
|
|
|
No docker, no proprietary firmware. Raise it only where the precondition
|
|
is genuinely absent -- never as a way to swallow a failure, which is the
|
|
exact confusion this class exists to end.
|
|
"""
|
|
|
|
|
|
class Fail(Exception):
|
|
"""This gate ran and the answer was no.
|
|
|
|
Also raised for a harness that is itself broken: a missing file, a bad
|
|
path, a docker that will not start. That merging is deliberate. The
|
|
alternative is a third state, and a third state is somewhere for "the
|
|
test did not really run" to hide again.
|
|
"""
|
|
|
|
|
|
# A file that exists in this repo and nowhere above it. bin/common.sh is
|
|
# sourced by every build script, so if it ever moves the build is broken
|
|
# anyway and a loud failure here is the least of it.
|
|
_MARKER = ("bin", "common.sh")
|
|
|
|
|
|
def repo_root(start=None):
|
|
"""The checkout root, found by looking rather than by counting.
|
|
|
|
Depth-independent by construction: this file can move to any directory in
|
|
the repo and it still resolves correctly, which is what makes the old
|
|
`cd "$(dirname "$0")/.."` class of bug impossible rather than merely
|
|
tested for.
|
|
"""
|
|
here = Path(start or __file__).resolve()
|
|
for d in (here,) + tuple(here.parents):
|
|
if (d.joinpath(*_MARKER)).is_file():
|
|
return d
|
|
raise Fail("not inside the repo: no %s above %s"
|
|
% ("/".join(_MARKER), here))
|