mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
Oleksandr ShyshatskyiandClaude Opus 5
f8b925e48f
docs/notes/85-packaging.md chose opkg in 2026-08 on one argument: every package
is cross-built here anyway, so the winner is whichever format is cheapest to
PRODUCE -- and .ipk had an upstream tool where .apk meant abuild, a keypair and
an Alpine host. Two lines of that comparison were wrong.
apk mkpkg is an applet of apk itself: -I key:value metadata, -F directory, no
keypair, no abuild, no Alpine host. The "making a package" row described
apk-tools 2. And the on-device client was counted a wash when it is better than
one -- openssl and zlib were already cross-built for CPython, and libarchive
existed in this tree only for opkg, so adopting apk REMOVES a cross-build.
What made it possible at all is that apk resolves its database relative to
--root and uses the same root as the file destination, while / here is a
read-only squashfs -- measured: neither /lib/apk nor /etc/apk can be created.
pkgs/3rdparty/apk-tools/prefix.patch splits the two. Upstream already carries
root_fd and dest_fd separately and merely aliases them in context.c, so the
patch defaults the database root to $MODDIR and leaves files extracting at /.
That is opkg's compile-time VARDIR plus `dest root /`, in apk's spelling.
THE FEED IS SIGNED NOW, which it never was: opkg signed the index and
bin/build-packages.sh signed nothing, so nothing was verified on a printer.
apk verifies with raw PEM public keys, not X.509 -- measured on the machine,
where every certificate directory is empty. ./bin/apk-keygen.sh makes an RSA
pair, APK_SIGN_KEY in config.env points at the private half, and the public
half both seeds the USB stick (so the first install can verify) and ships in
anvil-core (so the printer keeps trusting). An empty APK_SIGN_KEY still builds
an unsigned feed, deliberately: `make packages` has to run on a bare checkout
or the packaging lane stops being a CI gate.
WHAT IT COST is a carried patch, against an internal with one consumer, on a
branch mid v2->v3 transition, on an architecture with no upstream apk port to
share the rebasing. That is the standing risk and it does not go away.
Four bugs this shook out, all silent, all found by a gate rather than by
reading:
* The prune deleted seventeen packages it had just built. The all -> noarch
mapping went into the emitter and not into pkg_pkgfile, so the prune looked
for a name nothing wrote. `make packages` exited 0.
* Whitespace-separated Depends dropped nineteen packages. opkg's pkg_parse.c
splits that field on ',' and nothing else, so the whole list became one
dependency whose name contained spaces and resolved to nothing -- and opkg
neither warned nor failed. `make build` shipped a payload with no Tornado
in it. test_the_payload_is_dependency_closed is what catches that now.
* Ownership was recorded as `nobody`. apk resolves an owner to a NAME against
<--root>/etc/passwd and the build lane runs --user with no passwd entry, so
every file in every package would have installed as uid 65534, with no
error at build or install time. build-packages.sh writes a two-line
synthetic passwd naming the build user root and points --root at it.
* Four recipes carried versions apk cannot parse -- two spelled a commit
+git<sha> where the grammar's hash slot is ~, x264 had a snapshot name
whose -2245 reads as a malformed release, and helixscreen had
0.99.115.creator5.3 where apk allows one trailing letter and then only a
suffix from a closed set of nine words. All four were valid opkg versions,
which is why nothing had noticed. pkg_version_ok gates every recipe now.
A v3 .apk begins with ADBd magic and is not a tar stream, so there is no apk
equivalent of the `ar -p | gunzip | tar` that hand-unpacked anvil-opkg: you
cannot open a package without a package manager. bin/build-payload.py puts the
cross-built binary on the stick at /mnt/apk instead. It is the same build
anvil-apk-tools contains, that package installs normally, and the bootstrap
copy never lands.
Two gates retired rather than moved. apk's installed database is plain text
(F: sets a directory, R: names a file in it), so the payload-ownership check
reads one file where it used to glob info/*.list. And it carries no clock at
all -- apk records no install time and mkpkg sets no build-time -- so the
Installed-Time normalisation case-build-payload.sh did on every build is gone,
and the test asserts the ABSENCE of a t: line instead. Clock-free by
construction rather than by repair.
Also removed: PKG_SECTION, which 45 recipes declared and nothing read once
apk's schema had no field for it, and the `make apk-host` target -- the
packager is built on demand by bin/build-packages.sh and stamped on the pin,
the way pkg_toolchain and pkg_buildpython already work.
Verified from a cold work/host and work/packages: 47 packages built and signed,
`make build` installs 35 and verifies them against the shipped public key,
`make qa-static` 195 passed, `make qa-replica` 135 passed -- the printer
installing off a simulated USB stick and booting. The installed set was checked
against the declared closure of the eleven roots: 35 expected, 35 installed,
nothing extra and nothing missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013rbKEyLXrzSE6wm3iNApVp
108 lines
4.9 KiB
Python
108 lines
4.9 KiB
Python
"""The qa suite: pytest is the only framework.
|
|
|
|
This is the only suite. It decides whether a package bricks a printer. The
|
|
harness that used to stand beside it, and the host-side test/ tree it wrapped,
|
|
are both gone -- see docs/qa-migration.md.
|
|
|
|
WHAT IS DIFFERENT HERE
|
|
|
|
The suite this replaced ran one `docker run --privileged` per case script and
|
|
got one boolean back, because the assertions lived in POSIX sh inside the qemu
|
|
chroot. 6,280 lines of case-*.sh reported 13 bits between them. The rule here
|
|
is:
|
|
|
|
shell inside the chroot performs actions; python on the host asserts.
|
|
|
|
Shell in qemu is irreplaceable for DOING things -- the printer's busybox, its
|
|
tar, its OpenSSL, and the fact that our scripts survive its ash is a large part
|
|
of what the suite proves. It is a bad place to JUDGE things. Where a fact is
|
|
only observable inside the replica, qa/lib/replica.py exposes a typed probe for
|
|
it; a case script never branches on it.
|
|
|
|
LANES
|
|
|
|
static needs nothing but the checkout and the lint tools. milliseconds.
|
|
replica needs docker + qemu + the proprietary firmware. seconds to
|
|
minutes. The gates that decide whether a package bricks.
|
|
|
|
Select with `-m static` / `-m replica`, or by path.
|
|
|
|
SKIPS, AND WHY THERE IS NO FLAG CONTROLLING THEM
|
|
|
|
A missing tool, a missing daemon or a missing firmware image is a FAILURE, at
|
|
the point that needs it. Not a skip, and not a skip that some flag can be asked
|
|
to promote later.
|
|
|
|
This suite had a `--strict-skip=<lane|all>` for exactly one commit, and it was
|
|
the wrong shape. It was ALLOW_SKIP inverted -- and ALLOW_SKIP's defect was
|
|
never which direction it named things in, it was that THE DECISION LIVED IN THE
|
|
CI INVOCATION. An accept-list in ci.yml has to be edited whenever a gate moves;
|
|
a strict-list in ci.yml has to be remembered whenever a job is added. Both
|
|
degrade silently when someone gets the command line wrong, which is the same
|
|
disease as the printer-sim job that was gated on an unset secret and therefore
|
|
never ran once in its entire existence.
|
|
|
|
So the verdict lives with the knowledge. `shellcheck` missing means the machine
|
|
cannot run the check, and no configuration exists in which that is a complete
|
|
run, so the test fails and says how to fix it. Same for pyflakes, for the docker
|
|
CLI, for the daemon, and for the replica image.
|
|
|
|
A GATE THAT DID NOT RUN MUST NEVER LOOK GREEN -- and the way to guarantee
|
|
that is to not have a mode in which it can.
|
|
|
|
`pytest.skip` stays available for the case it is actually for: a question that
|
|
does not APPLY to this configuration, as opposed to one this machine happens to
|
|
be unequipped to ask. A test that only means something on a Creator5Pro is the
|
|
shape that earns it. "I could not find the tool" is not.
|
|
|
|
Three skips do not meet that bar: test_packages.py's payload questions read
|
|
work/modpayload-root, which only bin/payload.sh produces. Noted rather than
|
|
hidden -- and note that `make test` DELETES that directory when it clears up
|
|
after its fixture build, so running the old suite quietly costs the static
|
|
lane three tests. See docs/qa-migration.md.
|
|
"""
|
|
import os
|
|
import sys
|
|
|
|
import pytest
|
|
|
|
# qa/ on the path so `from lib.replica import Printer` works from any subdir.
|
|
# Not a package install: this tree is not shipped and never will be, and a
|
|
# setup.py for it would be one more thing that can disagree with reality.
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
|
|
LANES = ("static", "replica")
|
|
|
|
|
|
def pytest_configure(config):
|
|
for lane in LANES:
|
|
config.addinivalue_line(
|
|
"markers", "%s: the %s lane -- see qa/conftest.py" % (lane, lane))
|
|
_check_distribution(config)
|
|
|
|
|
|
def _check_distribution(config):
|
|
"""Refuse the xdist mode that would start a container per worker.
|
|
|
|
The `printer` fixture is module-scoped, and xdist's DEFAULT distribution
|
|
(`--dist load`) hands out individual tests. Four workers pulling tests
|
|
from one module each instantiate that module's fixture, so `-n 4` would
|
|
assemble four replicas of the same machine instead of one -- four binfmt
|
|
registrations, four mount layouts, and on a locally built replica four
|
|
stock installs under qemu. It would still pass, slowly, which is why it
|
|
needs catching here rather than in review: the symptom is a suite that
|
|
got mysteriously slower when it was parallelised.
|
|
|
|
`loadscope` groups by module (`loadfile` by file, `loadgroup` by an
|
|
explicit mark) -- any of those keeps one container per module, which is
|
|
the whole isolation model. Enforced rather than defaulted, because
|
|
silently rewriting somebody's -n flag is its own kind of surprise.
|
|
"""
|
|
dist = getattr(config.option, "dist", "no")
|
|
if dist in ("no", "loadscope", "loadfile", "loadgroup"):
|
|
return
|
|
raise pytest.UsageError(
|
|
"--dist=%s would start one replica per worker per module. Use "
|
|
"--dist=loadscope (or loadfile), which keeps each module on one "
|
|
"worker -- see _check_distribution in qa/conftest.py." % dist)
|