firmware

starred/firmware

0 Fork 0

mirror of https://github.com/Klipper4FlashForge/firmware.git synced 2026-10-22 13:22:16 -05:00

Commit Graph

Author SHA1 Message Date

Author	SHA1	Message	Date
Oleksandr ShyshatskyiandClaude Opus 5	f8b925e48f	Replace opkg with apk, and sign the feed docs/notes/85-packaging.md chose opkg in 2026-08 on one argument: every package is cross-built here anyway, so the winner is whichever format is cheapest to PRODUCE -- and .ipk had an upstream tool where .apk meant abuild, a keypair and an Alpine host. Two lines of that comparison were wrong. apk mkpkg is an applet of apk itself: -I key:value metadata, -F directory, no keypair, no abuild, no Alpine host. The "making a package" row described apk-tools 2. And the on-device client was counted a wash when it is better than one -- openssl and zlib were already cross-built for CPython, and libarchive existed in this tree only for opkg, so adopting apk REMOVES a cross-build. What made it possible at all is that apk resolves its database relative to --root and uses the same root as the file destination, while / here is a read-only squashfs -- measured: neither /lib/apk nor /etc/apk can be created. pkgs/3rdparty/apk-tools/prefix.patch splits the two. Upstream already carries root_fd and dest_fd separately and merely aliases them in context.c, so the patch defaults the database root to $MODDIR and leaves files extracting at /. That is opkg's compile-time VARDIR plus `dest root /`, in apk's spelling. THE FEED IS SIGNED NOW, which it never was: opkg signed the index and bin/build-packages.sh signed nothing, so nothing was verified on a printer. apk verifies with raw PEM public keys, not X.509 -- measured on the machine, where every certificate directory is empty. ./bin/apk-keygen.sh makes an RSA pair, APK_SIGN_KEY in config.env points at the private half, and the public half both seeds the USB stick (so the first install can verify) and ships in anvil-core (so the printer keeps trusting). An empty APK_SIGN_KEY still builds an unsigned feed, deliberately: `make packages` has to run on a bare checkout or the packaging lane stops being a CI gate. WHAT IT COST is a carried patch, against an internal with one consumer, on a branch mid v2->v3 transition, on an architecture with no upstream apk port to share the rebasing. That is the standing risk and it does not go away. Four bugs this shook out, all silent, all found by a gate rather than by reading: * The prune deleted seventeen packages it had just built. The all -> noarch mapping went into the emitter and not into pkg_pkgfile, so the prune looked for a name nothing wrote. `make packages` exited 0. * Whitespace-separated Depends dropped nineteen packages. opkg's pkg_parse.c splits that field on ',' and nothing else, so the whole list became one dependency whose name contained spaces and resolved to nothing -- and opkg neither warned nor failed. `make build` shipped a payload with no Tornado in it. test_the_payload_is_dependency_closed is what catches that now. * Ownership was recorded as `nobody`. apk resolves an owner to a NAME against <--root>/etc/passwd and the build lane runs --user with no passwd entry, so every file in every package would have installed as uid 65534, with no error at build or install time. build-packages.sh writes a two-line synthetic passwd naming the build user root and points --root at it. * Four recipes carried versions apk cannot parse -- two spelled a commit +git<sha> where the grammar's hash slot is ~, x264 had a snapshot name whose -2245 reads as a malformed release, and helixscreen had 0.99.115.creator5.3 where apk allows one trailing letter and then only a suffix from a closed set of nine words. All four were valid opkg versions, which is why nothing had noticed. pkg_version_ok gates every recipe now. A v3 .apk begins with ADBd magic and is not a tar stream, so there is no apk equivalent of the `ar -p \| gunzip \| tar` that hand-unpacked anvil-opkg: you cannot open a package without a package manager. bin/build-payload.py puts the cross-built binary on the stick at /mnt/apk instead. It is the same build anvil-apk-tools contains, that package installs normally, and the bootstrap copy never lands. Two gates retired rather than moved. apk's installed database is plain text (F: sets a directory, R: names a file in it), so the payload-ownership check reads one file where it used to glob info/*.list. And it carries no clock at all -- apk records no install time and mkpkg sets no build-time -- so the Installed-Time normalisation case-build-payload.sh did on every build is gone, and the test asserts the ABSENCE of a t: line instead. Clock-free by construction rather than by repair. Also removed: PKG_SECTION, which 45 recipes declared and nothing read once apk's schema had no field for it, and the `make apk-host` target -- the packager is built on demand by bin/build-packages.sh and stamped on the pin, the way pkg_toolchain and pkg_buildpython already work. Verified from a cold work/host and work/packages: 47 packages built and signed, `make build` installs 35 and verifies them against the shipped public key, `make qa-static` 195 passed, `make qa-replica` 135 passed -- the printer installing off a simulated USB stick and booting. The installed set was checked against the declared closure of the eleven roots: 35 expected, 35 installed, nothing extra and nothing missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rbKEyLXrzSE6wm3iNApVp	2026-08-31 14:32:07 +03:00
Oleksandr ShyshatskyiandClaude Opus 5	f67e7a269f	`ls pkg/` was thirty-eight entries and four of them mattered pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and carry no files of this repo move down a level into pkgs/3rdparty/. What is left at the top is the four components a person actually edits: anvil-core, klipper, moonraker, helixscreen. THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day somebody patches zlib nobody would agree which side it belongs on, and the answer would be a rename touching CI and docs decided by taste. "Does it carry files of ours" is a fact about the tree: it is exactly whether the recipe has a payload/, prog/ or seed/, checkable in one assertion, and test_recipe_layout.py now demands the move in both directions -- a 3rdparty recipe that grows one of the three must come up, and a top-level recipe with none of them must go down, or the top level slowly refills and the split stops meaning anything. ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id to its directory and nothing else spells either path -- not a recipe, not bin/build-packages.sh, not a test. Names stay unique across both levels because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare name; a duplicate would resolve to whichever level is searched first and build the wrong thing silently, so that is a test too. Recipes now source `. ./bin/common.sh` instead of counting `..` from $0. Depth-counting would have needed two spellings for two levels, and it was never buying anything: `. pkg/lib.sh` on the next line already required the working directory to be the repo root. A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files() expanded a list of globs and returned whatever matched, so "pkgs//build.sh" quietly stopped covering thirty-four recipes the moment they moved -- the lane stayed green, and the only visible sign was a test count dropping by 34 in a run nobody was counting. _files() now refuses a pattern that matches nothing. That is the same failure the ash-compatibility check had a guard for and the syntax check did not, which is why the guard reads the way it does. AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of .py helpers wholesale, so any test that imported one left a __pycache__ beside it -- gitignored, invisible in a diff, copied by cp -a like anything else. The package's contents depended on whether pytest had been run on the machine that built it: two builds of one commit, different sha256, cause nowhere in the tree. What it shipped was worthless as well as unstable, compiled by the build image's python 3.11 for a printer running the 3.13 this repo cross-builds. The sweep is in pkg_ship now, once, and the two hand-written copies in pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe that stages a tree of .py files, and the two recipes that had hit it had each solved it locally while the third, which had not, was the one shipping. Measured: the rename changes no package bytes. Same cached build trees, feed built with the diff stashed and unstashed: all 41 .ipk match. * with the bytecode sweep, anvil-core returns to exactly the bytes the previous commit produced. * no package in the feed contains a .pyc or a __pycache__. * make qa-static: 247 passed (236 before this work + 11 layout tests). * make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the synthetic stock package. make test-py: 181 passed. Not explained: anvil-python-dev produced two different sha256 from an UNCHANGED work/pkg/python at two points today, under code that was identical both times -- so it is not reproducible from a cached tree, and it is not this commit's doing (the stashed and unstashed builds above agree on it). Worth a look; the dev split in bin/build-packages.sh is where to start. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-28 20:04:20 +03:00

Oleksandr ShyshatskyiandClaude Opus 5

f8b925e48f

Replace opkg with apk, and sign the feed

docs/notes/85-packaging.md chose opkg in 2026-08 on one argument: every package
is cross-built here anyway, so the winner is whichever format is cheapest to
PRODUCE -- and .ipk had an upstream tool where .apk meant abuild, a keypair and
an Alpine host. Two lines of that comparison were wrong.

apk mkpkg is an applet of apk itself: -I key:value metadata, -F directory, no
keypair, no abuild, no Alpine host. The "making a package" row described
apk-tools 2. And the on-device client was counted a wash when it is better than
one -- openssl and zlib were already cross-built for CPython, and libarchive
existed in this tree only for opkg, so adopting apk REMOVES a cross-build.

What made it possible at all is that apk resolves its database relative to
--root and uses the same root as the file destination, while / here is a
read-only squashfs -- measured: neither /lib/apk nor /etc/apk can be created.
pkgs/3rdparty/apk-tools/prefix.patch splits the two. Upstream already carries
root_fd and dest_fd separately and merely aliases them in context.c, so the
patch defaults the database root to $MODDIR and leaves files extracting at /.
That is opkg's compile-time VARDIR plus `dest root /`, in apk's spelling.

THE FEED IS SIGNED NOW, which it never was: opkg signed the index and
bin/build-packages.sh signed nothing, so nothing was verified on a printer.
apk verifies with raw PEM public keys, not X.509 -- measured on the machine,
where every certificate directory is empty. ./bin/apk-keygen.sh makes an RSA
pair, APK_SIGN_KEY in config.env points at the private half, and the public
half both seeds the USB stick (so the first install can verify) and ships in
anvil-core (so the printer keeps trusting). An empty APK_SIGN_KEY still builds
an unsigned feed, deliberately: `make packages` has to run on a bare checkout
or the packaging lane stops being a CI gate.

WHAT IT COST is a carried patch, against an internal with one consumer, on a
branch mid v2->v3 transition, on an architecture with no upstream apk port to
share the rebasing. That is the standing risk and it does not go away.

Four bugs this shook out, all silent, all found by a gate rather than by
reading:

  * The prune deleted seventeen packages it had just built. The all -> noarch
    mapping went into the emitter and not into pkg_pkgfile, so the prune looked
    for a name nothing wrote. `make packages` exited 0.
  * Whitespace-separated Depends dropped nineteen packages. opkg's pkg_parse.c
    splits that field on ',' and nothing else, so the whole list became one
    dependency whose name contained spaces and resolved to nothing -- and opkg
    neither warned nor failed. `make build` shipped a payload with no Tornado
    in it. test_the_payload_is_dependency_closed is what catches that now.
  * Ownership was recorded as `nobody`. apk resolves an owner to a NAME against
    <--root>/etc/passwd and the build lane runs --user with no passwd entry, so
    every file in every package would have installed as uid 65534, with no
    error at build or install time. build-packages.sh writes a two-line
    synthetic passwd naming the build user root and points --root at it.
  * Four recipes carried versions apk cannot parse -- two spelled a commit
    +git<sha> where the grammar's hash slot is ~, x264 had a snapshot name
    whose -2245 reads as a malformed release, and helixscreen had
    0.99.115.creator5.3 where apk allows one trailing letter and then only a
    suffix from a closed set of nine words. All four were valid opkg versions,
    which is why nothing had noticed. pkg_version_ok gates every recipe now.

A v3 .apk begins with ADBd magic and is not a tar stream, so there is no apk
equivalent of the `ar -p | gunzip | tar` that hand-unpacked anvil-opkg: you
cannot open a package without a package manager. bin/build-payload.py puts the
cross-built binary on the stick at /mnt/apk instead. It is the same build
anvil-apk-tools contains, that package installs normally, and the bootstrap
copy never lands.

Two gates retired rather than moved. apk's installed database is plain text
(F: sets a directory, R: names a file in it), so the payload-ownership check
reads one file where it used to glob info/*.list. And it carries no clock at
all -- apk records no install time and mkpkg sets no build-time -- so the
Installed-Time normalisation case-build-payload.sh did on every build is gone,
and the test asserts the ABSENCE of a t: line instead. Clock-free by
construction rather than by repair.

Also removed: PKG_SECTION, which 45 recipes declared and nothing read once
apk's schema had no field for it, and the `make apk-host` target -- the
packager is built on demand by bin/build-packages.sh and stamped on the pin,
the way pkg_toolchain and pkg_buildpython already work.

Verified from a cold work/host and work/packages: 47 packages built and signed,
`make build` installs 35 and verifies them against the shipped public key,
`make qa-static` 195 passed, `make qa-replica` 135 passed -- the printer
installing off a simulated USB stick and booting. The installed set was checked
against the declared closure of the eleven roots: 35 expected, 35 installed,
nothing extra and nothing missing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013rbKEyLXrzSE6wm3iNApVp

2026-08-31 14:32:07 +03:00

Oleksandr ShyshatskyiandClaude Opus 5

f67e7a269f

ls pkg/ was thirty-eight entries and four of them mattered

pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and
carry no files of this repo move down a level into pkgs/3rdparty/. What is
left at the top is the four components a person actually edits: anvil-core,
klipper, moonraker, helixscreen.

THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory
rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day
somebody patches zlib nobody would agree which side it belongs on, and the
answer would be a rename touching CI and docs decided by taste. "Does it carry
files of ours" is a fact about the tree: it is exactly whether the recipe has
a payload/, prog/ or seed/, checkable in one assertion, and
test_recipe_layout.py now demands the move in both directions -- a 3rdparty
recipe that grows one of the three must come up, and a top-level recipe with
none of them must go down, or the top level slowly refills and the split stops
meaning anything.

ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id
to its directory and nothing else spells either path -- not a recipe, not
bin/build-packages.sh, not a test. Names stay unique across both levels
because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare
name; a duplicate would resolve to whichever level is searched first and
build the wrong thing silently, so that is a test too.

Recipes now source `. ./bin/common.sh` instead of counting `..` from $0.
Depth-counting would have needed two spellings for two levels, and it was
never buying anything: `. pkg/lib.sh` on the next line already required the
working directory to be the repo root.

A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files()
expanded a list of globs and returned whatever matched, so "pkgs/*/build.sh"
quietly stopped covering thirty-four recipes the moment they moved -- the lane
stayed green, and the only visible sign was a test count dropping by 34 in a
run nobody was counting. _files() now refuses a pattern that matches nothing.
That is the same failure the ash-compatibility check had a guard for and the
syntax check did not, which is why the guard reads the way it does.

AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of
.py helpers wholesale, so any test that imported one left a __pycache__ beside
it -- gitignored, invisible in a diff, copied by cp -a like anything else. The
package's contents depended on whether pytest had been run on the machine that
built it: two builds of one commit, different sha256, cause nowhere in the
tree. What it shipped was worthless as well as unstable, compiled by the build
image's python 3.11 for a printer running the 3.13 this repo cross-builds. The
sweep is in pkg_ship now, once, and the two hand-written copies in
pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe
that stages a tree of .py files, and the two recipes that had hit it had each
solved it locally while the third, which had not, was the one shipping.

Measured:
  * the rename changes no package bytes. Same cached build trees, feed built
    with the diff stashed and unstashed: all 41 .ipk match.
  * with the bytecode sweep, anvil-core returns to exactly the bytes the
    previous commit produced.
  * no package in the feed contains a .pyc or a __pycache__.
  * make qa-static: 247 passed (236 before this work + 11 layout tests).
  * make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the
    synthetic stock package. make test-py: 181 passed.

Not explained: anvil-python-dev produced two different sha256 from an
UNCHANGED work/pkg/python at two points today, under code that was identical
both times -- so it is not reproducible from a cached tree, and it is not this
commit's doing (the stashed and unstashed builds above agree on it). Worth a
look; the dev split in bin/build-packages.sh is where to start.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-28 20:04:20 +03:00