apk decides whether to upgrade from the version string alone. Three recipes
pin an upstream version -- anvil-moonraker on ${MOONRAKER_VERSION#v},
anvil-klipper on the pinned commit, anvil-helixscreen on its tag -- and each
ships a payload/ of ours beside that upstream tree. Our half changes without
the string moving, so the feed serves new bytes under the name every printer
already holds and `apk upgrade` finds nothing to do. #16 did exactly that to
moonraker.conf, the file that moves Moonraker's config to
/usr/data/anvil-data/config.
PKG_RELEASE is that string's other half, and it is the release date now:
pkg_release_stamp, monotone, with the same-day letter as a trailing digit so
20260827b sorts between 20260827 and 20260828. Nobody has to notice that a
payload changed, which is the only version of this that holds.
PKG_STAMP_EXTRA="$(pkg_payload_hash)" is not this and reads as though it were.
It hashes payload/ so the BUILD rebuilds; it says nothing to a printer. All
three recipes had it, and all three would have shipped stale.
test_recipe_layout.py makes it the rule for recipes added later: a pinned
recipe with a payload/ must take pkg_release_stamp, and the failure names the
recipe and the line to write.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
62 KiB
Packaging: apk vs opkg, and the migration between them
Whether to stop hand-building Python, Moonraker, s6 and libsodium into one tarball and start building packages instead. Written 2026-08-28, and the answer was opkg; revisited 2026-08-31 and reversed, because two lines of the comparison turned out to be wrong. The feed is apk now. The comparison is kept as written, because a decision is only reviewable next to what it was made on -- read the Decision section under it before leaning on the table.
Same rules as 80-s6-migration.md: each phase is independently shippable and
names the gate that proves it.
Why, in one paragraph
bin/patch.sh was 1,900 lines and eleven cross-builds deep, and its output was a
single 53MB anvil.tar.xz that the printer's installer unpacks whole. Changing
one line of anvil.conf ships all 53MB. There is no way to install just a new
Moonraker onto a printer you are debugging, no record on the machine of what
version of anything is on it, and the "remove what the last release installed"
problem has already been solved once by hand — installer/run-append.sh reads an
install manifest and prunes by it, which is a package manager's file database
with one package in it. The question is whether to keep growing that or to
adopt the format that already means it.
The fact that decides the comparison
Neither package manager brings any software with it. This is the whole argument and it is worth stating before the table, because every "apk vs opkg" discussion elsewhere on the internet is really about which distro's repository you want, and neither repository can be used here:
- Alpine has no MIPS port at all. Not "an old one" or "a community one" —
mips is not among the architectures Alpine builds. There is no
apkrepo to point at. - OpenWrt does have
mipsel_24kc, which is the same ISA and the same o32 ABI as this printer — and is built against musl. This printer's rootfs is glibc 2.29, and the reason that matters is written down at length inversions.env: klippy'sc_helper.sois dlopened by a glibc interpreter, so glibc is not a preference here. An OpenWrt package would unpack perfectly and produce a library nothing can load.
So every package will be cross-built by this repo either way. A package manager
buys packaging, versioning and installation. It does not buy software, and
adopting one does not make bin/patch.sh's cross-builds go away — it
reorganises them into units with names and versions.
That reframes the choice entirely: the winner is whichever format is cheapest to produce and install on a machine we control, not whichever has the better ecosystem.
The comparison
| opkg / .ipk | apk / .apk (Alpine) | |
|---|---|---|
| Upstream binaries we can use | none (musl) | none (no mips port) |
| Making a package | ar + two tar.gz + a 3-byte version file |
three concatenated gzip streams with an RSA signature over the control stream |
| ...and the tool that does it | opkg-build, upstream, 423 lines of POSIX sh |
realistically abuild + abuild-sign + a keypair, and abuild assumes an Alpine host |
| Indexing a feed | opkg-make-index, from the same upstream |
apk index, needs the signing key |
| Cross-building the packages | irrelevant; we stage a tree and hand it over | same, if you hand-roll; painful through abuild |
| On-device client | opkg + libarchive + zlib |
apk-tools + zlib + openssl |
| ...and what we already build | libarchive: nothing. New cross-build. | openssl 3.0.15 and zlib are already cross-built for CPython |
| Offline install into a prefix | --offline-root DIR, and opkg install ./file.ipk needs no feed at all |
apk add --root DIR --allow-untrusted ./file.apk |
| Signing | optional; signs the index, so packages need no signature each | expected; --allow-untrusted to skip |
| Format stability | unchanged for over a decade; also Yocto/OpenEmbedded's native format | apk-tools 3 introduced a new package format; v2/v3 coexistence is live churn |
| Home turf | embedded MIPS with a read-only rootfs — exactly this device class | a full musl distro root |
Two honest points for apk, since it is the more actively developed tool: its
world model (a declarative "these are the packages I want", reconciled on
every operation) is genuinely nicer than opkg's imperative install/remove, and
its runtime dependencies are ones this repo already cross-builds while
opkg's libarchive is not. If the deciding factor were the on-device client,
apk would win.
It is not. The deciding factor is that every package has to be produced here,
by us, on every build, in CI — and for .ipk there is an upstream tool that
does it, needing no keypair and no distro-specific build framework. For .apk
the equivalent is either implementing a signed three-stream format by hand or
importing Alpine's abuild into a Debian-based build image. That cost is paid
on every build and every recipe; the client is paid once.
The format-churn line matters more than it looks, too. A format with no users on our architecture and an in-progress v2→v3 transition is a thing to track forever for no benefit, since we can never consume anyone else's packages anyway.
Decision
apk, and the .apk format — apk-tools 3, cross-built here and carrying one
local patch, with apk mkpkg and apk mkndx as the packager.
This reverses the original decision, and the table above is what it was made
on. That decision was opkg and .ipk, driving upstream's opkg-build, and it
turned on one argument: every package is produced here on every build, so the
winner is whichever format is cheapest to produce, and .ipk had an upstream
tool where .apk meant abuild, a keypair and an Alpine host. Two lines of
the comparison were wrong, which the proof of concept below found out:
apk mkpkgis an applet of apk itself —-I key:valuemetadata,-Fdirectory, no keypair required, noabuild, no Alpine host. The "making a package" row described apk-tools 2.- The client was counted a wash. It is better than a wash: openssl and zlib were already cross-built for CPython, and libarchive existed in this tree only for opkg. Adopting apk removes a cross-build.
What made it possible at all is that apk resolves its database relative to
--root and uses the same root as the file destination, while / here is a
read-only squashfs. pkgs/3rdparty/apk-tools/prefix.patch splits the two —
upstream already carries root_fd and dest_fd separately and merely aliases
them — so the database lands under $MODDIR and files still extract at /.
That is opkg's compile-time VARDIR plus dest root /, in apk's spelling.
What it cost is a carried patch, against an internal with one consumer, on a branch mid v2→v3 transition, on an architecture with no upstream apk port to share the rebasing. That is the standing risk and it does not go away. Set against it: the feed is now signed and verified on the printer, which the opkg feed never was.
The migration itself is recorded below under "The migration, as it happened".
The apk proof of concept (built and measured; it is now the feed)
Run it with make packages PKG=apk-tools && ./tools/replica/run-apk-poc.py.
The recipe is pkgs/3rdparty/apk-tools/, the case is
tools/replica/printer/case-apk-poc.sh, and everything below happened on the
printer's real rootfs under qemu-mipsel rather than in an argument.
What was under test is not "is apk nicer" — it is the one thing the
comparison could not settle by reading: apk resolves its database, world,
arch and lock file relative to --root, and takes the same root as the
destination for package files. This printer's / is a read-only squashfs.
Measured first, because it decides the shape of everything else: /lib/apk
and /etc/apk cannot be created. mkdir and ln -s both return
Read-only file system, with a control write under /usr/data succeeding in
the same run. The mount table (docs/printer-replica.md) says why — / is
overlay (ro), /usr/prog and /usr/data are rw — and it is the same fact
that put the s6-rc compiled database under $MODDIR/etc rather than /etc
(docs/notes/80-s6-migration.md). So the symlink-into-$MODDIR idea is not
available: there is nowhere to put the symlink.
pkgs/3rdparty/apk-tools/prefix.patch, 102 lines, all in src/context.c.
Upstream already separates the two roots — apk_ctx carries root_fd and
dest_fd, database.c uses the first for the db and fs_fsys.c uses the
second for package files, and app_extract.c already overrides dest_fd by
itself. context.c merely aliases them (ac->dest_fd = ac->root_fd). The
patch defaults the database root to $MODDIR, opens dest_fd on /, and
sets APK_NO_CHROOT for that case so a maintainer script still sees the real
/usr/prog. That is opkg's compile-time VARDIR plus dest root /, in apk's
spelling.
It is inert unless the build asks for it: undefined,
APK_DEFAULT_DB_ROOT is "/" and every branch folds away to upstream's
behaviour. The prefix is passed in from the recipe rather than written into
the patch, so $MODDIR stays spelled once, in bin/common.sh — the trap
opkg's VARDIR, s6's libexecdir and execline's shebangdir all sprang.
The whole loop closed on mipsel, in one run of the case:
apk --version→apk-tools 3.0.7, compiled for mipsel, in a bare environment.--initdbwith no--rootwrote$MODDIR/lib/apk/db/{installed, scripts.tar.gz,triggers}and$MODDIR/etc/apk/{arch,world}, and/lib/apkand/etc/apkstill did not exist afterwards.apk mkpkgbuilt a package on the printer,apk addinstalled it, the file landed at its$MODDIR-absolute path,apk info -Llisted it, andapk delremoved it leaving nothing.
The two comparison lines this corrects
- "Making a package: three concatenated gzip streams with an RSA signature"
and "realistically
abuild+ a keypair, andabuildassumes an Alpine host". Wrong for v3.apk mkpkgis an applet of the same binary, takes-I key:valuemetadata and a-Fdirectory, needs no keypair, and produced a working package here — cross-built, running under qemu, on the printer. The argument that.ipkis cheaper to produce was the deciding one, and this is the half of it that does not hold. - "On-device client: apk-tools + zlib + openssl", counted as a wash. It is
better than a wash. openssl and zlib were already cross-built for CPython, so
PKG_BUILD_DEPENDSis two packages we already had — and libarchive exists in this tree only for opkg. Adopting apk deletes a cross-build rather than adding one.
Certificates: not for packages, only for https://
Worth separating, because "apk needs openssl" invites the wrong conclusion on a printer that has no trust store.
The printer has none — measured, in the same run. /etc/ssl/certs,
/usr/lib/ssl/certs, /usr/share/ca-certificates and $MODDIR/ssl/certs all
hold zero entries, and neither cert.pem exists. That is exactly what
tools/python/README.md already says under "The gap: there are no CA
certificates" — our openssl is configured --openssldir=$MODDIR/ssl and
nothing ships a bundle into it.
Package verification does not care. apk reads signing keys with
PEM_read_bio_PUBKEY (src/crypto_openssl.c) — raw public keys, not X.509,
so there is no chain to build and no CA store to consult. Measured end to end
on the printer, with every directory above empty:
- a package built with
apk mkpkg --sign-keyand its public half dropped in$MODDIR/etc/apk/keys/installed without--allow-untrusted; - the same package with the key removed was refused —
ERROR: UNTRUSTED signature, exit 99. The check is real, not vacuous.
So a signed feed on a stick, or over plain http://, needs no certificates at
all. This is also strictly better than what the feed has today: opkg signs the
index and our bin/build-packages.sh signs nothing, so nothing is verified
on the printer now.
https:// is the one thing that needs a bundle. libfetch defaults to
SSL_VERIFY_PEER; with no CA_CERT_FILE compiled in it falls back to
SSL_CTX_set_default_verify_paths(), which resolves to $MODDIR/ssl — empty
— so every verified connection fails. apk --no-check-certificate sets
SSL_VERIFY_NONE and turns that off. (Read from libfetch/common.c, not
measured: the replica has no network.)
That makes the certificate question a pre-existing gap this does not
change — the same ~200KB ca-certificates bundle Moonraker and CPython
already want — and not an apk problem. It only becomes one the day a feed is
served over https, and until then --no-check-certificate or http:// is the
same trade the interpreter already lives with.
What it cost, which is the honest other half
- The legacy Makefile assumes Alpine.
portability/—strlcpyand friends — is wired into the meson build only. Against glibc 2.29 the link fails onstrlcpyalone; the recipe compiles upstream's own shim as one object. Add-lpthread -ldl(separate libraries until glibc 2.34) and-latomic(mips32 has no 64-bit atomics, so__atomic_load_8is a library call, and nothing in the error message says "mips"). -l:libapk.aand--start-group.src/MakefilesaysLIBS_apk := -lapkwith bothlibapk.soandlibapk.apresent, so the binary comes out needing$MODDIR/libon the loader path — measured: bare, it dies withlibapk.so.3.0.0: cannot open shared object file. Linking the archive in instead means the orderingcmd_ldimposes ($(LIBS) before $(LIBS_apk)) starts to matter, which the shared build had hidden. The result is one 4.2MB binary whose everyNEEDEDis a rootfs library — the same shape asanvil-opkg, and all four confirmed present on the machine.--arch mipsel_xburst2atinitdb. The compiled-in arch is baremipsel, so a package built for this feed's architecture name is refused asuninstallable. Written into the database once, it is apk's answer to opkg'sarch mipsel_xburst2 10config line.- A carried patch, forever. This is the cost that does not go away.
dest_fdhas exactly one consumer in the tree; it is an internal, not an API, and upstream can collapse it back intoroot_fdwithout anyone calling that a breaking change. v3.0.7 is the tip of an actively developed branch mid v2→v3 transition, on an architecture with no upstream apk port to share the rebasing with. This repo bumped opkg 0.6.3→0.7.0 specifically to avoid carrying a patch.
Nothing installs anvil-apk-tools. The feed is still .ipk,
bin/payload.sh still installs it with opkg, and this recipe exists so the
next revision of the decision is made against measurements rather than against
the table above.
The migration, as it happened
Sequenced so that the producer and the payload assembler were never broken at
the same time. A PKG_FORMAT switch in bin/common.sh made each step land
alone — one build produced one format, there was never a dual feed — and it was
deleted with the opkg path.
A native apk had to exist first. apk mkpkg and apk mkndx run on the
build machine, so tools/apk-host/build.sh builds one from the same pinned
checkout, unpatched, into work/host/bin/apk. libssl-dev replaced
libarchive-dev as the one documented exception to versions.env, for the
same reason: a build tool that never leaves the container.
It is built ON DEMAND, and there is no target for it. bin/build-packages.sh
runs tools/apk-host/build.sh and that script caches on the pin, so the second
call is a stamp comparison. There was a make apk-host for a while, and it was
an anomaly: the cross toolchain is unpacked by pkg_toolchain and the
build-python compiled by pkg_buildpython, both on demand and both stamped.
The one tool that writes every package had no business being the one thing a
person had to remember to build first.
Then the metadata was made apk-legal while still under opkg, which is the step that de-risked everything after it:
Version:became1.2.3-r1. apk'sapk_version_validaterejects a bare-1; opkg reads the same string as upstream1.2.3revisionr1and is unbothered. Four recipes carried versions apk cannot parse at all —klipperandtimelapsespelled a commit+git<sha>where the grammar's hash slot is~,x264carried a snapshot name whose-2245reads as a malformed release, andhelixscreenhad0.99.115.creator5.3, where apk allows one trailing letter and then only a suffix from a closed set of nine words. All four were valid opkg versions, which is why nothing had noticed.pkg_version_oknow gates every recipe.PKG_ARCH=allemitsnoarch. apk interns that literal and compares against it, soallreads as an unknown architecture.- The filename moved behind one function,
pkg_pkgfile. It had been spelled in four places, which was safe only because the.ipkname separates name from version with_; the.apkname uses-, soanvil-python-*matchesanvil-python-pillow-11.0.0-r1.apkand a package that was never built looks present.
Two bugs this shook out, both silent, both found by a gate rather than by reading:
- The prune deleted seventeen packages it had just built. The
all→noarchmapping went into the emitter and not intopkg_pkgfile, so the prune's expected set named_all.ipk, did not find it, and removed the real files.make packagesexited 0. - Whitespace-separated
Dependsdropped nineteen packages. apk's list is whitespace-separated and opkg's is comma-separated, and stripping the commas for both formats looked harmless.pkg_parse.csplits that field on,and nothing else, so the whole list became one dependency whose name contained spaces, resolved to nothing, and opkg neither warned nor failed.make buildshipped a payload with no Tornado in it. The commas now come off insideemit_apkalone, andtest_the_payload_is_dependency_closedis what would catch it next time — nothing else asked, because the roots were right and a test that walks the payload cannot notice a file that is absent.
The bootstrap changed shape. A v3 .apk begins with ADBd magic and is not
a tar stream, so there is no apk equivalent of the ar -p | gunzip | tar that
unpacked anvil-opkg by hand — you cannot open a package without a package
manager. bin/build-payload.py puts the cross-built binary on the stick at
/mnt/apk instead. It is the same build anvil-apk-tools contains, that
package installs normally, and the bootstrap copy never lands.
Ownership needed a synthetic root. apk records owners by NAME against
<--root>/etc/passwd, and the build lane runs --user with no passwd entry —
so every file in every package was recorded nobody, silently, and would have
installed as uid 65534. bin/build-packages.sh writes a two-line passwd/group
naming the build user root and points --root at it.
test_a_package_is_owned_by_root_whoever_built_it and its inverse hold that
shut. --rootnode is not the flag for this; it is a deprecated alias for
--compat.
The feed is signed now, which it never was. opkg signs the index and
bin/build-packages.sh signed nothing, so nothing was verified on a printer.
./bin/apk-keygen.sh makes an RSA pair, APK_SIGN_KEY in config.env points
at the private half, and the public half both seeds the stick (so the first
install can verify) and ships in anvil-core (so the printer keeps trusting).
An empty APK_SIGN_KEY still builds an unsigned feed, deliberately: make packages has to run on a bare checkout or the packaging lane stops being a CI
gate. RSA or Ed25519 only — ECDSA signatures are randomised and two builds
of one tree would stop being byte-identical.
Where the key lives, on each of the two machines that build. On a
developer's, outside the checkout — ./bin/apk-keygen.sh ~/.anvil/anvil.rsa,
and APK_SIGN_KEY in the gitignored config.env — so neither half is
something the repository can commit. The build runs in a container that mounts
only the checkout, so the Makefile reads that path out of config.env and
bind-mounts the key's directory read-only; without that the key is visible in
the developer's shell and invisible to apk, which is a confusing enough
failure to have its own test.
In CI it is the APK_SIGN_KEY repository secret, holding the private key
PEM itself rather than a path. release.yml writes it to $RUNNER_TEMP
(outside the checkout, so no later step can archive it into a release), derives
the public half with openssl pkey -pubout rather than storing it as a second
secret that could drift out of step with the first, and names the file in
config.env. With the secret unset the release is built unsigned with a
warning rather than failing: a fork has no such secret, and a firmware that
cannot be built by someone else is worse than one whose feed is untrusted on a
stick the printer was already handed by hand.
Two gates retired rather than moved. apk's installed database is plain text
(F: sets a directory, R: names a file in it), so the ownership check reads
one file where it used to glob info/*.list. And it carries no clock at
all — apk records no install time and mkpkg sets no build-time — so the
Installed-Time normalisation case-build-payload.sh had to do on every build
is gone, and the test now asserts the absence of a t: line. Clock-free by
construction rather than by repair.
Facts established by measurement, not by reading
Each of these was run, not assumed. They constrain the phases below.
-
Our cross-built opkg installs our packages, on mipsel — measured when opkg was still a static musl binary. Run under
qemu-mipsel-staticagainst an--offline-root, it installed both packages of the day, reported them underlist-installed, listed their files, and removed both leaving nothing behind; the opkg it installed then ran and reported its own version. The whole loop, closed on the target architecture.This has not been re-run since opkg moved to dynamic glibc, and the command changed with it — a dynamic binary needs
qemu -L <sysroot>. See the end of phase 0. Nothing suggests it broke; nobody has checked. -
opkg bakes its state directory in at compile time. Its status file landed at
/usr/data/anvil/var/lib/opkgbecause it was configured--prefix=/usr/data/anvil—--offline-rootdoes not move it. Built with any other prefix it comes up believing nothing is installed and reinstalls the world. This is the third instance of the same trap on this printer, after s6'slibexecdirand execline'sshebangdir(both inversions.env). On this machine the--prefixis not a preference, and that is why$MODDIRnow lives inbin/common.shwhere a recipe cannot spell it differently. -
opkg 0.7.0, not 0.6.3, if you build against musl. Two files in 0.6.3 (
opkg_remove.c,opkg_archive.c) callbasename()without including<libgen.h>, getting away with it only because glibc's<string.h>also declares a differently-behavedbasename. musl does not, so gcc 14 stops at "implicit declaration" — and on a compiler that merely warned, the returned pointer would have been truncated through an implicitint. 0.7.0 adds both includes. That was the difference between a pin bump and carrying a patch. -
opkg will not configure without libarchive. Measured:
./configurehard-fails at thepkg-configcheck. There is no internal tar fallback any more. libarchive and zlib are therefore build dependencies, cut down to gzip-and-tar and linked into the static binary; neither ships as a file. -
libtool eats
-static. A program configuredLDFLAGS=-staticcomes out dynamically linked anyway, because libtool defines that flag to mean "prefer static libtool libraries" and does not pass it to gcc.-all-staticis the one that means it, and it cannot go through./configure(configure's own link probes call gcc directly, which rejects it), so it goes tomake. Nothing relies on this any more — opkg links glibc dynamically now and gets its static zlib and libarchive simply by there being no.soof either in the sysroot — but the check it motivated stayed, inverted:readelf -don the finished binary, expectinglibc.so.6and nothing of ours. The lesson is the check, not the flag. A link that is not what it was asked to be shows up as aNEEDEDentry nobody looks at, and then as a missing.soon a printer. -
opkg-utils has no release tarball, anywhere. Upstream's cgit has snapshots disabled (every format returns "Unsupported snapshot format"),
downloads.yoctoproject.orgcarries opkg but not opkg-utils, and the GitHub mirror atshr-project/opkg-utilslast saw a commit in 2012 and has no tags. So it is pinned by git commit — a hash of the whole tree, the same guarantee the sha256s give, in git's spelling — andbin/fetch-assets.shverifiesHEADagainst it and refuses a dirty checkout. -
The printer's busybox cannot be assumed to have
ar. It is 1.31.1 built small — notimeout, nonc, noionice, all measured on the replica. That is why the printer gets a real cross-built opkg (pkgs/3rdparty/opkg/), which reads.ipkthrough its own linked-in libarchive rather than shelling out to tools the busybox may not have. -
All four packages are reproducible.
rm -rf work/pkg work/packagesand a full recompile produce byte-identical.ipkfiles. That is a measurement of these packages and this toolchain, not a property of the build system — a package whose upstream embeds a timestamp will not have it, and the honest response is to record which ones do rather than to claim the fleet is reproducible. -
A static archive is not reproducible until it is made so, and this toolchain cannot do it alone.
arstores an mtime and the builder's uid/gid in every member header, plus another timestamp in the symbol index — none of whichSOURCE_DATE_EPOCHoropkg-build's-o 0 -g 0reaches, because they are inside a file the tarball merely contains. Measured: two cold builds ofpkgs/3rdparty/zlibproduced different sha256, every member reading1000/1000 Aug 28 15:10.objcopy -Dnormalises the member headers, but the cross binutils is 2.27 and itsranlib -Dwrites a fresh current timestamp into the index instead of zeroing it. The index is therefore rewritten with the build machine'sranlib(2.40 indocker/Dockerfile.build), which is version-sensitive and worth knowing before anyone changes the image. The proof it produces a valid index is downstream: opkg links against both archives, and a broken index fails that link. -
The ABI gate's
e_flagswhitelist was a proxy, and object files exposed it. It accepted exactly0x70001405and0x70001407, both measured from linked binaries — where crt startup objects setEF_MIPS_NOREORDER, so every executable and shared object read one or the other. Individual objects need not: libarchive'sxxhash.ois0x70001406, identical ABI, no NOREORDER, and the old gate called it wrong. The ABI is the high bits (0x70001400= arch32r2 | o32 | nan2008); the low three (noreorder, pic, cpic) say nothing about whether the kernel will exec the file. The gate now masks them, and reads every header rather than one —readelf -hon an archive prints one per member, 122 forlibarchive.a, and the single-line version handed a multi-line string to a comparison expecting one word. -
A cache stamp compared by one file and written by another will drift.
bin/fetch-assets.shcomparedwork/.s6/.versionagainst"$SKALIBS_VERSION $S6_VERSION"whilebin/patch.shwrote three fields into it. The test could therefore never be false: a 71MB toolchain was re-hashed on every run, downloaded on every cold one, and the comment above the condition described a fast path that had never once been taken.$S6_STAMPnow lives inbin/common.sh, the recipes compute theirs inpkg_stamp, andtest_no_cache_stamp_is_spelled_in_two_placeskeeps it that way.
The phases, as they were built (historical, and opkg-era)
Everything from here to "What this does not fix" describes how the feed was
BUILT UP, and it was built up on opkg. The structure it produced — a recipe per
cross-build, one recipe one source, the feed as the interface between recipes,
the payload assembled by installing that feed inside the replica — is all still
exactly true and is the part worth reading. The tool names in it are not: read
opkg-build as apk mkpkg, opkg-make-index as apk mkndx, CONTROL/control
as -I key:value, and .ipk as .apk. The migration section above lists what
actually changed.
Kept rather than rewritten because these sections are the record of why the layout is the shape it is, and a reason survives the tool it was first written about.
Phase 0 — the proof of concept (implemented)
Four packages, end to end, beside the existing build and changing nothing that ships.
make packages # or ./bin/build-packages.sh [name...]
work/packages/anvil-zlib_1.3.1-1_mipsel_xburst2.ipk
work/packages/anvil-libarchive_3.7.9-1_mipsel_xburst2.ipk
work/packages/anvil-libsodium_1.0.20-1_mipsel_xburst2.ipk
work/packages/anvil-opkg_0.7.0-1_mipsel_xburst2.ipk
work/packages/Packages{,.gz}
Needs no stock FlashForge package, which is most of the point: packaging has to be runnable in CI on a bare checkout, or the gate only runs where the proprietary firmware is and stops being a gate.
| file | what it is |
|---|---|
pkgs/lib.sh |
the part of a cross-build that is the same for every package |
pkgs/3rdparty/zlib/ |
the library that was cross-built twice, now built once |
pkgs/3rdparty/libarchive/ |
what opkg reads .ipk files with; builds against anvil-zlib |
pkgs/3rdparty/opkg/ |
opkg itself; builds against both of the above |
pkgs/3rdparty/libsodium/ |
build.sh + pkg.conf. bin/patch.sh section 5d's build, moved. |
bin/build-packages.sh |
orders the recipes, lays out each tree, drives opkg-build, indexes the feed |
qa/static/test_packages.py |
112 tests, no toolchain needed |
One recipe builds one package
This is the rule the layout exists to enforce, and it did not hold at first.
pkgs/3rdparty/opkg/build.sh used to unpack zlib, build it into a private sysroot,
unpack libarchive, build that against it, and only then build the binary it
shipped: one script, three libraries, one package. Section 5b still does the
same thing with skalibs and s6.
What that costs is not tidiness. A library built inside somebody else's recipe has no version, no package and no way to be reused, so the next consumer builds its own — zlib was cross-built twice in this tree, once in section 5c for CPython and once here for libarchive, from one pinned tarball with the same flags, and neither copy could see the other. Splitting them is what makes the feed the interface between recipes rather than an output nobody reads.
So a recipe now declares PKG_BUILD_DEPENDS, and pkg_deps fills its sysroot
by unpacking those packages out of the feed. Building against the package
rather than against the build tree is the part worth paying for: a package that
forgets to ship a header fails the next recipe's configure, on the build that
produced it.
opkg is not special, and nothing waits for it
The obvious objection to a feed-based build is that it needs a package manager
to install the build dependencies, and the package manager is itself a package.
It does not. opkg-unbuild is upstream's own inverse of opkg-build and comes
from the same pinned opkg-utils checkout, so a sysroot can be filled without
a working opkg existing anywhere. There is no bootstrap stage, no ordering rule
beyond declared dependencies, and no qemu in the build path.
bin/build-packages.sh topologically sorts the recipes and packages each one
before the next is built, because the next one reads it out of the feed.
Alphabetical order — what iterating pkgs/*/ gives you — puts libarchive before
the zlib it needs.
The split in pkgs/lib.sh
pkg_conf/pkg_stamp— a recipe's metadata and its cache key, both read frompkg.conf. The stamp includes the toolchain and, recursively, the stamps of everything the recipe builds against, so a zlib bump rebuilds libarchive and opkg with nobody maintaining a composite stamp by hand.pkg_order— dependency-first build order, and the closure of a single named recipe so thatPKG=opkgbuilds what opkg needs.pkg_begin/pkg_end— the stamped build cache, which is what letsbin/fetch-assets.shskip a 203MB toolchain download.pkg_toolchain— unpack, write the gcc wrappers that bake-EL -mnan=2008into the driver (autotools link lines do not all forwardCFLAGS), and then prove the wrapper emits the right ABI before anything is built on it. That step existed in section 5b and was missing from 5d; the copies had drifted before there was anywhere to put them.pkg_deps— unpack this recipe's build dependencies out of the feed withopkg-unbuildand pointCPPFLAGS,LDFLAGSand pkg-config at the result. The sysroot mirrors the printer, dependencies living under$MODDIRinside it, which is whyPKG_CONFIG_SYSROOT_DIRis set to the sysroot rather than emptied: the.pcfiles sayprefix=/usr/data/anviland that variable is what turns them into paths that exist on the build machine.pkg_autotools— configure, make, install into the staging tree.pkg_ship— copy out what the package contains, normalise static archives, strip ELF with the cross strip. What a package contains depends on what it is for: a library that ships to the printer ships its.so, and a library that exists to be built against ships headers,.aand.pc.
qa/static/test_packages.py gates this directly: a recipe that does not source
pkgs/lib.sh, spells its own -mnan=2008, unpacks its own toolchain, calls
./configure itself, or unpacks more than one source tarball fails the suite.
The one named exception is zlib, whose configure has never accepted --host.
A recipe that needs something pkgs/lib.sh cannot express should grow
pkgs/lib.sh — that is what the gate is for.
Not yet shared: bin/patch.sh section 5b (s6 and skalibs) still carries
its own copy of all of this, and is the remaining instance of one script
building two libraries. Phase 1 is what turns it into recipes.
The property everything rests on
A library is compiled once, whichever vehicle it ships in. bin/patch.sh
runs pkgs/3rdparty/libsodium/build.sh and stages its output into the payload exactly as
before, and bin/build-packages.sh packages the same tree. Section 5c now does
the same with zlib: it runs pkgs/3rdparty/zlib/build.sh and stages the result into
CPython's dependency sysroot instead of compiling its own copy. While that
holds, the tarball's copy and the package's copy cannot be different libraries
wearing one version number — and
test_the_package_and_the_payload_share_one_build asserts it rather than
trusting it.
Gate: pytest qa/static (127 tests, 29 of them packaging) plus make packages from a cold cache producing four byte-reproducible .ipk files and
an index. Both pass.
Not re-run since opkg moved to glibc: the end-to-end check where our own
cross-built opkg installs and removes this feed. It used to be a bare
qemu-mipsel-static invocation because opkg was a static musl binary; a
dynamic one needs a sysroot, so the command is now
qemu-mipsel-static -L work/.mips-toolchain/mips-gcc720-glibc229/mips-linux-gnu/libc \
work/pkg/opkg/bin/opkg --offline-root <tmpdir> install work/packages/*.ipk
(readelf -l says the interpreter is /lib/ld-linux-mipsn8.so.1, the nan2008
loader, which that sysroot provides and the printer's rootfs must too — the
same loader the interpreter already links.) Worth running before anyone leans
on this.
Phase 1 — a recipe per cross-build (~3–5 days)
Turn the rest of bin/patch.sh's builds into recipes, the way libsodium
became one —
klipper ✔ toolchange-config
python ✔ python-<18 packages> ✔
openssl ✔ libffi ✔ sqlite ✔ xz ✔ expat ✔ bzip2 ✔
zlib ✔ libarchive ✔ libsodium ✔ opkg ✔
skalibs ✔ execline ✔ s6 ✔ s6-rc ✔
mainsail ✔ moonraker ✔ helixscreen ✔ anvil-core ✔
Done: 38 recipes producing 41 packages.
Section 5b — the last place where one script built two libraries — is four
recipes (skalibs, execline, s6, s6-rc), and moving them off the musl
toolchain removed the second libc from this tree entirely. Section 5c, which
was 825 lines, is pkgs/3rdparty/python plus eighteen pkgs/3rdparty/python-* and 115 lines of
staging.
What splitting CPython actually bought. Not tidiness. The old section had
one cache and two stamps, and its own comment explained that a bumped Pillow
rebuilt the interpreter and all eighteen packages — because a wheel could only
be cross-built during the few minutes when an untrimmed staging tree, a private
sysroot of static libraries and a throwaway x86-64 build-python happened to
exist at once. Those are now, respectively: the feed's anvil-python-dev
package, each recipe's own sysroot, and pkg_buildpython's shared cache. None
of them is a passing moment, so make packages PKG=python-pillow is a Pillow
build and nothing else.
It also made eighteen invisible things visible. Each package now carries its
upstream version into the index, anvil-moonraker declares the thirteen it
imports, and a printer that does not want gcode thumbnails can leave the 500KB
of Pillow uninstalled — a decision that could not previously be expressed.
Klipper was the last one, and it needed one new knob. klippy/chelper has
no Makefile and never has: on a machine with a compiler klippy builds
c_helper.so at first run from the argument list in
klippy/chelper/__init__.py, and this printer has no compiler. So the build
system for that .so genuinely is one gcc line, which pkg_build could not
express — it always ran make. PKG_CC_SHARED is that line, appended to
$CC -shared -fPIC: the recipe states Klipper's own COMPILE_ARGS and
pkg_build keeps ownership of the compiler, so the ABI flags stay spelled in
exactly one place. A knob, not a verb, for the reason the header of
pkg_build gives about the three mechanisms with one user each.
Two things the recipe took with it. KLIPPER_FORK — the config.env knob
that pointed the build at a local checkout instead of the pin — is gone: a
recipe names its source exactly once, a checkout beside a pinned tarball is
the second source that rule forbids, and it is the same variable that let
v20260824 ship a klippy tree nobody had built. test/test-chelper.py had
already stopped falling back to reading KLIPPER_FORK out of config.env when
given no argument — that fallback printed SKIP: no KLIPPER_FORK configured, a
green line for a check that had not run — and the script itself is gone now,
deleted with the rest of test/.
The package installs the tree under $MODDIR/klipper, where nothing reads it
yet: klippy is still started by the stock /usr/prog/klipper/klipperDaemon,
so bin/patch.sh stages the recipe's output into the SOFTWARE component as
before. Phase 7 of docs/notes/80-s6-migration.md is what makes the installed
copy the live one. [Superseded: phase 7 landed. The klipper s6-rc service
execs $FF_PYTHON against $MODDIR/klipper/klippy, and the software-component
copy is gone.]
patch.sh keeps staging the payload from those same trees, so the shipped
tarball does not change. The work is mechanical; the risk is in the two builds
with the most measured constraints behind them (5b's s6 prefix, 5c's CPython
cross-build), which should go last — and both are exactly the shape
pkgs/lib.sh already handles, since it was written from them.
Gate, corrected. This used to read "the anvil.tar.xz built after this
phase is byte-identical to the one built before it". That cannot be the gate,
for two independent reasons, and stating it that way meant nobody could tell a
real regression from an expected difference:
- Moving s6 to dynamic glibc changes those binaries by design. A tarball that did not change would mean the change had not happened.
bin/pack.shbuildsanvil.tar.xzwith a baretar -cf, with no--sort=name,--mtimeor--owner. It carries filesystem order and per-file mtimes, so it has never been byte-reproducible and the stated gate could never have passed.
What is checked instead: the assembled work/modpayload-root tree — file list,
modes and content hashes — is unchanged except for the components a change
deliberately rebuilt. The .ipk files themselves ARE byte-reproducible and
that is checked directly: two cold make packages runs produce identical
archives. Measured over all 40 with work/pkg, the shared build-python and
the unpacked toolchain all deleted in between — so the x86-64 interpreter that
generates every mipsel extension module is itself rebuilt rather than reused.
A recipe owns its files, and the directory says how they ship
The recipes came first and the repository's own files stayed where they were:
one top-level payload/ tree organised by DESTINATION (init.d/, bin/,
etc/s6/, klipper/config/) plus an assets/ directory of three .conf
files. That was right when there was one owner. With 38 recipes it hid three
different kinds of file behind identical-looking paths, and the only thing
recording which package owned any of them was a thirty-line comment at the top
of pkgs/anvil-core/build.sh.
Files now live with the recipe that owns them, in one of three subtrees:
pkgs/<recipe>/payload/ staged into the .ipk, laid out as it lands under
$MODDIR -- payload/init.d/S60nginx installs as
$MODDIR/init.d/S60nginx and no recipe says so
pkgs/<recipe>/prog/ placed on /usr/prog by bin/patch.sh
pkgs/<recipe>/seed/ templated or seeded user state
Recipes themselves sit at two depths, and pkgs/lib.sh's pkg_dir is the
only thing that knows it: pkgs/<name>/ for a recipe that carries files of
this repo (four), pkgs/3rdparty/<name>/ for one that builds a pinned tarball
and carries none (thirty-four). That is for ls pkgs/ and nothing else — no
package, no dependency and no stamp knows which side a recipe is on. The line
is deliberately mechanical rather than "do we modify it", which drifts: the
day somebody patches zlib nobody would agree on which side it belongs, whereas
"does it carry files of ours" is a fact about the tree and a test can demand
the move.
installer/ holds the two files that are never files on a printer at all:
run-pre.sh and run-append.sh are spliced into FlashForge's own run.sh.
qa/static/test_recipe_layout.py fails if a fourth kind of directory appears,
which is the failure this layout is actually for — a misfiled file is caught
by a build, an unruled directory is caught by nothing.
prog/ and seed/ are the not-yet-packaged residue, and that is the point of
naming them. Both empty out on the way to the end state. seed/ got there
first and by a route this section did not predict: rather than a postinst
seeding anvil.conf.default when the real file is absent, anvil.conf was
removed outright and seed/ left anvil-core with it. prog/ empties when a
postinst run against a STAGING ROOT places
/usr/prog/PROGRAM/software/firmwareExe as a symlink into $MODDIR. The second is the interesting one, and it is only possible because
/usr/prog is written by a tarball this repo bakes, not by a flash we do not
control: the install can happen in the build container, which is what "the
tarball becomes a view of the feed" below already assumes. The trap there is
$IPKG_INSTROOT -- a postinst that writes a bare /usr/prog/... symlinks the
BUILD CONTAINER's root, the staging tree gets nothing, and the build stays
green.
Ownership followed, once the files could be seen. anvil-core had held
everything of ours for no better reason than being the first recipe written,
and with the files visible per-recipe the misfilings were obvious:
moonraker.conf went to pkgs/moonraker with the server it configures, the
ff-*.cfg to pkgs/klipper-config, and the two per-model chamber configs
became anvil-klipper-creator5-config and anvil-klipper-creator5pro-config
— which is the first use of Provides/Conflicts in this feed, on a virtual
anvil-klipper-chamber-config that either satisfies and neither may share.
The dependency runs from the model package to the shared one and not the
other way: opkg install anvil-klipper-config would otherwise resolve the
virtual name by picking whichever provider it found first, which is a coin
flip between two files describing different hardware. The model is the one
fact opkg cannot work out for itself, so it is the name you install.
The model stopped being a property of the build: two builds of one commit used
to produce different bytes under one version number, with nothing in the feed
or on the printer recording which machine they were for.
Two flags became honest as a side effect, and both are behaviour changes worth
knowing about. BUILD_TOOLCHANGE=0 used to ship the toolchanger's .cfg
anyway, because they rode in anvil-core while everything else the flag
controls was gated; it no longer does. BUILD_MOONRAKER=0 used to overwrite
the printer's moonraker.conf with one written for a server it was not going
to install; it no longer does, which is what docs/building.md already said
the flag meant.
What it cost and what it caught. Nothing shipped changed: all 41 packages
are byte-identical across the move. It did surface one live bug --
HelixScreen's printer_database.d entry was hashed into ANVIL-CORE's
PKG_STAMP_EXTRA, so editing the json rebuilt a package that does not contain
it and left the one that does sitting in the cache. pkg_payload_hash is now
the one way a recipe keys its own files, and a recipe cannot hash somebody
else's.
Phase 2 — the payload is the feed (assembly half: done)
Split in two, and only the first half is done.
The assembly half — done. bin/patch.sh no longer stages anything. It
installs the feed into a staging root and ships what lands there:
./bin/build-packages.sh # the feed
opkg --conf <generated> --force-postinstall \
install <the runtime set> # into work/modpayload-root
tar -C work/modpayload-root/usr/data/anvil # the payload
Eight sections that each ran a recipe and copied its build tree are gone, and
with them the hand-written PKG_DEV_FILES prune, three partial ABI gates and
the last pkg_out call. patch.sh went from 793 lines to about 700, and what
is left of it writes /usr/prog — FlashForge's tree, which no package of ours
may touch — plus the three files that are in the payload and in no package.
The on-printer half — not started. .install-manifest still exists and
installer/run-append.sh still deletes by it. Replacing it with opkg's own
.list files changes what happens on real printers during an upgrade and
rewrites what is now qa/replica/test_upgrade.py; it is a separate change
with a different risk profile. Keeping the halves apart is what made the first
one provable: on-printer behaviour is unchanged, so correctness reduced to a
payload diff.
Cancelled rather than done -- see 86-wipe-and-extract.md. $MODDIR is
deleted wholesale on update now, so a printer never deletes selectively and
needs no file database for the purpose — ours or opkg's. There was nothing
left for this half to replace .install-manifest with. The .list files
still ship and still describe the payload; nothing reads them at upgrade
time.
What it is assembled with
The printer's own opkg, on the printer's own filesystem. bin/patch.sh
runs bin/build-payload.py, which starts the replica, puts the feed on the
simulated stick and lets the machine's opkg install it. The tree is tarred
back out and becomes $MODDIR. Maintainer scripts therefore run where they
will run on a printer, and the install that ships is the install that was
tested.
The database is written by the same binary that will later read it on the machine — not merely by the same program built twice — which is what makes "phase 2 is a swap rather than a migration" true rather than an argument about format compatibility.
This is why make build needs the docker socket. The build lane runs on
RUNBUILD (Makefile), which is RUN plus the socket and the socket's own
gid, and still --user so the output belongs to you. A build that could not
reach the daemon could not assemble a payload.
A host opkg was the previous answer, and it is gone. pkg_buildopkg
cross-built an x86-64 opkg from the same pinned tarball, cached at
work/.opkg-host, and installed the feed into a staging root with
--offline-root. It worked, but everything about it was an imitation of the
machine: --prefix=$MODDIR was mandatory and easy to get wrong (opkg bakes
its state directory in at compile time — libopkg/Makefile.am:
-DVARDIR="@localstatedir@" — so any other prefix comes up believing nothing
is installed and reinstalls the world), --disable-shared was mandatory for
the same reason one layer down, and postinst scripts had to be forced because
nothing could execute a MIPS binary. The replica has none of those problems
because it is not imitating anything.
A chroot on the host was tried and rejected before either of those. It
needs root: unprivileged chroot is Operation not permitted and
unshare -Ur is blocked by Docker's seccomp profile. The payload then comes
out root-owned and the build-lane user cannot delete it, so the next
make build dies on its own rm -rf — the failure Makefile:48-61 exists
to prevent.
What the set is
bin/patch.sh names twelve packages and lets Depends bring the rest: 31
installed, 19 of them pulled in and marked Auto-Installed: yes. The feed is
indexed (src anvil file:$PKG_FEED plus opkg update), so this is the same
resolution an opkg install anvil-moonraker performs on a printer — which
means the dependency metadata is exercised by every build instead of only when
somebody tries it.
file: is answered before curl is reached at all
(libopkg/opkg_download.c:134), which is why a --disable-curl opkg can
still resolve a feed.
The gates reach here by absence. A PKG_WHEN-gated recipe leaves no
.ipk in the feed, so BUILD_HELIX=0 simply has no anvil-helixscreen to
install, with no BUILD_* flag restated outside the pkg.conf that owns it. The cost is that a misspelled root drops out
just as quietly, which is what test_the_payload_roots_name_real_packages
exists for. A missing dependency is still a hard error, raised by opkg.
Four roots are Recommends, a field opkg does not have.
anvil-python-pillow and anvil-python-preprocess-cancellation are
Moonraker's thumbnail path, which pkgs/moonraker/pkg.conf argues out of
Depends on the grounds that a printer without them still serves;
anvil-python-greenlet and anvil-python-cffi are klippy's, and
pkgs/klipper/pkg.conf cannot declare them while klippy still runs under
FlashForge's interpreter.
An earlier version enumerated the closure by hand, filtering
pkg_recipes. It worked and it hid the thing worth testing. It also had a trap
in it: -dev had to be filtered by package NAME, not by "has a runtime half",
because execline, s6 and CPython ship both halves while zlib, openssl,
sqlite, expat, libffi, xz, bzip2, libarchive and skalibs ship
only a dev package and say so by setting PKG_NAME=anvil-<x>-dev outright.
Asking the first question installed all nine — 40 packages and 2165 ELF
objects where there should be 31 and 154. Roots and Depends cannot make that
mistake: nothing depends on a -dev package.
The one version that cannot be left to opkg is anvil-core's. Its
PKG_VERSION is MOD_VER, which defaults to today's date, so a feed built
yesterday would resolve and install yesterday's anvil-core without
complaint. patch.sh checks for the expected filename before installing and
names ./bin/build-packages.sh if it is absent.
The recipes whose version is a pin have the opposite problem.
anvil-moonraker, anvil-klipper, anvil-helixscreen and anvil-timelapse
ship files of ours beside an upstream tree, and apk decides whether to upgrade
from the version string alone: a change under payload/ is new bytes under an
old name, so a printer already holding that version keeps what it has.
PKG_RELEASE is the field that moves — raise it when payload/ changes, and
back to 1 when the pin itself moves. PKG_STAMP_EXTRA="$(pkg_payload_hash)"
is a different mechanism for a different reader: it makes this build notice
the edit and rebuild. It does not make a printer notice it.
The model is the one fact opkg cannot work out for itself: both chamber
configs are built every time, they own the same config/printer.chamber.cfg,
and they Conflict. opkg refuses the pair with exit 255 — measured — so
$TARGET_MACHINE picks one.
The database ships
$MODDIR/var/lib/opkg/{status,info/*} is in the payload and in
.install-manifest. This is coherent with installer/run-append.sh: pass 1
deletes what the old manifest named, including the old database, and the new
tarball then extracts one that exactly describes the payload being extracted.
A .tgz flash is a full reset of /usr/data/anvil.
The alternatives are worse. Leaving it out defeats the point — an opkg with no
status believes nothing is installed. Keeping it out of the manifest only
is worse still: a package release N shipped and N+1 dropped would keep its
stanza forever while its files were gone.
Known gap. A printer that runs opkg install for something extra and then
flashes a .tgz keeps that package's files — they are in no manifest of ours
— but loses its stanza. That is inherent to running two install mechanisms at
once, and it dissolves when the manifest gives way to opkg's .list files.
Installed-Time is normalised after installing. opkg takes it from
time() and honours SOURCE_DATE_EPOCH only for a man-page date at configure
time, so two builds of one commit differed by one line per package — measured,
fixed, measured again.
Maintainer scripts: what we learned, and what it costs
opkg 0.7.0 does not run maintainer scripts under --offline-root
(libopkg/pkg.c:1339, opkg_cmd.c:342) unless --force-postinstall, and it
sets no IPKG_INSTROOT at all — the variable does not appear anywhere in
its source.
The install passes --force-postinstall, but only to make opkg mark packages
installed rather than unpacked: nothing runs opkg on the printer after
run-append.sh extracts the tarball, so the extraction is the install and a
database saying otherwise would make the first real opkg upgrade argue with
the filesystem. No recipe defines a maintainer script, so nothing runs.
This contradicts an assumption made earlier in this document. The plan for
firmwareExe and start.sh was to package them and let a postinst place them
on /usr/prog from a staging root. That does not work as written: the script
would run on the build host with no IPKG_INSTROOT to tell it where the root
is. The options when that step arrives are to set an environment variable of
our own and have the script read it, or to leave those two files as the
prog/ copies they are today. It needs its own design; it is not free.
The gate this cleared
A payload built from the parent commit and from the migration, on the synthetic stock fixture with the real vendored assets:
| files removed | 0 |
| content changes among 2778 shared files | 1 — .install-manifest, which lists the payload |
| mode changes | 0 |
| symlink changes | 0 |
| added | klipper/**, var/lib/opkg/**, bin/opkg |
| two builds in a row | byte-identical |
bin/verify.sh passed every check including the ship boundary; make test
ran build-packages → unpack → patch → pack → verify with 9 gates green.
(Measured while both existed. bin/verify.sh and make test have since been
retired -- the ship boundary is now qa/replica/test_what_ships.py, asked of
the installed machine.)
One bug fell out of it. PKG_EXCLUDE was applied with find -name, which
matches a basename at any depth, so .version deleted Mainsail's own
www/mainsail/.version along with the recipe stamp. anvil-mainsail had been
shipping without it, invisibly, because the payload was copied from
work/pkg where the file survived. Making the package's contents the
payload's contents is what surfaced it.
Still owed by this phase, and the allowlist in
test_every_payload_file_is_owned_by_a_package is the list: .install-manifest
(generated), config/moonraker-custom.conf (user state, and arguably
conffiles once the on-printer half lands), and opkg's own var, var/lib,
var/run. Five entries, three of them opkg's. anvil.conf was a sixth until
it was removed outright -- the test says so where the entry used to be, so it
cannot come back under an exemption.
Phase 3 — a feed over the network (the printer's half: done)
bin/build-packages.sh writes anvil.adb with apk mkndx, and signs both
it and every package when APK_SIGN_KEY is set. What was missing was the
other two thirds: a URL the printer asks, and a tree at that URL.
The printer now ships knowing where to look. bin/common.sh holds
FEED_URL="${FEED_URL:-http://reforge.8941973.xyz/apk}"
and pkgs/anvil-core generates $MODDIR/etc/apk/repositories from it, beside
the public key it already shipped. bin/publish-feed.sh writes the matching
tree — <arch>/anvil.adb and the packages beside it — and the release workflow
attaches it as reforge-apk-feed.tar.gz, so a host can be filled from any
release, including ones tagged before the host existed.
THE POINTER HAD TO SHIP FIRST, and that is why this landed with no host
running. A feed cannot tell a printer where the feed is. Every machine
installed without that file needs a USB stick before it can ever fetch
anything, so the file has to travel in the .tgz — which means shipping a URL
that 404s for as long as it takes to stand the host up. The cost of doing so
is bounded, and read out of apk's source rather than assumed: an unreachable
repository fails apk upgrade and nothing else -- apk_db_repository_check
is called from app_upgrade.c and not from app_add.c. apk add ./file.apk does not consult a
repository at all.
The entry names the index file rather than a directory, which is the same
shape the USB stick uses and is a grammar rather than a preference:
src/repoparser.c reads a line ending .adb or .tar.gz as an index in
place — packages as its siblings, named by the pkgname-spec mkndx wrote
into the index — and anything else as a directory to look for
<url>/<arch>/APKINDEX.tar.gz under. The v3 directory form would work too, but
only with mkndx --pkgname-spec '${arch}/${name}-${version}.apk'; one layout,
exercised by every payload build, is worth more than the convention.
The architecture is a directory in the URL and not a suffix on the file. Package file names carry no architecture, so two architectures served from one directory would be two files with one name.
The replica stopped writing that file. case-build-payload.sh used to
write its own etc/apk/repositories naming /mnt and delete it before
tarring; it now passes --repositories-file with a path on the stick, which
apk reads instead of the default (database.c: the default file and the
repositories.d directories are consulted only when no file is named). So
what ships is whatever anvil-core installed, and the build never reads it.
The one open piece is still certificates. libfetch defaults to
SSL_VERIFY_PEER and falls back to SSL_CTX_set_default_verify_paths(), which
resolves to $MODDIR/ssl — empty. That is the same gap tools/python/README.md
records for CPython, and the same ~200KB ca-certificates bundle closes both.
Until one ships, the feed is http:// and is not much weaker than it sounds:
every package is signed, and a signature is what actually decides whether a
printer installs it.
What is left after that is operational, not code: a host answering at
$FEED_URL, and a decision about who runs bin/publish-feed.sh — a release
step with a deploy secret, or a person with the release tarball.
What this does not fix, and is not meant to
- It does not shrink the build. Every package is still cross-compiled here, from a pinned and hashed source. That was never going to change; see the fact that decides the comparison.
- It does not replace
versions.env. Pins and hashes stay exactly where they are;pkg.confreads them rather than restating them. - It does not make the printer able to install other people's packages. The architecture string is chosen specifically to make that fail loudly.