firmware/docs/notes/85-packaging.md

Packaging: apk vs opkg, and the .ipk proof of concept

Whether to stop hand-building Python, Moonraker, s6 and libsodium into one tarball and start building packages instead. Written 2026-08-28. The comparison below is settled; the migration after it is a proposal, and only phase 0 is implemented.

Same rules as 80-s6-migration.md: each phase is independently shippable and names the gate that proves it.

Why, in one paragraph

bin/patch.sh is 1,900 lines and eleven cross-builds deep, and its output is a single 53MB anvil.tar.xz that the printer's installer unpacks whole. Changing one line of anvil.conf ships all 53MB. There is no way to install just a new Moonraker onto a printer you are debugging, no record on the machine of what version of anything is on it, and the "remove what the last release installed" problem has already been solved once by hand — installer/run-append.sh reads an install manifest and prunes by it, which is a package manager's file database with one package in it. The question is whether to keep growing that or to adopt the format that already means it.

The fact that decides the comparison

Neither package manager brings any software with it. This is the whole argument and it is worth stating before the table, because every "apk vs opkg" discussion elsewhere on the internet is really about which distro's repository you want, and neither repository can be used here:

• Alpine has no MIPS port at all. Not "an old one" or "a community one" — mips is not among the architectures Alpine builds. There is no apk repo to point at.
• OpenWrt does have mipsel_24kc, which is the same ISA and the same o32 ABI as this printer — and is built against musl. This printer's rootfs is glibc 2.29, and the reason that matters is written down at length in versions.env: klippy's c_helper.so is dlopened by a glibc interpreter, so glibc is not a preference here. An OpenWrt package would unpack perfectly and produce a library nothing can load.

So every package will be cross-built by this repo either way. A package manager buys packaging, versioning and installation. It does not buy software, and adopting one does not make bin/patch.sh's cross-builds go away — it reorganises them into units with names and versions.

That reframes the choice entirely: the winner is whichever format is cheapest to produce and install on a machine we control, not whichever has the better ecosystem.

The comparison

	opkg / .ipk	apk / .apk (Alpine)
Upstream binaries we can use	none (musl)	none (no mips port)
Making a package	ar + two tar.gz + a 3-byte version file	three concatenated gzip streams with an RSA signature over the control stream
...and the tool that does it	opkg-build, upstream, 423 lines of POSIX sh	realistically abuild + abuild-sign + a keypair, and abuild assumes an Alpine host
Indexing a feed	opkg-make-index, from the same upstream	apk index, needs the signing key
Cross-building the packages	irrelevant; we stage a tree and hand it over	same, if you hand-roll; painful through abuild
On-device client	opkg + libarchive + zlib	apk-tools + zlib + openssl
...and what we already build	libarchive: nothing. New cross-build.	openssl 3.0.15 and zlib are already cross-built for CPython
Offline install into a prefix	--offline-root DIR, and opkg install ./file.ipk needs no feed at all	apk add --root DIR --allow-untrusted ./file.apk
Signing	optional; signs the index, so packages need no signature each	expected; --allow-untrusted to skip
Format stability	unchanged for over a decade; also Yocto/OpenEmbedded's native format	apk-tools 3 introduced a new package format; v2/v3 coexistence is live churn
Home turf	embedded MIPS with a read-only rootfs — exactly this device class	a full musl distro root

Two honest points for apk, since it is the more actively developed tool: its world model (a declarative "these are the packages I want", reconciled on every operation) is genuinely nicer than opkg's imperative install/remove, and its runtime dependencies are ones this repo already cross-builds while opkg's libarchive is not. If the deciding factor were the on-device client, apk would win.

It is not. The deciding factor is that every package has to be produced here, by us, on every build, in CI — and for .ipk there is an upstream tool that does it, needing no keypair and no distro-specific build framework. For .apk the equivalent is either implementing a signed three-stream format by hand or importing Alpine's abuild into a Debian-based build image. That cost is paid on every build and every recipe; the client is paid once.

The format-churn line matters more than it looks, too. A format with no users on our architecture and an in-progress v2→v3 transition is a thing to track forever for no benefit, since we can never consume anyone else's packages anyway.

Decision

opkg, and the .ipk format, with opkg-utils as the packager — we drive upstream's opkg-build and opkg-make-index rather than assembling archives ourselves. An earlier revision of this work did hand-roll the ar-and-two-tarballs step; it worked, and it was still this repo re-deriving a format somebody else maintains, including the parts whose failure mode is a package that inspects fine and installs nowhere.

Facts established by measurement, not by reading

Each of these was run, not assumed. They constrain the phases below.

• Our cross-built opkg installs our packages, on mipsel — measured when opkg was still a static musl binary. Run under qemu-mipsel-static against an --offline-root, it installed both packages of the day, reported them under list-installed, listed their files, and removed both leaving nothing behind; the opkg it installed then ran and reported its own version. The whole loop, closed on the target architecture.

  This has not been re-run since opkg moved to dynamic glibc, and the command changed with it — a dynamic binary needs qemu -L <sysroot>. See the end of phase 0. Nothing suggests it broke; nobody has checked.

• opkg bakes its state directory in at compile time. Its status file landed at /usr/data/anvil/var/lib/opkg because it was configured --prefix=/usr/data/anvil — --offline-root does not move it. Built with any other prefix it comes up believing nothing is installed and reinstalls the world. This is the third instance of the same trap on this printer, after s6's libexecdir and execline's shebangdir (both in versions.env). On this machine the --prefix is not a preference, and that is why $MODDIR now lives in bin/common.sh where a recipe cannot spell it differently.

• opkg 0.7.0, not 0.6.3, if you build against musl. Two files in 0.6.3 (opkg_remove.c, opkg_archive.c) call basename() without including <libgen.h>, getting away with it only because glibc's <string.h> also declares a differently-behaved basename. musl does not, so gcc 14 stops at "implicit declaration" — and on a compiler that merely warned, the returned pointer would have been truncated through an implicit int. 0.7.0 adds both includes. That was the difference between a pin bump and carrying a patch.

• opkg will not configure without libarchive. Measured: ./configure hard-fails at the pkg-config check. There is no internal tar fallback any more. libarchive and zlib are therefore build dependencies, cut down to gzip-and-tar and linked into the static binary; neither ships as a file.

• libtool eats -static. A program configured LDFLAGS=-static comes out dynamically linked anyway, because libtool defines that flag to mean "prefer static libtool libraries" and does not pass it to gcc. -all-static is the one that means it, and it cannot go through ./configure (configure's own link probes call gcc directly, which rejects it), so it goes to make. Nothing relies on this any more — opkg links glibc dynamically now and gets its static zlib and libarchive simply by there being no .so of either in the sysroot — but the check it motivated stayed, inverted: readelf -d on the finished binary, expecting libc.so.6 and nothing of ours. The lesson is the check, not the flag. A link that is not what it was asked to be shows up as a NEEDED entry nobody looks at, and then as a missing .so on a printer.

• opkg-utils has no release tarball, anywhere. Upstream's cgit has snapshots disabled (every format returns "Unsupported snapshot format"), downloads.yoctoproject.org carries opkg but not opkg-utils, and the GitHub mirror at shr-project/opkg-utils last saw a commit in 2012 and has no tags. So it is pinned by git commit — a hash of the whole tree, the same guarantee the sha256s give, in git's spelling — and bin/fetch-assets.sh verifies HEAD against it and refuses a dirty checkout.

• The printer's busybox cannot be assumed to have ar. It is 1.31.1 built small — no timeout, no nc, no ionice, all measured on the replica. That is why the printer gets a real cross-built opkg (pkgs/3rdparty/opkg/), which reads .ipk through its own linked-in libarchive rather than shelling out to tools the busybox may not have.

• All four packages are reproducible. rm -rf work/pkg work/packages and a full recompile produce byte-identical .ipk files. That is a measurement of these packages and this toolchain, not a property of the build system — a package whose upstream embeds a timestamp will not have it, and the honest response is to record which ones do rather than to claim the fleet is reproducible.

• A static archive is not reproducible until it is made so, and this toolchain cannot do it alone. ar stores an mtime and the builder's uid/gid in every member header, plus another timestamp in the symbol index — none of which SOURCE_DATE_EPOCH or opkg-build's -o 0 -g 0 reaches, because they are inside a file the tarball merely contains. Measured: two cold builds of pkgs/3rdparty/zlib produced different sha256, every member reading 1000/1000 Aug 28 15:10. objcopy -D normalises the member headers, but the cross binutils is 2.27 and its ranlib -D writes a fresh current timestamp into the index instead of zeroing it. The index is therefore rewritten with the build machine's ranlib (2.40 in docker/Dockerfile.build), which is version-sensitive and worth knowing before anyone changes the image. The proof it produces a valid index is downstream: opkg links against both archives, and a broken index fails that link.

• The ABI gate's e_flags whitelist was a proxy, and object files exposed it. It accepted exactly 0x70001405 and 0x70001407, both measured from linked binaries — where crt startup objects set EF_MIPS_NOREORDER, so every executable and shared object read one or the other. Individual objects need not: libarchive's xxhash.o is 0x70001406, identical ABI, no NOREORDER, and the old gate called it wrong. The ABI is the high bits (0x70001400 = arch32r2 | o32 | nan2008); the low three (noreorder, pic, cpic) say nothing about whether the kernel will exec the file. The gate now masks them, and reads every header rather than one — readelf -h on an archive prints one per member, 122 for libarchive.a, and the single-line version handed a multi-line string to a comparison expecting one word.

• A cache stamp compared by one file and written by another will drift. bin/fetch-assets.sh compared work/.s6/.version against "$SKALIBS_VERSION $S6_VERSION" while bin/patch.sh wrote three fields into it. The test could therefore never be false: a 71MB toolchain was re-hashed on every run, downloaded on every cold one, and the comment above the condition described a fast path that had never once been taken. $S6_STAMP now lives in bin/common.sh, the recipes compute theirs in pkg_stamp, and test_no_cache_stamp_is_spelled_in_two_places keeps it that way.

Phase 0 — the proof of concept (implemented)

Four packages, end to end, beside the existing build and changing nothing that ships.

make packages            # or ./bin/build-packages.sh [name...]

work/packages/anvil-zlib_1.3.1-1_mipsel_xburst2.ipk
work/packages/anvil-libarchive_3.7.9-1_mipsel_xburst2.ipk
work/packages/anvil-libsodium_1.0.20-1_mipsel_xburst2.ipk
work/packages/anvil-opkg_0.7.0-1_mipsel_xburst2.ipk
work/packages/Packages{,.gz}

Needs no stock FlashForge package, which is most of the point: packaging has to be runnable in CI on a bare checkout, or the gate only runs where the proprietary firmware is and stops being a gate.

file	what it is
pkgs/lib.sh	the part of a cross-build that is the same for every package
pkgs/3rdparty/zlib/	the library that was cross-built twice, now built once
pkgs/3rdparty/libarchive/	what opkg reads .ipk files with; builds against anvil-zlib
pkgs/3rdparty/opkg/	opkg itself; builds against both of the above
pkgs/3rdparty/libsodium/	build.sh + pkg.conf. bin/patch.sh section 5d's build, moved.
bin/build-packages.sh	orders the recipes, lays out each tree, drives opkg-build, indexes the feed
qa/static/test_ipk.py	112 tests, no toolchain needed

One recipe builds one package

This is the rule the layout exists to enforce, and it did not hold at first. pkgs/3rdparty/opkg/build.sh used to unpack zlib, build it into a private sysroot, unpack libarchive, build that against it, and only then build the binary it shipped: one script, three libraries, one package. Section 5b still does the same thing with skalibs and s6.

What that costs is not tidiness. A library built inside somebody else's recipe has no version, no package and no way to be reused, so the next consumer builds its own — zlib was cross-built twice in this tree, once in section 5c for CPython and once here for libarchive, from one pinned tarball with the same flags, and neither copy could see the other. Splitting them is what makes the feed the interface between recipes rather than an output nobody reads.

So a recipe now declares PKG_BUILD_DEPENDS, and pkg_deps fills its sysroot by unpacking those packages out of the feed. Building against the package rather than against the build tree is the part worth paying for: a package that forgets to ship a header fails the next recipe's configure, on the build that produced it.

opkg is not special, and nothing waits for it

The obvious objection to a feed-based build is that it needs a package manager to install the build dependencies, and the package manager is itself a package. It does not. opkg-unbuild is upstream's own inverse of opkg-build and comes from the same pinned opkg-utils checkout, so a sysroot can be filled without a working opkg existing anywhere. There is no bootstrap stage, no ordering rule beyond declared dependencies, and no qemu in the build path.

bin/build-packages.sh topologically sorts the recipes and packages each one before the next is built, because the next one reads it out of the feed. Alphabetical order — what iterating pkgs/*/ gives you — puts libarchive before the zlib it needs.

The split in pkgs/lib.sh

• pkg_conf / pkg_stamp — a recipe's metadata and its cache key, both read from pkg.conf. The stamp includes the toolchain and, recursively, the stamps of everything the recipe builds against, so a zlib bump rebuilds libarchive and opkg with nobody maintaining a composite stamp by hand.
• pkg_order — dependency-first build order, and the closure of a single named recipe so that PKG=opkg builds what opkg needs.
• pkg_begin / pkg_end — the stamped build cache, which is what lets bin/fetch-assets.sh skip a 203MB toolchain download.
• pkg_toolchain — unpack, write the gcc wrappers that bake -EL -mnan=2008 into the driver (autotools link lines do not all forward CFLAGS), and then prove the wrapper emits the right ABI before anything is built on it. That step existed in section 5b and was missing from 5d; the copies had drifted before there was anywhere to put them.
• pkg_deps — unpack this recipe's build dependencies out of the feed with opkg-unbuild and point CPPFLAGS, LDFLAGS and pkg-config at the result. The sysroot mirrors the printer, dependencies living under $MODDIR inside it, which is why PKG_CONFIG_SYSROOT_DIR is set to the sysroot rather than emptied: the .pc files say prefix=/usr/data/anvil and that variable is what turns them into paths that exist on the build machine.
• pkg_autotools — configure, make, install into the staging tree.
• pkg_ship — copy out what the package contains, normalise static archives, strip ELF with the cross strip. What a package contains depends on what it is for: a library that ships to the printer ships its .so, and a library that exists to be built against ships headers, .a and .pc.

qa/static/test_ipk.py gates this directly: a recipe that does not source pkgs/lib.sh, spells its own -mnan=2008, unpacks its own toolchain, calls ./configure itself, or unpacks more than one source tarball fails the suite. The one named exception is zlib, whose configure has never accepted --host. A recipe that needs something pkgs/lib.sh cannot express should grow pkgs/lib.sh — that is what the gate is for.

Not yet shared: bin/patch.sh section 5b (s6 and skalibs) still carries its own copy of all of this, and is the remaining instance of one script building two libraries. Phase 1 is what turns it into recipes.

The property everything rests on

A library is compiled once, whichever vehicle it ships in. bin/patch.sh runs pkgs/3rdparty/libsodium/build.sh and stages its output into the payload exactly as before, and bin/build-packages.sh packages the same tree. Section 5c now does the same with zlib: it runs pkgs/3rdparty/zlib/build.sh and stages the result into CPython's dependency sysroot instead of compiling its own copy. While that holds, the tarball's copy and the package's copy cannot be different libraries wearing one version number — and test_the_package_and_the_payload_share_one_build asserts it rather than trusting it.

Gate: pytest qa/static (127 tests, 29 of them packaging) plus make packages from a cold cache producing four byte-reproducible .ipk files and an index. Both pass.

Not re-run since opkg moved to glibc: the end-to-end check where our own cross-built opkg installs and removes this feed. It used to be a bare qemu-mipsel-static invocation because opkg was a static musl binary; a dynamic one needs a sysroot, so the command is now

qemu-mipsel-static -L work/.mips-toolchain/mips-gcc720-glibc229/mips-linux-gnu/libc \
    work/pkg/opkg/bin/opkg --offline-root <tmpdir> install work/packages/*.ipk

(readelf -l says the interpreter is /lib/ld-linux-mipsn8.so.1, the nan2008 loader, which that sysroot provides and the printer's rootfs must too — the same loader the interpreter already links.) Worth running before anyone leans on this.

Phase 1 — a recipe per cross-build (~3–5 days)

Turn the rest of bin/patch.sh's builds into recipes, the way libsodium became one —

klipper ✔  toolchange-config
python ✔  python-<18 packages> ✔
openssl ✔  libffi ✔  sqlite ✔  xz ✔  expat ✔  bzip2 ✔
zlib ✔  libarchive ✔  libsodium ✔  opkg ✔
skalibs ✔  execline ✔  s6 ✔  s6-rc ✔
mainsail ✔  moonraker ✔  helixscreen ✔  anvil-core ✔

Done: 38 recipes producing 41 packages. Section 5b — the last place where one script built two libraries — is four recipes (skalibs, execline, s6, s6-rc), and moving them off the musl toolchain removed the second libc from this tree entirely. Section 5c, which was 825 lines, is pkgs/3rdparty/python plus eighteen pkgs/3rdparty/python-* and 115 lines of staging.

What splitting CPython actually bought. Not tidiness. The old section had one cache and two stamps, and its own comment explained that a bumped Pillow rebuilt the interpreter and all eighteen packages — because a wheel could only be cross-built during the few minutes when an untrimmed staging tree, a private sysroot of static libraries and a throwaway x86-64 build-python happened to exist at once. Those are now, respectively: the feed's anvil-python-dev package, each recipe's own sysroot, and pkg_buildpython's shared cache. None of them is a passing moment, so make packages PKG=python-pillow is a Pillow build and nothing else.

It also made eighteen invisible things visible. Each package now carries its upstream version into the index, anvil-moonraker declares the thirteen it imports, and a printer that does not want gcode thumbnails can leave the 500KB of Pillow uninstalled — a decision that could not previously be expressed.

Klipper was the last one, and it needed one new knob. klippy/chelper has no Makefile and never has: on a machine with a compiler klippy builds c_helper.so at first run from the argument list in klippy/chelper/__init__.py, and this printer has no compiler. So the build system for that .so genuinely is one gcc line, which pkg_build could not express — it always ran make. PKG_CC_SHARED is that line, appended to $CC -shared -fPIC: the recipe states Klipper's own COMPILE_ARGS and pkg_build keeps ownership of the compiler, so the ABI flags stay spelled in exactly one place. A knob, not a verb, for the reason the header of pkg_build gives about the three mechanisms with one user each.

Two things the recipe took with it. KLIPPER_FORK — the config.env knob that pointed the build at a local checkout instead of the pin — is gone: a recipe names its source exactly once, a checkout beside a pinned tarball is the second source that rule forbids, and it is the same variable that let v20260824 ship a klippy tree nobody had built. test/test-chelper.py had already stopped falling back to reading KLIPPER_FORK out of config.env when given no argument — that fallback printed SKIP: no KLIPPER_FORK configured, a green line for a check that had not run — and the script itself is gone now, deleted with the rest of test/.

The package installs the tree under $MODDIR/klipper, where nothing reads it yet: klippy is still started by the stock /usr/prog/klipper/klipperDaemon, so bin/patch.sh stages the recipe's output into the SOFTWARE component as before. Phase 7 of docs/notes/80-s6-migration.md is what makes the installed copy the live one. [Superseded: phase 7 landed. The klipper s6-rc service execs $FF_PYTHON against $MODDIR/klipper/klippy, and the software-component copy is gone.]

patch.sh keeps staging the payload from those same trees, so the shipped tarball does not change. The work is mechanical; the risk is in the two builds with the most measured constraints behind them (5b's s6 prefix, 5c's CPython cross-build), which should go last — and both are exactly the shape pkgs/lib.sh already handles, since it was written from them.

Gate, corrected. This used to read "the anvil.tar.xz built after this phase is byte-identical to the one built before it". That cannot be the gate, for two independent reasons, and stating it that way meant nobody could tell a real regression from an expected difference:

• Moving s6 to dynamic glibc changes those binaries by design. A tarball that did not change would mean the change had not happened.
• bin/pack.sh builds anvil.tar.xz with a bare tar -cf, with no --sort=name, --mtime or --owner. It carries filesystem order and per-file mtimes, so it has never been byte-reproducible and the stated gate could never have passed.

What is checked instead: the assembled work/modpayload-root tree — file list, modes and content hashes — is unchanged except for the components a change deliberately rebuilt. The .ipk files themselves ARE byte-reproducible and that is checked directly: two cold make packages runs produce identical archives. Measured over all 40 with work/pkg, the shared build-python and the unpacked toolchain all deleted in between — so the x86-64 interpreter that generates every mipsel extension module is itself rebuilt rather than reused.

A recipe owns its files, and the directory says how they ship

The recipes came first and the repository's own files stayed where they were: one top-level payload/ tree organised by DESTINATION (init.d/, bin/, etc/s6/, klipper/config/) plus an assets/ directory of three .conf files. That was right when there was one owner. With 38 recipes it hid three different kinds of file behind identical-looking paths, and the only thing recording which package owned any of them was a thirty-line comment at the top of pkgs/anvil-core/build.sh.

Files now live with the recipe that owns them, in one of three subtrees:

pkgs/<recipe>/payload/   staged into the .ipk, laid out as it lands under
                        $MODDIR -- payload/init.d/S60nginx installs as
                        $MODDIR/init.d/S60nginx and no recipe says so
pkgs/<recipe>/prog/      placed on /usr/prog by bin/patch.sh
pkgs/<recipe>/seed/      templated or seeded user state

Recipes themselves sit at two depths, and pkgs/lib.sh's pkg_dir is the only thing that knows it: pkgs/<name>/ for a recipe that carries files of this repo (four), pkgs/3rdparty/<name>/ for one that builds a pinned tarball and carries none (thirty-four). That is for ls pkgs/ and nothing else — no package, no dependency and no stamp knows which side a recipe is on. The line is deliberately mechanical rather than "do we modify it", which drifts: the day somebody patches zlib nobody would agree on which side it belongs, whereas "does it carry files of ours" is a fact about the tree and a test can demand the move.

installer/ holds the two files that are never files on a printer at all: run-pre.sh and run-append.sh are spliced into FlashForge's own run.sh. qa/static/test_recipe_layout.py fails if a fourth kind of directory appears, which is the failure this layout is actually for — a misfiled file is caught by a build, an unruled directory is caught by nothing.

prog/ and seed/ are the not-yet-packaged residue, and that is the point of naming them. Both empty out on the way to the end state. seed/ got there first and by a route this section did not predict: rather than a postinst seeding anvil.conf.default when the real file is absent, anvil.conf was removed outright and seed/ left anvil-core with it. prog/ empties when a postinst run against a STAGING ROOT places /usr/prog/PROGRAM/software/firmwareExe as a symlink into $MODDIR. The second is the interesting one, and it is only possible because /usr/prog is written by a tarball this repo bakes, not by a flash we do not control: the install can happen in the build container, which is what "the tarball becomes a view of the feed" below already assumes. The trap there is $IPKG_INSTROOT -- a postinst that writes a bare /usr/prog/... symlinks the BUILD CONTAINER's root, the staging tree gets nothing, and the build stays green.

Ownership followed, once the files could be seen. anvil-core had held everything of ours for no better reason than being the first recipe written, and with the files visible per-recipe the misfilings were obvious: moonraker.conf went to pkgs/moonraker with the server it configures, the ff-*.cfg to pkgs/klipper-config, and the two per-model chamber configs became anvil-klipper-creator5-config and anvil-klipper-creator5pro-config — which is the first use of Provides/Conflicts in this feed, on a virtual anvil-klipper-chamber-config that either satisfies and neither may share. The dependency runs from the model package to the shared one and not the other way: opkg install anvil-klipper-config would otherwise resolve the virtual name by picking whichever provider it found first, which is a coin flip between two files describing different hardware. The model is the one fact opkg cannot work out for itself, so it is the name you install. The model stopped being a property of the build: two builds of one commit used to produce different bytes under one version number, with nothing in the feed or on the printer recording which machine they were for.

Two flags became honest as a side effect, and both are behaviour changes worth knowing about. BUILD_TOOLCHANGE=0 used to ship the toolchanger's .cfg anyway, because they rode in anvil-core while everything else the flag controls was gated; it no longer does. BUILD_MOONRAKER=0 used to overwrite the printer's moonraker.conf with one written for a server it was not going to install; it no longer does, which is what docs/building.md already said the flag meant.

What it cost and what it caught. Nothing shipped changed: all 41 packages are byte-identical across the move. It did surface one live bug -- HelixScreen's printer_database.d entry was hashed into ANVIL-CORE's PKG_STAMP_EXTRA, so editing the json rebuilt a package that does not contain it and left the one that does sitting in the cache. pkg_payload_hash is now the one way a recipe keys its own files, and a recipe cannot hash somebody else's.

Phase 2 — the payload is the feed (assembly half: done)

Split in two, and only the first half is done.

The assembly half — done. bin/patch.sh no longer stages anything. It installs the feed into a staging root and ships what lands there:

./bin/build-packages.sh                       # the feed
opkg --conf <generated> --force-postinstall \
     install <the runtime set>                # into work/modpayload-root
tar -C work/modpayload-root/usr/data/anvil    # the payload

Eight sections that each ran a recipe and copied its build tree are gone, and with them the hand-written PKG_DEV_FILES prune, three partial ABI gates and the last pkg_out call. patch.sh went from 793 lines to about 700, and what is left of it writes /usr/prog — FlashForge's tree, which no package of ours may touch — plus the three files that are in the payload and in no package.

The on-printer half — not started. .install-manifest still exists and installer/run-append.sh still deletes by it. Replacing it with opkg's own .list files changes what happens on real printers during an upgrade and rewrites what is now qa/replica/test_upgrade.py; it is a separate change with a different risk profile. Keeping the halves apart is what made the first one provable: on-printer behaviour is unchanged, so correctness reduced to a payload diff.

Cancelled rather than done -- see 86-wipe-and-extract.md. $MODDIR is deleted wholesale on update now, so a printer never deletes selectively and needs no file database for the purpose — ours or opkg's. There was nothing left for this half to replace .install-manifest with. The .list files still ship and still describe the payload; nothing reads them at upgrade time.

What it is assembled with

The printer's own opkg, on the printer's own filesystem. bin/patch.sh runs bin/build-payload.py, which starts the replica, puts the feed on the simulated stick and lets the machine's opkg install it. The tree is tarred back out and becomes $MODDIR. Maintainer scripts therefore run where they will run on a printer, and the install that ships is the install that was tested.

The database is written by the same binary that will later read it on the machine — not merely by the same program built twice — which is what makes "phase 2 is a swap rather than a migration" true rather than an argument about format compatibility.

This is why make build needs the docker socket. The build lane runs on RUNBUILD (Makefile), which is RUN plus the socket and the socket's own gid, and still --user so the output belongs to you. A build that could not reach the daemon could not assemble a payload.

A host opkg was the previous answer, and it is gone. pkg_buildopkg cross-built an x86-64 opkg from the same pinned tarball, cached at work/.opkg-host, and installed the feed into a staging root with --offline-root. It worked, but everything about it was an imitation of the machine: --prefix=$MODDIR was mandatory and easy to get wrong (opkg bakes its state directory in at compile time — libopkg/Makefile.am: -DVARDIR="@localstatedir@" — so any other prefix comes up believing nothing is installed and reinstalls the world), --disable-shared was mandatory for the same reason one layer down, and postinst scripts had to be forced because nothing could execute a MIPS binary. The replica has none of those problems because it is not imitating anything.

A chroot on the host was tried and rejected before either of those. It needs root: unprivileged chroot is Operation not permitted and unshare -Ur is blocked by Docker's seccomp profile. The payload then comes out root-owned and the build-lane user cannot delete it, so the next make build dies on its own rm -rf — the failure Makefile:48-61 exists to prevent.

What the set is

bin/patch.sh names twelve packages and lets Depends bring the rest: 31 installed, 19 of them pulled in and marked Auto-Installed: yes. The feed is indexed (src anvil file:$PKG_FEED plus opkg update), so this is the same resolution an opkg install anvil-moonraker performs on a printer — which means the dependency metadata is exercised by every build instead of only when somebody tries it.

file: is answered before curl is reached at all (libopkg/opkg_download.c:134), which is why a --disable-curl opkg can still resolve a feed.

The gates reach here by absence. A PKG_WHEN-gated recipe leaves no .ipk in the feed, so BUILD_HELIX=0 simply has no anvil-helixscreen to install, with no BUILD_* flag restated outside the pkg.conf that owns it. The cost is that a misspelled root drops out just as quietly, which is what test_the_payload_roots_name_real_packages exists for. A missing dependency is still a hard error, raised by opkg.

Four roots are Recommends, a field opkg does not have. anvil-python-pillow and anvil-python-preprocess-cancellation are Moonraker's thumbnail path, which pkgs/moonraker/pkg.conf argues out of Depends on the grounds that a printer without them still serves; anvil-python-greenlet and anvil-python-cffi are klippy's, and pkgs/klipper/pkg.conf cannot declare them while klippy still runs under FlashForge's interpreter.

An earlier version enumerated the closure by hand, filtering pkg_recipes. It worked and it hid the thing worth testing. It also had a trap in it: -dev had to be filtered by package NAME, not by "has a runtime half", because execline, s6 and CPython ship both halves while zlib, openssl, sqlite, expat, libffi, xz, bzip2, libarchive and skalibs ship only a dev package and say so by setting PKG_NAME=anvil-<x>-dev outright. Asking the first question installed all nine — 40 packages and 2165 ELF objects where there should be 31 and 154. Roots and Depends cannot make that mistake: nothing depends on a -dev package.

The one version that cannot be left to opkg is anvil-core's. Its PKG_VERSION is MOD_VER, which defaults to today's date, so a feed built yesterday would resolve and install yesterday's anvil-core without complaint. patch.sh checks for the expected filename before installing and names ./bin/build-packages.sh if it is absent.

The model is the one fact opkg cannot work out for itself: both chamber configs are built every time, they own the same config/printer.chamber.cfg, and they Conflict. opkg refuses the pair with exit 255 — measured — so $TARGET_MACHINE picks one.

The database ships

$MODDIR/var/lib/opkg/{status,info/*} is in the payload and in .install-manifest. This is coherent with installer/run-append.sh: pass 1 deletes what the old manifest named, including the old database, and the new tarball then extracts one that exactly describes the payload being extracted. A .tgz flash is a full reset of /usr/data/anvil.

The alternatives are worse. Leaving it out defeats the point — an opkg with no status believes nothing is installed. Keeping it out of the manifest only is worse still: a package release N shipped and N+1 dropped would keep its stanza forever while its files were gone.

Known gap. A printer that runs opkg install for something extra and then flashes a .tgz keeps that package's files — they are in no manifest of ours — but loses its stanza. That is inherent to running two install mechanisms at once, and it dissolves when the manifest gives way to opkg's .list files.

Installed-Time is normalised after installing. opkg takes it from time() and honours SOURCE_DATE_EPOCH only for a man-page date at configure time, so two builds of one commit differed by one line per package — measured, fixed, measured again.

Maintainer scripts: what we learned, and what it costs

opkg 0.7.0 does not run maintainer scripts under --offline-root (libopkg/pkg.c:1339, opkg_cmd.c:342) unless --force-postinstall, and it sets no IPKG_INSTROOT at all — the variable does not appear anywhere in its source.

The install passes --force-postinstall, but only to make opkg mark packages installed rather than unpacked: nothing runs opkg on the printer after run-append.sh extracts the tarball, so the extraction is the install and a database saying otherwise would make the first real opkg upgrade argue with the filesystem. No recipe defines a maintainer script, so nothing runs.

This contradicts an assumption made earlier in this document. The plan for firmwareExe and start.sh was to package them and let a postinst place them on /usr/prog from a staging root. That does not work as written: the script would run on the build host with no IPKG_INSTROOT to tell it where the root is. The options when that step arrives are to set an environment variable of our own and have the script read it, or to leave those two files as the prog/ copies they are today. It needs its own design; it is not free.

The gate this cleared

A payload built from the parent commit and from the migration, on the synthetic stock fixture with the real vendored assets:

files removed	0
content changes among 2778 shared files	1 — .install-manifest, which lists the payload
mode changes	0
symlink changes	0
added	klipper/**, var/lib/opkg/**, bin/opkg
two builds in a row	byte-identical

bin/verify.sh passed every check including the ship boundary; make test ran build-packages → unpack → patch → pack → verify with 9 gates green. (Measured while both existed. bin/verify.sh and make test have since been retired -- the ship boundary is now qa/replica/test_what_ships.py, asked of the installed machine.)

One bug fell out of it. PKG_EXCLUDE was applied with find -name, which matches a basename at any depth, so .version deleted Mainsail's own www/mainsail/.version along with the recipe stamp. anvil-mainsail had been shipping without it, invisibly, because the payload was copied from work/pkg where the file survived. Making the package's contents the payload's contents is what surfaced it.

Still owed by this phase, and the allowlist in test_every_payload_file_is_owned_by_a_package is the list: .install-manifest (generated), config/moonraker-custom.conf (user state, and arguably conffiles once the on-printer half lands), and opkg's own var, var/lib, var/run. Five entries, three of them opkg's. anvil.conf was a sixth until it was removed outright -- the test says so where the entry used to be, so it cannot come back under an exemption.

Phase 3 — a feed (~2–3 days)

bin/build-packages.sh already writes Packages and Packages.gz with opkg-make-index, carrying both MD5 and sha256 per package (sha256 is not the default and has to be asked for). Serving that directory over HTTP from CI, teaching the printer an opkg.conf that points at it, and signing the index with usign makes partial over-the-network updates real: opkg upgrade moonraker instead of a 53MB tarball and a reboot.

Signing the index is what makes this safe, and it is one signature for the whole feed rather than one per package — the index carries each package's sha256, so a trusted index makes untrusted packages verifiable. Same argument versions.env makes about vendored tarballs.

Note that --disable-curl and --disable-gpg in pkgs/3rdparty/opkg/build.sh are what this phase turns back on; they are off now so that a static binary does not carry libcurl and OpenSSL for a capability nothing uses yet.

What this does not fix, and is not meant to

• It does not shrink the build. Every package is still cross-compiled here, from a pinned and hashed source. That was never going to change; see the fact that decides the comparison.
• It does not replace versions.env. Pins and hashes stay exactly where they are; pkg.conf reads them rather than restating them.
• It does not make the printer able to install other people's packages. The architecture string is chosen specifically to make that fail loudly.