klippy runs from $MODDIR, so patch.sh stops staging a second copy

The klipper s6-rc service already execs klippy under $FF_PYTHON -- our own
CPython 3.13 -- but it read the tree from /usr/prog/klipper/klippy: a copy
bin/patch.sh staged into the software component out of the very anvil-klipper
package that also installed it under $MODDIR. Two copies of one tree, and the
one that ran was the one a stock flash could overwrite.

One line moves it: KLIPPY=$MODDIR/klipper/klippy/klippy.py.

That deletes sections 1 and 2 of bin/patch.sh -- the cp -a of the klippy tree,
the chelper.tar wrapped around a .so the package already contained, the rm -rf
of the stock extras/ and kinematics/, and the ff_*.py copy. The software
component now carries firmwareExe, start.sh and run.sh and nothing else of
ours, which is the phase 7 end state in docs/notes/80-s6-migration.md.

It also lets pkgs/klipper declare the dependencies it has had since the
interpreter moved. cffi and greenlet reached the printer only through a
hardcoded list in patch.sh, so every build had them and no `opkg install
anvil-klipper` did; pyserial and jinja2 came only as anvil-moonraker's
Depends, so a BUILD_MOONRAKER=0 build shipped a klippy that dies the first
time it opens an MCU. Both are real Depends now and MOD_ROOTS drops to ten.

Also repointed: run-append.sh's __pycache__ sweep, which was clearing the tree
nothing imports while klippy writes bytecode into the one it does -- and those
runtime-created directories are in no .install-manifest, so the manifest
delete walks past them.

verify.sh reads the klippy tree and c_helper.so out of the payload (extracting
./klipper alone), and gains a check that the component does NOT carry a klippy
tree. New replica tests assert the service execs the payload tree under
$FF_PYTHON and that cffi/greenlet/serial/jinja2 import on it.

Comments in anvil-env.sh, verify.sh and pkgs/klipper/pkg.conf that still said
klippy runs on FlashForge's 3.8.2 are corrected; that stopped being true when
FF_PYTHON moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S

bin/patch.sh

@@ -56,14 +56,20 @@ say "payload: installing the feed with the printer's own opkg"
 # flag restated in this file. A missing DEPENDENCY is still an error, and opkg
 # raises it.
 #
-# The four loose python packages are Recommends, which opkg has no field for:
-# pillow and preprocess-cancellation are Moonraker's thumbnail path
-# (pkgs/moonraker argues them out of Depends), greenlet and cffi are klippy's,
-# which pkgs/klipper cannot declare while klippy still runs under
-# FlashForge's interpreter.
+# The two loose python packages are Recommends, which opkg has no field for:
+# pillow and preprocess-cancellation are Moonraker's thumbnail path, and
+# pkgs/moonraker argues them out of Depends because a Moonraker without them
+# serves every other endpoint.
+#
+# GREENLET AND CFFI WERE HERE TOO, for klippy, with a comment saying
+# pkgs/klipper could not declare them "while klippy still runs under
+# FlashForge's interpreter". It does not: the klipper service execs
+# $FF_PYTHON, so they are ordinary Depends of anvil-klipper now and this list
+# does not name them. That is the difference that matters -- listed here they
+# were installed by every build and by no `opkg install anvil-klipper`, which
+# is the one command that has to work on a printer.
 MOD_ROOTS="anvil-core anvil-opkg anvil-s6-rc anvil-klipper
            anvil-moonraker anvil-python-pillow anvil-python-preprocess-cancellation
-           anvil-python-greenlet anvil-python-cffi
            anvil-mainsail anvil-helixscreen anvil-busybox"
 
 # Named, not versioned: opkg reads the index and picks. The one version that
@@ -91,95 +97,44 @@ done
 say "payload: $(grep -c '^Package:' "$PAYLOAD_DIR/var/lib/opkg/status") packages installed (both chamber configs; the printer picks)"
 
 # ---------------------------------------------------------------- 1. Klipper
-# THE BUILD LIVES IN pkgs/klipper, and this section copies what it produced --
-# the arrangement sections 3, 4, 5 and 5d have had since each became a recipe,
-# and for the same reason: the .ipk and the tarball have to contain the same
-# klippy tree and the same c_helper.so, or they are two different Klippers
-# wearing one version number and no test can tell which a printer got.
+# NOTHING IS STAGED HERE ANY MORE. anvil-klipper installs the whole klippy
+# tree at $MODDIR/klipper/klippy and the klipper s6-rc service execs it there,
+# on our own $FF_PYTHON -- so the package is not merely the source of the
+# printer's Klipper, it IS the printer's Klipper.
 #
-# THERE IS NO STOCK PATH. BUILD_KLIPPER=stock kept FlashForge's v0.12 tree and
-# it is gone -- everything else the mod ships is written against the fork. The
-# flag's own history is why it went rather than gaining another guard: a
-# "fork" build silently falling back to stock is exactly what shipped as
-# v20260824-nova-kakhovka, where the 0.12-era overlay half-overwrote the v0.13
-# tree and klippy died at connect with "expects 4 arguments, got 3" -- a v0.12
-# extruder.py calling the v0.13 chelper cdef.
+# WHAT THIS SECTION USED TO BE, because the shape is worth not rebuilding: a
+# `cp -a` of the payload's klippy tree into the software component, a
+# chelper.tar wrapped around a c_helper.so the package already contained, an
+# `rm -rf` of the stock extras/ and kinematics/ the fork replaces, and a copy
+# of the ff_*.py toolchanger extras beside them. All four existed for one
+# reason -- klippy was started from /usr/prog/klipper/klippy by FlashForge's
+# klipperDaemon, so the tree had to be on the firmware partition, which meant
+# shipping a second copy of it in the software component and hoping the two
+# agreed. They are one copy now. See etc/s6-rc/source/klipper/run.
 #
-# THE TREE COMES OUT OF THE PAYLOAD, which is to say out of the installed
-# anvil-klipper package, not out of the recipe's build directory. That was the
-# last place this script reached into work/pkg, and removing it is what makes
-# "the tarball and the .ipk contain the same klippy" true by construction
-# rather than by two copies of one cp -a agreeing.
-_kl="$PAYLOAD_DIR/klipper"
-[ -d "$_kl/klippy" ] || pkg_die \
+# WHAT IS LEFT AT /usr/prog/klipper: FlashForge's own stock klippy, untouched
+# and unread. A stock flash restores it, which also replaces firmwareExe and
+# takes the mod down whatever else survived -- so it was never the fallback it
+# looked like. klipper_pri.sh and klipperDaemon are still read there: the
+# first is FlashForge's SCHED_FIFO helper, the second is our shim, and the
+# service greps KLIPPER_NICENESS out of it.
+#
+# THE ASSERTION STAYS. It is the whole of what this section still does, and it
+# is cheap: an anvil-klipper that did not install is a printer with no Klipper
+# at all now, with no component copy to mask it.
+[ -d "$PAYLOAD_DIR/klipper/klippy" ] || pkg_die \
     "the payload has no $MODDIR/klipper/klippy -- anvil-klipper did not install"
-
-# THE FORK GOES TO THE SOFTWARE COMPONENT, not to $MODDIR, and that is why
-# this is a copy rather than a line in the payload loop below. klippy is
-# started by the stock /usr/prog/klipper/klipperDaemon, so it has to be
-# where that binary looks. The package installs the same tree under
-# $MODDIR/klipper, which is inert until phase 7 of
-# docs/notes/80-s6-migration.md moves Klipper there and this mod starts it
-# itself; see pkgs/klipper/pkg.conf.
-#
-# Stock ships only a handful of klippy files as an overlay; the fork is a
-# different Klipper generation, so the WHOLE tree replaces it.
-rm -rf "$SOFTWARE_DIR/klipper/klippy"
-mkdir -p "$SOFTWARE_DIR/klipper"
-cp -a "$_kl/klippy" "$SOFTWARE_DIR/klipper/klippy"
-
-# chelper.tar IS THE STOCK INSTALLER'S VEHICLE, not a second build. The
-# software component's run.sh ends with
-# `tar -xf $WORK_DIR/klipper/chelper.tar -C /usr/prog/klipper/klippy/`
-# -- read out of a real stock package, so a package without it
-# extracts nothing over the .so already inside klippy/ -- harmless today,
-# and a missing file the moment FlashForge's installer stops copying
-# klippy/ wholesale. bin/verify.sh fails a fork package that lacks it.
-# Packing it here rather than in the recipe keeps it out of the .ipk,
-# where a tarball of a file the package already contains would be 30KB of
-# the same object under a second name.
-rm -rf work/.chelper
-mkdir -p work/.chelper/chelper
-cp -f "$_kl/klippy/chelper/c_helper.so" work/.chelper/chelper/c_helper.so
-tar -cf "$SOFTWARE_DIR/klipper/chelper.tar" -C work/.chelper chelper
-rm -rf work/.chelper
-
-# klippy/ now contains the fork's own extras+kinematics, so the stock
-# overlay dirs would only re-inject 0.12-era files on top. Drop them.
-# extras/ is recreated empty because section 2 puts ff_*.py in it.
-rm -rf "$SOFTWARE_DIR/klipper/extras" "$SOFTWARE_DIR/klipper/kinematics"
-mkdir -p "$SOFTWARE_DIR/klipper/extras"
-say "Klipper: fork tree from pinned commit ${KLIPPER_VERSION:0:8}"
+[ -f "$PAYLOAD_DIR/klipper/klippy/chelper/c_helper.so" ] || pkg_die \
+    "the payload has no $MODDIR/klipper/klippy/chelper/c_helper.so -- klippy cannot connect to an MCU without it"
+say "Klipper: fork tree from pinned commit ${KLIPPER_VERSION:0:8} (payload only)"
 
 # ----------------------------------------------------------- 2. Toolchanger
-# The configs are three packages, installed by section 0: anvil-klipper-config
-# for the ff-*.cfg and one of the two model packages for printer.chamber.cfg.
-# What is left here is the residue that cannot be a package -- klippy's extras
-# and printer.base.cfg go to /usr/prog, beside FlashForge's own Klipper tree,
-# and every path in a package of ours lands under $MODDIR.
-if [ "${BUILD_TOOLCHANGE:-1}" = "1" ]; then
-    say "Toolchange: ff_*.py + configs"
-    # OUT OF THE PAYLOAD, like section 1's klippy tree and for the same
-    # reason: anvil-klipper carries these now, so copying them from the
-    # recipe directory would be a second source for a file a package holds.
-    _ffx="$PAYLOAD_DIR/klipper/klippy/extras"
-    mkdir -p "$SOFTWARE_DIR/klipper/extras"
-    # Named by the recipe, not globbed out of the payload: the fork ships an
-    # ff_eddy.py of its own and `ff_*.py` cannot tell it from ours. It rides
-    # the klippy tree section 1 copied either way, so sweeping it up here
-    # would put the same file in the component twice.
-    for _f in pkgs/klipper/payload/klipper/klippy/extras/ff_*.py; do
-        cp -f "$_ffx/$(basename "$_f")" "$SOFTWARE_DIR/klipper/extras/"
-    done
-
-    # NO .cfg IS WRITTEN HERE: the software component's klipper/config is left
-    # exactly as FlashForge shipped it. anvil-klipper-config carries every
-    # config the mod owns and anvil-link-prog.sh symlinks them into
-    # /usr/data/config, which is where the stock run.sh puts that directory
-    # anyway. The ff_*.py above are what is left that cannot be a package.
-else
-    skip "Toolchange"
-fi
+# GONE, with section 1. The ff_*.py extras ride the klippy tree anvil-klipper
+# installs, and the ff-*.cfg are anvil-klipper-config's -- anvil-link-prog.sh
+# symlinks them into /usr/data/config, which is where printer.cfg looks. There
+# is no longer any part of the toolchanger that cannot be a package, so
+# BUILD_TOOLCHANGE is answered entirely by pkgs/klipper-config's PKG_WHEN and
+# this script does not restate it.
 
 # ------------------------------------------------------- 3. the user's seams
 # What is in the payload and in NO package, because being a package member

bin/verify.sh

@@ -82,22 +82,37 @@ if [ -f "$WORK/sw/app_startup.sh" ]; then
 fi
 
 # 7 ------------------------------------------------- MIPS ABI of any binaries
+# THE KLIPPY TREE IS IN THE PAYLOAD, NOT IN THE SOFTWARE COMPONENT. It used to
+# be checked at $WORK/sw/klipper -- a copy bin/patch.sh staged for FlashForge's
+# run.sh to place on the firmware partition, back when klipperDaemon started
+# klippy from there. The klipper s6-rc service execs $MODDIR/klipper/klippy on
+# $FF_PYTHON now, so the component copy and the chelper.tar that carried its
+# .so are both gone and this is where the one remaining tree lives.
+#
 # A package that ships no fork is a broken package, not an optional check: the
 # v20260824 release passed here precisely because everything below was
 # conditional on files that the broken build simply did not contain. It is
 # unconditional now for a second reason as well -- BUILD_KLIPPER is gone, so
 # there is no configuration in which a package legitimately has no klippy.
-if [ -f "$WORK/sw/klipper/klippy/chelper/__init__.py" ]; then
-    ok "package carries the fork klippy tree"
-else
-    bad "no klippy tree in the package -- this is the stock-overlay build that shipped as v20260824"
+# ./klipper alone, not the whole payload: the rest is Mainsail, HelixScreen
+# and a 30MB interpreter, and section 8 answers everything it needs from a
+# listing. The paths carry a leading ./ because bin/pack.sh tars `.` from
+# inside $PAYLOAD_DIR.
+PAYLOAD_TAR=$(ls -1 "$WORK"/anvil.tar.xz 2>/dev/null | head -n1)
+if [ -n "$PAYLOAD_TAR" ]; then
+    mkdir -p "$WORK/pl"
+    xz -dc "$PAYLOAD_TAR" 2>/dev/null | tar -xf - -C "$WORK/pl" ./klipper 2>/dev/null \
+        || tar -xf "$PAYLOAD_TAR" -C "$WORK/pl" ./klipper 2>/dev/null
 fi
-[ -f "$WORK/sw/klipper/chelper.tar" ] \
-    || bad "no chelper.tar in the package"
-if [ -f "$WORK/sw/klipper/chelper.tar" ]; then
-    mkdir -p "$WORK/ch" && tar -xf "$WORK/sw/klipper/chelper.tar" -C "$WORK/ch"
-    CHELPER=$(find "$WORK/ch" -name 'c_helper.so' | head -n1)
-    if [ -n "$CHELPER" ] && ! head -c 4 "$CHELPER" | grep -q ELF; then
+if [ -f "$WORK/pl/klipper/klippy/chelper/__init__.py" ]; then
+    ok "payload carries the fork klippy tree"
+else
+    bad "no klippy tree in the payload -- anvil-klipper did not install; this is the stock-overlay build that shipped as v20260824"
+fi
+CHELPER="$WORK/pl/klipper/klippy/chelper/c_helper.so"
+[ -f "$CHELPER" ] || bad "no c_helper.so in the payload klippy tree -- klippy cannot talk to an MCU"
+if [ -f "$CHELPER" ]; then
+    if ! head -c 4 "$CHELPER" | grep -q ELF; then
         warn "c_helper.so is not an ELF (synthetic fixture) -- ABI check skipped"
         CHELPER=""
     fi
@@ -110,15 +125,19 @@ if [ -f "$WORK/sw/klipper/chelper.tar" ]; then
         # The failure mode that reached a printer: a .so from older sources
         # than the klippy tree beside it. cffi resolves lazily, so only a
         # symbol-table check catches it before the machine does.
-        if [ -f "$WORK/sw/klipper/klippy/chelper/__init__.py" ]; then
-            if python3 "$ROOT/test/test-chelper.py" "$WORK/sw/klipper" >/dev/null 2>&1; then
-                ok "c_helper.so exports everything the shipped klippy declares"
-            else
-                bad "c_helper.so does not match the shipped klippy tree (stale build)"
-            fi
+        if python3 "$ROOT/test/test-chelper.py" "$WORK/pl/klipper" >/dev/null 2>&1; then
+            ok "c_helper.so exports everything the shipped klippy declares"
+        else
+            bad "c_helper.so does not match the shipped klippy tree (stale build)"
         fi
     fi
 fi
+# THE COMPONENT MUST NOT CARRY ONE. Two klippy trees under one version number
+# is the arrangement this release removed, and a stray one would be the copy a
+# stock flash leaves running.
+[ -e "$WORK/sw/klipper/klippy" ] \
+    && bad "the software component still carries a klippy tree -- bin/patch.sh section 1 is staging one again" \
+    || ok "no klippy tree in the software component (the payload is the only one)"
 
 # 8 ------------------------------------------------------------ mod payload
 if [ -f "$WORK/anvil.tar.xz" ]; then
@@ -188,11 +207,11 @@ if [ -f "$WORK/anvil.tar.xz" ]; then
                                           || bad "no s6 in the payload -- bin/patch.sh did not stage the cross-build"
     grep -q 'libexec/s6-ftrigrd' <<<"$LIST" && ok "s6-ftrigrd present in libexec (the waiting verbs can spawn it)" \
                                           || bad "no libexec/s6-ftrigrd -- s6-svwait and s6-svc -w will fail on the printer"
-    # CPython 3.13. anvil-env.sh points FF_PYTHON here -- Moonraker,
+    # CPython 3.13. anvil-env.sh points FF_PYTHON here -- Moonraker, klippy,
     # ff-startup.py, ffscreen.py and ff_mcu_bringup.py all run on it -- so a
-    # package missing it breaks the boot, not just a future release. klippy is
-    # not among FF_PYTHON's callers: it stays on FlashForge's 3.8.2, started by
-    # the klipper s6-rc service independently.
+    # package missing it breaks the boot, not just a future release. klippy IS
+    # among FF_PYTHON's callers: the klipper s6-rc service execs $FF_PYTHON
+    # against $MODDIR/klipper/klippy, and anvil-klipper declares it.
     #
     # Two checks, not one, and the second is the one worth having. The
     # interpreter can be present and perfectly runnable while _sqlite3 is
@@ -213,11 +232,9 @@ if [ -f "$WORK/anvil.tar.xz" ]; then
     # symlink CPython installs beside it. $MODDIR/bin is prepended to PATH by
     # anvil-env.sh (s6 needs that), so shipping one would silently put our
     # interpreter ahead of FlashForge's for every process that says `python3`
-    # -- an accidental switch, on a printer whose klippy still needs 3.8's
-    # site-packages (it is started separately, by FlashForge's own
-    # start.sh, and never goes through FF_PYTHON). Reads the way round it
-    # does for the same reason the dropbear check below does: its presence is
-    # the bug.
+    # -- an accidental switch, and the mod's own callers do not need it: every
+    # one of them names $FF_PYTHON. Reads the way round it does for the same
+    # reason the dropbear check below does: its presence is the bug.
     grep -qE '(^|/)bin/python3$' <<<"$LIST" \
         && bad "payload ships bin/python3 -- it would shadow FlashForge's interpreter on PATH" \
         || ok "no bin/python3 symlink (nothing shadows FlashForge's python3 on PATH)"

docs/building.md

@@ -321,7 +321,8 @@ test/           28 host-side unit tests over our own Python, and the two
   ffcfg.py        Klipper config parsing, as klippy's own parser sees it
   integration/    test_startup.py, test_tool_transform.py, test_ffscreen.py,
                   test_chamber.py, test_gcode.py -- see docs/testing.md
-  test-chelper.py in neither suite: bin/patch.sh and bin/verify.sh call it
+  test-chelper.py in neither suite: pkgs/klipper/build.sh and
+                  bin/verify.sh call it
 tools/replica/  THE REPLICA, which is a build tool as much as a test one --
                 bin/patch.sh assembles the payload inside it
   printer/        the machine: binfmt, the mount layout, its two Dockerfiles

docs/notes/20-klipper-fork.md

@@ -61,8 +61,13 @@ cdef and the caller both live in the tree; a stale .so shows up as a missing
 symbol (`AttributeError`) instead, since cffi resolves symbols lazily.
 There is now no fork build with no fork tree to refuse: `pkgs/klipper` has one
 source and `pkg_unpack` fails on a missing tarball. `verify.sh` still fails a
-fork package that lacks the klippy tree or `chelper.tar`, and the compile-
-from-shipped-sources rule makes a stale .so unrepresentable in a release.
+fork package that lacks the klippy tree or its `c_helper.so`, and the
+compile-from-shipped-sources rule makes a stale .so unrepresentable in a
+release. Both are read out of the PAYLOAD now: the klippy tree moved to
+`$MODDIR/klipper/klippy` — the `klipper` s6-rc service execs it there, on our
+own CPython 3.13 — so the software-component copy and the `chelper.tar` that
+carried its .so onto the firmware partition are both gone. One tree, in one
+package.
 
 ## Custom commands (by area)
 

docs/notes/80-s6-migration.md

@@ -318,14 +318,33 @@ has to catch that class of failure.
 
 ## Phase 7 -- own Klipper
 
-**Now reachable: phase 6 is done.** The interpreter it needs is `pkgs/3rdparty/python`,
-built with the Ingenic glibc toolchain -- the requirement the paragraph above
-calls fatal if got wrong, since klippy `dlopen`s a glibc `c_helper.so`.
+**DONE.** The interpreter it needed is `pkgs/3rdparty/python`, built with the
+Ingenic glibc toolchain -- the requirement the paragraph above calls fatal if
+got wrong, since klippy `dlopen`s a glibc `c_helper.so`.
 
-Ship the klippy tree into `$MODDIR` instead of
-handing it to FlashForge's `run.sh`; replace `klipperDaemon` with an s6 `run`
-script (the command line is quoted above); turn `S70klipper`'s retry loop into
-an s6 restart policy plus readiness. `checkEboard`, `libmcu-bare.bin` and
+It landed in two halves and the gap between them is the part worth recording.
+The `klipperDaemon` replacement, the s6 `run` script and the move onto
+`$FF_PYTHON` went in together; the klippy TREE stayed at
+`/usr/prog/klipper/klippy` for a while afterwards, staged there by
+`bin/patch.sh` out of the very `anvil-klipper` package that also installed it
+under `$MODDIR`. So the printer ran our Klipper, on our Python, out of a copy
+on the firmware partition that a stock flash could overwrite -- and because
+nothing declared it, `anvil-klipper` carried no dependency on the interpreter
+it had already moved to. `KLIPPY=$MODDIR/klipper/klippy/klippy.py` closed
+both: it deleted sections 1 and 2 of `bin/patch.sh` (~85 lines, the software
+component's whole `klipper/` directory and the `chelper.tar` that fed stock
+`run.sh`) and let `pkgs/klipper` declare `anvil-python`, `-cffi`, `-greenlet`,
+`-pyserial` and `-jinja2` for real. Until then `cffi` and `greenlet` reached
+the printer only via a hardcoded list in `patch.sh`, and `pyserial` and
+`jinja2` only as `anvil-moonraker`'s Depends -- so a `BUILD_MOONRAKER=0` build
+shipped a klippy that dies the first time it opens an MCU.
+
+**The lesson, stated once:** moving what a program RUNS ON without moving what
+it RUNS leaves the dependency graph lying, and nothing fails until a
+configuration nobody builds gets built.
+
+Still open: `S70klipper`'s retry loop is not yet an s6 restart policy plus
+readiness. `checkEboard`, `libmcu-bare.bin` and
 `cmd_mcu` stay where they are -- they are version-matched to the firmware and
 reading them from the firmware partition is correct.
 

docs/testing.md

@@ -67,7 +67,7 @@ whose subject had become the fake it was built on, or whose failure the printer
 reports loudly the first time. What is kept is the mistakes that RUN
 PERFECTLY and are wrong.
 
-`test/test-chelper.py` sits outside both suites: `bin/patch.sh` and
+`test/test-chelper.py` sits outside both suites: `pkgs/klipper/build.sh` and
 `bin/verify.sh` call it directly at build time.
 
 ## The replica is a tool, not a test

installer/run-append.sh

@@ -345,6 +345,17 @@ sync
 
 # Stale bytecode from the previous Klipper generation is a silent import-time
 # landmine; the stock run.sh only clears four of the __pycache__ dirs.
+#
+# $MODDIR, NOT /usr/prog: the klipper s6-rc service execs
+# $MODDIR/klipper/klippy, so that is the tree whose .pyc files get imported.
+# It also needs the sweep MORE than /usr/prog ever did -- klippy WRITES those
+# directories at runtime on a writable /usr/data, and a path created after
+# install is in no .install-manifest, so the manifest delete above walks past
+# it and leaves last release's bytecode beside this release's source.
+find $MODDIR/klipper/klippy -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null
+# The stock tree is swept too, and only because it costs one line: nothing
+# imports it any more, but a machine that has been through several releases
+# has bytecode there from when something did.
 find /usr/prog/klipper/klippy -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null
 sync
 

pkgs/anvil-core/payload/anvil-env.sh

@@ -87,18 +87,24 @@ export LD_LIBRARY_PATH
 #
 # WHO ELSE THIS MOVES. grep FF_PYTHON payload/ bin/ before touching this line
 # again -- the answer changes as callers are added. Today it is Moonraker,
-# ff-startup.py, ffscreen.py and ff_mcu_bringup.py (the MCU bootloader
-# handshake start.sh runs before Klipper). The last three are stdlib-only
+# KLIPPY, ff-startup.py, ffscreen.py and ff_mcu_bringup.py (the MCU bootloader
+# handshake that runs before Klipper). Three of those five are stdlib-only
 # (os/sys/termios/time/argparse/json/subprocess/urllib -- checked against
-# CPython's own removed-in-3.13 list), so the only one with any C-extension
-# surface is Moonraker, and that is the one measured above.
+# CPython's own removed-in-3.13 list); the two with C-extension surface are
+# Moonraker, measured above, and klippy.
 #
-# WHO THIS DOES NOT MOVE. Klipper is not on this list. It is started by
-# FlashForge's own /usr/prog/klipper/start.sh, hardcoded to
-# /usr/prog/Python-3.8.2/bin/python3, independently of FF_PYTHON -- see
-# init.d/S70klipper's own header. klippy's numpy gap
-# (extras/stepper_resonance_tester.py) is therefore not this switch's problem;
-# it stays open as a separate, smaller item.
+# KLIPPY IS ON THIS LIST NOW. This block used to say it was not -- that
+# FlashForge's own start.sh started it, hardcoded to
+# /usr/prog/Python-3.8.2/bin/python3. Both halves are gone:
+# etc/s6-rc/source/klipper/run execs $FF_PYTHON against $MODDIR/klipper/klippy,
+# and prog/start.sh is a `s6-rc -u change klipper` and nothing else. klippy
+# reaches c_helper.so through _cffi_backend, which is the reason this
+# interpreter had to be a glibc build; anvil-klipper declares cffi, greenlet,
+# pyserial and jinja2 against it.
+#
+# klippy's numpy gap (extras/stepper_resonance_tester.py) IS this switch's
+# problem now, and stays open as a separate, smaller item: the module guards
+# its own import, so the printer runs without it and loses resonance testing.
 #
 # FlashForge's tree is not touched either way: nothing here writes to
 # /usr/prog, and everything of ours lives under /usr/data/anvil like every

pkgs/anvil-core/payload/etc/s6-rc/source/klipper/run

@@ -11,18 +11,32 @@
 # here as well would hand the same serial ports to two processes at once.
 MODDIR=/usr/data/anvil
 
-# klippy is FlashForge's Python 3.8.2 on the firmware partition, linked against
-# libraries in four directories nothing on the base rootfs knows about.
-# s6-supervise hands this script the scanner's environment, which is already
-# correct on the boot path -- but a service must not work only because the
-# shell that started the tree happened to be set up right.
+# anvil-env.sh names the interpreter ($FF_PYTHON) and puts the mod's four
+# library directories on LD_LIBRARY_PATH. s6-supervise hands this script the
+# scanner's environment, which is already correct on the boot path -- but a
+# service must not work only because the shell that started the tree happened
+# to be set up right.
 if [ -f $MODDIR/anvil-env.sh ]; then
     . $MODDIR/anvil-env.sh
 else
     echo "klipper/run: !! no $MODDIR/anvil-env.sh -- klippy will not find its libraries"
 fi
 
-KLIPPY=/usr/prog/klipper/klippy/klippy.py
+# BOTH HALVES ARE OURS AND BOTH ARE UNDER $MODDIR: the klippy tree anvil-klipper
+# installs, on the CPython 3.13 anvil-python installs. There is no /usr/prog in
+# this command line any more.
+#
+# The tree used to be read from /usr/prog/klipper/klippy -- a copy bin/patch.sh
+# staged into the software component out of this very package, so that
+# FlashForge's run.sh would place it on the firmware partition. Two copies of
+# one tree, and the one that ran was the one a stock flash could overwrite.
+# anvil-klipper is the only source now, which is also what lets this package
+# declare the interpreter it actually needs: see pkgs/klipper/pkg.conf.
+#
+# THE INTERPRETER IS NOT FLASHFORGE'S 3.8.2 and has not been since $FF_PYTHON
+# moved to 3.13. klippy reaches c_helper.so through _cffi_backend, which is why
+# pkgs/3rdparty/python is a glibc build and not a musl one.
+KLIPPY=$MODDIR/klipper/klippy/klippy.py
 PRINTER_CFG=/usr/data/config/printer.cfg
 PRINTER_LOG=/usr/data/logs/printer.log
 UDS=/tmp/uds

pkgs/klipper/pkg.conf

@@ -23,20 +23,36 @@ PKG_RELEASE=1
 PKG_SECTION=base
 PKG_EXCLUDE=".version"
 
-# NO DEPENDS ON THE INTERPRETER, and the reason is where klippy runs rather
-# than what it imports. The stock /usr/prog/klipper/klipperDaemon starts
-# klippy under FlashForge's own interpreter -- bin/patch.sh copies this tree
-# into the SOFTWARE component, not into $MODDIR -- so naming anvil-python here
-# would declare a dependency the running system does not have. It becomes a
-# real Depends when phase 7 of docs/notes/80-s6-migration.md moves Klipper
-# under $MODDIR and this mod starts klippy itself.
+# THE INTERPRETER IS A DEPENDS NOW. This block used to say the opposite --
+# "the reason is where klippy runs rather than what it imports", the stock
+# klipperDaemon starting klippy under FlashForge's 3.8.2 while bin/patch.sh
+# copied the tree into the software component -- and it named phase 7 of
+# docs/notes/80-s6-migration.md as the change that would make it false. That
+# change has happened: etc/s6-rc/source/klipper/run execs $FF_PYTHON against
+# $MODDIR/klipper/klippy, so klippy is on our CPython 3.13 and the tree it
+# runs is the one THIS package installs.
 #
-# anvil-klipper-config is not optional in that way: printer.base.cfg is the
-# file printer.cfg includes and every other config hangs off it, so a klippy
-# tree without it is a Klipper that cannot start. Naming it in bin/patch.sh's
-# MOD_ROOTS instead would answer the dependency in the one place a printer's
-# own `opkg install anvil-klipper` never reads.
-PKG_DEPENDS="anvil-klipper-config"
+# THE LIST IS scripts/klippy-requirements.txt MINUS WHAT IS NOT REACHABLE
+# HERE. python-can is not packaged (no CAN transport on this machine) and
+# numpy is not either, which costs extras/stepper_resonance_tester.py and
+# nothing else -- both are import-guarded by the modules that want them.
+# markupsafe and setuptools are not named because anvil-python-jinja2 declares
+# the first and nothing here needs the second at import.
+#
+# WHAT THIS FIXES, and it is not bookkeeping. cffi and greenlet were installed
+# by a hardcoded list in bin/patch.sh, so every build had them and no `opkg
+# install anvil-klipper` did. pyserial and jinja2 were worse: they arrived
+# only as anvil-moonraker's Depends, so a BUILD_MOONRAKER=0 build shipped a
+# klippy that imports serialhdl and dies the first time it opens an MCU.
+#
+# anvil-klipper-config is not optional either: printer.base.cfg is the file
+# printer.cfg includes and every other config hangs off it, so a klippy tree
+# without it is a Klipper that cannot start. Naming any of these in
+# bin/patch.sh's MOD_ROOTS instead would answer the dependency in the one
+# place a printer's own `opkg install anvil-klipper` never reads.
+PKG_DEPENDS="anvil-klipper-config, anvil-python, anvil-python-cffi,
+             anvil-python-greenlet, anvil-python-pyserial,
+             anvil-python-jinja2"
 PKG_BUILD_DEPENDS=""
 
 # NO PKG_WHEN. The fork is the only Klipper, and nothing here is optional --

qa/replica/test_install.py

@@ -58,7 +58,11 @@ FE = "/usr/prog/PROGRAM/software/firmwareExe"
 APP = "/usr/prog/app_startup.sh"
 SOURCE = MODDIR + "/etc/s6-rc/source"
 DB = MODDIR + "/etc/s6-rc/compiled/current"
-CHELPER = "/usr/prog/klipper/klippy/chelper/c_helper.so"
+# The klipper s6-rc service execs $FF_PYTHON against $MODDIR/klipper/klippy,
+# so this is the .so that actually gets dlopened. /usr/prog/klipper/klippy
+# still holds FlashForge's stock tree; nothing reads it.
+KLIPPY_DIR = MODDIR + "/klipper/klippy"
+CHELPER = KLIPPY_DIR + "/chelper/c_helper.so"
 INSTALL_LOG = "/usr/data/anvil-install.log"
 
 
@@ -194,8 +198,45 @@ def test_klipper_depends_on_the_mcu_bringup(box):
 # --------------------------------------------------------------- klippy itself
 
 def test_klippy_is_present(box):
-    assert box.file("/usr/prog/klipper/klippy/klippy.py").exists, (
-        "klippy.py missing -- there is nothing to start")
+    """In the payload, which is the only copy: anvil-klipper installs it and
+    the s6 service execs it there. There is no component copy to fall back
+    on any more, so this failing means the printer has no Klipper at all."""
+    assert box.file(KLIPPY_DIR + "/klippy.py").exists, (
+        "%s/klippy.py missing -- anvil-klipper did not install and there is "
+        "nothing to start" % KLIPPY_DIR)
+
+
+def test_the_klipper_service_runs_the_payload_tree_on_our_python(box):
+    """The regression that would silently restore the old arrangement: a run
+    script pointing back at /usr/prog gets its klippy from a stock flash, not
+    from the package, and no other test here would notice."""
+    # The compiled servicedir is what s6-supervise actually execs; the source
+    # tree is the fallback so this still says something useful on a machine
+    # where the database did not compile.
+    body = box.file(DB + "/servicedirs/klipper/run").text \
+        or box.file(SOURCE + "/klipper/run").text
+    assert body, "no klipper run script in either the database or the source tree"
+    assert "/usr/prog/klipper/klippy" not in body, (
+        "the klipper service still execs klippy out of /usr/prog -- that tree "
+        "is FlashForge's stock 0.12 and a stock flash owns it")
+    # The run script spells it $MODDIR, unexpanded, so match the tail rather
+    # than the absolute path this file computes.
+    assert "klipper/klippy/klippy.py" in body, (
+        "the klipper service does not exec %s/klippy.py" % KLIPPY_DIR)
+    assert "$FF_PYTHON" in body, (
+        "the klipper service does not run klippy under $FF_PYTHON")
+
+
+def test_klippy_imports_resolve_on_our_python(box):
+    """cffi, greenlet, pyserial and jinja2 are anvil-klipper's Depends. They
+    used to arrive as anvil-moonraker's, or from a hardcoded list in
+    bin/patch.sh -- so a BUILD_MOONRAKER=0 build shipped a klippy that dies
+    the first time it opens an MCU."""
+    for mod in ("cffi", "greenlet", "serial", "jinja2"):
+        r = box.sh(". %s/anvil-env.sh; $FF_PYTHON -c 'import %s' 2>&1"
+                   % (MODDIR, mod))
+        assert r.ok, (
+            "klippy's %s does not import on FF_PYTHON: %s" % (mod, r.text))
 
 
 def _bytes_at(box, path, skip, count):