mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
e3d1bdcdf7788bc5fef967d9d1428aa2db123f03
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr ShyshatskyiandClaude Opus 5
|
e3d1bdcdf7 |
The payload is the feed installed, not the build trees copied
bin/patch.sh ran every recipe and then cp -a'd its build tree into
work/modpayload. Eight sections of it, each a second description of what the
.ipk that recipe produces already contains -- the same duplication the recipes
themselves removed one level down, one level further up.
It now installs the feed. pkg_buildopkg's x86-64 opkg gets one invocation with
the whole selected set and an --offline-root, and what lands there is the
payload. "What does this release install?" is answerable with
`opkg list-installed` against the payload itself, and the tarball and the
package cannot describe different bytes because the tarball is made of the
package.
THE SET COMES FROM pkg_recipes, which is already PKG_WHEN-gated -- so
BUILD_HELIX, BUILD_MAINSAIL, BUILD_MOONRAKER, BUILD_TOOLCHANGE and
BUILD_KLIPPER are read in the pkg.conf that owns each and nowhere else,
instead of being restated by an `if` here that could drift from it.
-dev IS FILTERED BY PACKAGE NAME, and the first attempt got this wrong in a
way worth recording. execline, s6 and CPython ship both halves and pkg_ipk
names the runtime one; zlib, openssl, sqlite, expat, libffi, xz, bzip2,
libarchive and skalibs ship ONLY a dev package and say so by setting
PKG_NAME=anvil-<x>-dev outright. Asking "does this recipe have a runtime half"
installed all nine: 40 packages and 2165 ELF objects where there should be 31
and 154.
--force-postinstall, because opkg skips configuration under an offline root
and would leave every package Status: unpacked. Nothing runs opkg on the
printer after run-append.sh extracts the tarball -- the extraction IS the
install -- so a database saying otherwise would make the first real `opkg
upgrade` argue with the filesystem. It runs nothing today: no recipe defines a
maintainer script, and opkg 0.7.0 would not run one here anyway (it sets no
IPKG_INSTROOT; see libopkg/pkg.c:1339).
Installed-Time is normalised afterwards. opkg takes it from the clock and
honours SOURCE_DATE_EPOCH only for a man-page date, so two builds of one
commit differed by one line per package -- measured, then fixed, then measured
again.
Gone with the staging: the hand-written PKG_DEV_FILES prune (the dev half is
not installed, so bin/verify.sh's "no headers" check now passes because they
were never there), three partial ABI gates in favour of one over the finished
payload, and the last pkg_out call -- section 1 reads klippy out of the
installed anvil-klipper.
The payload gains anvil-klipper's tree at $MODDIR/klipper and bin/opkg. The
first is inert until phase 7 of docs/notes/80-s6-migration.md and is 2.4MB of
duplicate until then; the second is route 1 of the bootstrap in
docs/notes/85-packaging.md -- the first payload that can install its own
successor.
VERIFIED by building the payload from the parent commit and from this one, on
the synthetic stock fixture with the real vendored assets:
files removed 0
content changes among the 2778 shared files 1 (.install-manifest,
which lists the payload)
mode changes 0
symlink changes 0
added klipper/**, var/lib/opkg/** (63), bin/opkg, and the
www/mainsail/.version the previous commit stopped deleting
two builds in a row byte-identical
bin/verify.sh passes every check including the ship boundary; qa-static 255;
make test 9 gates, 0 failed; test-py 181.
test/run-tests.py builds the feed as the first step of the fixture lane, into
a PKG_FEED of its own -- sharing work/packages let the MOD_VER=ci build rename
every dated package and the prune delete the rest on the way past, measured as
44 becoming 43.
qa/static/test_ipk.py's share-one-build test is inverted rather than deleted:
the property was "one build, two vehicles" and is now "one vehicle", so what
it asserts is that nobody reintroduces the second path -- no recipe run from
patch.sh, no pkg_out, no work/pkg, and opkg rather than pkgs/ipk-install,
which stays the printer's hand-repair tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8cc92e8266 |
A wheel could only be built while three scratch trees happened to coexist
That is why bin/patch.sh section 5c was 825 lines and why a Pillow bump
rebuilt CPython. Cross-building an extension module needs the untrimmed
staging tree (INCLUDEPY points into it), the private sysroot of static C
libraries (pillow links zlib, cffi links libffi) and an x86-64 build-python
of the same version -- and all three existed only mid-section, so the wheels
had to be built there too, on a cache with two stamps that invalidated each
other.
Each of those three is now a durable thing: anvil-python-dev carries the
headers, pkg_deps fills each recipe's own sysroot from the feed, and
pkg_buildpython caches one build-python for all nineteen recipes. So the
coupling goes away with them -- `make packages PKG=python-pillow` builds
Pillow.
pkg/python -> anvil-python + anvil-python-dev
pkg/python-<name> x18 -> anvil-python-distro, anvil-python-pillow, ...
Three verbs in pkg/lib.sh, shared by all nineteen rather than one per awkward
project: pkg_buildpython provides the build-python, pkg_pytarget points its
sysconfig at the target headers in the sysroot, pkg_pywheel builds one wheel
and unpacks it into the staging site-packages. greenlet's gcc 7.2 designated
initializers, lmdb's LMDB_FORCE_CPYTHON and pillow's build_ext options are
recipe lines and a PKG_PY_SETUP_ARGS knob, not three more verbs.
Eighteen versions that were invisible are now in the feed index, anvil-moonraker
declares the thirteen packages it imports, and a printer that does not want
gcode thumbnails can leave Pillow uninstalled.
PY_STAMP, PYPKG_STAMP and PY_BUILD are gone: pkg_stamp derives the key from
PKG_BUILD_DEPENDS, and bin/fetch-assets.sh asks pkg_needs instead of
re-deriving a stamp bin/patch.sh also derived. Two of its four toolchain
conditions were that second spelling.
Measured:
* 40 packages; 959 site-packages files, none owned by two packages
* 12 mipsel extension modules -- the count the old gate demanded
* every Depends resolves inside the feed (24 declare one, 0 unsatisfiable)
* two cold builds byte-identical, with work/pkg, work/.py-host and the
unpacked toolchain deleted in between
* 233 static tests, 4 new: the PYPKG_LIST/recipe agreement both ways, no
version pinned twice, patch.sh builds every package, and no Depends
naming a package no recipe produces
Not verified here: the replica suite. It needs the stock FlashForge package,
which the packaging lane deliberately does without.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
99706bb5f7 |
s6 was legacy-NaN. The kernel this printer runs refuses that at exec().
The Bootlin swap two commits ago (musl.cc -> mips32el) fixed a
reachability problem and kept a correctness one: mips32el is plain
mips32r1, whose musl crt/libc is legacy-NaN by construction and cannot
be made to emit nan2008 output at all -- forcing -mnan=2008 on it fails
to LINK ("mixing -mnan=2008 module with previous -mnan=legacy modules").
So every s6 binary this repo has shipped read e_flags=0x1007, defended
in five separate places as correct on the theory that a supervisor
doing no floating point cannot care about NaN encoding.
True of the arithmetic. False of exec(): docs/how-it-works.md already
documented, for a different binary, that "the Ingenic X2000 kernel
returns ENOEXEC for legacy-NaN MIPS" -- the ABI check happens before a
single instruction of the program's own runs, which means it was never
about what s6 computes. None of the replica gates that "measured s6
working on the printer" ever caught this, because none of them ran on
the printer: qemu-mipsel-static's user-mode emulation does not enforce
the same ABI check a real kernel's binfmt loader does. A build that
passed case-supervisor.sh, case-services.sh, case-moonraker313-s6.sh and
a full release run was still shipping a supervisor that a real X2000
kernel may refuse to start at all.
THE FIX. Bootlin also publishes mips32r5el: MIPS Release 5 has no
legacy-NaN silicon to be compatible with, so its musl crt/libc is
nan2008 by construction, and its default codegen already targets
mips32r2. Verified outside the repo before touching anything: fetched
the tarball, cross-compiled skalibs+s6 with it, every binary read
nan2008/o32/mips32r2, and s6-svscan ran and supervised under
qemu-mipsel-static. Section 5b now applies the same gcc-wrapper
discipline 5c already uses for the Ingenic toolchain -- -EL -mnan=2008
-march=mips32r2 baked into the driver, not left to CFLAGS, with a probe
gate before building anything on top -- because "the toolchain's default
happens to be right" is exactly the kind of assumption this bug was.
mips_abi_gate moved earlier in the file so 5b can use it, and s6 is IN
that gate now rather than exempt from it -- exempt is how this shipped
wrong for as long as it did. The gate itself grew a real relaxation
alongside the real tightening: a static EXEC from this musl toolchain
legitimately carries EF_MIPS_PIC (musl's crt is unconditionally -fPIC;
no combination of -static/-no-pie/-fno-PIC removes it, measured) and so
reads the DYN flag value anyway, which is a property of the toolchain
and not a sign of anything wrong -- so the gate now accepts either
0x70001405 or 0x70001407 for any object, rather than picking one from
the ELF Type.
Also bumped S6_STAMP to include MUSL_TOOLCHAIN_FILE: without that, a
checkout with an already-built work/.s6 from before this commit would
read its own stale legacy-NaN tree as current and never rebuild it.
Full real build: "s6: 14 ELF objects are nan2008/o32/mips32r2 -- good".
case-supervisor.sh green on the replica with the new binaries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c870d8417d |
FF_PYTHON moves to our own 3.13, and CI stops rehearsing around it
The gate this switch was waiting on now exists and passes:
case-moonraker313-s6.sh drives Moonraker through the real boot path --
S40s6's scandir, S62moonraker, readiness gated on :7125 actually
listening, a kill -9 respawn, a stop that stays stopped -- on the
interpreter this commit ships. anvil-env.sh now points FF_PYTHON at
$MODDIR/bin/python3.13 and drops /usr/prog/libsodium/lib from
ANVIL_LIBS, since nothing left running under FF_PYTHON imports libnacl
from there any more (Moonraker carries its own libsodium; klippy is not
on this list at all -- it stays on FlashForge's 3.8.2, started
independently by /usr/prog/klipper/start.sh).
Three real bugs surfaced chasing this through the replica, none of them
about the switch itself:
* The Phase-1 manifest-based install can only delete files it once
tracked, so an init.d/S60web surviving from a release that predates
the manifest system would never be cleaned -- caught by
case-install.sh, which had never run against real stock packages
before today. run-append.sh now clears $MODDIR/init.d unconditionally
on every update: those scripts are 100% mod-owned and get executed,
so a diff against tracked history is not a strong enough guarantee.
* case-moonraker313-s6.sh's own readiness-ordering check flaked once:
its bind-time sampler polled at 1-second resolution while s6-svwait
is event-driven and returns within a fraction of a second, so
`date +%s` truncation could show readiness before the bind though the
real events were in order. Tightened to 0.2s.
* case-mcu-bringup.sh and case-boot-screen.sh source anvil-env.sh but
named their own directory variable $MOD, not $MODDIR -- harmless
while FF_PYTHON was a literal path, and silently wrong the moment it
started reading $MODDIR. Aliased.
Every other test/integration file touched here is the same story:
FF_PYTHON's callers were hardcoded to FlashForge's 3.8.2 throughout the
suite, and each one either now stages our interpreter as a tarball
(gates.py's mcu_bringup/boot_screen/moonraker, matching the pattern
already used for python()/moonraker313_s6) or asserts against the new
target. case-libpath.sh's premise changes with it: the three ANVIL_LIBS
entries that remain are read as what klippy's process tree still needs,
not what every mod process needs -- Moonraker maps zero /usr/prog
libraries now, measured.
payload/anvil-service.sh's SVC_S6_SCANDIR line tripped test_paths.py's
absolute-path regex, which reads a closing brace immediately followed by
a suffix as a stock rootfs path in its own right; split onto its own
line so a word character precedes it instead, which the regex already
excludes.
bin/fetch-assets.sh had a real latent bug of its own, independent of
this switch: the Ingenic toolchain fetch was gated on BUILD_KLIPPER=fork
alone, a condition that predates CPython 3.13's unconditional need for
the same compiler. A BUILD_KLIPPER=stock build from a clean vendor/
would reach the Python cross-build with sources and no compiler. Also
added curl --retry -- musl.cc flaked mid-download in CI while this was
being tracked down.
.github/workflows/ci.yml's fixture-only `test` job now actually
cross-builds s6 and CPython 3.13 instead of never exercising that path:
it fetches just those two build inputs (Mainsail/HelixScreen/Klipper
stay stubbed, unchanged) and caches vendor/, work/.s6, work/.py313 and
work/.sodium on versions.env.
Full suite, replica required, against real stock packages fetched for
this run: 23 passed, 0 failed, 0 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
a8a46e21d2 |
The build makes the packages Moonraker runs on, from hashed sources
Section 5c grows a second half and section 5d appears. The 18 packages and libsodium that two spikes proved are now built by bin/patch.sh from pins in versions.env, and land in the payload at lib/python3.13/site-packages and lib/libsodium.so. WHERE THE PACKAGE BUILD HAD TO GO, and it is not obvious: inside 5c, against the UNTRIMMED interpreter stage, before the trim runs. The headers INCLUDEPY names, the static dependency libraries pillow and cffi link against, and the x86-64 build-python all exist only at that moment -- the trim deletes include/ and config-3.13-* and then removes the staging tree entirely. The alternative was caching a second untrimmed ~200MB tree. The price of not doing that is that either stamp going stale rebuilds both halves, which is written down at the stamp rather than left to be discovered. The build no longer touches PyPI. pip comes from --with-ensurepip=install on the build-python, out of the same pinned CPython tarball, and every pip wheel runs --no-index --no-build-isolation --no-binary :all: against hashed sdists in vendor/. That last flag is not caution: during the spike pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado, three wrong-architecture objects sailed through the build, and only the ABI gate caught them. Which is why the gate now reaches the STAGED PAYLOAD rather than the build cache. It walked $PY_BUILD/bin and $PY_BUILD/lib, so libsodium and the twelve extension modules staged straight into $MOD_PAYLOAD would have gone unexamined. 69 ELF objects now, plus libsodium's resolved .so. Deliberately not all of $MOD_PAYLOAD: s6 is musl-built and correctly reads e_flags=0x1007, mips1 and legacy-NaN, which is right for a static binary that does no floating point and is measured working on the printer. It has its own gate in 5b. That distinction is recorded in patch.sh, because a sweep that assumes every object must be nan2008 flags all thirteen s6 binaries and looks like a disaster. Two pins carry reasons that cost someone an afternoon: pillow is 11.0.0 because 10.3.0 cannot be built by ANY 3.13 -- its setup.py execs and then reads locals()["__version__"], which PEP 667 broke -- and streaming-form-data is 1.19.1 because Moonraker's own pin ships pregenerated Cython calling the four-argument _PyLong_AsByteArray, which grew a fifth parameter in 3.13. docker/Dockerfile.build needs zlib1g-dev and patch, and the image must be rebuilt. The first is a genuinely nasty one: a build-python without zlib cannot unpack a wheel, or install pip, and ensurepip fails INSIDE a `make install` that reports success. patch.sh now asserts zlib and pip on the build-python rather than failing later and further away. The install manifest goes from 1872 paths to 2940 and case-upgrade.sh still passes, which is the check that the deletion loop scales. Note that the "70 to 728" in the commit that landed the interpreter was wrong -- 728 predates the interpreter itself; HEAD's own patch.sh produces 1872 here, measured. anvil.tar.xz grows 1.6MB packed, about 15MB on the printer. FF_PYTHON still points at FlashForge's 3.8.2 and /usr/prog/libsodium/lib is still in ANVIL_LIBS. Both comments now say what changed -- the packages exist -- and what has not: nothing has run through S62moonraker under s6 on 3.13. The two lines move together, in the commit that has that gate behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6a750f6bcc |
Moonraker serves on the 3.13 we built, with nothing from /usr/prog
The previous commit proved every component IMPORTS. Importing is not serving,
so this runs it. case-moonraker313.sh starts Moonraker on the exact command
line payload/etc/s6/moonraker/run execs -- only with our interpreter --
against the mod's own assets/moonraker.conf, and asks the printer what
happened:
* bound :7125 in 3s, with /proc/PID/cmdline confirming our interpreter and
our entry point rather than a coincidence
* GET /server/info answers 200 with 23 components loaded and none failed.
/machine/system_info, a different component reading this box's own /proc,
reports cpu: mips and python 3.13.7 [GCC 7.2.0]
* the database really works, which is the entire reason for the exercise: a
POST wrote a namespace, a GET read it back, the namespace appears beside
Moonraker's own, and data.mdb is 147456 bytes on disk -- through lmdb's
CPython extension, not the compile-at-import cffi fallback that would try
to invoke a compiler this printer does not have
* still alive and still answering 120 seconds later, with no tracebacks in
its log, and SIGTERM released the port in 2s
* the RUNNING process maps zero libraries under /usr/prog, with
LD_LIBRARY_PATH completely unset -- not merely purged of /usr/prog
* negative controls: nothing on :7125 beforehand, and with site-packages
moved aside it never binds at all
libsodium was the last /usr/prog string in the picture and is now ours.
libsodium 1.0.20, cross-built with the Ingenic toolchain in 24 seconds, gated
at 0x70001407 nan2008/o32/mips32r2, needing only libpthread, libc and the
loader, 406KB stripped into $MODDIR/lib. It computes rather than merely
loading: sodium_version_string is 1.0.20, its BLAKE2b-256 matches CPython's
own _blake2 on the same kernel, and libnacl.sign.Signer/Verifier -- the exact
ed25519 pair authorization.py imports -- signs, verifies, and rejects a
tampered message.
It costs anvil-env.sh nothing, which was not obvious and had to be measured.
libnacl's third fallback builds a path out of its own __file__, so from
$MODDIR/lib/python3.13/site-packages/libnacl it resolves $MODDIR/lib/libsodium.so
by absolute path -- working with LD_LIBRARY_PATH unset entirely. Recorded in
the README, because it works only because the prefix happens to contain the
string "lib": a prefix without it would break this silently.
klippy was scanned rather than guessed at. Its module-scope third-party
imports are cffi, greenlet, jinja2, pyserial and numpy; four of the five are
built and proven, and cffi already dlopens the printer's real c_helper.so.
numpy is at module scope only in extras/stepper_resonance_tester.py, which
Klipper loads on demand, and it is a multi-day problem that costs input-shaper
calibration rather than printing. There is no removed-stdlib usage anywhere in
klippy -- no imp, distutils, asyncore, cgi or getargspec.
What is left is packaging, not capability, and the README says so in order.
The sharp one: section 5c's trim deletes include/ and config-3.13-*, which the
package build needs, so the package build has to run inside 5c against the
untrimmed stage. And the ABI gate walks the build cache rather than the
payload, so anything staged straight into $MOD_PAYLOAD would bypass it --
coverage, not logic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
da65764b86 |
Moonraker's whole component set imports on the 3.13 we built
Nineteen packages cross-built for mipsel, twelve of them C extensions, every one gated at nan2008/o32/mips32r2. Not wired into bin/patch.sh yet -- this preserves the build and, more importantly, what it cost, the way tools/supervisor/ does for s6. The measurement lives in test/integration/printer/case-pyext.sh rather than in a paragraph. Two results carry the phase. The first: 3.13's cffi dlopens the printer's OWN /usr/prog/klipper/klippy/chelper/c_helper.so, resolves stepcompress_alloc, steppersync_alloc and itersolve_generate_steps, and calls one. That is phase 7's central question -- whether an interpreter we built can drive FlashForge's C helper -- answered by doing it, and it is the whole reason the interpreter had to be glibc rather than musl. The second: all 16 Moonraker components import on 3.13, with the list taken from CORE_COMPONENTS plus moonraker.conf rather than written down. The negative control is that moving the tree aside fails every third-party import and the database component. The package list is deliberately NOT moonraker-requirements.txt, which installs every optional component's dependencies. It is the closure of what the ENABLED set imports plus klippy's, so no apprise, ldap3, paho-mqtt or zeroconf. msgspec and uvloop are skipped on purpose: Moonraker guards both with contextlib.suppress(ImportError) and falls back to the stdlib. greenlet was the expected blocker and blocked for an unexpected reason. Its MIPS stack-switching assembly was never a problem; greenlet 3.x is C++ and writes its PyTypeObjects as designated initializers, which gcc 7.2 takes only in declaration order with no gaps. The exact rule was probed rather than guessed -- in-order contiguous fine, a trailing gap fine, an interior or leading gap refused -- and fill-designators.py writes the skipped fields back as explicit zeros, reading field order from the target's own headers and refusing to guess when a designator is not in them. Semantically a no-op: static storage duration, those fields were already zero. It switches out and back, in order, on the printer. Three traps worth having in the log. pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado on the first run -- three wrong-architecture .so files sailed through the build and only the ABI gate caught them, so --no-binary :all: is mandatory here rather than cautious. lmdb chose its cffi backend and produced a py3-none-any wheel with mdb.c inside, which is phase 6's recorded trap rebuilt from scratch: on the printer that path invokes a compiler that does not exist. And pillow 10.3.0 cannot be built by any 3.13, cross or not, because its setup.py execs and then reads locals()["__version__"], which PEP 667 broke. One dependency is unsolved and named rather than buried: libnacl ctypes-loads libsodium, which exists on this box only at /usr/prog/libsodium/lib. Measured both ways. So the authorization component keeps the last /usr/prog string in the picture until libsodium is cross-built or that one path entry is kept. What is left before FF_PYTHON could switch is written down in the README, and the honest remaining unknown is at the top of it: every component imports and the entry point loads, but importing is not serving. Nothing has bound :7125 on 3.13 yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |