mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 12:52:16 -05:00
worktree-timelapse-test-gcode
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr ShyshatskyiandClaude Opus 5
|
d898d105e0 |
The Moonraker pin was frozen by a missing _sqlite3 we now build ourselves
MOONRAKER_VERSION was 9d0d09d, the last commit before 80c7620 moved the
database from lmdb to sqlite, because FlashForge's python 3.8.2 has no
_sqlite3 and every release after it died loading the database component.
This repo builds and ships its own CPython 3.13 now, and it has one. The pin
is v0.11.0 -- a release tag, not a sha, so PKG_VERSION is just 0.11.0 and no
longer the Debian-style 0.8.0+git<sha> snapshot string. GitHub strips the
leading "v" from the directory inside /archive/<tag>.tar.gz, which is what
${MOONRAKER_VERSION#v} in the recipe is for.
The dependency set is the part that actually moved. Read against 0.11.0's
CORE_COMPONENTS plus the sections moonraker.conf enables:
dbus-next became dbus-fast. dbus_manager is a core component and imports
it at module scope, so this is not optional whatever [machine] provider
says. Its sdist cythonises nine modules; SKIP_CYTHON=1 ships the python
they were generated from and keeps the package ARCH all. The wheel still
comes out platform-tagged -- poetry-core marks the distribution impure
for declaring a build script at all -- but it holds 29 .py and no .so,
and the content is what ships.
importlib-metadata is new, imported at module scope by
utils/source_info.py, and brings zipp. Both build through setuptools_scm,
so PYPKG_HOST_LIST grew packaging and setuptools-scm. That list is now
install order: pkg_buildpython pip-installs each with --no-deps against
no index, flit-core builds packaging, setuptools_scm imports it.
pyserial-asyncio is gone. Nothing in 0.11.0 imports serial_asyncio;
utils/async_serial.py drives pyserial over asyncio itself. pyserial stays
-- klippy needs it and declares it.
LMDB STAYS, for a reason that is not the old one. 0.11.0 converts an
existing data.mdb to sqlite on first start and imports lmdb to read it. With
the module missing it tries to pip install one, and a printer has no index
to install from -- the old database would be dropped in silence. It is the
upgrade path now, not the database, and the comment on WANTED_MODULES says
so.
inotify-simple STAYS AT 1.3.5 though 0.11.0 pins ==2.0.1: 2.0.1 declares its
licence the PEP 639 way and needs setuptools>=77 to build, against the 75.6.0
pinned here for lmdb's sake. It fails at metadata generation, not at import,
which is the kind of thing that reads as a broken recipe later. 1.3.5 carries
every name 0.11.0 asks of it, and moonraker.conf sets file_system_observer:
none, so the import is all that is asked. jinja2 goes to 3.1.6 to match.
Not taken: paho-mqtt, zeroconf, apprise, ldap3 and python-periphery, whose
components this printer does not enable, and msgspec and uvloop, which
Moonraker imports under contextlib.suppress(ImportError).
Verified: every changed sha256 downloads and matches at the URL
fetch-assets.sh builds; all five affected wheels build offline
(--no-index --no-build-isolation) on a real 3.13 with the host backends
installed in the declared order; qa/static 122 passed, including the gates
that cross-check PYPKG_LIST against the recipes and every PKG_DEPENDS
against the feed. The mipsel cross-build and `make qa-replica` have not run
yet, and versions.env still says a bump is gated on the latter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012oRdgKM7scrEsr6waAByaY
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
2e9f2214a0 |
Merge master into the comment trim
Master moved twelve commits while this branch ran: patch.sh became
payload.sh, run-pre.sh and run-append.sh became runFirmwareExe.sh, the
install manifest and anvil.conf are gone, and versions.env was trimmed
harder there than here (53 comment lines against 185).
Resolved so the merge contributes comments and nothing else. Verified
mechanically: every one of the 270 files in the tree now carries
exactly master's code, and no file this branch touched is resurrected.
- master's deletions accepted: bin/patch.sh, bin/verify.sh,
installer/run-{pre,append}.sh, anvil.conf.in
- versions.env: master's, which is the better trim
- 26 files master only renamed through: this branch's trim kept, with
patch.sh -> payload.sh applied
- 14 files master rewrote to describe the new installer: master's
text taken wholesale, to be re-trimmed separately
- stale references to the manifest and run-append.sh corrected in
pkgs/3rdparty/busybox/build.sh and pkgs/moonraker/build.sh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMMpQTajN7RDJySrhQ2cTE
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b98552386b |
No FlashForge component ships; patch.sh is payload.sh
The software component existed to give the mod one exec hook. It is not needed for that any more, so a release now carries our installer and our payload and nothing else: 26MB smaller, 21MB of which was FlashForge's own firmwareExe travelling back to the partition it was already on, along with their klippy tree, their wifi module and their configs. WHAT IT GIVES BACK. /usr/prog/PROGRAM/software is no longer rewritten, so the version directory there stays FlashForge's with their genuine firmwareExe inside it -- and app_startup.sh's five-second watchdog, which copies one out of that directory when nothing by that name is running, has something to copy again. A printer whose payload fails to install now comes up on the stock UI instead of a blank screen. That recovery has been broken since we started deleting their binary out of a component we shipped; docs/how-it-works.md claimed it worked throughout. THE ROOT PASSWORD moves with it, because it used to ride in the component's shadow file. bin/pack.sh bakes ROOT_PW_HASH into the installer, single-quoted -- an unquoted $6$... would have the printer's shell expand $6 as a positional parameter and ship a truncated hash -- and the installer writes it to /usr/prog/etc/shadow. The random-password path asks one question instead of four: FlashForge ship the same root hash on every machine, so a live shadow still carrying it means nobody has set a password and it is ours to set. The .prev-root-hash recording, the shipped-vs-live comparison and the restore-it- afterwards step were all machinery for surviving a file that nothing overwrites any more, and are gone. pack.sh checks the hardcoded constant against the stock package it built from, because getting it wrong is silent and total: every printer would read as "already set" and none would ever get a password. bin/patch.sh is bin/payload.sh. It patched nothing any more -- no splice, no shadow edit, no firmwareExe deletion -- and what is left is the payload: the feed installed by the printer's own opkg, the seeded config, the manifest. Its sections are renumbered 1-5 with no gaps, and the five tombstone comments marking where removed sections used to be are gone with them. test_recipe_layout.py loses six of its nine tests. Each compared the tree against a description of the tree -- no fourth directory kind, each kind still used, 3rdparty carries no files of ours, installer/ holds exactly two named files -- and none could go red for a reason a printer would notice, while all six went red the moment the layout legitimately changed. The two that bite are kept: a payload/ path escaping $MODDIR, and a recipe name at both levels, which makes pkg_dir silently build the wrong package. SW_VER is deleted from config.env.example and both workflows: it stamped a component that is not built. Gates: 112 static tests pass. The generated installer is verified after substitution -- all four lines present, and `sh -n` over the result, because a hash with a stray character in it would otherwise reach a printer as a syntax error whose only symptom is a package that installs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
9b5b562a22 |
Comments: keep the contract and the trap, drop the narration
A sweep over every commented file in the tree. What is kept is what a reader needs to change the code safely: what a thing is, its signature, the invariants other files depend on, and the traps that fail silently (find's non-zero exit under set -e, s6's baked-in libexecdir, the -lm that decides whether _sqlite3 exists, opkg's compile-time prefix). What goes is the narration around it: incident histories, arguments against alternatives already rejected, restatements of the code below, and paragraphs re-explaining a rule three files already state. One paragraph in pkgs/3rdparty/opkg/build.sh was duplicated outright. Comment lines 7971 -> 6158. Every change is comment-only: no non-comment line differs anywhere in the 86 files, checked mechanically. Commented-out config directives (#fast_feed: 30000 and friends) are documentation of defaults and are kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NMMpQTajN7RDJySrhQ2cTE |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
4ce928f160 |
The last two artefact ABI gates go; the toolchain check stays
Follow-on to |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
214a4e58b9 |
anvil-core owns firmwareExe and start.sh; kill the 3.8.2 myth
Three things, all downstream of klippy having moved to $MODDIR and our own
CPython 3.13.
SECTIONS 8 AND 9 ARE GONE. Both files install at $MODDIR/prog/ and
anvil-link-prog.sh points the stock paths at them -- from run-append.sh on a
flash and from anvil-core's postinst on `opkg upgrade anvil-core`. Nothing
execs either between run.sh distributing the component and link-prog running,
so the component copy bought no ordering. It cost, though: our firmwareExe
with no payload behind it logs "no anvil-env.sh", finds no s6-svscan and then
`while true; do sleep 3600; done` -- a printer that boots to a blank screen for
ever and never reaches app_startup.sh's own recovery. Absent, that recovery
restores a firmwareExe from a version directory. So patch.sh now REMOVES both
from the component: the stock component carries them, and deleting the two cp
lines alone would have shipped FlashForge's originals.
SECTION 5c IS GONE. It did no work -- two du prints and a comment whose first
line was the falsehood below.
THE FALSEHOOD. "klippy runs on the stock 3.8.2, started by
/usr/prog/klipper/start.sh" had spread well past patch.sh, and it had reached
things people follow:
- docs/toolchange.md told owners to scp the ff_*.py extras to
/usr/prog/klipper/klippy/extras/ -- the wrong partition -- and warned that
an OTA would delete them there. Same wrong path in three ff-*.cfg headers
and in ff_toolchange.py.
- versions.env labelled our CPython 3.13 and its eighteen packages
"SHIPPED BUT NOT USED".
- docs/how-it-works.md said Moonraker runs on python3.8, contradicting line
126 of the same file.
All corrected. Historical notes keep their history and get a [Superseded: ...]
marker instead -- including 80-s6-migration.md's "s6-rc-compile is not
shipped", which pkgs/3rdparty/s6-rc/build.sh contradicts deliberately.
Found while reviewing: every docs/ reference to the ff-*.cfg pointed at
pkgs/anvil-core/payload/config/, which does not exist -- they are
anvil-klipper-config's. 28 references across 9 files, several of them broken
markdown links on the published site. All 24 relative links under docs/ now
resolve.
Gates: verify.sh syntax-checks prog/start.sh and prog/firmwareExe out of the
payload (firmwareExe was never syntax-checked before) and fails if the
component carries either. A new replica test asserts both stock paths are
symlinks into $MODDIR/prog and do not dangle. One existing replica test stubbed
firmwareExe by redirecting into $FE, which now writes THROUGH the symlink into
the packaged wrapper; it moves the link aside instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6aa622d2ff |
6,280 lines of case scripts reported 13 bits. They are 118 tests now.
test/run-tests.py is gone, and with it the bespoke harness that wrapped
pytest, re-parsed its JUnit XML, and drove thirteen replica case scripts to
get one boolean back from each. Every gate it ran is either a module under
qa/replica/ or deliberately retired.
PORTED. case-install.sh (579 lines, 1 bit) is test_install.py, 22 tests, asked
of the machine the printer's own app_startup.sh built rather than one the
harness assembled. case-upgrade.sh is 21, case-supervisor.sh 31,
case-boot-screen.sh 17, case-mcu-bringup.sh 8, and case-nginx.sh plus
case-moonraker313-s6.sh are test_web.py's 7.
RETIRED, subject deleted by the s6-rc switch: case-services.sh, case-nginx.sh,
case-moonraker.sh, case-moonraker313-s6.sh, case-camera.sh and
case-priority.sh drove $MODDIR/init.d/S* or sourced anvil-service.sh, neither
of which exists. Decided against real output -- `make test-services` failed
with "the payload ships no anvil-service.sh". case-pyext.sh and
case-moonraker313.sh went too: no gate, no target, no CI step, never run once.
RETIRED as a coverage decision: case-python.sh (test_web.py runs real Moonraker
on that interpreter, which is harder), case-libpath.sh, and case-recovery.sh.
The last is the brick-safety rollback and has no replacement; it is recorded in
docs/qa-migration.md rather than left to be discovered.
THREE ASSERTIONS HAD STOPPED MEANING ANYTHING, and each had been green for
months:
* case-mcu-bringup.sh's negative control had INVERTED. It asserted the
bring-up fails without LD_LIBRARY_PATH; CPython is --disable-shared now and
needs no library path, so the case's own skip branch was the one firing.
* case-upgrade.sh's headline assertion was VACUOUS. run-append.sh:150 does
`rm -rf $MODDIR/init.d` unconditionally, outside the manifest branch, so
"the update removed the stale S60web" passed regardless of the manifest
logic it was written to test.
* case-boot-screen.sh CONFIRMED ITS OWN ARITHMETIC, computing expected pixel
packing by calling ffscreen._pixel inside the replica -- a swapped R/B
matched itself.
case-install.sh's boot-log grep is not reproduced for the same reason: the log
line names every service in the boot set including the ones that then fail, so
grepping it cannot go red. The compiled database is asked instead.
TWO REAL BUGS FOUND ON THE WAY.
assemble.sh attached /stick.img with `losetup` and wrote the device to
/stick.loop, which NOTHING HAS EVER READ. Loop devices are the host's, not the
container's, so every USB_STICK=1 run leaked one permanently and a WSL2 kernel
has thirteen. Once they were gone every replica run failed with "failed to
setup loop device", which looks like a docker fault and is not. Detached now in
entrypoint.sh on the way out, and in qa's stop() for the held-open containers
that never take that path.
test_startup.py was 43 tests red: it built its argparse.Namespace by hand and
had never been taught --only-bringup / --no-bringup, and its FakeKlipper still
modelled start.sh plus a separate stop when klipper is restarted with one
`s6-svc -wr -t`.
CI AND RELEASE. The replica job was getting its package as a side effect of
run-tests.py running unpack/patch/pack; that is an explicit step now, as the
comment there predicted it would have to be. release.yml tested the exact files
it ships through sim-install.py, whose exit status the loop was discarding --
it is `REAL_PKG=<pkg> pytest ./qa/replica` per model now, 22 named results each.
WHAT IS NOT TESTED ANY MORE: bin/unpack.sh, patch.sh, pack.sh and verify.sh.
The packaging build on a synthetic stock package went with the runner and has
no counterpart in qa/. That is the path producing the .tgz a user flashes.
make-stock-fixture.sh is kept as the input a replacement needs.
Measured: qa/static 246 passed, qa/replica 110 passed and 8 blocked on hardware
the replica has not got (no /dev/ttyS*, no /dev/video*, so ok-all is
unreachable), test/ pytest 185 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BTAXgjtF8GdhqSxvdXCFW
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e3d1bdcdf7 |
The payload is the feed installed, not the build trees copied
bin/patch.sh ran every recipe and then cp -a'd its build tree into
work/modpayload. Eight sections of it, each a second description of what the
.ipk that recipe produces already contains -- the same duplication the recipes
themselves removed one level down, one level further up.
It now installs the feed. pkg_buildopkg's x86-64 opkg gets one invocation with
the whole selected set and an --offline-root, and what lands there is the
payload. "What does this release install?" is answerable with
`opkg list-installed` against the payload itself, and the tarball and the
package cannot describe different bytes because the tarball is made of the
package.
THE SET COMES FROM pkg_recipes, which is already PKG_WHEN-gated -- so
BUILD_HELIX, BUILD_MAINSAIL, BUILD_MOONRAKER, BUILD_TOOLCHANGE and
BUILD_KLIPPER are read in the pkg.conf that owns each and nowhere else,
instead of being restated by an `if` here that could drift from it.
-dev IS FILTERED BY PACKAGE NAME, and the first attempt got this wrong in a
way worth recording. execline, s6 and CPython ship both halves and pkg_ipk
names the runtime one; zlib, openssl, sqlite, expat, libffi, xz, bzip2,
libarchive and skalibs ship ONLY a dev package and say so by setting
PKG_NAME=anvil-<x>-dev outright. Asking "does this recipe have a runtime half"
installed all nine: 40 packages and 2165 ELF objects where there should be 31
and 154.
--force-postinstall, because opkg skips configuration under an offline root
and would leave every package Status: unpacked. Nothing runs opkg on the
printer after run-append.sh extracts the tarball -- the extraction IS the
install -- so a database saying otherwise would make the first real `opkg
upgrade` argue with the filesystem. It runs nothing today: no recipe defines a
maintainer script, and opkg 0.7.0 would not run one here anyway (it sets no
IPKG_INSTROOT; see libopkg/pkg.c:1339).
Installed-Time is normalised afterwards. opkg takes it from the clock and
honours SOURCE_DATE_EPOCH only for a man-page date, so two builds of one
commit differed by one line per package -- measured, then fixed, then measured
again.
Gone with the staging: the hand-written PKG_DEV_FILES prune (the dev half is
not installed, so bin/verify.sh's "no headers" check now passes because they
were never there), three partial ABI gates in favour of one over the finished
payload, and the last pkg_out call -- section 1 reads klippy out of the
installed anvil-klipper.
The payload gains anvil-klipper's tree at $MODDIR/klipper and bin/opkg. The
first is inert until phase 7 of docs/notes/80-s6-migration.md and is 2.4MB of
duplicate until then; the second is route 1 of the bootstrap in
docs/notes/85-packaging.md -- the first payload that can install its own
successor.
VERIFIED by building the payload from the parent commit and from this one, on
the synthetic stock fixture with the real vendored assets:
files removed 0
content changes among the 2778 shared files 1 (.install-manifest,
which lists the payload)
mode changes 0
symlink changes 0
added klipper/**, var/lib/opkg/** (63), bin/opkg, and the
www/mainsail/.version the previous commit stopped deleting
two builds in a row byte-identical
bin/verify.sh passes every check including the ship boundary; qa-static 255;
make test 9 gates, 0 failed; test-py 181.
test/run-tests.py builds the feed as the first step of the fixture lane, into
a PKG_FEED of its own -- sharing work/packages let the MOD_VER=ci build rename
every dated package and the prune delete the rest on the way past, measured as
44 becoming 43.
qa/static/test_ipk.py's share-one-build test is inverted rather than deleted:
the property was "one build, two vehicles" and is now "one vehicle", so what
it asserts is that nobody reintroduces the second path -- no recipe run from
patch.sh, no pkg_out, no work/pkg, and opkg rather than pkgs/ipk-install,
which stays the printer's hand-repair tool.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8cc92e8266 |
A wheel could only be built while three scratch trees happened to coexist
That is why bin/patch.sh section 5c was 825 lines and why a Pillow bump
rebuilt CPython. Cross-building an extension module needs the untrimmed
staging tree (INCLUDEPY points into it), the private sysroot of static C
libraries (pillow links zlib, cffi links libffi) and an x86-64 build-python
of the same version -- and all three existed only mid-section, so the wheels
had to be built there too, on a cache with two stamps that invalidated each
other.
Each of those three is now a durable thing: anvil-python-dev carries the
headers, pkg_deps fills each recipe's own sysroot from the feed, and
pkg_buildpython caches one build-python for all nineteen recipes. So the
coupling goes away with them -- `make packages PKG=python-pillow` builds
Pillow.
pkg/python -> anvil-python + anvil-python-dev
pkg/python-<name> x18 -> anvil-python-distro, anvil-python-pillow, ...
Three verbs in pkg/lib.sh, shared by all nineteen rather than one per awkward
project: pkg_buildpython provides the build-python, pkg_pytarget points its
sysconfig at the target headers in the sysroot, pkg_pywheel builds one wheel
and unpacks it into the staging site-packages. greenlet's gcc 7.2 designated
initializers, lmdb's LMDB_FORCE_CPYTHON and pillow's build_ext options are
recipe lines and a PKG_PY_SETUP_ARGS knob, not three more verbs.
Eighteen versions that were invisible are now in the feed index, anvil-moonraker
declares the thirteen packages it imports, and a printer that does not want
gcode thumbnails can leave Pillow uninstalled.
PY_STAMP, PYPKG_STAMP and PY_BUILD are gone: pkg_stamp derives the key from
PKG_BUILD_DEPENDS, and bin/fetch-assets.sh asks pkg_needs instead of
re-deriving a stamp bin/patch.sh also derived. Two of its four toolchain
conditions were that second spelling.
Measured:
* 40 packages; 959 site-packages files, none owned by two packages
* 12 mipsel extension modules -- the count the old gate demanded
* every Depends resolves inside the feed (24 declare one, 0 unsatisfiable)
* two cold builds byte-identical, with work/pkg, work/.py-host and the
unpacked toolchain deleted in between
* 233 static tests, 4 new: the PYPKG_LIST/recipe agreement both ways, no
version pinned twice, patch.sh builds every package, and no Depends
naming a package no recipe produces
Not verified here: the replica suite. It needs the stock FlashForge package,
which the packaging lane deliberately does without.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
99706bb5f7 |
s6 was legacy-NaN. The kernel this printer runs refuses that at exec().
The Bootlin swap two commits ago (musl.cc -> mips32el) fixed a
reachability problem and kept a correctness one: mips32el is plain
mips32r1, whose musl crt/libc is legacy-NaN by construction and cannot
be made to emit nan2008 output at all -- forcing -mnan=2008 on it fails
to LINK ("mixing -mnan=2008 module with previous -mnan=legacy modules").
So every s6 binary this repo has shipped read e_flags=0x1007, defended
in five separate places as correct on the theory that a supervisor
doing no floating point cannot care about NaN encoding.
True of the arithmetic. False of exec(): docs/how-it-works.md already
documented, for a different binary, that "the Ingenic X2000 kernel
returns ENOEXEC for legacy-NaN MIPS" -- the ABI check happens before a
single instruction of the program's own runs, which means it was never
about what s6 computes. None of the replica gates that "measured s6
working on the printer" ever caught this, because none of them ran on
the printer: qemu-mipsel-static's user-mode emulation does not enforce
the same ABI check a real kernel's binfmt loader does. A build that
passed case-supervisor.sh, case-services.sh, case-moonraker313-s6.sh and
a full release run was still shipping a supervisor that a real X2000
kernel may refuse to start at all.
THE FIX. Bootlin also publishes mips32r5el: MIPS Release 5 has no
legacy-NaN silicon to be compatible with, so its musl crt/libc is
nan2008 by construction, and its default codegen already targets
mips32r2. Verified outside the repo before touching anything: fetched
the tarball, cross-compiled skalibs+s6 with it, every binary read
nan2008/o32/mips32r2, and s6-svscan ran and supervised under
qemu-mipsel-static. Section 5b now applies the same gcc-wrapper
discipline 5c already uses for the Ingenic toolchain -- -EL -mnan=2008
-march=mips32r2 baked into the driver, not left to CFLAGS, with a probe
gate before building anything on top -- because "the toolchain's default
happens to be right" is exactly the kind of assumption this bug was.
mips_abi_gate moved earlier in the file so 5b can use it, and s6 is IN
that gate now rather than exempt from it -- exempt is how this shipped
wrong for as long as it did. The gate itself grew a real relaxation
alongside the real tightening: a static EXEC from this musl toolchain
legitimately carries EF_MIPS_PIC (musl's crt is unconditionally -fPIC;
no combination of -static/-no-pie/-fno-PIC removes it, measured) and so
reads the DYN flag value anyway, which is a property of the toolchain
and not a sign of anything wrong -- so the gate now accepts either
0x70001405 or 0x70001407 for any object, rather than picking one from
the ELF Type.
Also bumped S6_STAMP to include MUSL_TOOLCHAIN_FILE: without that, a
checkout with an already-built work/.s6 from before this commit would
read its own stale legacy-NaN tree as current and never rebuild it.
Full real build: "s6: 14 ELF objects are nan2008/o32/mips32r2 -- good".
case-supervisor.sh green on the replica with the new binaries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c870d8417d |
FF_PYTHON moves to our own 3.13, and CI stops rehearsing around it
The gate this switch was waiting on now exists and passes:
case-moonraker313-s6.sh drives Moonraker through the real boot path --
S40s6's scandir, S62moonraker, readiness gated on :7125 actually
listening, a kill -9 respawn, a stop that stays stopped -- on the
interpreter this commit ships. anvil-env.sh now points FF_PYTHON at
$MODDIR/bin/python3.13 and drops /usr/prog/libsodium/lib from
ANVIL_LIBS, since nothing left running under FF_PYTHON imports libnacl
from there any more (Moonraker carries its own libsodium; klippy is not
on this list at all -- it stays on FlashForge's 3.8.2, started
independently by /usr/prog/klipper/start.sh).
Three real bugs surfaced chasing this through the replica, none of them
about the switch itself:
* The Phase-1 manifest-based install can only delete files it once
tracked, so an init.d/S60web surviving from a release that predates
the manifest system would never be cleaned -- caught by
case-install.sh, which had never run against real stock packages
before today. run-append.sh now clears $MODDIR/init.d unconditionally
on every update: those scripts are 100% mod-owned and get executed,
so a diff against tracked history is not a strong enough guarantee.
* case-moonraker313-s6.sh's own readiness-ordering check flaked once:
its bind-time sampler polled at 1-second resolution while s6-svwait
is event-driven and returns within a fraction of a second, so
`date +%s` truncation could show readiness before the bind though the
real events were in order. Tightened to 0.2s.
* case-mcu-bringup.sh and case-boot-screen.sh source anvil-env.sh but
named their own directory variable $MOD, not $MODDIR -- harmless
while FF_PYTHON was a literal path, and silently wrong the moment it
started reading $MODDIR. Aliased.
Every other test/integration file touched here is the same story:
FF_PYTHON's callers were hardcoded to FlashForge's 3.8.2 throughout the
suite, and each one either now stages our interpreter as a tarball
(gates.py's mcu_bringup/boot_screen/moonraker, matching the pattern
already used for python()/moonraker313_s6) or asserts against the new
target. case-libpath.sh's premise changes with it: the three ANVIL_LIBS
entries that remain are read as what klippy's process tree still needs,
not what every mod process needs -- Moonraker maps zero /usr/prog
libraries now, measured.
payload/anvil-service.sh's SVC_S6_SCANDIR line tripped test_paths.py's
absolute-path regex, which reads a closing brace immediately followed by
a suffix as a stock rootfs path in its own right; split onto its own
line so a word character precedes it instead, which the regex already
excludes.
bin/fetch-assets.sh had a real latent bug of its own, independent of
this switch: the Ingenic toolchain fetch was gated on BUILD_KLIPPER=fork
alone, a condition that predates CPython 3.13's unconditional need for
the same compiler. A BUILD_KLIPPER=stock build from a clean vendor/
would reach the Python cross-build with sources and no compiler. Also
added curl --retry -- musl.cc flaked mid-download in CI while this was
being tracked down.
.github/workflows/ci.yml's fixture-only `test` job now actually
cross-builds s6 and CPython 3.13 instead of never exercising that path:
it fetches just those two build inputs (Mainsail/HelixScreen/Klipper
stay stubbed, unchanged) and caches vendor/, work/.s6, work/.py313 and
work/.sodium on versions.env.
Full suite, replica required, against real stock packages fetched for
this run: 23 passed, 0 failed, 0 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
a8a46e21d2 |
The build makes the packages Moonraker runs on, from hashed sources
Section 5c grows a second half and section 5d appears. The 18 packages and libsodium that two spikes proved are now built by bin/patch.sh from pins in versions.env, and land in the payload at lib/python3.13/site-packages and lib/libsodium.so. WHERE THE PACKAGE BUILD HAD TO GO, and it is not obvious: inside 5c, against the UNTRIMMED interpreter stage, before the trim runs. The headers INCLUDEPY names, the static dependency libraries pillow and cffi link against, and the x86-64 build-python all exist only at that moment -- the trim deletes include/ and config-3.13-* and then removes the staging tree entirely. The alternative was caching a second untrimmed ~200MB tree. The price of not doing that is that either stamp going stale rebuilds both halves, which is written down at the stamp rather than left to be discovered. The build no longer touches PyPI. pip comes from --with-ensurepip=install on the build-python, out of the same pinned CPython tarball, and every pip wheel runs --no-index --no-build-isolation --no-binary :all: against hashed sdists in vendor/. That last flag is not caution: during the spike pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado, three wrong-architecture objects sailed through the build, and only the ABI gate caught them. Which is why the gate now reaches the STAGED PAYLOAD rather than the build cache. It walked $PY_BUILD/bin and $PY_BUILD/lib, so libsodium and the twelve extension modules staged straight into $MOD_PAYLOAD would have gone unexamined. 69 ELF objects now, plus libsodium's resolved .so. Deliberately not all of $MOD_PAYLOAD: s6 is musl-built and correctly reads e_flags=0x1007, mips1 and legacy-NaN, which is right for a static binary that does no floating point and is measured working on the printer. It has its own gate in 5b. That distinction is recorded in patch.sh, because a sweep that assumes every object must be nan2008 flags all thirteen s6 binaries and looks like a disaster. Two pins carry reasons that cost someone an afternoon: pillow is 11.0.0 because 10.3.0 cannot be built by ANY 3.13 -- its setup.py execs and then reads locals()["__version__"], which PEP 667 broke -- and streaming-form-data is 1.19.1 because Moonraker's own pin ships pregenerated Cython calling the four-argument _PyLong_AsByteArray, which grew a fifth parameter in 3.13. docker/Dockerfile.build needs zlib1g-dev and patch, and the image must be rebuilt. The first is a genuinely nasty one: a build-python without zlib cannot unpack a wheel, or install pip, and ensurepip fails INSIDE a `make install` that reports success. patch.sh now asserts zlib and pip on the build-python rather than failing later and further away. The install manifest goes from 1872 paths to 2940 and case-upgrade.sh still passes, which is the check that the deletion loop scales. Note that the "70 to 728" in the commit that landed the interpreter was wrong -- 728 predates the interpreter itself; HEAD's own patch.sh produces 1872 here, measured. anvil.tar.xz grows 1.6MB packed, about 15MB on the printer. FF_PYTHON still points at FlashForge's 3.8.2 and /usr/prog/libsodium/lib is still in ANVIL_LIBS. Both comments now say what changed -- the packages exist -- and what has not: nothing has run through S62moonraker under s6 on 3.13. The two lines move together, in the commit that has that gate behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6a750f6bcc |
Moonraker serves on the 3.13 we built, with nothing from /usr/prog
The previous commit proved every component IMPORTS. Importing is not serving,
so this runs it. case-moonraker313.sh starts Moonraker on the exact command
line payload/etc/s6/moonraker/run execs -- only with our interpreter --
against the mod's own assets/moonraker.conf, and asks the printer what
happened:
* bound :7125 in 3s, with /proc/PID/cmdline confirming our interpreter and
our entry point rather than a coincidence
* GET /server/info answers 200 with 23 components loaded and none failed.
/machine/system_info, a different component reading this box's own /proc,
reports cpu: mips and python 3.13.7 [GCC 7.2.0]
* the database really works, which is the entire reason for the exercise: a
POST wrote a namespace, a GET read it back, the namespace appears beside
Moonraker's own, and data.mdb is 147456 bytes on disk -- through lmdb's
CPython extension, not the compile-at-import cffi fallback that would try
to invoke a compiler this printer does not have
* still alive and still answering 120 seconds later, with no tracebacks in
its log, and SIGTERM released the port in 2s
* the RUNNING process maps zero libraries under /usr/prog, with
LD_LIBRARY_PATH completely unset -- not merely purged of /usr/prog
* negative controls: nothing on :7125 beforehand, and with site-packages
moved aside it never binds at all
libsodium was the last /usr/prog string in the picture and is now ours.
libsodium 1.0.20, cross-built with the Ingenic toolchain in 24 seconds, gated
at 0x70001407 nan2008/o32/mips32r2, needing only libpthread, libc and the
loader, 406KB stripped into $MODDIR/lib. It computes rather than merely
loading: sodium_version_string is 1.0.20, its BLAKE2b-256 matches CPython's
own _blake2 on the same kernel, and libnacl.sign.Signer/Verifier -- the exact
ed25519 pair authorization.py imports -- signs, verifies, and rejects a
tampered message.
It costs anvil-env.sh nothing, which was not obvious and had to be measured.
libnacl's third fallback builds a path out of its own __file__, so from
$MODDIR/lib/python3.13/site-packages/libnacl it resolves $MODDIR/lib/libsodium.so
by absolute path -- working with LD_LIBRARY_PATH unset entirely. Recorded in
the README, because it works only because the prefix happens to contain the
string "lib": a prefix without it would break this silently.
klippy was scanned rather than guessed at. Its module-scope third-party
imports are cffi, greenlet, jinja2, pyserial and numpy; four of the five are
built and proven, and cffi already dlopens the printer's real c_helper.so.
numpy is at module scope only in extras/stepper_resonance_tester.py, which
Klipper loads on demand, and it is a multi-day problem that costs input-shaper
calibration rather than printing. There is no removed-stdlib usage anywhere in
klippy -- no imp, distutils, asyncore, cgi or getargspec.
What is left is packaging, not capability, and the README says so in order.
The sharp one: section 5c's trim deletes include/ and config-3.13-*, which the
package build needs, so the package build has to run inside 5c against the
untrimmed stage. And the ABI gate walks the build cache rather than the
payload, so anything staged straight into $MOD_PAYLOAD would bypass it --
coverage, not logic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
da65764b86 |
Moonraker's whole component set imports on the 3.13 we built
Nineteen packages cross-built for mipsel, twelve of them C extensions, every one gated at nan2008/o32/mips32r2. Not wired into bin/patch.sh yet -- this preserves the build and, more importantly, what it cost, the way tools/supervisor/ does for s6. The measurement lives in test/integration/printer/case-pyext.sh rather than in a paragraph. Two results carry the phase. The first: 3.13's cffi dlopens the printer's OWN /usr/prog/klipper/klippy/chelper/c_helper.so, resolves stepcompress_alloc, steppersync_alloc and itersolve_generate_steps, and calls one. That is phase 7's central question -- whether an interpreter we built can drive FlashForge's C helper -- answered by doing it, and it is the whole reason the interpreter had to be glibc rather than musl. The second: all 16 Moonraker components import on 3.13, with the list taken from CORE_COMPONENTS plus moonraker.conf rather than written down. The negative control is that moving the tree aside fails every third-party import and the database component. The package list is deliberately NOT moonraker-requirements.txt, which installs every optional component's dependencies. It is the closure of what the ENABLED set imports plus klippy's, so no apprise, ldap3, paho-mqtt or zeroconf. msgspec and uvloop are skipped on purpose: Moonraker guards both with contextlib.suppress(ImportError) and falls back to the stdlib. greenlet was the expected blocker and blocked for an unexpected reason. Its MIPS stack-switching assembly was never a problem; greenlet 3.x is C++ and writes its PyTypeObjects as designated initializers, which gcc 7.2 takes only in declaration order with no gaps. The exact rule was probed rather than guessed -- in-order contiguous fine, a trailing gap fine, an interior or leading gap refused -- and fill-designators.py writes the skipped fields back as explicit zeros, reading field order from the target's own headers and refusing to guess when a designator is not in them. Semantically a no-op: static storage duration, those fields were already zero. It switches out and back, in order, on the printer. Three traps worth having in the log. pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado on the first run -- three wrong-architecture .so files sailed through the build and only the ABI gate caught them, so --no-binary :all: is mandatory here rather than cautious. lmdb chose its cffi backend and produced a py3-none-any wheel with mdb.c inside, which is phase 6's recorded trap rebuilt from scratch: on the printer that path invokes a compiler that does not exist. And pillow 10.3.0 cannot be built by any 3.13, cross or not, because its setup.py execs and then reads locals()["__version__"], which PEP 667 broke. One dependency is unsolved and named rather than buried: libnacl ctypes-loads libsodium, which exists on this box only at /usr/prog/libsodium/lib. Measured both ways. So the authorization component keeps the last /usr/prog string in the picture until libsodium is cross-built or that one path entry is kept. What is left before FF_PYTHON could switch is written down in the README, and the honest remaining unknown is at the top of it: every component imports and the entry point loads, but importing is not serving. Nothing has bound :7125 on 3.13 yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |