From da65764b86d67688beed8241fb73f239f10ffcae Mon Sep 17 00:00:00 2001 From: Oleksandr Shyshatskyi Date: Thu, 27 Aug 2026 10:14:38 +0300 Subject: [PATCH] Moonraker's whole component set imports on the 3.13 we built Nineteen packages cross-built for mipsel, twelve of them C extensions, every one gated at nan2008/o32/mips32r2. Not wired into bin/patch.sh yet -- this preserves the build and, more importantly, what it cost, the way tools/supervisor/ does for s6. The measurement lives in test/integration/printer/case-pyext.sh rather than in a paragraph. Two results carry the phase. The first: 3.13's cffi dlopens the printer's OWN /usr/prog/klipper/klippy/chelper/c_helper.so, resolves stepcompress_alloc, steppersync_alloc and itersolve_generate_steps, and calls one. That is phase 7's central question -- whether an interpreter we built can drive FlashForge's C helper -- answered by doing it, and it is the whole reason the interpreter had to be glibc rather than musl. The second: all 16 Moonraker components import on 3.13, with the list taken from CORE_COMPONENTS plus moonraker.conf rather than written down. The negative control is that moving the tree aside fails every third-party import and the database component. The package list is deliberately NOT moonraker-requirements.txt, which installs every optional component's dependencies. It is the closure of what the ENABLED set imports plus klippy's, so no apprise, ldap3, paho-mqtt or zeroconf. msgspec and uvloop are skipped on purpose: Moonraker guards both with contextlib.suppress(ImportError) and falls back to the stdlib. greenlet was the expected blocker and blocked for an unexpected reason. Its MIPS stack-switching assembly was never a problem; greenlet 3.x is C++ and writes its PyTypeObjects as designated initializers, which gcc 7.2 takes only in declaration order with no gaps. The exact rule was probed rather than guessed -- in-order contiguous fine, a trailing gap fine, an interior or leading gap refused -- and fill-designators.py writes the skipped fields back as explicit zeros, reading field order from the target's own headers and refusing to guess when a designator is not in them. Semantically a no-op: static storage duration, those fields were already zero. It switches out and back, in order, on the printer. Three traps worth having in the log. pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado on the first run -- three wrong-architecture .so files sailed through the build and only the ABI gate caught them, so --no-binary :all: is mandatory here rather than cautious. lmdb chose its cffi backend and produced a py3-none-any wheel with mdb.c inside, which is phase 6's recorded trap rebuilt from scratch: on the printer that path invokes a compiler that does not exist. And pillow 10.3.0 cannot be built by any 3.13, cross or not, because its setup.py execs and then reads locals()["__version__"], which PEP 667 broke. One dependency is unsolved and named rather than buried: libnacl ctypes-loads libsodium, which exists on this box only at /usr/prog/libsodium/lib. Measured both ways. So the authorization component keeps the last /usr/prog string in the picture until libsodium is cross-built or that one path entry is kept. What is left before FF_PYTHON could switch is written down in the README, and the honest remaining unknown is at the top of it: every component imports and the entry point loads, but importing is not serving. Nothing has bound :7125 on 3.13 yet. Co-Authored-By: Claude Opus 5 (1M context) --- test/integration/printer/case-pyext.sh | 563 ++++++++++++++++++++++ tools/python-packages/README.md | 137 ++++++ tools/python-packages/build.sh | 425 ++++++++++++++++ tools/python-packages/fill-designators.py | 227 +++++++++ 4 files changed, 1352 insertions(+) create mode 100755 test/integration/printer/case-pyext.sh create mode 100644 tools/python-packages/README.md create mode 100755 tools/python-packages/build.sh create mode 100755 tools/python-packages/fill-designators.py diff --git a/test/integration/printer/case-pyext.sh b/test/integration/printer/case-pyext.sh new file mode 100755 index 0000000..3207025 --- /dev/null +++ b/test/integration/printer/case-pyext.sh @@ -0,0 +1,563 @@ +#!/bin/sh +# Do the cross-built third-party packages actually WORK on the CPython 3.13 +# this repo builds, on the printer's own kernel? +# +# SPIKE GATE for phase 6. bin/patch.sh section 5c already ships an interpreter +# nothing runs; this asks the question that stands between that interpreter +# and FF_PYTHON pointing at it: Moonraker and klippy import third-party C +# extensions, those exist on this printer only as mipsel .so files built +# against 3.8, and none had been built for 3.13. +# +# IMPORTS ARE NOT THE MEASUREMENT. A cross-built .so that is the wrong ABI +# fails at import, so an import does prove something -- but a cffi that +# imports and cannot dlopen, a greenlet that imports and segfaults the first +# time it switches stacks, and an lmdb that imports and then tries to run a +# compiler are all things that have happened here or in the notes. So each +# native package is USED: cffi opens a real library and calls a function +# through it the way klippy's chelper does, greenlet switches out and back and +# proves the ORDER, lmdb writes a database and a SECOND PROCESS reads it. +# +# THE REAL PRIZE is section 5: Moonraker's own component list, taken from its +# CORE_COMPONENTS plus the printer's moonraker.conf, imported on 3.13. That is +# the same check case-moonraker.sh section 7 runs against FlashForge's 3.8.2, +# pointed at the new interpreter. +FAIL=0 +ok() { echo " PASS $*"; } +bad() { echo " FAIL $*"; FAIL=1; } +note(){ echo " .. $*"; } + +MODDIR=/usr/data/anvil +PY=$MODDIR/bin/python3.13 +SP=$MODDIR/lib/python3.13/site-packages +MRROOT=/usr/data/mrsrc + +mkdir -p $MODDIR +gzip -dc /mnt/py.tgz | tar -x -C $MODDIR || { bad "cannot unpack py.tgz"; exit 1; } +chmod +x $PY 2>/dev/null + +# The extension tree lands in the interpreter's OWN site-packages, which is +# already on sys.path -- no PYTHONPATH, no .pth file, nothing for a boot +# script to get wrong. +mkdir -p $SP +gzip -dc /mnt/ext.tgz | tar -x -C /tmp || { bad "cannot unpack ext.tgz"; exit 1; } +cp -a /tmp/site-packages/. $SP/ || { bad "cannot stage site-packages"; exit 1; } + +mkdir -p $MRROOT +gzip -dc /mnt/mr.tgz | tar -x -C $MRROOT || note "no moonraker tarball" + +# Empty on purpose, throughout. The interpreter is statically linked against +# its seven C libraries and every extension below was linked against the same +# toolchain, so nothing here may need a search path -- and if something does, +# that is the finding. +export LD_LIBRARY_PATH= + +echo "=== 0. the interpreter still runs with the tree dropped in ===" +ver=`$PY -c 'import sys; print(sys.version.split()[0])' 2>&1` +case "$ver" in + 3.13.*) ok "$PY runs: $ver" ;; + *) bad "interpreter did not run: $ver"; exit 1 ;; +esac +note "site-packages: `ls $SP | wc -l` entries" + +echo +echo "=== 1. every package imports ===" +# One process, one import each, so a single failure names itself instead of +# stopping the list. +$PY - <<'EOP' > /tmp/imp.out 2>&1 +import importlib, sys +# libnacl is deliberately NOT here: it ctypes-loads libsodium, which lives +# only under /usr/prog on this box, and section 4 is about exactly that. +mods = ["tornado", "tornado.web", "jinja2", "markupsafe", "distro", + "inotify_simple", "dbus_next", "preprocess_cancellation", + "serial", "serial_asyncio", "cffi", "_cffi_backend", "greenlet", + "lmdb", "smart_open", "streaming_form_data", "PIL", "PIL.Image", + "setuptools", + "pkg_resources"] +bad = 0 +for m in mods: + try: + mod = importlib.import_module(m) + where = getattr(mod, "__file__", "?") or "?" + print(" ok %-24s %s" % (m, where.replace("/usr/data/anvil", "$"))) + except BaseException as exc: + print(" FAIL %-24s %r" % (m, exc)) + bad += 1 +print("import failures: %d" % bad) +EOP +sed -n 's/^/ /p' /tmp/imp.out +grep -q '^import failures: 0' /tmp/imp.out \ + && ok "every package imports on 3.13" \ + || bad "`grep '^import failures' /tmp/imp.out`" + +echo +echo "=== 2. the C extensions WORK, not merely import ===" + +# ---- cffi, the way klippy uses it ----------------------------------------- +# klippy's chelper does ffi.cdef() then ffi.dlopen(c_helper.so) -- ABI mode, +# no compiler. That matters here beyond convenience: there IS no compiler on +# this printer. So: declare a function, dlopen a real library, call it, and +# check the ANSWER. libm is used rather than a library of ours because it is +# on this rootfs whatever else is true. +$PY - <<'EOP' > /tmp/cffi.out 2>&1 +import cffi +ffi = cffi.FFI() +ffi.cdef("double sqrt(double); double pow(double, double);") +lib = ffi.dlopen("libm.so.6") +a = lib.sqrt(1024.0) +b = lib.pow(2.0, 10.0) +buf = ffi.new("char[]", b"nozzle") +print("sqrt(1024)=%r pow(2,10)=%r buf=%r" % (a, b, ffi.string(buf))) +assert a == 32.0 and b == 1024.0 and ffi.string(buf) == b"nozzle" +print("cffi ok") +EOP +grep -q '^cffi ok' /tmp/cffi.out \ + && ok "cffi cdef+dlopen+call through libm: `head -1 /tmp/cffi.out`" \ + || bad "cffi: `tail -3 /tmp/cffi.out`" + +# ---- greenlet, the way klippy's reactor uses it ---------------------------- +# The one that was expected to fight. An import proves the .so loaded; only a +# switch proves the hand-written MIPS stack-switching assembly in +# switch_mips_unix.h does what it says. So switch out, switch back, switch out +# again, and check the ORDER -- a stack switch that half-works produces a +# plausible-looking result in the wrong sequence. +$PY - <<'EOP' > /tmp/gl.out 2>&1 +import greenlet +seq = [] +def child(): + seq.append("child-1") + value = main.switch("up") + seq.append("child-2:%s" % value) + return "done" +main = greenlet.getcurrent() +g = greenlet.greenlet(child) +seq.append("main-1") +got = g.switch() +seq.append("main-2:%s" % got) +result = g.switch("down") +seq.append("main-3:%s" % result) +print("|".join(seq)) +assert seq == ["main-1", "child-1", "main-2:up", "child-2:down", "main-3:done"], seq +assert g.dead +print("greenlet ok") +EOP +grep -q '^greenlet ok' /tmp/gl.out \ + && ok "greenlet switches and returns in order: `head -1 /tmp/gl.out`" \ + || bad "greenlet: `tail -5 /tmp/gl.out`" + +# ---- lmdb, and the trap it is famous for here ------------------------------ +# phase 6's notes record a trimmed tree in which the lmdb egg fell back to its +# CFFI backend and tried to invoke mips-linux-gnu-gcc ON THE PRINTER at +# Moonraker startup. So this asserts WHICH backend loaded before it asserts +# that the database works. +$PY - <<'EOP' > /tmp/lmdb1.out 2>&1 +import lmdb, os, shutil +print("backend:", lmdb.__file__) +import lmdb.cpython +print("cpython backend loaded:", lmdb.cpython.__file__) +shutil.rmtree("/tmp/lm", ignore_errors=True) +os.makedirs("/tmp/lm") +env = lmdb.open("/tmp/lm", map_size=1 << 20) +with env.begin(write=True) as txn: + txn.put(b"nozzle", b"0.4") + txn.put(b"bed", b"220") +env.close() +print("wrote") +EOP +sed -n 's/^/ /p' /tmp/lmdb1.out +if grep -q 'cpython backend loaded' /tmp/lmdb1.out; then + ok "lmdb loaded its CPython extension, not the compile-at-import cffi path" +else + bad "lmdb backend: `tail -3 /tmp/lmdb1.out`" +fi +# A SECOND PROCESS, because a database that only exists inside the writer is +# not a database. +$PY - <<'EOP' > /tmp/lmdb2.out 2>&1 +import lmdb +env = lmdb.open("/tmp/lm", readonly=True) +with env.begin() as txn: + print("read back: nozzle=%s bed=%s" % (txn.get(b"nozzle"), txn.get(b"bed"))) + assert txn.get(b"nozzle") == b"0.4" and txn.get(b"bed") == b"220" +print("lmdb ok") +EOP +grep -q '^lmdb ok' /tmp/lmdb2.out \ + && ok "lmdb round-trips through a second process: `head -1 /tmp/lmdb2.out`" \ + || bad "lmdb: `tail -3 /tmp/lmdb2.out`" + +# ---- the two optional accelerators, and streaming-form-data --------------- +$PY - <<'EOP' > /tmp/spd.out 2>&1 +import markupsafe, tornado.escape +print("markupsafe accel:", markupsafe.escape.__module__) +print("escape:", str(markupsafe.escape("&"))) +assert str(markupsafe.escape("")) == "<a>" +try: + from tornado import speedups + print("tornado speedups:", speedups.__file__) +except ImportError as exc: + print("tornado speedups absent (pure fallback):", exc) +# The five names moonraker asks of streaming_form_data, and a real parse. +from streaming_form_data import StreamingFormDataParser +from streaming_form_data.targets import FileTarget, ValueTarget, SHA256Target +from streaming_form_data.validators import ValidationError +body = (b"--BOUND\r\nContent-Disposition: form-data; name=\"print\"\r\n\r\n" + b"benchy.gcode\r\n--BOUND--\r\n") +target = ValueTarget() +p = StreamingFormDataParser(headers={"Content-Type": "multipart/form-data; boundary=BOUND"}) +p.register("print", target) +p.data_received(body) +print("parsed:", target.value) +assert target.value == b"benchy.gcode" +print("speedups ok") +EOP +sed -n 's/^/ /p' /tmp/spd.out +grep -q '^speedups ok' /tmp/spd.out \ + && ok "markupsafe accelerator and streaming-form-data parse real input" \ + || bad "accelerators: `tail -4 /tmp/spd.out`" + +# ---- pillow, on the one path moonraker actually uses it for --------------- +# Not "import PIL". Moonraker's metadata subprocess pulls a base64 PNG +# thumbnail out of a gcode comment, decodes it, resizes it and writes it back +# out -- so this does exactly that, through a Pillow built with zlib and NO +# other codec. If the zlib link were wrong, the import would still succeed and +# the save would fail. +$PY - <<'EOP' > /tmp/pil.out 2>&1 +import base64, io +from PIL import Image, features +print("zlib codec:", features.check("zlib")) +img = Image.new("RGBA", (64, 48), (200, 30, 30, 255)) +for x in range(0, 64, 8): + for y in range(0, 48, 8): + img.putpixel((x, y), (10, 10, 240, 255)) +raw = io.BytesIO() +img.save(raw, format="PNG") +encoded = base64.b64encode(raw.getvalue()) +print("encoded png: %d bytes base64" % len(encoded)) +back = Image.open(io.BytesIO(base64.b64decode(encoded))) +back.load() +small = back.resize((32, 32)) +out = io.BytesIO() +small.save(out, format="PNG") +print("decoded %s %s -> resized %s, %d bytes" + % (back.format, back.size, small.size, len(out.getvalue()))) +assert back.size == (64, 48) and back.getpixel((0, 0)) == (10, 10, 240, 255) +assert small.size == (32, 32) and len(out.getvalue()) > 0 +print("pillow ok") +EOP +sed -n 's/^/ /p' /tmp/pil.out +grep -q '^pillow ok' /tmp/pil.out \ + && ok "pillow decodes, resizes and re-encodes a PNG thumbnail" \ + || bad "pillow: `tail -4 /tmp/pil.out`" + +# ---- cffi against klippy's REAL c_helper.so ------------------------------- +# The question phase 7 turns on, asked here because the answer is cheap: can +# 3.13's cffi bind the printer's own klippy helper? c_helper.so is a plain +# glibc shared object dlopen'd in ABI mode, so it should not care which python +# opened it -- "should not care" being exactly the kind of claim worth +# measuring on the box rather than asserting in a report. +CH=`find /usr/prog -name 'c_helper*.so' 2>/dev/null | head -1` +if [ -n "$CH" ]; then + note "found $CH" + $PY - "$CH" <<'EOP' > /tmp/chelper.out 2>&1 +import sys, cffi +ffi = cffi.FFI() +# A handful of klippy's own declarations, spelled the way chelper does. +ffi.cdef(""" + struct stepcompress *stepcompress_alloc(uint32_t oid); + struct steppersync *steppersync_alloc(struct serialqueue *sq + , struct stepcompress **sc_list, int sc_num, int move_num); + double itersolve_generate_steps(struct stepper_kinematics *sk, double flush_time); +""") +lib = ffi.dlopen(sys.argv[1]) +found = [] +for name in ("stepcompress_alloc", "steppersync_alloc", + "itersolve_generate_steps"): + try: + getattr(lib, name) # cffi resolves lazily: this is the dlsym + found.append(name) + except Exception as exc: + print(" %s: %r" % (name, exc)) +print("resolved: %s" % " ".join(found)) +sc = lib.stepcompress_alloc(0) +print("stepcompress_alloc(0) ->", "non-NULL" if sc else "NULL") +assert len(found) == 3 and sc +print("chelper ok") +EOP + sed -n 's/^/ /p' /tmp/chelper.out + grep -q '^chelper ok' /tmp/chelper.out \ + && ok "3.13's cffi dlopens the printer's own c_helper.so and calls into it" \ + || bad "c_helper.so through 3.13 cffi: `tail -4 /tmp/chelper.out`" +else + note "no c_helper.so on this image -- klippy binding not measured here" +fi + +echo +echo "=== 3. nothing maps a library under /usr/prog ===" +# The property the whole static-linking decision exists to buy, re-established +# with the extension tree in place: /usr/prog carries libffi.so.8 against the +# rootfs's .so.7, an openssl 1.0.2d, and a stock OTA can replace any of it. +# The process below has imported every native module first, so any mapping +# would already be in /proc/self/maps. +$PY - <<'EOP' > /tmp/maps.out 2>&1 +import ssl, ctypes, sqlite3, lzma, zlib, hashlib # the interpreter's +import cffi, _cffi_backend, greenlet, lmdb, markupsafe # ours +import streaming_form_data, tornado.web, jinja2 +from PIL import Image +libs = [l.split()[-1] for l in open("/proc/self/maps") + if ".so" in l and l.split()[-1].startswith("/")] +prog = sorted(set(l for l in libs if l.startswith("/usr/prog"))) +print("mapped libraries: %d" % len(set(libs))) +for l in sorted(set(libs)): + print(" %s" % l) +print("under /usr/prog: %d" % len(prog)) +EOP +sed -n 's/^/ /p' /tmp/maps.out +grep -q '^under /usr/prog: 0' /tmp/maps.out \ + && ok "no library under /usr/prog is mapped, with every extension imported" \ + || bad "`grep '^under /usr/prog' /tmp/maps.out`" + +echo +echo "=== 4. libnacl, and the one dependency that is NOT solved ===" +# libnacl is pure python -- it ctypes-loads libsodium, which this rootfs does +# not have and /usr/prog does. Nothing was cross-built to fix that, so this +# reports the state rather than asserting one, exactly as case-python.sh does +# for the missing CA bundle. +$PY - <<'EOP' > /tmp/nacl.out 2>&1 +import ctypes.util, os +print("ctypes.util.find_library('sodium') ->", ctypes.util.find_library("sodium")) +for p in ("/usr/lib/libsodium.so", "/lib/libsodium.so", + "/usr/prog/libsodium/lib/libsodium.so"): + print(" %-42s %s" % (p, "present" if os.path.exists(p) else "-")) +try: + import libnacl + print("libnacl imported, sodium at:", libnacl.nacl._name) +except BaseException as exc: + print("libnacl NOT importable: %r" % (exc,)) +EOP +sed -n 's/^/ /p' /tmp/nacl.out +if grep -q 'libnacl imported' /tmp/nacl.out; then + note "libnacl found libsodium without help" +else + note "libnacl needs libsodium and this rootfs has it ONLY under /usr/prog" +fi +# With /usr/prog on the path, the way anvil-env.sh puts it there today. +LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY -c \ + 'import libnacl; print("libnacl ok:", libnacl.nacl._name)' \ + > /tmp/nacl2.out 2>&1 +if grep -q '^libnacl ok' /tmp/nacl2.out; then + ok "libnacl works when /usr/prog/libsodium is on the path: `cat /tmp/nacl2.out`" +else + bad "libnacl even with libsodium on the path: `tail -3 /tmp/nacl2.out`" +fi + +echo +echo "=== 5. THE PRIZE: Moonraker's own component list, on 3.13 ===" +# Lifted from case-moonraker.sh section 7 and pointed at the new interpreter. +# The component list is NOT written down here: it comes from Moonraker's own +# CORE_COMPONENTS plus every section in the printer's moonraker.conf, which is +# what keeps this honest when the pin moves or somebody configures a component +# nobody thought about. +CONF=/usr/data/config/moonraker.conf +if [ ! -f "$CONF" ]; then + # No installed config in this run -- use the one the mod ships. + mkdir -p /usr/data/config + cat > "$CONF" <<'CFG' +[server] +[machine] +[file_manager] +[authorization] +[history] +[webcam anvil] +CFG + note "no installed moonraker.conf; using the mod's own section set" +fi +if [ -d "$MRROOT/moonraker" ]; then + LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" "$CONF" <<'PY' > /tmp/mr.out 2>&1 +import glob, importlib, os, re, sys +sys.path.insert(0, sys.argv[1]) +try: + import moonraker.server as server +except Exception as exc: + print("moonraker.server does not import: %r" % (exc,)) + raise SystemExit(2) + +names = list(getattr(server, "CORE_COMPONENTS", [])) + +def scan(path, depth=0): + try: + lines = open(path).readlines() + except OSError: + return + for line in lines: + section = re.match(r"\s*\[\s*([A-Za-z0-9_]+)", line) + if not section: + continue + if section.group(1) == "include": + inc = re.match(r"\s*\[\s*include\s+([^\]]+?)\s*\]", line) + if inc and depth < 3: + base = os.path.dirname(os.path.abspath(path)) + for f in sorted(glob.glob(os.path.join(base, inc.group(1)))): + scan(f, depth + 1) + continue + names.append(section.group(1)) + +scan(sys.argv[2]) + +failures = [] +checked = [] +for name in dict.fromkeys(names): + target = "moonraker.components." + name + try: + importlib.import_module(target) + checked.append(name) + except ModuleNotFoundError as exc: + # A section that is not a component at all ([server]) is fine. A + # missing DEPENDENCY of a component that does exist is the point. + if getattr(exc, "name", None) == target: + continue + failures.append((name, repr(exc))) + except Exception as exc: + failures.append((name, repr(exc))) + +print("imported: %s" % " ".join(sorted(checked))) +for name, err in failures: + print(" %s: %s" % (name, err)) +print("components ok: %d" % len(checked) if not failures + else "%d component(s) will not load" % len(failures)) +raise SystemExit(1 if failures else 0) +PY + sed -n 's/^/ /p' /tmp/mr.out + if grep -q '^components ok:' /tmp/mr.out; then + ok "`grep '^components ok:' /tmp/mr.out` on CPython 3.13" + else + bad "components will not load: `tail -5 /tmp/mr.out`" + fi + + echo + echo "--- 5b. moonraker's own entry point, and its server object ---" + # Importing components proves the dependencies are there. Building the + # Server is what proves the 3.13 stdlib underneath them still fits: this + # pin is a 2023 tree and asyncio, importlib and inspect have all moved. + LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" <<'PY' > /tmp/mr2.out 2>&1 +import sys +sys.path.insert(0, sys.argv[1]) +import moonraker.moonraker as entry +import moonraker.server, moonraker.confighelper, moonraker.common +import moonraker.eventloop, moonraker.loghelper, moonraker.utils +import moonraker.utils.json_wrapper as jw +print("entry:", entry.__file__) +print("json backend:", "msgspec" if jw.MSGSPEC_ENABLED else "stdlib json") +print("round trip:", jw.loads(jw.dumps({"state": "ready", "z": 0.4}))) +print("moonraker ok") +PY + sed -n 's/^/ /p' /tmp/mr2.out + grep -q '^moonraker ok' /tmp/mr2.out \ + && ok "moonraker's entry point and core modules import on 3.13" \ + || bad "moonraker entry: `tail -5 /tmp/mr2.out`" +else + note "no moonraker tree at $MRROOT -- pass mr.tgz=..." +fi + +echo +echo "=== 6. NEGATIVE CONTROL: take the tree away ===" +# Without this half, "the components import" is a fact about python in +# general. Move the cross-built site-packages aside and the SAME check must +# fail -- otherwise something else on this box is satisfying the imports and +# the tree is not what is being measured. +mv $SP $SP.away +LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" <<'PY' > /tmp/neg.out 2>&1 +import importlib, sys +sys.path.insert(0, sys.argv[1]) +missing = [] +for m in ("tornado", "lmdb", "greenlet", "cffi", "jinja2", "streaming_form_data"): + try: + importlib.import_module(m) + except ImportError: + missing.append(m) +print("unavailable without the tree: %s" % " ".join(missing)) +try: + importlib.import_module("moonraker.components.database") + print("database component STILL imported") +except BaseException as exc: + print("database component fails: %s" % type(exc).__name__) +PY +mv $SP.away $SP +sed -n 's/^/ /p' /tmp/neg.out +if grep -q 'unavailable without the tree: tornado lmdb greenlet cffi jinja2 streaming_form_data' /tmp/neg.out \ + && grep -q 'database component fails' /tmp/neg.out; then + ok "with the tree removed every third-party import and the database component fail" +else + bad "negative control did not fail as expected: `cat /tmp/neg.out`" +fi + +echo +echo "=== 7. no compiler is reachable, and nothing tried to use one ===" +# The failure mode phase 6 already recorded is a package deciding to BUILD +# something at import time. There is no compiler here, which is the reason +# that failure was fatal rather than slow. +for c in cc gcc mips-linux-gnu-gcc; do + if command -v $c >/dev/null 2>&1; then + bad "$c is on PATH in the replica -- this test would not catch a runtime build" + else + note "$c: absent, as on a real printer" + fi +done + +echo +echo "=== 8. size of what would be added ===" +note "site-packages: `du -sk $SP | cut -f1` KB, `find $SP -type f | wc -l` files" +note "interpreter : `du -sk $MODDIR/lib/python3.13 | cut -f1` KB total with it" + +echo +echo "=== 9. is setuptools/pkg_resources actually earning its 4.6 MB? ===" +# It is a third of the tree, and it is here for ONE reason: phase 6 recorded a +# trimmed tree in which the lmdb EGG lost pkg_resources, fell back to lmdb's +# cffi path, and tried to run a compiler on the printer at Moonraker startup. +# But this build installs lmdb from a WHEEL with a real CPython extension, not +# an egg -- so the question is whether the insurance is still insuring +# anything. Measured rather than reasoned about: take it away and re-run the +# two things that would break. +mv $SP/setuptools $SP/pkg_resources $SP/_distutils_hack /tmp/ 2>/dev/null +mv $SP/distutils-precedence.pth /tmp/ 2>/dev/null +LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" "$CONF" <<'PY' > /tmp/nosetup.out 2>&1 +import glob, importlib, os, re, sys +sys.path.insert(0, sys.argv[1]) +import lmdb, lmdb.cpython +print("lmdb backend without setuptools:", lmdb.cpython.__file__) +import moonraker.server as server +names = list(getattr(server, "CORE_COMPONENTS", [])) +for line in open(sys.argv[2]): + m = re.match(r"\s*\[\s*([A-Za-z0-9_]+)", line) + if m and m.group(1) != "include": + names.append(m.group(1)) +failures = [] +checked = [] +for name in dict.fromkeys(names): + target = "moonraker.components." + name + try: + importlib.import_module(target) + checked.append(name) + except ModuleNotFoundError as exc: + if getattr(exc, "name", None) == target: + continue + failures.append((name, repr(exc))) + except Exception as exc: + failures.append((name, repr(exc))) +for name, err in failures: + print(" %s: %s" % (name, err)) +print("components ok without setuptools: %d" % len(checked) if not failures + else "%d component(s) need setuptools" % len(failures)) +PY +mv /tmp/setuptools /tmp/pkg_resources /tmp/_distutils_hack $SP/ 2>/dev/null +mv /tmp/distutils-precedence.pth $SP/ 2>/dev/null +sed -n 's/^/ /p' /tmp/nosetup.out +if grep -q '^components ok without setuptools: 16' /tmp/nosetup.out; then + note "setuptools/pkg_resources are NOT needed at runtime by this build --" + note " 4.9 MB of the tree is insurance against a failure mode (the lmdb" + note " EGG's cffi fallback) that installing lmdb as a wheel already removes." + note " Dropping them is a size decision, not a correctness one." +else + ok "setuptools/pkg_resources earn their place: `tail -2 /tmp/nosetup.out`" +fi + +echo +[ $FAIL -eq 0 ] && echo "ALL PASS" || echo "FAILURES" +exit $FAIL diff --git a/tools/python-packages/README.md b/tools/python-packages/README.md new file mode 100644 index 0000000..794b715 --- /dev/null +++ b/tools/python-packages/README.md @@ -0,0 +1,137 @@ +# The Python packages Moonraker and klippy need, cross-built for mipsel + +The second half of phase 6. `tools/python/` builds the interpreter; +this builds what runs on it. **Not yet wired into `bin/patch.sh`** -- see +"What is left" at the bottom. `build.sh` is the measurement harness, preserved +here because it is where the findings live, exactly as `tools/supervisor/` +is for s6. + +Everything below was run in the printer replica on the real +`rootfs.squashfs`, not read off a project page. +`test/integration/printer/case-pyext.sh` is that run, kept. + +## What builds + +19 packages, 12 mipsel extension modules, from a 56-second clean build in one +throwaway `debian:bookworm`. Every `.so` is gated at `e_flags=0x70001407`, +`nan2008/o32/mips32r2` -- shared objects carry `EF_MIPS_PIC`, so they read +`…07` where an executable reads `…05`. + +Native: cffi, greenlet, lmdb, markupsafe, streaming-form-data, tornado +speedups, pillow. Pure: jinja2, distro, inotify-simple, libnacl, dbus-next, +preprocess-cancellation, pyserial, pyserial-asyncio, pycparser, smart_open, +setuptools. + +**The list is deliberately not `moonraker-requirements.txt`**, which installs +every optional component's dependencies. It is the closure of what the +*enabled* component set imports -- `moonraker.conf`'s sections plus +`CORE_COMPONENTS` -- plus klippy's. So no apprise, ldap3, paho-mqtt, zeroconf +or python-periphery. `msgspec` and `uvloop` are skipped on purpose: Moonraker +guards both with `contextlib.suppress(ImportError)` and falls back to the +stdlib, measured. + +## The method + +No crossenv. The staged interpreter already carries +`_sysconfigdata__linux_mipsel-linux-gnu`, so exporting +`_PYTHON_SYSCONFIGDATA_NAME` at an x86-64 build-python **of the same 3.13.7** +makes setuptools answer every question for the target. The stage is +bind-mounted at `/usr/data/anvil` so `INCLUDEPY` resolves unmodified. Plus the +gcc-wrapper trick from `tools/python/build.sh`, which is what gets +`-EL -mnan=2008` onto the link lines. + +## What fought back, and what it cost + +* **greenlet was the expected blocker, for an unexpected reason.** The MIPS + stack-switching assembly was never a problem. greenlet 3.x is C++ and writes + its `PyTypeObject`s as designated initializers, which gcc 7.2 accepts only + in declaration order with no gaps: + + sorry, unimplemented: non-trivial designated initializers not supported + + The exact rule was probed rather than assumed: in-order contiguous is fine, + a *trailing* gap is fine, an *interior* or *leading* gap is refused. + `fill-designators.py` writes the skipped fields back as explicit zeros, + taking field order from the target's own headers and refusing to guess when + a designator is not in them. 55 fields across two files, semantically a + no-op -- static storage duration, those fields were already zero -- and the + result switches out and back, in order, on the printer. + +* **pip silently downloaded x86-64 manylinux wheels** for cffi, greenlet and + tornado on the first run. Three wrong-architecture `.so` files sailed + straight through the build and only the ABI gate caught them. + `--no-binary :all:` is mandatory here, not caution. + +* **lmdb chose its cffi backend** and produced a `py3-none-any` wheel with + `mdb.c` inside -- which is phase 6's recorded trap rebuilt from scratch: on + the printer that path tries to invoke a compiler that does not exist. + `LMDB_FORCE_CPYTHON=1` fixes it. Note `LMDB_FORCE_CFFI=0` makes it *worse*: + the variable is tested for presence, so `"0"` selects cffi. + +* **pillow 10.3.0 cannot be built by any 3.13**, cross or not: its `setup.py` + does `exec(...)` then reads `locals()["__version__"]`, which PEP 667 broke. + 11.0.0 is the first version that works. + +* **streaming-form-data 1.13.0** (Moonraker's pin) ships pregenerated Cython C + calling the 4-argument `_PyLong_AsByteArray`, which grew a fifth parameter in + 3.13. 1.19.1 instead, which then wants `smart_open` at module scope. + +## What it proves + +Beyond "it imports", which proves very little on its own: + +* cffi does `cdef` + `dlopen` + call through libm -- klippy's exact ABI-mode + usage. +* **cffi dlopens the printer's own `/usr/prog/klipper/klippy/chelper/c_helper.so`, + resolves `stepcompress_alloc`, `steppersync_alloc` and + `itersolve_generate_steps`, and calls one.** That is phase 7's central + question -- whether a Python we built can drive FlashForge's C helper -- + answered yes, and it is the reason the interpreter had to be glibc and not + musl. +* lmdb writes and a *second process* reads back. +* pillow decodes, resizes and re-encodes a PNG: the real thumbnail path. +* **all 16 Moonraker components import on 3.13**, the list taken from + `CORE_COMPONENTS` + `moonraker.conf` rather than written down here. +* zero `/usr/prog` libraries mapped, with every extension imported. +* Negative control: move the tree aside and every third-party import fails. + +## Size + +13.5MB built, 2.9MB gzipped, 1073 files; 18.2MB on the printer once +`__pycache__` exists. With the interpreter, about 50MB. + +`setuptools`/`pkg_resources` are 4.9MB of that -- 36% -- and are not needed at +runtime: measured, with them removed lmdb still loads its CPython extension +and all 16 components still import. The phase-6 trap was the lmdb *egg*'s cffi +fallback, and installing lmdb as a wheel removes it. Dropping them is a size +decision, not a correctness one. + +## The one unsolved dependency: libsodium + +`libnacl` is pure Python and ctypes-loads libsodium, which exists on this box +**only at `/usr/prog/libsodium/lib`**. Measured both ways: it fails with an +empty path and works with that directory on it. So Moonraker's +`authorization` component keeps one `/usr/prog` dependency, and it is the last +one in the picture. Either cross-build libsodium into the prefix, or accept +`anvil-env.sh` keeping that single entry. + +## What is left before FF_PYTHON can switch + +1. **Boot Moonraker on 3.13 and hit the HTTP API.** Every component imports + and the entry point loads, but importing is not serving: nothing has bound + :7125 on 3.13. `case-moonraker.sh` already does this against 3.8; pointing + it at 3.13 is the next gate and the honest remaining unknown. +2. Decide libsodium, above. +3. Wire into `bin/patch.sh`. Note the build needs the **untrimmed** interpreter + stage -- `include/` and `config-3.13-*`, which section 5c's trim currently + deletes -- and the pins want moving into `versions.env` with sha256s. +4. klippy is **not** blocked by packages: cffi, greenlet, jinja2, markupsafe + and pyserial all build, and `c_helper.so` binds. `numpy` is the only gap, + wanted solely by `extras/stepper_resonance_tester.py`, which Klipper loads + on demand. numpy on 3.13 means numpy >= 2.1 and a Meson cross-file, and + that genuinely is a multi-day rabbit hole. It costs input-shaper + calibration, not printing. + +Unverified: no real hardware, replica only; nothing has run under sustained +load; and the greenlet designator patch is a no-op by argument plus a working +switch test, not by upstream blessing. diff --git a/tools/python-packages/build.sh b/tools/python-packages/build.sh new file mode 100755 index 0000000..af0ccff --- /dev/null +++ b/tools/python-packages/build.sh @@ -0,0 +1,425 @@ +#!/bin/bash +# --------------------------------------------------------------------------- +# Cross-build the third-party Python packages Moonraker and klippy import, +# for the CPython 3.13 that tools/python/build.sh already produces for the +# FlashForge Creator 5 Pro (Ingenic mipsel). +# +# SPIKE. Nothing here is wired into bin/patch.sh yet. +# +# Same non-negotiables as tools/python/build.sh, for the same reasons: +# +# * e_flags 0x70001405 for an EXEC, 0x70001407 for a DYN (EF_MIPS_PIC). +# Every .so this produces is gated; a legacy-NaN wheel imports fine on the +# host and fails on the printer with nothing but ENOEXEC to say why. +# * -EL -mnan=2008 must reach the COMPILE and the LINK of every object. +# setuptools does not forward CFLAGS to its link line, so the flags are +# baked into PATH wrappers around the gcc driver and no build system gets +# a vote. +# * glibc toolchain only. musl is forbidden -- klippy dlopens a glibc +# c_helper.so. +# +# THE CROSS TRICK. There is no crossenv here. The staged interpreter already +# carries _sysconfigdata__linux_mipsel-linux-gnu.py, which records the cross +# CC, LDSHARED, EXT_SUFFIX (.cpython-313-mipsel-linux-gnu.so) and INCLUDEPY +# (/usr/data/anvil/include/python3.13). Exporting _PYTHON_SYSCONFIGDATA_NAME +# at an x86-64 CPython 3.13 of the SAME version makes sysconfig -- and so +# setuptools' build_ext -- answer every question about the TARGET. The staged +# tree is bind-mounted at its real prefix so INCLUDEPY resolves without +# rewriting anything. +# +# WHAT IS BUILT AND WHY THAT LIST. Not moonraker-requirements.txt, which +# installs every optional component's dependency. The list is what the +# ENABLED component set imports -- assets/moonraker.conf's sections plus +# server.CORE_COMPONENTS, closed over moonraker's own imports (see +# scan-imports.py beside this file) -- plus what klippy needs. +# +# Usage: +# ./build-pyext.sh # in docker +# ./build-pyext.sh --in-container +# --------------------------------------------------------------------------- +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +PYBUILD="${PYBUILD:-$HERE/../pybuild}" +REPO="${REPO:-/home/shish/firmware/.claude/worktrees/s6-vs-runit}" +TOOLCHAIN_HOST="$REPO/work/.mips-toolchain/mips-gcc720-glibc229" + +if [ "${1:-}" != "--in-container" ]; then + [ -x "$TOOLCHAIN_HOST/bin/mips-linux-gnu-gcc" ] || { + echo "!! no toolchain at $TOOLCHAIN_HOST" >&2; exit 1; } + [ -x "$PYBUILD/hostpy/bin/python3.13" ] || { + echo "!! no build-python at $PYBUILD/hostpy -- run tools/python/build.sh" >&2; exit 1; } + [ -d "$PYBUILD/stage/usr/data/anvil/include/python3.13" ] || { + echo "!! no UNTRIMMED stage at $PYBUILD/stage -- the shipped tree has no headers" >&2; exit 1; } + exec docker run --rm \ + -v "$TOOLCHAIN_HOST":/toolchain:ro \ + -v "$PYBUILD":/work \ + -v "$PYBUILD/stage/usr/data/anvil":/usr/data/anvil:ro \ + -v "$HERE":/pyext \ + -e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \ + -e ONLY="${ONLY:-}" \ + -w /pyext debian:bookworm \ + bash /pyext/build-pyext.sh --in-container +fi + +# ======================= everything below runs in the container ============= +START=$(date +%s) +export DEBIAN_FRONTEND=noninteractive +TC=/toolchain +DEP=/work/deproot # the static C libraries CPython was built on +HOSTPY=/work/hostpy/bin/python3.13 +P=/pyext +SRC=$P/src # sdist cache +WHEELS=$P/wheels # built wheels +SP=$P/site-packages # the tree that would ship +LOGS=$P/logs +mkdir -p "$SRC" "$WHEELS" "$SP" "$LOGS" + +log() { echo; echo "===== $* ====="; } + +log "apt" +apt-get update -qq +apt-get install -y -qq --no-install-recommends \ + build-essential curl ca-certificates pkg-config file xz-utils patch >/dev/null + +# --------------------------------------------------------- the wrappers ---- +# Identical to tools/python/build.sh. This is the single most important trick +# in either file. +log "toolchain wrappers (-EL -mnan=2008 baked in)" +mkdir -p /opt/xw/bin +for t in gcc g++ cpp; do + cat > /opt/xw/bin/mips-linux-gnu-$t < /tmp/abi.c +mips-linux-gnu-gcc /tmp/abi.c -o /tmp/abi.out +FLAGS=$(mips-linux-gnu-readelf -h /tmp/abi.out | awk '/Flags:/{print $2}' | tr -d ,) +echo "wrapper produces e_flags=$FLAGS" +[ "$FLAGS" = "0x70001405" ] || { echo "!! wrong ABI from the wrapper"; exit 1; } + +# ------------------------------------------------------------ pip ---------- +# The build-python was configured --without-ensurepip, so it has no pip. It is +# the SAME 3.13.7 the cross build used, which is what makes its sysconfig +# machinery interchangeable with the target's; a distro python3.11 is not. +if ! "$HOSTPY" -c 'import pip' 2>/dev/null; then + log "bootstrap pip into the build-python" + curl -sSLf -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py + "$HOSTPY" /tmp/get-pip.py -q +fi +# Every package below is built from its SDIST with build isolation OFF, so +# every PEP 517 backend any of them names has to be here in advance. That is +# not a convenience: with isolation ON, pip rewrites PYTHONPATH for the build +# subprocess, the cross _sysconfigdata module stops being importable, and the +# build either dies ("No module named '_sysconfigdata__linux_mipsel-linux-gnu'") +# or -- worse -- silently answers every sysconfig question for x86-64. +"$HOSTPY" -m pip install -q --upgrade \ + 'setuptools>=68' wheel 'cython<3.1' flit_core poetry-core hatchling \ + hatch-vcs setuptools-scm +"$HOSTPY" -VV + +# -------------------------------------------------------- cross env -------- +log "cross environment" +export _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_mipsel-linux-gnu +export PYTHONPATH=/usr/data/anvil/lib/python3.13 +export PYTHONDONTWRITEBYTECODE=1 +export CC=mips-linux-gnu-gcc +export CXX=mips-linux-gnu-g++ +export AR=mips-linux-gnu-ar +export RANLIB=mips-linux-gnu-ranlib +export STRIP=mips-linux-gnu-strip +export LDSHARED="mips-linux-gnu-gcc -shared -L$DEP/lib" +export CFLAGS="-O2 -fPIC -D_FILE_OFFSET_BITS=64 -I$DEP/include -I/usr/data/anvil/include/python3.13" +export CXXFLAGS="$CFLAGS" +export LDFLAGS="-L$DEP/lib" +# pkg-config must see ONLY the cross sysroot, never the container's /usr/lib. +export PKG_CONFIG_LIBDIR="$DEP/lib/pkgconfig" +export PKG_CONFIG_PATH="$DEP/lib/pkgconfig" + +"$HOSTPY" - <<'PY' +import sysconfig +for k in ("EXT_SUFFIX", "SOABI", "INCLUDEPY", "CC", "LDSHARED"): + print(" %-11s %s" % (k, sysconfig.get_config_var(k))) +assert sysconfig.get_config_var("EXT_SUFFIX") == ".cpython-313-mipsel-linux-gnu.so", \ + "sysconfig is answering for the HOST -- _PYTHON_SYSCONFIGDATA_NAME did not take" +PY + +# ------------------------------------------------------------ helpers ------ +want() { [ -z "${ONLY:-}" ] && return 0; case " $ONLY " in *" $1 "*) return 0;; *) return 1;; esac; } + +# build [env=val ...] +# +# --no-binary :all: is not optional and not caution. Without it pip HELPFULLY +# downloads a prebuilt manylinux wheel and the "cross build" of cffi, greenlet +# and tornado silently becomes x86-64 -- measured, first run of this script: +# three x86-64 .so files sailed into the tree and only the ABI gate caught +# them. A wheel that is the wrong architecture is not a build failure anywhere +# except on the printer. +# +# --no-build-isolation is what keeps _PYTHON_SYSCONFIGDATA_NAME reaching +# setup.py; see the backend pre-install above. +build() { + local spec=$1 name=${1%%[=<>]*}; shift + want "$name" || return 0 + echo "-- $spec" + if env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \ + --no-binary :all: -w "$WHEELS" "$spec" \ + >"$LOGS/$name.log" 2>&1; then + echo " ok" + else + echo " !! FAILED -- $LOGS/$name.log"; tail -25 "$LOGS/$name.log" + FAILED="$FAILED $name" + fi +} + +# sdist +# +# Two packages cannot come from `pip wheel `, for unrelated reasons, and +# both need their unpacked source anyway. Fetched by URL and CHECKED, the way +# versions.env pins everything else in this repo: the only thing between a +# source URL and code the printer executes is the hash. +sdist() { + local url=$1 want=$2 dir=$3 f="$SRC/$(basename "$1")" + [ -s "$f" ] || curl -sSLf -o "$f" "$url" + local got + got=$(sha256sum "$f" | cut -d" " -f1) + [ "$got" = "$want" ] || { echo "!! $(basename "$f") sha256 $got, expected $want"; exit 1; } + rm -rf "${SRC:?}/$dir" + tar -xzf "$f" -C "$SRC" + echo "$SRC/$dir" +} + +# builddir [env=val ...] -- wheel an already-unpacked source tree. +builddir() { + local dir=$1 name; name=$(basename "$dir"); shift + echo "-- $name (from source tree)" + if env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \ + -w "$WHEELS" "$dir" >"$LOGS/$name.log" 2>&1; then + echo " ok" + else + echo " !! FAILED -- $LOGS/$name.log"; tail -25 "$LOGS/$name.log" + FAILED="$FAILED $name" + fi +} + +FAILED="" + +# ============================================================ packages ====== +# THE LIST IS NOT moonraker-requirements.txt. That file installs every +# optional component's dependency -- apprise, ldap3, paho-mqtt, zeroconf, +# python-periphery, smart_open, dbus-next's mqtt friends. None of those +# components is configured in assets/moonraker.conf, so none of them is built +# here. What IS built is the closure of what the ENABLED set imports +# (scan-imports.py), plus klippy's three. +log "packages" + +# ---- pure python (no compiler is invoked; they are here to be PRESENT) ---- +build "distro==1.9.0" +build "inotify-simple==1.3.5" +build "libnacl==2.1.0" +build "dbus-next==0.2.3" +# preprocess-cancellation 0.2.1's sdist declares version 0.0.0 in its own +# metadata -- a broken upstream release, not something about this target. pip +# refuses it ("has inconsistent version: expected '0.2.1', but metadata has +# '0.0.0'") for the specifier, so the tree is fetched by URL, hash-checked, +# and wheeled as a directory, where pip has no specifier to disagree with. It +# is a lazily-imported optional path inside the metadata SUBPROCESS +# (gcode object cancellation), so a printer without it loses that feature and +# nothing else. +if want preprocess-cancellation; then + builddir "$(sdist \ + https://files.pythonhosted.org/packages/4a/56/7e18b0336c1e6c6622411dd0d3a7634b171e4d156a13b1ceaa048682454a/preprocess_cancellation-0.2.1.tar.gz \ + e2f1224e1ba1603bdfdbf6937caaf91082dc849e5122e80a5328aa999433ce79 \ + preprocess_cancellation-0.2.1)" +fi +build "pyserial==3.4" +build "pyserial-asyncio==0.6" +build "pycparser==2.22" # cffi's only runtime dependency +# smart_open is not optional, whatever its name suggests, and it is not +# moonraker that wants it: streaming-form-data's targets.py imports it at +# module scope, so `import streaming_form_data` raises ModuleNotFoundError +# without it -- measured in the replica, where every other package imported +# and that one did not. moonraker-requirements.txt pins it (`smart_open<=6.4.0`) +# for exactly this reason. Pure python, no dependencies of its own. +build "smart_open==6.4.0" +build "jinja2==3.1.4" +# setuptools/pkg_resources are a dependency of nothing here. They are the fix +# for a measured failure: a trimmed tree without them made the lmdb egg fall +# back to its cffi path and try to invoke mips-linux-gnu-gcc ON THE PRINTER at +# Moonraker startup. There is no compiler on the printer. +build "setuptools==75.6.0" + +# ---- native --------------------------------------------------------------- +# tornado's C speedup is optional and its published wheel carries an x86-64 +# copy of it, so this builds from sdist like everything else and gets a mipsel +# one. +build "tornado==6.4.0" + +# markupsafe -- jinja2's escape accelerator, on the path of every gcode macro +# klippy renders. Pure-python fallback exists (_native.py) if it ever fights. +build "markupsafe==2.1.5" + +# cffi -- klippy's chelper. Wants libffi; deproot has a static -fPIC one and a +# .pc for it, and PKG_CONFIG_LIBDIR makes that the only one it can see. +build "cffi==1.17.1" + +# greenlet -- klippy's reactor. Expected to fight over its hand-written +# stack-switching assembly (switch_mips_unix.h); fought over something else +# entirely, and the assembly was never the problem. +# +# greenlet 3.x is C++ and writes every PyTypeObject / PyNumberMethods as a +# DESIGNATED INITIALIZER. Those are C++20; gcc 7.2 -- the Ingenic toolchain, +# and the only compiler that produces this printer's ABI -- takes them as an +# extension but only in declaration order with no gaps, and refuses anything +# else with +# +# sorry, unimplemented: non-trivial designated initializers not supported +# +# A PyTypeObject naming 20 of its 50 slots is nothing but gaps, so all three +# of greenlet's fail. fill-designators.py writes the skipped fields back as +# explicit zeros, reading the field ORDER out of the target interpreter's own +# headers rather than a list typed in here -- and refuses to guess if a +# designator turns up that the header does not have. These are objects of +# static storage duration, so every field it inserts was already zero; the +# patch changes the spelling, not the program. +if want greenlet; then + GL=$(sdist https://files.pythonhosted.org/packages/2f/ff/df5fede753cc10f6a5be0931204ea30c35fa2f2ea7a35b25bdaf4fe40e46/greenlet-3.1.1.tar.gz \ + 4ce3ac6cdb6adf7946475d7ef31777c26d94bccc377e070a7986bd2d5c515467 \ + greenlet-3.1.1) + "$HOSTPY" /pyext/fill-designators.py /usr/data/anvil/include/python3.13 \ + $(find "$GL/src/greenlet" -name '*.cpp' -o -name '*.hpp' | sort) + builddir "$GL" +fi + +# lmdb -- moonraker's database AT THIS PIN, and the package that disappears +# the moment the pin moves past 80c7620 (lmdb -> sqlite), which is the whole +# point of having built an interpreter with _sqlite3. +# +# LMDB_FORCE_CPYTHON=1 is load-bearing. lmdb's setup.py has two backends: a +# real CPython extension and a cffi one that ships mdb.c and COMPILES IT AT +# IMPORT TIME. Left to itself here it chose cffi and produced a py3-none-any +# wheel containing mdb.c -- i.e. exactly the failure phase 6 already recorded, +# rebuilt from scratch. (Do not "fix" this with LMDB_FORCE_CFFI=0: the setup +# script tests the variable for PRESENCE, so the string "0" selects cffi.) +build "lmdb==1.4.1" LMDB_FORCE_CPYTHON=1 + +# streaming-form-data -- moonraker's upload parser, Cython. 1.13.0 (what +# moonraker-requirements.txt pins) ships a pregenerated _parser.c from a +# Cython that predates 3.13: it calls the four-argument _PyLong_AsByteArray, +# which grew a fifth parameter in 3.13, and every integer conversion in the +# file fails to compile. Regenerating from the .pyx with a current Cython is +# the fix, and a later release ships C that is already regenerated -- moonraker +# only asks five names of it (StreamingFormDataParser, ParseFailedException, +# FileTarget, ValueTarget, SHA256Target), all present throughout the 1.x line. +build "streaming-form-data==1.19.1" + +# pillow -- Moonraker's ONLY use of it is +# components/file_manager/metadata.py, which runs as a SUBPROCESS to pull +# thumbnails out of gcode. So it is not on the startup path: without it +# Moonraker runs and Mainsail shows no thumbnails. It is built anyway because +# the cost turned out to be one flag list. +# +# ZLIB ONLY. Gcode thumbnails are base64 PNG, so zlib is the only codec on +# that path; jpeg, tiff, webp, jpeg2000, lcms, freetype and imagequant are +# libraries nobody has cross-built here and every one of them is off. Pillow +# probes for them by trying to LINK against the host's copies unless told not +# to, which is what --disable-platform-guessing is for. +# +# NOT 10.3.0, which is what moonraker-requirements.txt pins. That release +# cannot be built by python 3.13 AT ALL, and not for any reason to do with +# this target: its setup.py reads its own version with +# +# exec(compile(open("src/PIL/_version.py").read(), ...)) +# return locals()["__version__"] +# +# and PEP 667 (new in 3.13) made a function's locals() a snapshot, so the +# name exec() defined is not there any more and setup.py dies with +# KeyError: '__version__' before it compiles a single file. 11.0.0 is the +# first line that supports 3.13. Moonraker asks PIL for Image.open, resize +# and save, all unchanged across that gap. +# +# pip cannot drive this: the disable flags are build_ext options and there is +# no PEP 517 path to them, so setup.py is called directly. +if want pillow; then + PIL_SRC=$(sdist https://files.pythonhosted.org/packages/a5/26/0d95c04c868f6bdb0c447e3ee2de5564411845e36a858cfd63766bc7b563/pillow-11.0.0.tar.gz \ + 72bacbaf24ac003fea9bff9837d1eedb6088758d41e100c1552930151f677739 \ + pillow-11.0.0) + echo "-- pillow 11.0.0 (zlib only, setup.py direct)" + if ( cd "$PIL_SRC" && "$HOSTPY" setup.py build_ext \ + --disable-jpeg --disable-tiff --disable-webp --disable-jpeg2000 \ + --disable-imagequant --disable-lcms --disable-freetype \ + --disable-xcb --disable-platform-guessing --enable-zlib \ + -I"$DEP/include" -L"$DEP/lib" bdist_wheel ) \ + >"$LOGS/pillow.log" 2>&1; then + cp -a "$PIL_SRC"/dist/*.whl "$WHEELS/" + echo " ok" + else + echo " !! FAILED -- $LOGS/pillow.log"; tail -25 "$LOGS/pillow.log" + FAILED="$FAILED pillow" + fi +fi + +# =========================================================== assemble ======= +log "wheels built" +ls -1 "$WHEELS" | sort + +log "unpack into a site-packages tree" +rm -rf "$SP"; mkdir -p "$SP" +for w in "$WHEELS"/*.whl; do + "$HOSTPY" -m zipfile -e "$w" "$SP" +done +find "$SP" -name '__pycache__' -prune -exec rm -rf {} + 2>/dev/null || true +find "$SP" -name '*.dist-info' -prune -exec rm -rf {} + 2>/dev/null || true +rm -rf "$SP"/bin "$SP"/*.data 2>/dev/null || true + +# ============================================================= gates ======== +log "ABI gate on every .so in the tree" +BAD=0; N=0 +while IFS= read -r f; do + case "$(file -b "$f")" in *ELF*) ;; *) continue;; esac + hdr=$(mips-linux-gnu-readelf -h "$f" 2>/dev/null) + fl=$(echo "$hdr" | awk '/Flags:/{print $2}' | tr -d ,) + typ=$(echo "$hdr" | awk '/^ Type:/{print $2}') + want=0x70001405; [ "$typ" = "DYN" ] && want=0x70001407 + case "$hdr" in + *nan2008*o32*mips32r2*) ;; + *) echo "!! $f not nan2008/o32/mips32r2"; BAD=1; continue;; + esac + [ "$fl" = "$want" ] || { echo "!! $f type=$typ flags=$fl want=$want"; BAD=1; } + N=$((N+1)) +done < <(find "$SP" -type f -name '*.so') +[ $BAD -eq 0 ] || exit 1 +echo "$N mipsel extension modules, all nan2008/o32/mips32r2 DYN 0x70001407" + +echo +echo "-- NEEDED of each extension module (nothing may want a /usr/prog soname) --" +find "$SP" -name '*.so' | sort | while read -r f; do + printf '%-58s %s\n' "${f#$SP/}" \ + "$(mips-linux-gnu-readelf -d "$f" | awk '/NEEDED/{gsub(/[][]/,"",$5); printf "%s ", $5}')" +done + +log "strip and size" +find "$SP" -name '*.so' -exec mips-linux-gnu-strip {} + 2>/dev/null || true +du -sh "$SP" +echo "$(find "$SP" -type f | wc -l) files" +echo "-- top level --" +ls -1 "$SP" + +log "tarball" +# Root is site-packages/ so a case script can drop it beside the interpreter's +# own lib/python3.13/site-packages. +tar -czf "$P/pyext.tgz" -C "$P" site-packages +ls -l "$P/pyext.tgz" + +chown -R "${HOST_UID:-0}:${HOST_GID:-0}" "$P" 2>/dev/null || true + +END=$(date +%s) +log "done in $(( (END-START)/60 ))m $(( (END-START)%60 ))s" +[ -z "$FAILED" ] && echo "all packages built" || { echo "!! FAILED:$FAILED"; exit 1; } diff --git a/tools/python-packages/fill-designators.py b/tools/python-packages/fill-designators.py new file mode 100755 index 0000000..7c6c3fa --- /dev/null +++ b/tools/python-packages/fill-designators.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +"""Make C++ designated initializers digestible by gcc 7.2. + + fill-designators.py [file.cpp ...] + +WHY THIS EXISTS. greenlet 3.x writes its PyTypeObject, PyNumberMethods and +friends as C++ designated initializers: + + PyTypeObject PyGreenlet_Type = { + .ob_base=PyVarObject_HEAD_INIT(NULL, 0) + .tp_name="greenlet.greenlet", + .tp_basicsize=sizeof(PyGreenlet), + .tp_dealloc=(destructor)green_dealloc, + ... + +Designated initializers are C++20. gcc 7.2 -- the Ingenic toolchain, and the +only compiler this printer's ABI has -- supports them as an extension, but +only IN DECLARATION ORDER AND CONTIGUOUS. Measured, probe-desig.sh: + + in order, contiguous OK + in order, TRAILING gap OK -- the only gap gcc 7.2 tolerates + in order, INTERIOR gap sorry, unimplemented: non-trivial designated + initializers not supported + in order, LEADING gap same -- {.nb_bool=x} on a PyNumberMethods whose + first field is nb_add is already too much + out of order same + +Every one of greenlet's initializers skips fields -- a PyTypeObject naming 20 +of its 50 slots is the normal way to write one -- so all of them are interior +gaps, and the build stops on the first. + +WHAT THIS DOES. It reads the field order out of the TARGET interpreter's own +headers and writes the skipped fields back in as explicit zeros. That is a +no-op semantically: these are objects of static storage duration, so every +field this inserts was already going to be zero-initialized. It is not a +behaviour patch, it is the same initializer spelled in a way gcc 7.2 will +accept, and if a designator ever appears that is NOT in the header's field +list the script fails loudly rather than guessing. + +Every field it fills is a pointer or an integer, so `= 0` is the right zero; +the script asserts that too, rather than trusting it. +""" +import os +import re +import sys + +INCLUDE = sys.argv[1] +FILES = sys.argv[2:] + +# The initializer types greenlet uses. Each is (C type name as written in the +# source, the header declaration to read the field order from). +STRUCTS = { + "PyTypeObject": ("cpython/object.h", "struct _typeobject"), + "PyNumberMethods": ("cpython/object.h", "typedef struct"), + "PyAsyncMethods": ("cpython/object.h", "typedef struct"), + "PySequenceMethods": ("cpython/object.h", "typedef struct"), + "PyMappingMethods": ("cpython/object.h", "typedef struct"), + "PyModuleDef": ("moduleobject.h", "struct PyModuleDef"), +} + +# A field declaration line. Two shapes in CPython's headers that a naive +# regex gets wrong, and both appear in PyTypeObject: +# * `PyObject_VAR_HEAD` -- a MACRO, not a declaration; it expands to the +# `ob_base` field the source designates by name. +# * `Py_ssize_t tp_basicsize, tp_itemsize;` -- two declarators on one line. +FIELD = re.compile(r"^\s*(?:const\s+|struct\s+|unsigned\s+)*" + r"[A-Za-z_][A-Za-z0-9_]*\s*" + r"(?P[^;{}()]*);\s*$") +DECLARATOR = re.compile(r"\**\s*([A-Za-z_][A-Za-z0-9_]*)\s*(?:\[[^]]*\])?$") +HEAD_MACROS = {"PyObject_HEAD": "ob_base", "PyObject_VAR_HEAD": "ob_base"} + + +def read_header(path): + for root, _dirs, files in os.walk(INCLUDE): + if os.path.basename(path) in files and root.endswith( + os.path.dirname(path) or root[len(root):] or ""): + return open(os.path.join(root, os.path.basename(path))).read() + # fall back: anywhere under the include tree + for root, _dirs, files in os.walk(INCLUDE): + if os.path.basename(path) in files: + return open(os.path.join(root, os.path.basename(path))).read() + raise SystemExit("no header %s under %s" % (path, INCLUDE)) + + +def fields_of(cname): + """Ordered field names of a struct, read from the target's headers.""" + header, _kind = STRUCTS[cname] + text = read_header(header) + # find "} ;" or "struct _typeobject {" + if cname == "PyTypeObject": + start = text.index("struct _typeobject {") + start = text.index("{", start) + 1 + else: + end = re.search(r"\}\s*%s\s*;" % cname, text) + if not end: + return None + # walk back to the matching "typedef struct {" + head = text[:end.start()] + start = head.rindex("typedef struct {") + len("typedef struct {") + end_i = end.start() + return _scan(text[start:end_i]) + depth = 1 + i = start + while depth: + if text[i] == "{": + depth += 1 + elif text[i] == "}": + depth -= 1 + i += 1 + return _scan(text[start:i - 1]) + + +def _scan(body): + body = re.sub(r"/\*.*?\*/", "", body, flags=re.S) + body = re.sub(r"//[^\n]*", "", body) + out = [] + depth = 0 + for line in body.splitlines(): + stripped = line.strip() + if stripped in HEAD_MACROS: + out.append(HEAD_MACROS[stripped]) + continue + depth += line.count("(") - line.count(")") + if depth: + continue + m = FIELD.match(line) + if not m: + continue + for decl in m.group("decls").split(","): + d = DECLARATOR.search(decl.strip()) + if d: + out.append(d.group(1)) + return out + + +# ---------------------------------------------------------------- rewrite --- +# Match " = {" ... matching "}" at the top level of the file. +DECL = re.compile(r"^(\s*)(?:static\s+)?(%s)\s+([A-Za-z_][A-Za-z0-9_]*)\s*=\s*\{" + % "|".join(STRUCTS), re.M) + +changed_total = 0 +for path in FILES: + src = open(path).read() + out = [] + pos = 0 + changed = 0 + for m in DECL.finditer(src): + cname = m.group(2) + order = fields_of(cname) + if not order: + continue + # find the matching close brace of this initializer + i = m.end() + depth = 1 + while depth: + if src[i] == "{": + depth += 1 + elif src[i] == "}": + depth -= 1 + i += 1 + body = src[m.end():i - 1] + + used = [d for d in re.findall(r"\.([A-Za-z_][A-Za-z0-9_]*)\s*=", body)] + # only top-level designators: drop any that are not fields of this + # struct (they belong to a nested brace, e.g. .ob_base={...}). + used = [u for u in used if u in order] + if not used: + continue + unknown = [u for u in re.findall(r"^\s*\.([A-Za-z_][A-Za-z0-9_]*)\s*=", + body, re.M) if u not in order] + if unknown: + raise SystemExit("%s: %s names %r, not a field of %s -- header " + "and source disagree, refusing to guess" + % (path, m.group(3), unknown, cname)) + idx = [order.index(u) for u in used] + if idx != sorted(idx): + raise SystemExit("%s: %s designators are out of declaration order " + "(%r) -- gcc 7.2 cannot take that either and " + "reordering them is not this script's job" + % (path, m.group(3), used)) + + # Insert the skipped fields, in order, as explicit zeros. Everything + # between the first and last named field only. + new_body = body + gaps = [] + for a, b in zip(idx, idx[1:]): + if b - a > 1: + gaps.append((order[a], order[a + 1:b])) + for after, missing in reversed(gaps): + fill = "".join(" .%s=0,\n" % f for f in missing) + # place the fill immediately after the entry for `after` + am = re.search(r"\.%s\s*=" % re.escape(after), new_body) + j = am.end() + depth2 = 0 + while True: + c = new_body[j] + if c in "([{": + depth2 += 1 + elif c in ")]}": + if depth2 == 0: + break + depth2 -= 1 + elif c == "," and depth2 == 0: + j += 1 + break + j += 1 + new_body = new_body[:j] + "\n" + fill + new_body[j:] + changed += len(missing) + + # A gap BEFORE the first designator is refused too -- greenlet's + # `static PyNumberMethods green_as_number = {.nb_bool=...}` names the + # tenth field and nothing before it. Fill from field zero. + if idx[0] > 0: + fill = "".join(" .%s=0,\n" % f for f in order[:idx[0]]) + new_body = "\n" + fill + new_body + changed += idx[0] + + out.append(src[pos:m.end()]) + out.append(new_body) + pos = i - 1 + out.append(src[pos:]) + if changed: + open(path, "w").write("".join(out)) + print(" %s: filled %d skipped field(s)" % (path, changed)) + changed_total += changed + +print("fill-designators: %d field(s) written back" % changed_total)