mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
Moonraker's whole component set imports on the 3.13 we built
Nineteen packages cross-built for mipsel, twelve of them C extensions, every one gated at nan2008/o32/mips32r2. Not wired into bin/patch.sh yet -- this preserves the build and, more importantly, what it cost, the way tools/supervisor/ does for s6. The measurement lives in test/integration/printer/case-pyext.sh rather than in a paragraph. Two results carry the phase. The first: 3.13's cffi dlopens the printer's OWN /usr/prog/klipper/klippy/chelper/c_helper.so, resolves stepcompress_alloc, steppersync_alloc and itersolve_generate_steps, and calls one. That is phase 7's central question -- whether an interpreter we built can drive FlashForge's C helper -- answered by doing it, and it is the whole reason the interpreter had to be glibc rather than musl. The second: all 16 Moonraker components import on 3.13, with the list taken from CORE_COMPONENTS plus moonraker.conf rather than written down. The negative control is that moving the tree aside fails every third-party import and the database component. The package list is deliberately NOT moonraker-requirements.txt, which installs every optional component's dependencies. It is the closure of what the ENABLED set imports plus klippy's, so no apprise, ldap3, paho-mqtt or zeroconf. msgspec and uvloop are skipped on purpose: Moonraker guards both with contextlib.suppress(ImportError) and falls back to the stdlib. greenlet was the expected blocker and blocked for an unexpected reason. Its MIPS stack-switching assembly was never a problem; greenlet 3.x is C++ and writes its PyTypeObjects as designated initializers, which gcc 7.2 takes only in declaration order with no gaps. The exact rule was probed rather than guessed -- in-order contiguous fine, a trailing gap fine, an interior or leading gap refused -- and fill-designators.py writes the skipped fields back as explicit zeros, reading field order from the target's own headers and refusing to guess when a designator is not in them. Semantically a no-op: static storage duration, those fields were already zero. It switches out and back, in order, on the printer. Three traps worth having in the log. pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado on the first run -- three wrong-architecture .so files sailed through the build and only the ABI gate caught them, so --no-binary :all: is mandatory here rather than cautious. lmdb chose its cffi backend and produced a py3-none-any wheel with mdb.c inside, which is phase 6's recorded trap rebuilt from scratch: on the printer that path invokes a compiler that does not exist. And pillow 10.3.0 cannot be built by any 3.13, cross or not, because its setup.py execs and then reads locals()["__version__"], which PEP 667 broke. One dependency is unsolved and named rather than buried: libnacl ctypes-loads libsodium, which exists on this box only at /usr/prog/libsodium/lib. Measured both ways. So the authorization component keeps the last /usr/prog string in the picture until libsodium is cross-built or that one path entry is kept. What is left before FF_PYTHON could switch is written down in the README, and the honest remaining unknown is at the top of it: every component imports and the entry point loads, but importing is not serving. Nothing has bound :7125 on 3.13 yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oleksandr ShyshatskyiandClaude Opus 5 committed
1 parent
813973620c
commit
da65764b86
4 files changed
+1352
No files matched your search
Executable
+563
@@ -0,0 +1,563 @@
|
||||
#!/bin/sh
|
||||
# Do the cross-built third-party packages actually WORK on the CPython 3.13
|
||||
# this repo builds, on the printer's own kernel?
|
||||
#
|
||||
# SPIKE GATE for phase 6. bin/patch.sh section 5c already ships an interpreter
|
||||
# nothing runs; this asks the question that stands between that interpreter
|
||||
# and FF_PYTHON pointing at it: Moonraker and klippy import third-party C
|
||||
# extensions, those exist on this printer only as mipsel .so files built
|
||||
# against 3.8, and none had been built for 3.13.
|
||||
#
|
||||
# IMPORTS ARE NOT THE MEASUREMENT. A cross-built .so that is the wrong ABI
|
||||
# fails at import, so an import does prove something -- but a cffi that
|
||||
# imports and cannot dlopen, a greenlet that imports and segfaults the first
|
||||
# time it switches stacks, and an lmdb that imports and then tries to run a
|
||||
# compiler are all things that have happened here or in the notes. So each
|
||||
# native package is USED: cffi opens a real library and calls a function
|
||||
# through it the way klippy's chelper does, greenlet switches out and back and
|
||||
# proves the ORDER, lmdb writes a database and a SECOND PROCESS reads it.
|
||||
#
|
||||
# THE REAL PRIZE is section 5: Moonraker's own component list, taken from its
|
||||
# CORE_COMPONENTS plus the printer's moonraker.conf, imported on 3.13. That is
|
||||
# the same check case-moonraker.sh section 7 runs against FlashForge's 3.8.2,
|
||||
# pointed at the new interpreter.
|
||||
FAIL=0
|
||||
ok() { echo " PASS $*"; }
|
||||
bad() { echo " FAIL $*"; FAIL=1; }
|
||||
note(){ echo " .. $*"; }
|
||||
|
||||
MODDIR=/usr/data/anvil
|
||||
PY=$MODDIR/bin/python3.13
|
||||
SP=$MODDIR/lib/python3.13/site-packages
|
||||
MRROOT=/usr/data/mrsrc
|
||||
|
||||
mkdir -p $MODDIR
|
||||
gzip -dc /mnt/py.tgz | tar -x -C $MODDIR || { bad "cannot unpack py.tgz"; exit 1; }
|
||||
chmod +x $PY 2>/dev/null
|
||||
|
||||
# The extension tree lands in the interpreter's OWN site-packages, which is
|
||||
# already on sys.path -- no PYTHONPATH, no .pth file, nothing for a boot
|
||||
# script to get wrong.
|
||||
mkdir -p $SP
|
||||
gzip -dc /mnt/ext.tgz | tar -x -C /tmp || { bad "cannot unpack ext.tgz"; exit 1; }
|
||||
cp -a /tmp/site-packages/. $SP/ || { bad "cannot stage site-packages"; exit 1; }
|
||||
|
||||
mkdir -p $MRROOT
|
||||
gzip -dc /mnt/mr.tgz | tar -x -C $MRROOT || note "no moonraker tarball"
|
||||
|
||||
# Empty on purpose, throughout. The interpreter is statically linked against
|
||||
# its seven C libraries and every extension below was linked against the same
|
||||
# toolchain, so nothing here may need a search path -- and if something does,
|
||||
# that is the finding.
|
||||
export LD_LIBRARY_PATH=
|
||||
|
||||
echo "=== 0. the interpreter still runs with the tree dropped in ==="
|
||||
ver=`$PY -c 'import sys; print(sys.version.split()[0])' 2>&1`
|
||||
case "$ver" in
|
||||
3.13.*) ok "$PY runs: $ver" ;;
|
||||
*) bad "interpreter did not run: $ver"; exit 1 ;;
|
||||
esac
|
||||
note "site-packages: `ls $SP | wc -l` entries"
|
||||
|
||||
echo
|
||||
echo "=== 1. every package imports ==="
|
||||
# One process, one import each, so a single failure names itself instead of
|
||||
# stopping the list.
|
||||
$PY - <<'EOP' > /tmp/imp.out 2>&1
|
||||
import importlib, sys
|
||||
# libnacl is deliberately NOT here: it ctypes-loads libsodium, which lives
|
||||
# only under /usr/prog on this box, and section 4 is about exactly that.
|
||||
mods = ["tornado", "tornado.web", "jinja2", "markupsafe", "distro",
|
||||
"inotify_simple", "dbus_next", "preprocess_cancellation",
|
||||
"serial", "serial_asyncio", "cffi", "_cffi_backend", "greenlet",
|
||||
"lmdb", "smart_open", "streaming_form_data", "PIL", "PIL.Image",
|
||||
"setuptools",
|
||||
"pkg_resources"]
|
||||
bad = 0
|
||||
for m in mods:
|
||||
try:
|
||||
mod = importlib.import_module(m)
|
||||
where = getattr(mod, "__file__", "?") or "?"
|
||||
print(" ok %-24s %s" % (m, where.replace("/usr/data/anvil", "$")))
|
||||
except BaseException as exc:
|
||||
print(" FAIL %-24s %r" % (m, exc))
|
||||
bad += 1
|
||||
print("import failures: %d" % bad)
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/imp.out
|
||||
grep -q '^import failures: 0' /tmp/imp.out \
|
||||
&& ok "every package imports on 3.13" \
|
||||
|| bad "`grep '^import failures' /tmp/imp.out`"
|
||||
|
||||
echo
|
||||
echo "=== 2. the C extensions WORK, not merely import ==="
|
||||
|
||||
# ---- cffi, the way klippy uses it -----------------------------------------
|
||||
# klippy's chelper does ffi.cdef() then ffi.dlopen(c_helper.so) -- ABI mode,
|
||||
# no compiler. That matters here beyond convenience: there IS no compiler on
|
||||
# this printer. So: declare a function, dlopen a real library, call it, and
|
||||
# check the ANSWER. libm is used rather than a library of ours because it is
|
||||
# on this rootfs whatever else is true.
|
||||
$PY - <<'EOP' > /tmp/cffi.out 2>&1
|
||||
import cffi
|
||||
ffi = cffi.FFI()
|
||||
ffi.cdef("double sqrt(double); double pow(double, double);")
|
||||
lib = ffi.dlopen("libm.so.6")
|
||||
a = lib.sqrt(1024.0)
|
||||
b = lib.pow(2.0, 10.0)
|
||||
buf = ffi.new("char[]", b"nozzle")
|
||||
print("sqrt(1024)=%r pow(2,10)=%r buf=%r" % (a, b, ffi.string(buf)))
|
||||
assert a == 32.0 and b == 1024.0 and ffi.string(buf) == b"nozzle"
|
||||
print("cffi ok")
|
||||
EOP
|
||||
grep -q '^cffi ok' /tmp/cffi.out \
|
||||
&& ok "cffi cdef+dlopen+call through libm: `head -1 /tmp/cffi.out`" \
|
||||
|| bad "cffi: `tail -3 /tmp/cffi.out`"
|
||||
|
||||
# ---- greenlet, the way klippy's reactor uses it ----------------------------
|
||||
# The one that was expected to fight. An import proves the .so loaded; only a
|
||||
# switch proves the hand-written MIPS stack-switching assembly in
|
||||
# switch_mips_unix.h does what it says. So switch out, switch back, switch out
|
||||
# again, and check the ORDER -- a stack switch that half-works produces a
|
||||
# plausible-looking result in the wrong sequence.
|
||||
$PY - <<'EOP' > /tmp/gl.out 2>&1
|
||||
import greenlet
|
||||
seq = []
|
||||
def child():
|
||||
seq.append("child-1")
|
||||
value = main.switch("up")
|
||||
seq.append("child-2:%s" % value)
|
||||
return "done"
|
||||
main = greenlet.getcurrent()
|
||||
g = greenlet.greenlet(child)
|
||||
seq.append("main-1")
|
||||
got = g.switch()
|
||||
seq.append("main-2:%s" % got)
|
||||
result = g.switch("down")
|
||||
seq.append("main-3:%s" % result)
|
||||
print("|".join(seq))
|
||||
assert seq == ["main-1", "child-1", "main-2:up", "child-2:down", "main-3:done"], seq
|
||||
assert g.dead
|
||||
print("greenlet ok")
|
||||
EOP
|
||||
grep -q '^greenlet ok' /tmp/gl.out \
|
||||
&& ok "greenlet switches and returns in order: `head -1 /tmp/gl.out`" \
|
||||
|| bad "greenlet: `tail -5 /tmp/gl.out`"
|
||||
|
||||
# ---- lmdb, and the trap it is famous for here ------------------------------
|
||||
# phase 6's notes record a trimmed tree in which the lmdb egg fell back to its
|
||||
# CFFI backend and tried to invoke mips-linux-gnu-gcc ON THE PRINTER at
|
||||
# Moonraker startup. So this asserts WHICH backend loaded before it asserts
|
||||
# that the database works.
|
||||
$PY - <<'EOP' > /tmp/lmdb1.out 2>&1
|
||||
import lmdb, os, shutil
|
||||
print("backend:", lmdb.__file__)
|
||||
import lmdb.cpython
|
||||
print("cpython backend loaded:", lmdb.cpython.__file__)
|
||||
shutil.rmtree("/tmp/lm", ignore_errors=True)
|
||||
os.makedirs("/tmp/lm")
|
||||
env = lmdb.open("/tmp/lm", map_size=1 << 20)
|
||||
with env.begin(write=True) as txn:
|
||||
txn.put(b"nozzle", b"0.4")
|
||||
txn.put(b"bed", b"220")
|
||||
env.close()
|
||||
print("wrote")
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/lmdb1.out
|
||||
if grep -q 'cpython backend loaded' /tmp/lmdb1.out; then
|
||||
ok "lmdb loaded its CPython extension, not the compile-at-import cffi path"
|
||||
else
|
||||
bad "lmdb backend: `tail -3 /tmp/lmdb1.out`"
|
||||
fi
|
||||
# A SECOND PROCESS, because a database that only exists inside the writer is
|
||||
# not a database.
|
||||
$PY - <<'EOP' > /tmp/lmdb2.out 2>&1
|
||||
import lmdb
|
||||
env = lmdb.open("/tmp/lm", readonly=True)
|
||||
with env.begin() as txn:
|
||||
print("read back: nozzle=%s bed=%s" % (txn.get(b"nozzle"), txn.get(b"bed")))
|
||||
assert txn.get(b"nozzle") == b"0.4" and txn.get(b"bed") == b"220"
|
||||
print("lmdb ok")
|
||||
EOP
|
||||
grep -q '^lmdb ok' /tmp/lmdb2.out \
|
||||
&& ok "lmdb round-trips through a second process: `head -1 /tmp/lmdb2.out`" \
|
||||
|| bad "lmdb: `tail -3 /tmp/lmdb2.out`"
|
||||
|
||||
# ---- the two optional accelerators, and streaming-form-data ---------------
|
||||
$PY - <<'EOP' > /tmp/spd.out 2>&1
|
||||
import markupsafe, tornado.escape
|
||||
print("markupsafe accel:", markupsafe.escape.__module__)
|
||||
print("escape:", str(markupsafe.escape("<a href='x'>&")))
|
||||
assert str(markupsafe.escape("<a>")) == "<a>"
|
||||
try:
|
||||
from tornado import speedups
|
||||
print("tornado speedups:", speedups.__file__)
|
||||
except ImportError as exc:
|
||||
print("tornado speedups absent (pure fallback):", exc)
|
||||
# The five names moonraker asks of streaming_form_data, and a real parse.
|
||||
from streaming_form_data import StreamingFormDataParser
|
||||
from streaming_form_data.targets import FileTarget, ValueTarget, SHA256Target
|
||||
from streaming_form_data.validators import ValidationError
|
||||
body = (b"--BOUND\r\nContent-Disposition: form-data; name=\"print\"\r\n\r\n"
|
||||
b"benchy.gcode\r\n--BOUND--\r\n")
|
||||
target = ValueTarget()
|
||||
p = StreamingFormDataParser(headers={"Content-Type": "multipart/form-data; boundary=BOUND"})
|
||||
p.register("print", target)
|
||||
p.data_received(body)
|
||||
print("parsed:", target.value)
|
||||
assert target.value == b"benchy.gcode"
|
||||
print("speedups ok")
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/spd.out
|
||||
grep -q '^speedups ok' /tmp/spd.out \
|
||||
&& ok "markupsafe accelerator and streaming-form-data parse real input" \
|
||||
|| bad "accelerators: `tail -4 /tmp/spd.out`"
|
||||
|
||||
# ---- pillow, on the one path moonraker actually uses it for ---------------
|
||||
# Not "import PIL". Moonraker's metadata subprocess pulls a base64 PNG
|
||||
# thumbnail out of a gcode comment, decodes it, resizes it and writes it back
|
||||
# out -- so this does exactly that, through a Pillow built with zlib and NO
|
||||
# other codec. If the zlib link were wrong, the import would still succeed and
|
||||
# the save would fail.
|
||||
$PY - <<'EOP' > /tmp/pil.out 2>&1
|
||||
import base64, io
|
||||
from PIL import Image, features
|
||||
print("zlib codec:", features.check("zlib"))
|
||||
img = Image.new("RGBA", (64, 48), (200, 30, 30, 255))
|
||||
for x in range(0, 64, 8):
|
||||
for y in range(0, 48, 8):
|
||||
img.putpixel((x, y), (10, 10, 240, 255))
|
||||
raw = io.BytesIO()
|
||||
img.save(raw, format="PNG")
|
||||
encoded = base64.b64encode(raw.getvalue())
|
||||
print("encoded png: %d bytes base64" % len(encoded))
|
||||
back = Image.open(io.BytesIO(base64.b64decode(encoded)))
|
||||
back.load()
|
||||
small = back.resize((32, 32))
|
||||
out = io.BytesIO()
|
||||
small.save(out, format="PNG")
|
||||
print("decoded %s %s -> resized %s, %d bytes"
|
||||
% (back.format, back.size, small.size, len(out.getvalue())))
|
||||
assert back.size == (64, 48) and back.getpixel((0, 0)) == (10, 10, 240, 255)
|
||||
assert small.size == (32, 32) and len(out.getvalue()) > 0
|
||||
print("pillow ok")
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/pil.out
|
||||
grep -q '^pillow ok' /tmp/pil.out \
|
||||
&& ok "pillow decodes, resizes and re-encodes a PNG thumbnail" \
|
||||
|| bad "pillow: `tail -4 /tmp/pil.out`"
|
||||
|
||||
# ---- cffi against klippy's REAL c_helper.so -------------------------------
|
||||
# The question phase 7 turns on, asked here because the answer is cheap: can
|
||||
# 3.13's cffi bind the printer's own klippy helper? c_helper.so is a plain
|
||||
# glibc shared object dlopen'd in ABI mode, so it should not care which python
|
||||
# opened it -- "should not care" being exactly the kind of claim worth
|
||||
# measuring on the box rather than asserting in a report.
|
||||
CH=`find /usr/prog -name 'c_helper*.so' 2>/dev/null | head -1`
|
||||
if [ -n "$CH" ]; then
|
||||
note "found $CH"
|
||||
$PY - "$CH" <<'EOP' > /tmp/chelper.out 2>&1
|
||||
import sys, cffi
|
||||
ffi = cffi.FFI()
|
||||
# A handful of klippy's own declarations, spelled the way chelper does.
|
||||
ffi.cdef("""
|
||||
struct stepcompress *stepcompress_alloc(uint32_t oid);
|
||||
struct steppersync *steppersync_alloc(struct serialqueue *sq
|
||||
, struct stepcompress **sc_list, int sc_num, int move_num);
|
||||
double itersolve_generate_steps(struct stepper_kinematics *sk, double flush_time);
|
||||
""")
|
||||
lib = ffi.dlopen(sys.argv[1])
|
||||
found = []
|
||||
for name in ("stepcompress_alloc", "steppersync_alloc",
|
||||
"itersolve_generate_steps"):
|
||||
try:
|
||||
getattr(lib, name) # cffi resolves lazily: this is the dlsym
|
||||
found.append(name)
|
||||
except Exception as exc:
|
||||
print(" %s: %r" % (name, exc))
|
||||
print("resolved: %s" % " ".join(found))
|
||||
sc = lib.stepcompress_alloc(0)
|
||||
print("stepcompress_alloc(0) ->", "non-NULL" if sc else "NULL")
|
||||
assert len(found) == 3 and sc
|
||||
print("chelper ok")
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/chelper.out
|
||||
grep -q '^chelper ok' /tmp/chelper.out \
|
||||
&& ok "3.13's cffi dlopens the printer's own c_helper.so and calls into it" \
|
||||
|| bad "c_helper.so through 3.13 cffi: `tail -4 /tmp/chelper.out`"
|
||||
else
|
||||
note "no c_helper.so on this image -- klippy binding not measured here"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== 3. nothing maps a library under /usr/prog ==="
|
||||
# The property the whole static-linking decision exists to buy, re-established
|
||||
# with the extension tree in place: /usr/prog carries libffi.so.8 against the
|
||||
# rootfs's .so.7, an openssl 1.0.2d, and a stock OTA can replace any of it.
|
||||
# The process below has imported every native module first, so any mapping
|
||||
# would already be in /proc/self/maps.
|
||||
$PY - <<'EOP' > /tmp/maps.out 2>&1
|
||||
import ssl, ctypes, sqlite3, lzma, zlib, hashlib # the interpreter's
|
||||
import cffi, _cffi_backend, greenlet, lmdb, markupsafe # ours
|
||||
import streaming_form_data, tornado.web, jinja2
|
||||
from PIL import Image
|
||||
libs = [l.split()[-1] for l in open("/proc/self/maps")
|
||||
if ".so" in l and l.split()[-1].startswith("/")]
|
||||
prog = sorted(set(l for l in libs if l.startswith("/usr/prog")))
|
||||
print("mapped libraries: %d" % len(set(libs)))
|
||||
for l in sorted(set(libs)):
|
||||
print(" %s" % l)
|
||||
print("under /usr/prog: %d" % len(prog))
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/maps.out
|
||||
grep -q '^under /usr/prog: 0' /tmp/maps.out \
|
||||
&& ok "no library under /usr/prog is mapped, with every extension imported" \
|
||||
|| bad "`grep '^under /usr/prog' /tmp/maps.out`"
|
||||
|
||||
echo
|
||||
echo "=== 4. libnacl, and the one dependency that is NOT solved ==="
|
||||
# libnacl is pure python -- it ctypes-loads libsodium, which this rootfs does
|
||||
# not have and /usr/prog does. Nothing was cross-built to fix that, so this
|
||||
# reports the state rather than asserting one, exactly as case-python.sh does
|
||||
# for the missing CA bundle.
|
||||
$PY - <<'EOP' > /tmp/nacl.out 2>&1
|
||||
import ctypes.util, os
|
||||
print("ctypes.util.find_library('sodium') ->", ctypes.util.find_library("sodium"))
|
||||
for p in ("/usr/lib/libsodium.so", "/lib/libsodium.so",
|
||||
"/usr/prog/libsodium/lib/libsodium.so"):
|
||||
print(" %-42s %s" % (p, "present" if os.path.exists(p) else "-"))
|
||||
try:
|
||||
import libnacl
|
||||
print("libnacl imported, sodium at:", libnacl.nacl._name)
|
||||
except BaseException as exc:
|
||||
print("libnacl NOT importable: %r" % (exc,))
|
||||
EOP
|
||||
sed -n 's/^/ /p' /tmp/nacl.out
|
||||
if grep -q 'libnacl imported' /tmp/nacl.out; then
|
||||
note "libnacl found libsodium without help"
|
||||
else
|
||||
note "libnacl needs libsodium and this rootfs has it ONLY under /usr/prog"
|
||||
fi
|
||||
# With /usr/prog on the path, the way anvil-env.sh puts it there today.
|
||||
LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY -c \
|
||||
'import libnacl; print("libnacl ok:", libnacl.nacl._name)' \
|
||||
> /tmp/nacl2.out 2>&1
|
||||
if grep -q '^libnacl ok' /tmp/nacl2.out; then
|
||||
ok "libnacl works when /usr/prog/libsodium is on the path: `cat /tmp/nacl2.out`"
|
||||
else
|
||||
bad "libnacl even with libsodium on the path: `tail -3 /tmp/nacl2.out`"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== 5. THE PRIZE: Moonraker's own component list, on 3.13 ==="
|
||||
# Lifted from case-moonraker.sh section 7 and pointed at the new interpreter.
|
||||
# The component list is NOT written down here: it comes from Moonraker's own
|
||||
# CORE_COMPONENTS plus every section in the printer's moonraker.conf, which is
|
||||
# what keeps this honest when the pin moves or somebody configures a component
|
||||
# nobody thought about.
|
||||
CONF=/usr/data/config/moonraker.conf
|
||||
if [ ! -f "$CONF" ]; then
|
||||
# No installed config in this run -- use the one the mod ships.
|
||||
mkdir -p /usr/data/config
|
||||
cat > "$CONF" <<'CFG'
|
||||
[server]
|
||||
[machine]
|
||||
[file_manager]
|
||||
[authorization]
|
||||
[history]
|
||||
[webcam anvil]
|
||||
CFG
|
||||
note "no installed moonraker.conf; using the mod's own section set"
|
||||
fi
|
||||
if [ -d "$MRROOT/moonraker" ]; then
|
||||
LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" "$CONF" <<'PY' > /tmp/mr.out 2>&1
|
||||
import glob, importlib, os, re, sys
|
||||
sys.path.insert(0, sys.argv[1])
|
||||
try:
|
||||
import moonraker.server as server
|
||||
except Exception as exc:
|
||||
print("moonraker.server does not import: %r" % (exc,))
|
||||
raise SystemExit(2)
|
||||
|
||||
names = list(getattr(server, "CORE_COMPONENTS", []))
|
||||
|
||||
def scan(path, depth=0):
|
||||
try:
|
||||
lines = open(path).readlines()
|
||||
except OSError:
|
||||
return
|
||||
for line in lines:
|
||||
section = re.match(r"\s*\[\s*([A-Za-z0-9_]+)", line)
|
||||
if not section:
|
||||
continue
|
||||
if section.group(1) == "include":
|
||||
inc = re.match(r"\s*\[\s*include\s+([^\]]+?)\s*\]", line)
|
||||
if inc and depth < 3:
|
||||
base = os.path.dirname(os.path.abspath(path))
|
||||
for f in sorted(glob.glob(os.path.join(base, inc.group(1)))):
|
||||
scan(f, depth + 1)
|
||||
continue
|
||||
names.append(section.group(1))
|
||||
|
||||
scan(sys.argv[2])
|
||||
|
||||
failures = []
|
||||
checked = []
|
||||
for name in dict.fromkeys(names):
|
||||
target = "moonraker.components." + name
|
||||
try:
|
||||
importlib.import_module(target)
|
||||
checked.append(name)
|
||||
except ModuleNotFoundError as exc:
|
||||
# A section that is not a component at all ([server]) is fine. A
|
||||
# missing DEPENDENCY of a component that does exist is the point.
|
||||
if getattr(exc, "name", None) == target:
|
||||
continue
|
||||
failures.append((name, repr(exc)))
|
||||
except Exception as exc:
|
||||
failures.append((name, repr(exc)))
|
||||
|
||||
print("imported: %s" % " ".join(sorted(checked)))
|
||||
for name, err in failures:
|
||||
print(" %s: %s" % (name, err))
|
||||
print("components ok: %d" % len(checked) if not failures
|
||||
else "%d component(s) will not load" % len(failures))
|
||||
raise SystemExit(1 if failures else 0)
|
||||
PY
|
||||
sed -n 's/^/ /p' /tmp/mr.out
|
||||
if grep -q '^components ok:' /tmp/mr.out; then
|
||||
ok "`grep '^components ok:' /tmp/mr.out` on CPython 3.13"
|
||||
else
|
||||
bad "components will not load: `tail -5 /tmp/mr.out`"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "--- 5b. moonraker's own entry point, and its server object ---"
|
||||
# Importing components proves the dependencies are there. Building the
|
||||
# Server is what proves the 3.13 stdlib underneath them still fits: this
|
||||
# pin is a 2023 tree and asyncio, importlib and inspect have all moved.
|
||||
LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" <<'PY' > /tmp/mr2.out 2>&1
|
||||
import sys
|
||||
sys.path.insert(0, sys.argv[1])
|
||||
import moonraker.moonraker as entry
|
||||
import moonraker.server, moonraker.confighelper, moonraker.common
|
||||
import moonraker.eventloop, moonraker.loghelper, moonraker.utils
|
||||
import moonraker.utils.json_wrapper as jw
|
||||
print("entry:", entry.__file__)
|
||||
print("json backend:", "msgspec" if jw.MSGSPEC_ENABLED else "stdlib json")
|
||||
print("round trip:", jw.loads(jw.dumps({"state": "ready", "z": 0.4})))
|
||||
print("moonraker ok")
|
||||
PY
|
||||
sed -n 's/^/ /p' /tmp/mr2.out
|
||||
grep -q '^moonraker ok' /tmp/mr2.out \
|
||||
&& ok "moonraker's entry point and core modules import on 3.13" \
|
||||
|| bad "moonraker entry: `tail -5 /tmp/mr2.out`"
|
||||
else
|
||||
note "no moonraker tree at $MRROOT -- pass mr.tgz=..."
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== 6. NEGATIVE CONTROL: take the tree away ==="
|
||||
# Without this half, "the components import" is a fact about python in
|
||||
# general. Move the cross-built site-packages aside and the SAME check must
|
||||
# fail -- otherwise something else on this box is satisfying the imports and
|
||||
# the tree is not what is being measured.
|
||||
mv $SP $SP.away
|
||||
LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" <<'PY' > /tmp/neg.out 2>&1
|
||||
import importlib, sys
|
||||
sys.path.insert(0, sys.argv[1])
|
||||
missing = []
|
||||
for m in ("tornado", "lmdb", "greenlet", "cffi", "jinja2", "streaming_form_data"):
|
||||
try:
|
||||
importlib.import_module(m)
|
||||
except ImportError:
|
||||
missing.append(m)
|
||||
print("unavailable without the tree: %s" % " ".join(missing))
|
||||
try:
|
||||
importlib.import_module("moonraker.components.database")
|
||||
print("database component STILL imported")
|
||||
except BaseException as exc:
|
||||
print("database component fails: %s" % type(exc).__name__)
|
||||
PY
|
||||
mv $SP.away $SP
|
||||
sed -n 's/^/ /p' /tmp/neg.out
|
||||
if grep -q 'unavailable without the tree: tornado lmdb greenlet cffi jinja2 streaming_form_data' /tmp/neg.out \
|
||||
&& grep -q 'database component fails' /tmp/neg.out; then
|
||||
ok "with the tree removed every third-party import and the database component fail"
|
||||
else
|
||||
bad "negative control did not fail as expected: `cat /tmp/neg.out`"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== 7. no compiler is reachable, and nothing tried to use one ==="
|
||||
# The failure mode phase 6 already recorded is a package deciding to BUILD
|
||||
# something at import time. There is no compiler here, which is the reason
|
||||
# that failure was fatal rather than slow.
|
||||
for c in cc gcc mips-linux-gnu-gcc; do
|
||||
if command -v $c >/dev/null 2>&1; then
|
||||
bad "$c is on PATH in the replica -- this test would not catch a runtime build"
|
||||
else
|
||||
note "$c: absent, as on a real printer"
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "=== 8. size of what would be added ==="
|
||||
note "site-packages: `du -sk $SP | cut -f1` KB, `find $SP -type f | wc -l` files"
|
||||
note "interpreter : `du -sk $MODDIR/lib/python3.13 | cut -f1` KB total with it"
|
||||
|
||||
echo
|
||||
echo "=== 9. is setuptools/pkg_resources actually earning its 4.6 MB? ==="
|
||||
# It is a third of the tree, and it is here for ONE reason: phase 6 recorded a
|
||||
# trimmed tree in which the lmdb EGG lost pkg_resources, fell back to lmdb's
|
||||
# cffi path, and tried to run a compiler on the printer at Moonraker startup.
|
||||
# But this build installs lmdb from a WHEEL with a real CPython extension, not
|
||||
# an egg -- so the question is whether the insurance is still insuring
|
||||
# anything. Measured rather than reasoned about: take it away and re-run the
|
||||
# two things that would break.
|
||||
mv $SP/setuptools $SP/pkg_resources $SP/_distutils_hack /tmp/ 2>/dev/null
|
||||
mv $SP/distutils-precedence.pth /tmp/ 2>/dev/null
|
||||
LD_LIBRARY_PATH=/usr/prog/libsodium/lib $PY - "$MRROOT" "$CONF" <<'PY' > /tmp/nosetup.out 2>&1
|
||||
import glob, importlib, os, re, sys
|
||||
sys.path.insert(0, sys.argv[1])
|
||||
import lmdb, lmdb.cpython
|
||||
print("lmdb backend without setuptools:", lmdb.cpython.__file__)
|
||||
import moonraker.server as server
|
||||
names = list(getattr(server, "CORE_COMPONENTS", []))
|
||||
for line in open(sys.argv[2]):
|
||||
m = re.match(r"\s*\[\s*([A-Za-z0-9_]+)", line)
|
||||
if m and m.group(1) != "include":
|
||||
names.append(m.group(1))
|
||||
failures = []
|
||||
checked = []
|
||||
for name in dict.fromkeys(names):
|
||||
target = "moonraker.components." + name
|
||||
try:
|
||||
importlib.import_module(target)
|
||||
checked.append(name)
|
||||
except ModuleNotFoundError as exc:
|
||||
if getattr(exc, "name", None) == target:
|
||||
continue
|
||||
failures.append((name, repr(exc)))
|
||||
except Exception as exc:
|
||||
failures.append((name, repr(exc)))
|
||||
for name, err in failures:
|
||||
print(" %s: %s" % (name, err))
|
||||
print("components ok without setuptools: %d" % len(checked) if not failures
|
||||
else "%d component(s) need setuptools" % len(failures))
|
||||
PY
|
||||
mv /tmp/setuptools /tmp/pkg_resources /tmp/_distutils_hack $SP/ 2>/dev/null
|
||||
mv /tmp/distutils-precedence.pth $SP/ 2>/dev/null
|
||||
sed -n 's/^/ /p' /tmp/nosetup.out
|
||||
if grep -q '^components ok without setuptools: 16' /tmp/nosetup.out; then
|
||||
note "setuptools/pkg_resources are NOT needed at runtime by this build --"
|
||||
note " 4.9 MB of the tree is insurance against a failure mode (the lmdb"
|
||||
note " EGG's cffi fallback) that installing lmdb as a wheel already removes."
|
||||
note " Dropping them is a size decision, not a correctness one."
|
||||
else
|
||||
ok "setuptools/pkg_resources earn their place: `tail -2 /tmp/nosetup.out`"
|
||||
fi
|
||||
|
||||
echo
|
||||
[ $FAIL -eq 0 ] && echo "ALL PASS" || echo "FAILURES"
|
||||
exit $FAIL
|
||||
@@ -0,0 +1,137 @@
|
||||
# The Python packages Moonraker and klippy need, cross-built for mipsel
|
||||
|
||||
The second half of phase 6. `tools/python/` builds the interpreter;
|
||||
this builds what runs on it. **Not yet wired into `bin/patch.sh`** -- see
|
||||
"What is left" at the bottom. `build.sh` is the measurement harness, preserved
|
||||
here because it is where the findings live, exactly as `tools/supervisor/`
|
||||
is for s6.
|
||||
|
||||
Everything below was run in the printer replica on the real
|
||||
`rootfs.squashfs`, not read off a project page.
|
||||
`test/integration/printer/case-pyext.sh` is that run, kept.
|
||||
|
||||
## What builds
|
||||
|
||||
19 packages, 12 mipsel extension modules, from a 56-second clean build in one
|
||||
throwaway `debian:bookworm`. Every `.so` is gated at `e_flags=0x70001407`,
|
||||
`nan2008/o32/mips32r2` -- shared objects carry `EF_MIPS_PIC`, so they read
|
||||
`…07` where an executable reads `…05`.
|
||||
|
||||
Native: cffi, greenlet, lmdb, markupsafe, streaming-form-data, tornado
|
||||
speedups, pillow. Pure: jinja2, distro, inotify-simple, libnacl, dbus-next,
|
||||
preprocess-cancellation, pyserial, pyserial-asyncio, pycparser, smart_open,
|
||||
setuptools.
|
||||
|
||||
**The list is deliberately not `moonraker-requirements.txt`**, which installs
|
||||
every optional component's dependencies. It is the closure of what the
|
||||
*enabled* component set imports -- `moonraker.conf`'s sections plus
|
||||
`CORE_COMPONENTS` -- plus klippy's. So no apprise, ldap3, paho-mqtt, zeroconf
|
||||
or python-periphery. `msgspec` and `uvloop` are skipped on purpose: Moonraker
|
||||
guards both with `contextlib.suppress(ImportError)` and falls back to the
|
||||
stdlib, measured.
|
||||
|
||||
## The method
|
||||
|
||||
No crossenv. The staged interpreter already carries
|
||||
`_sysconfigdata__linux_mipsel-linux-gnu`, so exporting
|
||||
`_PYTHON_SYSCONFIGDATA_NAME` at an x86-64 build-python **of the same 3.13.7**
|
||||
makes setuptools answer every question for the target. The stage is
|
||||
bind-mounted at `/usr/data/anvil` so `INCLUDEPY` resolves unmodified. Plus the
|
||||
gcc-wrapper trick from `tools/python/build.sh`, which is what gets
|
||||
`-EL -mnan=2008` onto the link lines.
|
||||
|
||||
## What fought back, and what it cost
|
||||
|
||||
* **greenlet was the expected blocker, for an unexpected reason.** The MIPS
|
||||
stack-switching assembly was never a problem. greenlet 3.x is C++ and writes
|
||||
its `PyTypeObject`s as designated initializers, which gcc 7.2 accepts only
|
||||
in declaration order with no gaps:
|
||||
|
||||
sorry, unimplemented: non-trivial designated initializers not supported
|
||||
|
||||
The exact rule was probed rather than assumed: in-order contiguous is fine,
|
||||
a *trailing* gap is fine, an *interior* or *leading* gap is refused.
|
||||
`fill-designators.py` writes the skipped fields back as explicit zeros,
|
||||
taking field order from the target's own headers and refusing to guess when
|
||||
a designator is not in them. 55 fields across two files, semantically a
|
||||
no-op -- static storage duration, those fields were already zero -- and the
|
||||
result switches out and back, in order, on the printer.
|
||||
|
||||
* **pip silently downloaded x86-64 manylinux wheels** for cffi, greenlet and
|
||||
tornado on the first run. Three wrong-architecture `.so` files sailed
|
||||
straight through the build and only the ABI gate caught them.
|
||||
`--no-binary :all:` is mandatory here, not caution.
|
||||
|
||||
* **lmdb chose its cffi backend** and produced a `py3-none-any` wheel with
|
||||
`mdb.c` inside -- which is phase 6's recorded trap rebuilt from scratch: on
|
||||
the printer that path tries to invoke a compiler that does not exist.
|
||||
`LMDB_FORCE_CPYTHON=1` fixes it. Note `LMDB_FORCE_CFFI=0` makes it *worse*:
|
||||
the variable is tested for presence, so `"0"` selects cffi.
|
||||
|
||||
* **pillow 10.3.0 cannot be built by any 3.13**, cross or not: its `setup.py`
|
||||
does `exec(...)` then reads `locals()["__version__"]`, which PEP 667 broke.
|
||||
11.0.0 is the first version that works.
|
||||
|
||||
* **streaming-form-data 1.13.0** (Moonraker's pin) ships pregenerated Cython C
|
||||
calling the 4-argument `_PyLong_AsByteArray`, which grew a fifth parameter in
|
||||
3.13. 1.19.1 instead, which then wants `smart_open` at module scope.
|
||||
|
||||
## What it proves
|
||||
|
||||
Beyond "it imports", which proves very little on its own:
|
||||
|
||||
* cffi does `cdef` + `dlopen` + call through libm -- klippy's exact ABI-mode
|
||||
usage.
|
||||
* **cffi dlopens the printer's own `/usr/prog/klipper/klippy/chelper/c_helper.so`,
|
||||
resolves `stepcompress_alloc`, `steppersync_alloc` and
|
||||
`itersolve_generate_steps`, and calls one.** That is phase 7's central
|
||||
question -- whether a Python we built can drive FlashForge's C helper --
|
||||
answered yes, and it is the reason the interpreter had to be glibc and not
|
||||
musl.
|
||||
* lmdb writes and a *second process* reads back.
|
||||
* pillow decodes, resizes and re-encodes a PNG: the real thumbnail path.
|
||||
* **all 16 Moonraker components import on 3.13**, the list taken from
|
||||
`CORE_COMPONENTS` + `moonraker.conf` rather than written down here.
|
||||
* zero `/usr/prog` libraries mapped, with every extension imported.
|
||||
* Negative control: move the tree aside and every third-party import fails.
|
||||
|
||||
## Size
|
||||
|
||||
13.5MB built, 2.9MB gzipped, 1073 files; 18.2MB on the printer once
|
||||
`__pycache__` exists. With the interpreter, about 50MB.
|
||||
|
||||
`setuptools`/`pkg_resources` are 4.9MB of that -- 36% -- and are not needed at
|
||||
runtime: measured, with them removed lmdb still loads its CPython extension
|
||||
and all 16 components still import. The phase-6 trap was the lmdb *egg*'s cffi
|
||||
fallback, and installing lmdb as a wheel removes it. Dropping them is a size
|
||||
decision, not a correctness one.
|
||||
|
||||
## The one unsolved dependency: libsodium
|
||||
|
||||
`libnacl` is pure Python and ctypes-loads libsodium, which exists on this box
|
||||
**only at `/usr/prog/libsodium/lib`**. Measured both ways: it fails with an
|
||||
empty path and works with that directory on it. So Moonraker's
|
||||
`authorization` component keeps one `/usr/prog` dependency, and it is the last
|
||||
one in the picture. Either cross-build libsodium into the prefix, or accept
|
||||
`anvil-env.sh` keeping that single entry.
|
||||
|
||||
## What is left before FF_PYTHON can switch
|
||||
|
||||
1. **Boot Moonraker on 3.13 and hit the HTTP API.** Every component imports
|
||||
and the entry point loads, but importing is not serving: nothing has bound
|
||||
:7125 on 3.13. `case-moonraker.sh` already does this against 3.8; pointing
|
||||
it at 3.13 is the next gate and the honest remaining unknown.
|
||||
2. Decide libsodium, above.
|
||||
3. Wire into `bin/patch.sh`. Note the build needs the **untrimmed** interpreter
|
||||
stage -- `include/` and `config-3.13-*`, which section 5c's trim currently
|
||||
deletes -- and the pins want moving into `versions.env` with sha256s.
|
||||
4. klippy is **not** blocked by packages: cffi, greenlet, jinja2, markupsafe
|
||||
and pyserial all build, and `c_helper.so` binds. `numpy` is the only gap,
|
||||
wanted solely by `extras/stepper_resonance_tester.py`, which Klipper loads
|
||||
on demand. numpy on 3.13 means numpy >= 2.1 and a Meson cross-file, and
|
||||
that genuinely is a multi-day rabbit hole. It costs input-shaper
|
||||
calibration, not printing.
|
||||
|
||||
Unverified: no real hardware, replica only; nothing has run under sustained
|
||||
load; and the greenlet designator patch is a no-op by argument plus a working
|
||||
switch test, not by upstream blessing.
|
||||
Executable
+425
@@ -0,0 +1,425 @@
|
||||
#!/bin/bash
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-build the third-party Python packages Moonraker and klippy import,
|
||||
# for the CPython 3.13 that tools/python/build.sh already produces for the
|
||||
# FlashForge Creator 5 Pro (Ingenic mipsel).
|
||||
#
|
||||
# SPIKE. Nothing here is wired into bin/patch.sh yet.
|
||||
#
|
||||
# Same non-negotiables as tools/python/build.sh, for the same reasons:
|
||||
#
|
||||
# * e_flags 0x70001405 for an EXEC, 0x70001407 for a DYN (EF_MIPS_PIC).
|
||||
# Every .so this produces is gated; a legacy-NaN wheel imports fine on the
|
||||
# host and fails on the printer with nothing but ENOEXEC to say why.
|
||||
# * -EL -mnan=2008 must reach the COMPILE and the LINK of every object.
|
||||
# setuptools does not forward CFLAGS to its link line, so the flags are
|
||||
# baked into PATH wrappers around the gcc driver and no build system gets
|
||||
# a vote.
|
||||
# * glibc toolchain only. musl is forbidden -- klippy dlopens a glibc
|
||||
# c_helper.so.
|
||||
#
|
||||
# THE CROSS TRICK. There is no crossenv here. The staged interpreter already
|
||||
# carries _sysconfigdata__linux_mipsel-linux-gnu.py, which records the cross
|
||||
# CC, LDSHARED, EXT_SUFFIX (.cpython-313-mipsel-linux-gnu.so) and INCLUDEPY
|
||||
# (/usr/data/anvil/include/python3.13). Exporting _PYTHON_SYSCONFIGDATA_NAME
|
||||
# at an x86-64 CPython 3.13 of the SAME version makes sysconfig -- and so
|
||||
# setuptools' build_ext -- answer every question about the TARGET. The staged
|
||||
# tree is bind-mounted at its real prefix so INCLUDEPY resolves without
|
||||
# rewriting anything.
|
||||
#
|
||||
# WHAT IS BUILT AND WHY THAT LIST. Not moonraker-requirements.txt, which
|
||||
# installs every optional component's dependency. The list is what the
|
||||
# ENABLED component set imports -- assets/moonraker.conf's sections plus
|
||||
# server.CORE_COMPONENTS, closed over moonraker's own imports (see
|
||||
# scan-imports.py beside this file) -- plus what klippy needs.
|
||||
#
|
||||
# Usage:
|
||||
# ./build-pyext.sh # in docker
|
||||
# ./build-pyext.sh --in-container
|
||||
# ---------------------------------------------------------------------------
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
PYBUILD="${PYBUILD:-$HERE/../pybuild}"
|
||||
REPO="${REPO:-/home/shish/firmware/.claude/worktrees/s6-vs-runit}"
|
||||
TOOLCHAIN_HOST="$REPO/work/.mips-toolchain/mips-gcc720-glibc229"
|
||||
|
||||
if [ "${1:-}" != "--in-container" ]; then
|
||||
[ -x "$TOOLCHAIN_HOST/bin/mips-linux-gnu-gcc" ] || {
|
||||
echo "!! no toolchain at $TOOLCHAIN_HOST" >&2; exit 1; }
|
||||
[ -x "$PYBUILD/hostpy/bin/python3.13" ] || {
|
||||
echo "!! no build-python at $PYBUILD/hostpy -- run tools/python/build.sh" >&2; exit 1; }
|
||||
[ -d "$PYBUILD/stage/usr/data/anvil/include/python3.13" ] || {
|
||||
echo "!! no UNTRIMMED stage at $PYBUILD/stage -- the shipped tree has no headers" >&2; exit 1; }
|
||||
exec docker run --rm \
|
||||
-v "$TOOLCHAIN_HOST":/toolchain:ro \
|
||||
-v "$PYBUILD":/work \
|
||||
-v "$PYBUILD/stage/usr/data/anvil":/usr/data/anvil:ro \
|
||||
-v "$HERE":/pyext \
|
||||
-e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
|
||||
-e ONLY="${ONLY:-}" \
|
||||
-w /pyext debian:bookworm \
|
||||
bash /pyext/build-pyext.sh --in-container
|
||||
fi
|
||||
|
||||
# ======================= everything below runs in the container =============
|
||||
START=$(date +%s)
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
TC=/toolchain
|
||||
DEP=/work/deproot # the static C libraries CPython was built on
|
||||
HOSTPY=/work/hostpy/bin/python3.13
|
||||
P=/pyext
|
||||
SRC=$P/src # sdist cache
|
||||
WHEELS=$P/wheels # built wheels
|
||||
SP=$P/site-packages # the tree that would ship
|
||||
LOGS=$P/logs
|
||||
mkdir -p "$SRC" "$WHEELS" "$SP" "$LOGS"
|
||||
|
||||
log() { echo; echo "===== $* ====="; }
|
||||
|
||||
log "apt"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq --no-install-recommends \
|
||||
build-essential curl ca-certificates pkg-config file xz-utils patch >/dev/null
|
||||
|
||||
# --------------------------------------------------------- the wrappers ----
|
||||
# Identical to tools/python/build.sh. This is the single most important trick
|
||||
# in either file.
|
||||
log "toolchain wrappers (-EL -mnan=2008 baked in)"
|
||||
mkdir -p /opt/xw/bin
|
||||
for t in gcc g++ cpp; do
|
||||
cat > /opt/xw/bin/mips-linux-gnu-$t <<EOF
|
||||
#!/bin/sh
|
||||
exec $TC/bin/mips-linux-gnu-$t -EL -mnan=2008 "\$@"
|
||||
EOF
|
||||
chmod +x /opt/xw/bin/mips-linux-gnu-$t
|
||||
done
|
||||
for t in ar as ld nm objcopy objdump ranlib readelf strip strings size; do
|
||||
ln -sf $TC/bin/mips-linux-gnu-$t /opt/xw/bin/mips-linux-gnu-$t
|
||||
done
|
||||
export PATH=/opt/xw/bin:$PATH
|
||||
|
||||
echo 'int main(void){return 0;}' > /tmp/abi.c
|
||||
mips-linux-gnu-gcc /tmp/abi.c -o /tmp/abi.out
|
||||
FLAGS=$(mips-linux-gnu-readelf -h /tmp/abi.out | awk '/Flags:/{print $2}' | tr -d ,)
|
||||
echo "wrapper produces e_flags=$FLAGS"
|
||||
[ "$FLAGS" = "0x70001405" ] || { echo "!! wrong ABI from the wrapper"; exit 1; }
|
||||
|
||||
# ------------------------------------------------------------ pip ----------
|
||||
# The build-python was configured --without-ensurepip, so it has no pip. It is
|
||||
# the SAME 3.13.7 the cross build used, which is what makes its sysconfig
|
||||
# machinery interchangeable with the target's; a distro python3.11 is not.
|
||||
if ! "$HOSTPY" -c 'import pip' 2>/dev/null; then
|
||||
log "bootstrap pip into the build-python"
|
||||
curl -sSLf -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py
|
||||
"$HOSTPY" /tmp/get-pip.py -q
|
||||
fi
|
||||
# Every package below is built from its SDIST with build isolation OFF, so
|
||||
# every PEP 517 backend any of them names has to be here in advance. That is
|
||||
# not a convenience: with isolation ON, pip rewrites PYTHONPATH for the build
|
||||
# subprocess, the cross _sysconfigdata module stops being importable, and the
|
||||
# build either dies ("No module named '_sysconfigdata__linux_mipsel-linux-gnu'")
|
||||
# or -- worse -- silently answers every sysconfig question for x86-64.
|
||||
"$HOSTPY" -m pip install -q --upgrade \
|
||||
'setuptools>=68' wheel 'cython<3.1' flit_core poetry-core hatchling \
|
||||
hatch-vcs setuptools-scm
|
||||
"$HOSTPY" -VV
|
||||
|
||||
# -------------------------------------------------------- cross env --------
|
||||
log "cross environment"
|
||||
export _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_mipsel-linux-gnu
|
||||
export PYTHONPATH=/usr/data/anvil/lib/python3.13
|
||||
export PYTHONDONTWRITEBYTECODE=1
|
||||
export CC=mips-linux-gnu-gcc
|
||||
export CXX=mips-linux-gnu-g++
|
||||
export AR=mips-linux-gnu-ar
|
||||
export RANLIB=mips-linux-gnu-ranlib
|
||||
export STRIP=mips-linux-gnu-strip
|
||||
export LDSHARED="mips-linux-gnu-gcc -shared -L$DEP/lib"
|
||||
export CFLAGS="-O2 -fPIC -D_FILE_OFFSET_BITS=64 -I$DEP/include -I/usr/data/anvil/include/python3.13"
|
||||
export CXXFLAGS="$CFLAGS"
|
||||
export LDFLAGS="-L$DEP/lib"
|
||||
# pkg-config must see ONLY the cross sysroot, never the container's /usr/lib.
|
||||
export PKG_CONFIG_LIBDIR="$DEP/lib/pkgconfig"
|
||||
export PKG_CONFIG_PATH="$DEP/lib/pkgconfig"
|
||||
|
||||
"$HOSTPY" - <<'PY'
|
||||
import sysconfig
|
||||
for k in ("EXT_SUFFIX", "SOABI", "INCLUDEPY", "CC", "LDSHARED"):
|
||||
print(" %-11s %s" % (k, sysconfig.get_config_var(k)))
|
||||
assert sysconfig.get_config_var("EXT_SUFFIX") == ".cpython-313-mipsel-linux-gnu.so", \
|
||||
"sysconfig is answering for the HOST -- _PYTHON_SYSCONFIGDATA_NAME did not take"
|
||||
PY
|
||||
|
||||
# ------------------------------------------------------------ helpers ------
|
||||
want() { [ -z "${ONLY:-}" ] && return 0; case " $ONLY " in *" $1 "*) return 0;; *) return 1;; esac; }
|
||||
|
||||
# build <spec> [env=val ...]
|
||||
#
|
||||
# --no-binary :all: is not optional and not caution. Without it pip HELPFULLY
|
||||
# downloads a prebuilt manylinux wheel and the "cross build" of cffi, greenlet
|
||||
# and tornado silently becomes x86-64 -- measured, first run of this script:
|
||||
# three x86-64 .so files sailed into the tree and only the ABI gate caught
|
||||
# them. A wheel that is the wrong architecture is not a build failure anywhere
|
||||
# except on the printer.
|
||||
#
|
||||
# --no-build-isolation is what keeps _PYTHON_SYSCONFIGDATA_NAME reaching
|
||||
# setup.py; see the backend pre-install above.
|
||||
build() {
|
||||
local spec=$1 name=${1%%[=<>]*}; shift
|
||||
want "$name" || return 0
|
||||
echo "-- $spec"
|
||||
if env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \
|
||||
--no-binary :all: -w "$WHEELS" "$spec" \
|
||||
>"$LOGS/$name.log" 2>&1; then
|
||||
echo " ok"
|
||||
else
|
||||
echo " !! FAILED -- $LOGS/$name.log"; tail -25 "$LOGS/$name.log"
|
||||
FAILED="$FAILED $name"
|
||||
fi
|
||||
}
|
||||
|
||||
# sdist <url> <sha256> <dir-name>
|
||||
#
|
||||
# Two packages cannot come from `pip wheel <spec>`, for unrelated reasons, and
|
||||
# both need their unpacked source anyway. Fetched by URL and CHECKED, the way
|
||||
# versions.env pins everything else in this repo: the only thing between a
|
||||
# source URL and code the printer executes is the hash.
|
||||
sdist() {
|
||||
local url=$1 want=$2 dir=$3 f="$SRC/$(basename "$1")"
|
||||
[ -s "$f" ] || curl -sSLf -o "$f" "$url"
|
||||
local got
|
||||
got=$(sha256sum "$f" | cut -d" " -f1)
|
||||
[ "$got" = "$want" ] || { echo "!! $(basename "$f") sha256 $got, expected $want"; exit 1; }
|
||||
rm -rf "${SRC:?}/$dir"
|
||||
tar -xzf "$f" -C "$SRC"
|
||||
echo "$SRC/$dir"
|
||||
}
|
||||
|
||||
# builddir <dir> [env=val ...] -- wheel an already-unpacked source tree.
|
||||
builddir() {
|
||||
local dir=$1 name; name=$(basename "$dir"); shift
|
||||
echo "-- $name (from source tree)"
|
||||
if env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \
|
||||
-w "$WHEELS" "$dir" >"$LOGS/$name.log" 2>&1; then
|
||||
echo " ok"
|
||||
else
|
||||
echo " !! FAILED -- $LOGS/$name.log"; tail -25 "$LOGS/$name.log"
|
||||
FAILED="$FAILED $name"
|
||||
fi
|
||||
}
|
||||
|
||||
FAILED=""
|
||||
|
||||
# ============================================================ packages ======
|
||||
# THE LIST IS NOT moonraker-requirements.txt. That file installs every
|
||||
# optional component's dependency -- apprise, ldap3, paho-mqtt, zeroconf,
|
||||
# python-periphery, smart_open, dbus-next's mqtt friends. None of those
|
||||
# components is configured in assets/moonraker.conf, so none of them is built
|
||||
# here. What IS built is the closure of what the ENABLED set imports
|
||||
# (scan-imports.py), plus klippy's three.
|
||||
log "packages"
|
||||
|
||||
# ---- pure python (no compiler is invoked; they are here to be PRESENT) ----
|
||||
build "distro==1.9.0"
|
||||
build "inotify-simple==1.3.5"
|
||||
build "libnacl==2.1.0"
|
||||
build "dbus-next==0.2.3"
|
||||
# preprocess-cancellation 0.2.1's sdist declares version 0.0.0 in its own
|
||||
# metadata -- a broken upstream release, not something about this target. pip
|
||||
# refuses it ("has inconsistent version: expected '0.2.1', but metadata has
|
||||
# '0.0.0'") for the specifier, so the tree is fetched by URL, hash-checked,
|
||||
# and wheeled as a directory, where pip has no specifier to disagree with. It
|
||||
# is a lazily-imported optional path inside the metadata SUBPROCESS
|
||||
# (gcode object cancellation), so a printer without it loses that feature and
|
||||
# nothing else.
|
||||
if want preprocess-cancellation; then
|
||||
builddir "$(sdist \
|
||||
https://files.pythonhosted.org/packages/4a/56/7e18b0336c1e6c6622411dd0d3a7634b171e4d156a13b1ceaa048682454a/preprocess_cancellation-0.2.1.tar.gz \
|
||||
e2f1224e1ba1603bdfdbf6937caaf91082dc849e5122e80a5328aa999433ce79 \
|
||||
preprocess_cancellation-0.2.1)"
|
||||
fi
|
||||
build "pyserial==3.4"
|
||||
build "pyserial-asyncio==0.6"
|
||||
build "pycparser==2.22" # cffi's only runtime dependency
|
||||
# smart_open is not optional, whatever its name suggests, and it is not
|
||||
# moonraker that wants it: streaming-form-data's targets.py imports it at
|
||||
# module scope, so `import streaming_form_data` raises ModuleNotFoundError
|
||||
# without it -- measured in the replica, where every other package imported
|
||||
# and that one did not. moonraker-requirements.txt pins it (`smart_open<=6.4.0`)
|
||||
# for exactly this reason. Pure python, no dependencies of its own.
|
||||
build "smart_open==6.4.0"
|
||||
build "jinja2==3.1.4"
|
||||
# setuptools/pkg_resources are a dependency of nothing here. They are the fix
|
||||
# for a measured failure: a trimmed tree without them made the lmdb egg fall
|
||||
# back to its cffi path and try to invoke mips-linux-gnu-gcc ON THE PRINTER at
|
||||
# Moonraker startup. There is no compiler on the printer.
|
||||
build "setuptools==75.6.0"
|
||||
|
||||
# ---- native ---------------------------------------------------------------
|
||||
# tornado's C speedup is optional and its published wheel carries an x86-64
|
||||
# copy of it, so this builds from sdist like everything else and gets a mipsel
|
||||
# one.
|
||||
build "tornado==6.4.0"
|
||||
|
||||
# markupsafe -- jinja2's escape accelerator, on the path of every gcode macro
|
||||
# klippy renders. Pure-python fallback exists (_native.py) if it ever fights.
|
||||
build "markupsafe==2.1.5"
|
||||
|
||||
# cffi -- klippy's chelper. Wants libffi; deproot has a static -fPIC one and a
|
||||
# .pc for it, and PKG_CONFIG_LIBDIR makes that the only one it can see.
|
||||
build "cffi==1.17.1"
|
||||
|
||||
# greenlet -- klippy's reactor. Expected to fight over its hand-written
|
||||
# stack-switching assembly (switch_mips_unix.h); fought over something else
|
||||
# entirely, and the assembly was never the problem.
|
||||
#
|
||||
# greenlet 3.x is C++ and writes every PyTypeObject / PyNumberMethods as a
|
||||
# DESIGNATED INITIALIZER. Those are C++20; gcc 7.2 -- the Ingenic toolchain,
|
||||
# and the only compiler that produces this printer's ABI -- takes them as an
|
||||
# extension but only in declaration order with no gaps, and refuses anything
|
||||
# else with
|
||||
#
|
||||
# sorry, unimplemented: non-trivial designated initializers not supported
|
||||
#
|
||||
# A PyTypeObject naming 20 of its 50 slots is nothing but gaps, so all three
|
||||
# of greenlet's fail. fill-designators.py writes the skipped fields back as
|
||||
# explicit zeros, reading the field ORDER out of the target interpreter's own
|
||||
# headers rather than a list typed in here -- and refuses to guess if a
|
||||
# designator turns up that the header does not have. These are objects of
|
||||
# static storage duration, so every field it inserts was already zero; the
|
||||
# patch changes the spelling, not the program.
|
||||
if want greenlet; then
|
||||
GL=$(sdist https://files.pythonhosted.org/packages/2f/ff/df5fede753cc10f6a5be0931204ea30c35fa2f2ea7a35b25bdaf4fe40e46/greenlet-3.1.1.tar.gz \
|
||||
4ce3ac6cdb6adf7946475d7ef31777c26d94bccc377e070a7986bd2d5c515467 \
|
||||
greenlet-3.1.1)
|
||||
"$HOSTPY" /pyext/fill-designators.py /usr/data/anvil/include/python3.13 \
|
||||
$(find "$GL/src/greenlet" -name '*.cpp' -o -name '*.hpp' | sort)
|
||||
builddir "$GL"
|
||||
fi
|
||||
|
||||
# lmdb -- moonraker's database AT THIS PIN, and the package that disappears
|
||||
# the moment the pin moves past 80c7620 (lmdb -> sqlite), which is the whole
|
||||
# point of having built an interpreter with _sqlite3.
|
||||
#
|
||||
# LMDB_FORCE_CPYTHON=1 is load-bearing. lmdb's setup.py has two backends: a
|
||||
# real CPython extension and a cffi one that ships mdb.c and COMPILES IT AT
|
||||
# IMPORT TIME. Left to itself here it chose cffi and produced a py3-none-any
|
||||
# wheel containing mdb.c -- i.e. exactly the failure phase 6 already recorded,
|
||||
# rebuilt from scratch. (Do not "fix" this with LMDB_FORCE_CFFI=0: the setup
|
||||
# script tests the variable for PRESENCE, so the string "0" selects cffi.)
|
||||
build "lmdb==1.4.1" LMDB_FORCE_CPYTHON=1
|
||||
|
||||
# streaming-form-data -- moonraker's upload parser, Cython. 1.13.0 (what
|
||||
# moonraker-requirements.txt pins) ships a pregenerated _parser.c from a
|
||||
# Cython that predates 3.13: it calls the four-argument _PyLong_AsByteArray,
|
||||
# which grew a fifth parameter in 3.13, and every integer conversion in the
|
||||
# file fails to compile. Regenerating from the .pyx with a current Cython is
|
||||
# the fix, and a later release ships C that is already regenerated -- moonraker
|
||||
# only asks five names of it (StreamingFormDataParser, ParseFailedException,
|
||||
# FileTarget, ValueTarget, SHA256Target), all present throughout the 1.x line.
|
||||
build "streaming-form-data==1.19.1"
|
||||
|
||||
# pillow -- Moonraker's ONLY use of it is
|
||||
# components/file_manager/metadata.py, which runs as a SUBPROCESS to pull
|
||||
# thumbnails out of gcode. So it is not on the startup path: without it
|
||||
# Moonraker runs and Mainsail shows no thumbnails. It is built anyway because
|
||||
# the cost turned out to be one flag list.
|
||||
#
|
||||
# ZLIB ONLY. Gcode thumbnails are base64 PNG, so zlib is the only codec on
|
||||
# that path; jpeg, tiff, webp, jpeg2000, lcms, freetype and imagequant are
|
||||
# libraries nobody has cross-built here and every one of them is off. Pillow
|
||||
# probes for them by trying to LINK against the host's copies unless told not
|
||||
# to, which is what --disable-platform-guessing is for.
|
||||
#
|
||||
# NOT 10.3.0, which is what moonraker-requirements.txt pins. That release
|
||||
# cannot be built by python 3.13 AT ALL, and not for any reason to do with
|
||||
# this target: its setup.py reads its own version with
|
||||
#
|
||||
# exec(compile(open("src/PIL/_version.py").read(), ...))
|
||||
# return locals()["__version__"]
|
||||
#
|
||||
# and PEP 667 (new in 3.13) made a function's locals() a snapshot, so the
|
||||
# name exec() defined is not there any more and setup.py dies with
|
||||
# KeyError: '__version__' before it compiles a single file. 11.0.0 is the
|
||||
# first line that supports 3.13. Moonraker asks PIL for Image.open, resize
|
||||
# and save, all unchanged across that gap.
|
||||
#
|
||||
# pip cannot drive this: the disable flags are build_ext options and there is
|
||||
# no PEP 517 path to them, so setup.py is called directly.
|
||||
if want pillow; then
|
||||
PIL_SRC=$(sdist https://files.pythonhosted.org/packages/a5/26/0d95c04c868f6bdb0c447e3ee2de5564411845e36a858cfd63766bc7b563/pillow-11.0.0.tar.gz \
|
||||
72bacbaf24ac003fea9bff9837d1eedb6088758d41e100c1552930151f677739 \
|
||||
pillow-11.0.0)
|
||||
echo "-- pillow 11.0.0 (zlib only, setup.py direct)"
|
||||
if ( cd "$PIL_SRC" && "$HOSTPY" setup.py build_ext \
|
||||
--disable-jpeg --disable-tiff --disable-webp --disable-jpeg2000 \
|
||||
--disable-imagequant --disable-lcms --disable-freetype \
|
||||
--disable-xcb --disable-platform-guessing --enable-zlib \
|
||||
-I"$DEP/include" -L"$DEP/lib" bdist_wheel ) \
|
||||
>"$LOGS/pillow.log" 2>&1; then
|
||||
cp -a "$PIL_SRC"/dist/*.whl "$WHEELS/"
|
||||
echo " ok"
|
||||
else
|
||||
echo " !! FAILED -- $LOGS/pillow.log"; tail -25 "$LOGS/pillow.log"
|
||||
FAILED="$FAILED pillow"
|
||||
fi
|
||||
fi
|
||||
|
||||
# =========================================================== assemble =======
|
||||
log "wheels built"
|
||||
ls -1 "$WHEELS" | sort
|
||||
|
||||
log "unpack into a site-packages tree"
|
||||
rm -rf "$SP"; mkdir -p "$SP"
|
||||
for w in "$WHEELS"/*.whl; do
|
||||
"$HOSTPY" -m zipfile -e "$w" "$SP"
|
||||
done
|
||||
find "$SP" -name '__pycache__' -prune -exec rm -rf {} + 2>/dev/null || true
|
||||
find "$SP" -name '*.dist-info' -prune -exec rm -rf {} + 2>/dev/null || true
|
||||
rm -rf "$SP"/bin "$SP"/*.data 2>/dev/null || true
|
||||
|
||||
# ============================================================= gates ========
|
||||
log "ABI gate on every .so in the tree"
|
||||
BAD=0; N=0
|
||||
while IFS= read -r f; do
|
||||
case "$(file -b "$f")" in *ELF*) ;; *) continue;; esac
|
||||
hdr=$(mips-linux-gnu-readelf -h "$f" 2>/dev/null)
|
||||
fl=$(echo "$hdr" | awk '/Flags:/{print $2}' | tr -d ,)
|
||||
typ=$(echo "$hdr" | awk '/^ Type:/{print $2}')
|
||||
want=0x70001405; [ "$typ" = "DYN" ] && want=0x70001407
|
||||
case "$hdr" in
|
||||
*nan2008*o32*mips32r2*) ;;
|
||||
*) echo "!! $f not nan2008/o32/mips32r2"; BAD=1; continue;;
|
||||
esac
|
||||
[ "$fl" = "$want" ] || { echo "!! $f type=$typ flags=$fl want=$want"; BAD=1; }
|
||||
N=$((N+1))
|
||||
done < <(find "$SP" -type f -name '*.so')
|
||||
[ $BAD -eq 0 ] || exit 1
|
||||
echo "$N mipsel extension modules, all nan2008/o32/mips32r2 DYN 0x70001407"
|
||||
|
||||
echo
|
||||
echo "-- NEEDED of each extension module (nothing may want a /usr/prog soname) --"
|
||||
find "$SP" -name '*.so' | sort | while read -r f; do
|
||||
printf '%-58s %s\n' "${f#$SP/}" \
|
||||
"$(mips-linux-gnu-readelf -d "$f" | awk '/NEEDED/{gsub(/[][]/,"",$5); printf "%s ", $5}')"
|
||||
done
|
||||
|
||||
log "strip and size"
|
||||
find "$SP" -name '*.so' -exec mips-linux-gnu-strip {} + 2>/dev/null || true
|
||||
du -sh "$SP"
|
||||
echo "$(find "$SP" -type f | wc -l) files"
|
||||
echo "-- top level --"
|
||||
ls -1 "$SP"
|
||||
|
||||
log "tarball"
|
||||
# Root is site-packages/ so a case script can drop it beside the interpreter's
|
||||
# own lib/python3.13/site-packages.
|
||||
tar -czf "$P/pyext.tgz" -C "$P" site-packages
|
||||
ls -l "$P/pyext.tgz"
|
||||
|
||||
chown -R "${HOST_UID:-0}:${HOST_GID:-0}" "$P" 2>/dev/null || true
|
||||
|
||||
END=$(date +%s)
|
||||
log "done in $(( (END-START)/60 ))m $(( (END-START)%60 ))s"
|
||||
[ -z "$FAILED" ] && echo "all packages built" || { echo "!! FAILED:$FAILED"; exit 1; }
|
||||
Executable
+227
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Make C++ designated initializers digestible by gcc 7.2.
|
||||
|
||||
fill-designators.py <include/python3.13> <file.cpp> [file.cpp ...]
|
||||
|
||||
WHY THIS EXISTS. greenlet 3.x writes its PyTypeObject, PyNumberMethods and
|
||||
friends as C++ designated initializers:
|
||||
|
||||
PyTypeObject PyGreenlet_Type = {
|
||||
.ob_base=PyVarObject_HEAD_INIT(NULL, 0)
|
||||
.tp_name="greenlet.greenlet",
|
||||
.tp_basicsize=sizeof(PyGreenlet),
|
||||
.tp_dealloc=(destructor)green_dealloc,
|
||||
...
|
||||
|
||||
Designated initializers are C++20. gcc 7.2 -- the Ingenic toolchain, and the
|
||||
only compiler this printer's ABI has -- supports them as an extension, but
|
||||
only IN DECLARATION ORDER AND CONTIGUOUS. Measured, probe-desig.sh:
|
||||
|
||||
in order, contiguous OK
|
||||
in order, TRAILING gap OK -- the only gap gcc 7.2 tolerates
|
||||
in order, INTERIOR gap sorry, unimplemented: non-trivial designated
|
||||
initializers not supported
|
||||
in order, LEADING gap same -- {.nb_bool=x} on a PyNumberMethods whose
|
||||
first field is nb_add is already too much
|
||||
out of order same
|
||||
|
||||
Every one of greenlet's initializers skips fields -- a PyTypeObject naming 20
|
||||
of its 50 slots is the normal way to write one -- so all of them are interior
|
||||
gaps, and the build stops on the first.
|
||||
|
||||
WHAT THIS DOES. It reads the field order out of the TARGET interpreter's own
|
||||
headers and writes the skipped fields back in as explicit zeros. That is a
|
||||
no-op semantically: these are objects of static storage duration, so every
|
||||
field this inserts was already going to be zero-initialized. It is not a
|
||||
behaviour patch, it is the same initializer spelled in a way gcc 7.2 will
|
||||
accept, and if a designator ever appears that is NOT in the header's field
|
||||
list the script fails loudly rather than guessing.
|
||||
|
||||
Every field it fills is a pointer or an integer, so `= 0` is the right zero;
|
||||
the script asserts that too, rather than trusting it.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
INCLUDE = sys.argv[1]
|
||||
FILES = sys.argv[2:]
|
||||
|
||||
# The initializer types greenlet uses. Each is (C type name as written in the
|
||||
# source, the header declaration to read the field order from).
|
||||
STRUCTS = {
|
||||
"PyTypeObject": ("cpython/object.h", "struct _typeobject"),
|
||||
"PyNumberMethods": ("cpython/object.h", "typedef struct"),
|
||||
"PyAsyncMethods": ("cpython/object.h", "typedef struct"),
|
||||
"PySequenceMethods": ("cpython/object.h", "typedef struct"),
|
||||
"PyMappingMethods": ("cpython/object.h", "typedef struct"),
|
||||
"PyModuleDef": ("moduleobject.h", "struct PyModuleDef"),
|
||||
}
|
||||
|
||||
# A field declaration line. Two shapes in CPython's headers that a naive
|
||||
# regex gets wrong, and both appear in PyTypeObject:
|
||||
# * `PyObject_VAR_HEAD` -- a MACRO, not a declaration; it expands to the
|
||||
# `ob_base` field the source designates by name.
|
||||
# * `Py_ssize_t tp_basicsize, tp_itemsize;` -- two declarators on one line.
|
||||
FIELD = re.compile(r"^\s*(?:const\s+|struct\s+|unsigned\s+)*"
|
||||
r"[A-Za-z_][A-Za-z0-9_]*\s*"
|
||||
r"(?P<decls>[^;{}()]*);\s*$")
|
||||
DECLARATOR = re.compile(r"\**\s*([A-Za-z_][A-Za-z0-9_]*)\s*(?:\[[^]]*\])?$")
|
||||
HEAD_MACROS = {"PyObject_HEAD": "ob_base", "PyObject_VAR_HEAD": "ob_base"}
|
||||
|
||||
|
||||
def read_header(path):
|
||||
for root, _dirs, files in os.walk(INCLUDE):
|
||||
if os.path.basename(path) in files and root.endswith(
|
||||
os.path.dirname(path) or root[len(root):] or ""):
|
||||
return open(os.path.join(root, os.path.basename(path))).read()
|
||||
# fall back: anywhere under the include tree
|
||||
for root, _dirs, files in os.walk(INCLUDE):
|
||||
if os.path.basename(path) in files:
|
||||
return open(os.path.join(root, os.path.basename(path))).read()
|
||||
raise SystemExit("no header %s under %s" % (path, INCLUDE))
|
||||
|
||||
|
||||
def fields_of(cname):
|
||||
"""Ordered field names of a struct, read from the target's headers."""
|
||||
header, _kind = STRUCTS[cname]
|
||||
text = read_header(header)
|
||||
# find "} <cname>;" or "struct _typeobject {"
|
||||
if cname == "PyTypeObject":
|
||||
start = text.index("struct _typeobject {")
|
||||
start = text.index("{", start) + 1
|
||||
else:
|
||||
end = re.search(r"\}\s*%s\s*;" % cname, text)
|
||||
if not end:
|
||||
return None
|
||||
# walk back to the matching "typedef struct {"
|
||||
head = text[:end.start()]
|
||||
start = head.rindex("typedef struct {") + len("typedef struct {")
|
||||
end_i = end.start()
|
||||
return _scan(text[start:end_i])
|
||||
depth = 1
|
||||
i = start
|
||||
while depth:
|
||||
if text[i] == "{":
|
||||
depth += 1
|
||||
elif text[i] == "}":
|
||||
depth -= 1
|
||||
i += 1
|
||||
return _scan(text[start:i - 1])
|
||||
|
||||
|
||||
def _scan(body):
|
||||
body = re.sub(r"/\*.*?\*/", "", body, flags=re.S)
|
||||
body = re.sub(r"//[^\n]*", "", body)
|
||||
out = []
|
||||
depth = 0
|
||||
for line in body.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped in HEAD_MACROS:
|
||||
out.append(HEAD_MACROS[stripped])
|
||||
continue
|
||||
depth += line.count("(") - line.count(")")
|
||||
if depth:
|
||||
continue
|
||||
m = FIELD.match(line)
|
||||
if not m:
|
||||
continue
|
||||
for decl in m.group("decls").split(","):
|
||||
d = DECLARATOR.search(decl.strip())
|
||||
if d:
|
||||
out.append(d.group(1))
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- rewrite ---
|
||||
# Match "<Type> <name> = {" ... matching "}" at the top level of the file.
|
||||
DECL = re.compile(r"^(\s*)(?:static\s+)?(%s)\s+([A-Za-z_][A-Za-z0-9_]*)\s*=\s*\{"
|
||||
% "|".join(STRUCTS), re.M)
|
||||
|
||||
changed_total = 0
|
||||
for path in FILES:
|
||||
src = open(path).read()
|
||||
out = []
|
||||
pos = 0
|
||||
changed = 0
|
||||
for m in DECL.finditer(src):
|
||||
cname = m.group(2)
|
||||
order = fields_of(cname)
|
||||
if not order:
|
||||
continue
|
||||
# find the matching close brace of this initializer
|
||||
i = m.end()
|
||||
depth = 1
|
||||
while depth:
|
||||
if src[i] == "{":
|
||||
depth += 1
|
||||
elif src[i] == "}":
|
||||
depth -= 1
|
||||
i += 1
|
||||
body = src[m.end():i - 1]
|
||||
|
||||
used = [d for d in re.findall(r"\.([A-Za-z_][A-Za-z0-9_]*)\s*=", body)]
|
||||
# only top-level designators: drop any that are not fields of this
|
||||
# struct (they belong to a nested brace, e.g. .ob_base={...}).
|
||||
used = [u for u in used if u in order]
|
||||
if not used:
|
||||
continue
|
||||
unknown = [u for u in re.findall(r"^\s*\.([A-Za-z_][A-Za-z0-9_]*)\s*=",
|
||||
body, re.M) if u not in order]
|
||||
if unknown:
|
||||
raise SystemExit("%s: %s names %r, not a field of %s -- header "
|
||||
"and source disagree, refusing to guess"
|
||||
% (path, m.group(3), unknown, cname))
|
||||
idx = [order.index(u) for u in used]
|
||||
if idx != sorted(idx):
|
||||
raise SystemExit("%s: %s designators are out of declaration order "
|
||||
"(%r) -- gcc 7.2 cannot take that either and "
|
||||
"reordering them is not this script's job"
|
||||
% (path, m.group(3), used))
|
||||
|
||||
# Insert the skipped fields, in order, as explicit zeros. Everything
|
||||
# between the first and last named field only.
|
||||
new_body = body
|
||||
gaps = []
|
||||
for a, b in zip(idx, idx[1:]):
|
||||
if b - a > 1:
|
||||
gaps.append((order[a], order[a + 1:b]))
|
||||
for after, missing in reversed(gaps):
|
||||
fill = "".join(" .%s=0,\n" % f for f in missing)
|
||||
# place the fill immediately after the entry for `after`
|
||||
am = re.search(r"\.%s\s*=" % re.escape(after), new_body)
|
||||
j = am.end()
|
||||
depth2 = 0
|
||||
while True:
|
||||
c = new_body[j]
|
||||
if c in "([{":
|
||||
depth2 += 1
|
||||
elif c in ")]}":
|
||||
if depth2 == 0:
|
||||
break
|
||||
depth2 -= 1
|
||||
elif c == "," and depth2 == 0:
|
||||
j += 1
|
||||
break
|
||||
j += 1
|
||||
new_body = new_body[:j] + "\n" + fill + new_body[j:]
|
||||
changed += len(missing)
|
||||
|
||||
# A gap BEFORE the first designator is refused too -- greenlet's
|
||||
# `static PyNumberMethods green_as_number = {.nb_bool=...}` names the
|
||||
# tenth field and nothing before it. Fill from field zero.
|
||||
if idx[0] > 0:
|
||||
fill = "".join(" .%s=0,\n" % f for f in order[:idx[0]])
|
||||
new_body = "\n" + fill + new_body
|
||||
changed += idx[0]
|
||||
|
||||
out.append(src[pos:m.end()])
|
||||
out.append(new_body)
|
||||
pos = i - 1
|
||||
out.append(src[pos:])
|
||||
if changed:
|
||||
open(path, "w").write("".join(out))
|
||||
print(" %s: filled %d skipped field(s)" % (path, changed))
|
||||
changed_total += changed
|
||||
|
||||
print("fill-designators: %d field(s) written back" % changed_total)
|
||||
Reference in new issue
Block a user