starred/firmware
1
0 Fork 0
mirror of https://github.com/Klipper4FlashForge/firmware.git synced 2026-10-21 21:02:16 -05:00
Code Issues Packages Projects Releases 8 Wiki Activity

Moonraker serves on the 3.13 we built, with nothing from /usr/prog

Browse Source 
The previous commit proved every component IMPORTS. Importing is not serving,
so this runs it. case-moonraker313.sh starts Moonraker on the exact command
line payload/etc/s6/moonraker/run execs -- only with our interpreter --
against the mod's own assets/moonraker.conf, and asks the printer what
happened:

  * bound :7125 in 3s, with /proc/PID/cmdline confirming our interpreter and
    our entry point rather than a coincidence
  * GET /server/info answers 200 with 23 components loaded and none failed.
    /machine/system_info, a different component reading this box's own /proc,
    reports cpu: mips and python 3.13.7 [GCC 7.2.0]
  * the database really works, which is the entire reason for the exercise: a
    POST wrote a namespace, a GET read it back, the namespace appears beside
    Moonraker's own, and data.mdb is 147456 bytes on disk -- through lmdb's
    CPython extension, not the compile-at-import cffi fallback that would try
    to invoke a compiler this printer does not have
  * still alive and still answering 120 seconds later, with no tracebacks in
    its log, and SIGTERM released the port in 2s
  * the RUNNING process maps zero libraries under /usr/prog, with
    LD_LIBRARY_PATH completely unset -- not merely purged of /usr/prog
  * negative controls: nothing on :7125 beforehand, and with site-packages
    moved aside it never binds at all

libsodium was the last /usr/prog string in the picture and is now ours.
libsodium 1.0.20, cross-built with the Ingenic toolchain in 24 seconds, gated
at 0x70001407 nan2008/o32/mips32r2, needing only libpthread, libc and the
loader, 406KB stripped into $MODDIR/lib. It computes rather than merely
loading: sodium_version_string is 1.0.20, its BLAKE2b-256 matches CPython's
own _blake2 on the same kernel, and libnacl.sign.Signer/Verifier -- the exact
ed25519 pair authorization.py imports -- signs, verifies, and rejects a
tampered message.

It costs anvil-env.sh nothing, which was not obvious and had to be measured.
libnacl's third fallback builds a path out of its own __file__, so from
$MODDIR/lib/python3.13/site-packages/libnacl it resolves $MODDIR/lib/libsodium.so
by absolute path -- working with LD_LIBRARY_PATH unset entirely. Recorded in
the README, because it works only because the prefix happens to contain the
string "lib": a prefix without it would break this silently.

klippy was scanned rather than guessed at. Its module-scope third-party
imports are cffi, greenlet, jinja2, pyserial and numpy; four of the five are
built and proven, and cffi already dlopens the printer's real c_helper.so.
numpy is at module scope only in extras/stepper_resonance_tester.py, which
Klipper loads on demand, and it is a multi-day problem that costs input-shaper
calibration rather than printing. There is no removed-stdlib usage anywhere in
klippy -- no imp, distutils, asyncore, cgi or getargspec.

What is left is packaging, not capability, and the README says so in order.
The sharp one: section 5c's trim deletes include/ and config-3.13-*, which the
package build needs, so the package build has to run inside 5c against the
untrimmed stage. And the ABI gate walks the build cache rather than the
payload, so anything staged straight into $MOD_PAYLOAD would bypass it --
coverage, not logic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oleksandr ShyshatskyiandClaude Opus 5 committed 2026-08-27 10:41:24 +03:00
1 parent da65764b86
commit 6a750f6bcc
3 files changed
+708 -16
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

test/integration/printer/case-moonraker313.sh
Executable
+487
View File
@@ -0,0 +1,487 @@
#!/bin/sh
# Does Moonraker actually SERVE on the CPython 3.13 this repo cross-builds?
#
# THE DECISIVE GATE for phase 6. case-pyext.sh proved that all 16 components
# IMPORT on 3.13. Importing is not serving: nothing had ever bound :7125 on
# this interpreter. An import exercises module-level code and nothing else --
# it does not open an lmdb environment, does not construct a tornado
# Application, does not run an asyncio event loop on this kernel, and does not
# touch a socket. Every one of those is a separate way for a cross-built
# interpreter to fail, and every one of them is between here and FF_PYTHON
# pointing at 3.13.
#
# It also settles the last /usr/prog string. libnacl is pure Python and
# ctypes-loads libsodium, which on this firmware exists ONLY at
# /usr/prog/libsodium/lib -- so Moonraker's `authorization` component has kept
# one dependency on FlashForge's tree through the whole exercise. sodium.tgz
# is a libsodium cross-built into the mod's own prefix; section 1 takes
# /usr/prog away and makes libnacl work anyway.
#
# The approach is case-moonraker.sh's, deliberately: stage the tree, start it
# the way the shipped `run` script starts it (`python moonraker.py -d
# /usr/data`), wait for the PORT rather than for the process, and then ask the
# HTTP API. What is NOT reused is the s6/S62moonraker machinery -- that is
# already gated against 3.8 and re-running it here would measure init scripts,
# not the interpreter. This runs the entry point directly, with the same
# arguments and the same data path, so that anything that fails is the
# interpreter, the extensions or Moonraker itself.
FAIL=0
ok() { echo " PASS $*"; }
bad() { echo " FAIL $*"; FAIL=1; }
note(){ echo " .. $*"; }
skip(){ echo " SKIP $*"; }
MODDIR=/usr/data/anvil
PY=$MODDIR/bin/python3.13
SP=$MODDIR/lib/python3.13/site-packages
MRROOT=/usr/data/mrsrc
MAIN=$MRROOT/moonraker/moonraker.py
DATA=/usr/data
PORT=7125
SODIUM_PROG=/usr/prog/libsodium/lib
# ---- stage ----------------------------------------------------------------
mkdir -p $MODDIR
gzip -dc /mnt/py.tgz | tar -x -C $MODDIR || { bad "cannot unpack py.tgz"; exit 1; }
chmod +x $PY 2>/dev/null
mkdir -p $SP
gzip -dc /mnt/ext.tgz | tar -x -C /tmp || { bad "cannot unpack ext.tgz"; exit 1; }
cp -a /tmp/site-packages/. $SP/ || { bad "cannot stage site-packages"; exit 1; }
# libsodium goes into the SAME prefix lib/ the stdlib lives under. That is not
# only tidiness -- see section 1c for why the path matters to libnacl.
gzip -dc /mnt/sodium.tgz | tar -x -C $MODDIR || { bad "cannot unpack sodium.tgz"; exit 1; }
mkdir -p $MRROOT
gzip -dc /mnt/mr.tgz | tar -x -C $MRROOT || { bad "cannot unpack mr.tgz"; exit 1; }
# The mod's own moonraker.conf, from assets/, at the path moonraker -d
# /usr/data resolves to. Not a config invented here: a config the mod ships.
gzip -dc /mnt/conf.tgz | tar -x -C $DATA || { bad "cannot unpack conf.tgz"; exit 1; }
mkdir -p /usr/data/logs /usr/data/tmp
[ -x "$PY" ] || { bad "no interpreter at $PY"; exit 1; }
[ -f "$MAIN" ] || { bad "no moonraker.py at $MAIN"; exit 1; }
note "interpreter: `$PY -c 'import sys;print(sys.version.split()[0])' 2>&1`"
note "libsodium staged: `ls -l $MODDIR/lib/libsodium.so* 2>/dev/null | wc -l` names"
echo
echo "=== 1. libsodium: the last /usr/prog dependency ==="
# ---- 1a. the negative control, FIRST ---------------------------------------
# Everything below is worthless unless libnacl genuinely cannot find a
# libsodium when neither copy is reachable. If it can, some third libsodium is
# on this rootfs and sections 1b/1c are measuring nothing.
mv $MODDIR/lib/libsodium.so /tmp/ls.so 2>/dev/null
mv $MODDIR/lib/libsodium.so.26 /tmp/ls.so.26 2>/dev/null
mv $MODDIR/lib/libsodium.so.26.2.0 /tmp/ls.so.full 2>/dev/null
# The sentinel is printed with a leading newline and matched anchored, because
# CPython 3.13 echoes the failing SOURCE LINE into its traceback -- a `-c`
# program that contains the word it greps for passes its own negative control.
# That is not hypothetical; it is what the first run of this section did.
env -u LD_LIBRARY_PATH $PY -c \
'import libnacl; print("\nSENTINEL-loaded", libnacl.nacl._name)' \
> /tmp/nacl-neg.out 2>&1
if grep -q '^SENTINEL-loaded' /tmp/nacl-neg.out; then
bad "negative control: libnacl found a libsodium with ours moved aside and no path: `cat /tmp/nacl-neg.out`"
else
ok "negative control: with ours moved aside and /usr/prog off the path, libnacl fails -- `sed -n '$p' /tmp/nacl-neg.out`"
fi
mv /tmp/ls.so.full $MODDIR/lib/libsodium.so.26.2.0 2>/dev/null
mv /tmp/ls.so.26 $MODDIR/lib/libsodium.so.26 2>/dev/null
mv /tmp/ls.so $MODDIR/lib/libsodium.so 2>/dev/null
# ---- 1b. ours, on LD_LIBRARY_PATH, with /usr/prog NOWHERE ------------------
LD_LIBRARY_PATH=$MODDIR/lib $PY -c \
'import libnacl; print("libnacl ok:", libnacl.nacl._name)' \
> /tmp/nacl-ours.out 2>&1
if grep -q '^libnacl ok' /tmp/nacl-ours.out; then
ok "libnacl loads OUR libsodium with \$MODDIR/lib on LD_LIBRARY_PATH: `cat /tmp/nacl-ours.out`"
else
bad "libnacl with \$MODDIR/lib on the path: `tail -3 /tmp/nacl-ours.out`"
fi
# ---- 1c. and with NO path at all -------------------------------------------
# libnacl's own search is worth reading rather than assuming. After
# ctypes.util.find_library and a bare dlopen("libsodium.so") it tries
#
# libpath = __file__[0 : __file__.find("lib") + 3] + "/libsodium.so"
#
# and __file__ here is $MODDIR/lib/python3.13/site-packages/libnacl/__init__.py
# -- so the first "lib" in that path IS the prefix's own lib/, and the fallback
# resolves to $MODDIR/lib/libsodium.so by ABSOLUTE path. Which means putting
# libsodium in the prefix may cost anvil-env.sh nothing at all. Measured, not
# assumed, because it is a coincidence of the prefix name and a future prefix
# without "lib" in it would break it silently.
env -u LD_LIBRARY_PATH $PY -c \
'import libnacl; print("libnacl ok:", libnacl.nacl._name)' \
> /tmp/nacl-nopath.out 2>&1
if grep -q '^libnacl ok' /tmp/nacl-nopath.out; then
ok "libnacl finds it with LD_LIBRARY_PATH UNSET too: `cat /tmp/nacl-nopath.out`"
SODIUM_NEEDS_PATH=0
else
note "with no LD_LIBRARY_PATH libnacl does not find it: `tail -2 /tmp/nacl-nopath.out`"
note "so \$MODDIR/lib has to go on LD_LIBRARY_PATH -- an anvil-env.sh change"
SODIUM_NEEDS_PATH=1
fi
# ---- 1d. it is not merely loaded, it computes ------------------------------
# A dlopen that succeeds and a crypto library that works are different claims:
# a wrong-ABI .so fails at dlopen, but a miscompiled one loads and returns
# rubbish. So run a real round trip through it and check the ANSWER.
LD_LIBRARY_PATH=$MODDIR/lib $PY - <<'EOP' > /tmp/nacl-use.out 2>&1
import binascii, ctypes
import libnacl
import libnacl.sign
libnacl.nacl.sodium_version_string.restype = ctypes.c_char_p
print("sodium_version_string(): %s" % libnacl.nacl.sodium_version_string().decode())
# A KNOWN-ANSWER test, because a dlopen that succeeds and a crypto library
# that is correct are different claims -- a miscompiled one loads and returns
# rubbish. crypto_generichash is BLAKE2b at libsodium's default 32-byte
# output, and the ANSWER is not written down here: it is computed by CPython's
# own _blake2 extension, in this process, on this kernel. Two independent
# mipsel builds of the same algorithm agreeing is a stronger statement than a
# constant pasted from a spec -- and a constant pasted from a spec is how the
# first version of this check failed, against a correct library, because 512
# is the algorithm's default and 32 bytes is libsodium's.
import hashlib
got = binascii.hexlify(libnacl.crypto_generichash(b"nozzle at 210")).decode()
want = hashlib.blake2b(b"nozzle at 210", digest_size=32).hexdigest()
print("blake2b-256: libsodium=%s" % got)
print(" cpython =%s" % want)
assert got == want, (got, want)
# And then EXACTLY what moonraker/components/authorization.py imports and
# uses: libnacl.sign.Signer / Verifier, the ed25519 pair behind its JWTs.
signer = libnacl.sign.Signer()
sig = signer.signature(b"nozzle at 210")
ver = libnacl.sign.Verifier(signer.hex_vk())
assert ver.verify(sig + b"nozzle at 210") == b"nozzle at 210"
try:
ver.verify(sig + b"nozzle at 211")
print("FORGERY ACCEPTED")
except ValueError:
print("ed25519 sign/verify ok, and a tampered message is rejected")
print("libsodium works")
EOP
if grep -q '^libsodium works' /tmp/nacl-use.out; then
ok "and it computes -- not merely loads:"
grep -v '^libsodium works' /tmp/nacl-use.out | sed 's/^/ /'
else
bad "libsodium loaded but does not work:"
sed 's/^/ /' /tmp/nacl-use.out | tail -8
fi
# ---- 1e. the authorization component, with /usr/prog unreachable -----------
# The component that caused the original outage, on the new interpreter, with
# the library it needs coming from our prefix.
LD_LIBRARY_PATH=$MODDIR/lib $PY -c "
import sys; sys.path.insert(0, '$MRROOT')
import moonraker.components.authorization as a
print('authorization imported from', a.__file__)
" > /tmp/auth.out 2>&1
grep -q '^authorization imported' /tmp/auth.out \
&& ok "moonraker's authorization component imports on 3.13 with no /usr/prog libsodium" \
|| bad "authorization: `tail -3 /tmp/auth.out`"
echo
echo "=== 2. THE DECISIVE ONE: does it SERVE? ==="
# The environment the server gets: LD_LIBRARY_PATH is UNSET, not merely
# purged of /usr/prog. That is the strongest form of the claim and the one
# worth measuring -- the interpreter is statically linked against its seven C
# libraries, every extension was linked by the same toolchain, and section 1c
# showed libnacl resolving the prefix path by itself. If anything in the
# server needs a search path, this is where it says so.
# TMPDIR is off the ramdisk, exactly as payload/etc/s6/moonraker/run sets it.
MRENV="TMPDIR=/usr/data/tmp"
mrun() { env -u LD_LIBRARY_PATH $MRENV "$@"; }
port_listening() {
awk -v p=":`printf '%04X' $PORT`" '$2 ~ p"$" && $4 == "0A" { f = 1 }
END { exit !f }' /proc/net/tcp 2>/dev/null
}
# ---- 2a. negative control: nothing is listening yet ------------------------
if port_listening; then
bad "negative control: something is ALREADY listening on :$PORT"
else
ok "negative control: nothing is listening on :$PORT before we start"
fi
# ---- 2b. start it ----------------------------------------------------------
# Exactly the command line payload/etc/s6/moonraker/run execs, with our
# interpreter substituted for FF_PYTHON and the log named so it cannot be
# confused with a 3.8 run.
LOG=/usr/data/logs/moonraker313.log
rm -f $LOG
echo " .. starting: $PY $MAIN -d $DATA -l $LOG"
# NOT `mrun ... &`. Backgrounding a shell FUNCTION makes busybox sh fork a
# subshell and hand back the SUBSHELL's pid, which still carries the case
# script's own cmdline -- so every check below reads /proc of the wrong
# process, the SIGTERM at the end kills the wrong process, and the server
# survives into the next section. Measured, and it cost a replica run. A
# simple command is exec'd by the background child, so $! is the server.
env -u LD_LIBRARY_PATH $MRENV $PY $MAIN -d $DATA -l $LOG > /tmp/mr-stdout.txt 2>&1 &
MRPID=$!
note "pid $MRPID"
# And prove that, rather than trusting it: a pid whose cmdline is this script
# is the bug above, not a server.
note "pid $MRPID cmdline: `tr '\0' ' ' < /proc/$MRPID/cmdline 2>/dev/null`"
waited=0
while [ $waited -lt 240 ]; do
port_listening && break
kill -0 $MRPID 2>/dev/null || break
sleep 3
waited=$((waited + 3))
done
if ! kill -0 $MRPID 2>/dev/null; then
bad "moonraker EXITED before binding :$PORT (after ${waited}s)"
echo " ---- stdout/stderr ----"
sed 's/^/ /' /tmp/mr-stdout.txt | tail -n 40
echo " ---- $LOG ----"
tail -n 60 $LOG 2>/dev/null | sed 's/^/ /'
echo
echo "RESULT: FAIL (moonraker does not serve on 3.13)"
exit 1
fi
if port_listening; then
ok "moonraker BOUND :$PORT after ${waited}s, under $PY"
else
bad "moonraker is still alive after ${waited}s but never bound :$PORT"
tail -n 40 $LOG 2>/dev/null | sed 's/^/ /'
fi
# ---- 2c. it is our interpreter holding the socket --------------------------
CMD=`tr '\0' ' ' < /proc/$MRPID/cmdline 2>/dev/null`
case "$CMD" in
*"$PY"*) ok "the process is running under $PY" ;;
*) bad "the process is '$CMD', not our interpreter" ;;
esac
case "$CMD" in
*"$MAIN"*) ok "and it is running $MAIN" ;;
*) bad "it is not running $MAIN: $CMD" ;;
esac
# ---- 2d. the API answers ---------------------------------------------------
# The HTTP client is our own 3.13, which is convenient and also honest: if the
# client cannot make an HTTP request the interpreter has a bigger problem than
# Moonraker. Every call goes through one helper so a failure names the URL.
cat > /tmp/get.py <<'EOP'
import json, sys, urllib.request
url = sys.argv[1]
data = None
method = "GET"
if len(sys.argv) > 2:
method = "POST"
data = sys.argv[2].encode()
req = urllib.request.Request(url, data=data, method=method,
headers={"Content-Type": "application/json"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
body = r.read().decode()
print("HTTP %d" % r.status)
print(body)
except Exception as exc:
print("REQUEST FAILED: %r" % (exc,))
sys.exit(1)
EOP
get() { mrun $PY /tmp/get.py "http://127.0.0.1:$PORT$1" ${2:+"$2"}; }
# /server/info -- the endpoint every UI hits first, and the one that names the
# components the server actually LOADED. That list is the point: a component
# that imports but throws in its constructor never reaches it.
get /server/info > /tmp/info.out 2>&1
if grep -q '^HTTP 200' /tmp/info.out; then
ok "GET /server/info answers 200"
mrun $PY - <<'EOP' > /tmp/info2.out 2>&1
import json
raw = open("/tmp/info.out").read().split("\n", 1)[1]
d = json.loads(raw)["result"]
comps = sorted(d.get("components", []))
fail = sorted(d.get("failed_components", []))
print("klippy_connected=%s klippy_state=%s api_version=%s" % (
d.get("klippy_connected"), d.get("klippy_state"), d.get("api_version_string")))
print("moonraker_version=%s" % d.get("moonraker_version"))
print("components(%d): %s" % (len(comps), ", ".join(comps)))
print("failed_components(%d): %s" % (len(fail), ", ".join(map(str, fail))))
print("warnings: %s" % (d.get("warnings"),))
EOP
sed 's/^/ /' /tmp/info2.out
NCOMP=`sed -n 's/^components(\([0-9]*\)).*/\1/p' /tmp/info2.out`
[ -n "$NCOMP" ] && [ "$NCOMP" -ge 16 ] \
&& ok "the JSON names $NCOMP loaded components" \
|| bad "server/info reports only '$NCOMP' components"
grep -q '^failed_components(0)' /tmp/info2.out \
&& ok "no failed components" \
|| bad "`grep '^failed_components' /tmp/info2.out`"
else
bad "GET /server/info: `tail -2 /tmp/info.out`"
fi
# A SECOND endpoint, and deliberately one served by a different component:
# /server/info is the Server object itself, /machine/system_info is the
# `machine` component reading this box's /proc and /sys. Two endpoints from
# two components is the difference between "tornado routes" and "Moonraker
# runs".
get /machine/system_info > /tmp/sys.out 2>&1
if grep -q '^HTTP 200' /tmp/sys.out; then
ok "GET /machine/system_info answers 200"
mrun $PY - <<'EOP' 2>&1 | sed 's/^/ /'
import json
raw = open("/tmp/sys.out").read().split("\n", 1)[1]
si = json.loads(raw)["result"]["system_info"]
cpu = si.get("cpu_info", {})
print("cpu: %s x%s %s" % (cpu.get("processor"), cpu.get("cpu_count"),
cpu.get("model", "")[:40]))
print("total_memory: %s %s" % (cpu.get("total_memory"), cpu.get("memory_units")))
print("python: %s" % (si.get("python", {}).get("version_string"),))
print("keys: %s" % ", ".join(sorted(si)))
EOP
else
bad "GET /machine/system_info: `tail -2 /tmp/sys.out`"
fi
# A third, from `webcam` -- the component the mod's moonraker.conf configures
# and the reason the pin is what it is.
get /server/webcams/list > /tmp/cam.out 2>&1
grep -q '^HTTP 200' /tmp/cam.out \
&& ok "GET /server/webcams/list answers 200 (`sed -n '2p' /tmp/cam.out | cut -c1-90`)" \
|| note "GET /server/webcams/list: `tail -1 /tmp/cam.out`"
# ---- 2e. THE DATABASE, which is the whole reason for the 3.13 exercise -----
# This pin keeps its database in lmdb (v0.9.0 is where Moonraker moved to
# sqlite, and MOONRAKER_VERSION is deliberately the last commit before that).
# So what is being proved here is the lmdb C extension driving a real
# environment from inside the running server -- write over HTTP, read back
# over HTTP, and then look at the files on disk.
DBNS=anvil_gate
get /server/database/item "{\"namespace\":\"$DBNS\",\"key\":\"phase6.probe\",\"value\":{\"nozzle\":210,\"ok\":true}}" \
> /tmp/db-w.out 2>&1
if grep -q '^HTTP 200' /tmp/db-w.out; then
ok "POST /server/database/item wrote to namespace '$DBNS'"
else
bad "database write: `tail -2 /tmp/db-w.out`"
fi
get "/server/database/item?namespace=$DBNS&key=phase6.probe" > /tmp/db-r.out 2>&1
if grep -q '^HTTP 200' /tmp/db-r.out && grep -q '210' /tmp/db-r.out; then
ok "GET reads it back: `sed -n '2p' /tmp/db-r.out | cut -c1-100`"
else
bad "database read: `tail -2 /tmp/db-r.out`"
fi
get /server/database/list > /tmp/db-l.out 2>&1
grep -q "$DBNS" /tmp/db-l.out \
&& ok "and the namespace is listed: `sed -n '2p' /tmp/db-l.out | cut -c1-110`" \
|| bad "namespace not listed: `tail -1 /tmp/db-l.out`"
# On disk. An lmdb environment is data.mdb + lock.mdb, and a data.mdb of
# nonzero size is the claim "there is a database" made without asking the
# process that wrote it.
if [ -f $DATA/database/data.mdb ]; then
ok "lmdb environment on disk: `ls -l $DATA/database/data.mdb | awk '{print $5}'` bytes at $DATA/database/data.mdb"
else
note "no $DATA/database/data.mdb -- looking for one:"
find $DATA -name 'data.mdb' 2>/dev/null | sed 's/^/ /'
bad "no lmdb data.mdb where the database component should have made one"
fi
# And the lmdb the SERVER is using is the CPython extension, not the cffi
# fallback that would try to run a compiler on this printer.
mrun $PY -c \
'import lmdb, lmdb.cpython; print("lmdb backend:", lmdb.__file__, lmdb.version())' \
2>&1 | sed 's/^/ /'
# ---- 2f. zero /usr/prog, read off the RUNNING process ----------------------
# Not off a python that imported some modules -- off the pid that is holding
# :7125, with every component loaded and the database open. This is the claim
# the whole phase is for.
echo " .. /proc/$MRPID/maps:"
awk '{print $NF}' /proc/$MRPID/maps 2>/dev/null | grep '^/' | sort -u > /tmp/maps.txt
sed 's/^/ /' /tmp/maps.txt
NPROG=`grep -c '^/usr/prog' /tmp/maps.txt`
[ "$NPROG" = 0 ] \
&& ok "the RUNNING moonraker maps 0 libraries under /usr/prog" \
|| bad "$NPROG libraries under /usr/prog are mapped by the running moonraker"
if grep -q "^$MODDIR/lib/libsodium" /tmp/maps.txt; then
ok "and libsodium is mapped from the prefix: `grep libsodium /tmp/maps.txt`"
else
bad "our libsodium is NOT mapped by the running server -- authorization did not need it, or found another"
fi
# ---- 2g. it stays up -------------------------------------------------------
# A process that binds a port, answers once and exits is not a server. Two
# minutes is not an endurance test and is not claimed as one; it is long
# enough to catch an event loop that dies on its first idle timer, a component
# whose periodic callback throws, and the proc_stats sampler, which runs every
# second and touches /proc on this kernel.
BEFORE=`date +%s`
n=0
while [ $n -lt 120 ]; do
sleep 10
n=$((n + 10))
kill -0 $MRPID 2>/dev/null || break
done
if kill -0 $MRPID 2>/dev/null; then
ok "still alive ${n}s later (pid $MRPID)"
port_listening && ok "and still listening on :$PORT" \
|| bad "alive but no longer listening on :$PORT"
get /server/info > /tmp/info3.out 2>&1
grep -q '^HTTP 200' /tmp/info3.out \
&& ok "and still answers /server/info after ${n}s" \
|| bad "stopped answering after ${n}s: `tail -1 /tmp/info3.out`"
# proc_stats runs a 1Hz sampler; if it were throwing, the log would say so
# every second. Reading the log for tracebacks is the cheap way to catch a
# component that is failing quietly behind a working endpoint.
NTB=`grep -c 'Traceback' $LOG 2>/dev/null`
[ "$NTB" = 0 ] \
&& ok "no tracebacks in $LOG" \
|| note "$NTB traceback(s) in the log:"
[ "$NTB" = 0 ] || grep -A6 'Traceback' $LOG | head -40 | sed 's/^/ /'
else
bad "moonraker DIED during the ${n}s soak"
tail -n 40 $LOG | sed 's/^/ /'
fi
# ---- 2h. the shutdown, and the log ----------------------------------------
echo " .. log head:"
head -n 25 $LOG 2>/dev/null | sed 's/^/ /'
kill -TERM $MRPID 2>/dev/null
w=0
while [ $w -lt 30 ] && kill -0 $MRPID 2>/dev/null; do sleep 2; w=$((w + 2)); done
kill -0 $MRPID 2>/dev/null && { kill -9 $MRPID 2>/dev/null; note "needed SIGKILL"; } \
|| ok "SIGTERM stopped it in ${w}s"
sleep 3
port_listening && bad ":$PORT still bound after the server exited" \
|| ok ":$PORT released"
echo
echo "=== 3. negative control: it is OUR tree that is doing this ==="
# Move site-packages aside and start the same command line again. If it still
# comes up, then FlashForge's 3.8 site-packages are somehow being reached and
# every PASS above belongs to somebody else's build.
mv $SP $SP.away
env -u LD_LIBRARY_PATH $MRENV $PY $MAIN -d $DATA -l /usr/data/logs/mr-neg.log > /tmp/mr-neg.txt 2>&1 &
NEGPID=$!
w=0
while [ $w -lt 60 ]; do
port_listening && break
kill -0 $NEGPID 2>/dev/null || break
sleep 3; w=$((w + 3))
done
if port_listening; then
bad "negative control: moonraker bound :$PORT with site-packages moved away"
kill -9 $NEGPID 2>/dev/null
else
ok "negative control: with site-packages moved away it never binds -- `tail -2 /tmp/mr-neg.txt /usr/data/logs/mr-neg.log 2>/dev/null | grep -i 'error\|no module' | head -1`"
fi
kill -9 $NEGPID 2>/dev/null
mv $SP.away $SP
echo
if [ "$SODIUM_NEEDS_PATH" = 1 ]; then
echo "anvil-env.sh WOULD NEED: $MODDIR/lib on LD_LIBRARY_PATH"
else
echo "anvil-env.sh needs NO new LD_LIBRARY_PATH entry (libnacl resolves the prefix path itself)"
fi
echo "and $SODIUM_PROG can come OFF the list."
echo
[ "$FAIL" = 0 ] && echo "RESULT: PASS" || echo "RESULT: FAIL"
exit $FAIL
tools/python-packages/README.md
+74 -16
View File
@@ -106,26 +106,84 @@ and all 16 components still import. The phase-6 trap was the lmdb *egg*'s cffi
fallback, and installing lmdb as a wheel removes it. Dropping them is a size
decision, not a correctness one.
## The one unsolved dependency: libsodium
## Moonraker SERVES on 3.13 -- settled, not assumed
`libnacl` is pure Python and ctypes-loads libsodium, which exists on this box
**only at `/usr/prog/libsodium/lib`**. Measured both ways: it fails with an
empty path and works with that directory on it. So Moonraker's
`authorization` component keeps one `/usr/prog` dependency, and it is the last
one in the picture. Either cross-build libsodium into the prefix, or accept
`anvil-env.sh` keeping that single entry.
Importing is not serving, so it was run. `test/integration/printer/case-moonraker313.sh`
starts Moonraker on the exact command line `payload/etc/s6/moonraker/run`
execs, only with our interpreter, against the mod's own `assets/moonraker.conf`:
* bound :7125 in 3s, `/proc/PID/cmdline` confirming our interpreter and our
entry point
* `GET /server/info` 200 with **23 components loaded and 0 failed**;
`/machine/system_info` (a different component, reading this box's /proc)
reports `cpu: mips`, `python 3.13.7 [GCC 7.2.0]`
* the **database really works**: POST wrote a namespace, GET read it back, the
namespace appears beside Moonraker's own, and `data.mdb` is 147456 bytes on
disk -- through `lmdb/cpython.cpython-313-mipsel-linux-gnu.so`, not the
compile-at-import cffi fallback
* alive and still answering **120 seconds later**, zero tracebacks in its log,
and SIGTERM released the port in 2s
* **the running process maps 0 libraries under `/usr/prog`**, with
`LD_LIBRARY_PATH` completely unset -- not merely purged of `/usr/prog`
* negative controls: nothing on :7125 beforehand, and with `site-packages`
moved aside it never binds
What is still unproven is endurance and integration, not capability: no real
hardware, no live klippy attached (`klippy_connected=False` throughout, so the
klippy_apis paths loaded but were never exercised), no upload through
streaming_form_data, no websocket client, and it has not been run through
`S62moonraker` under s6 on 3.13 -- the entry point was driven directly, on
purpose, so that a failure would be the interpreter and not an init script.
## libsodium: built into the prefix, and it costs anvil-env.sh nothing
`build-libsodium.sh` cross-builds libsodium 1.0.20 with the Ingenic toolchain
(24s), gated at `0x70001407` nan2008/o32/mips32r2, SONAME `libsodium.so.26`,
needing only libpthread, libc and the loader. 406KB stripped, into
`$MODDIR/lib`.
It does not need a new `LD_LIBRARY_PATH` entry, which is the pleasant part.
libnacl's third fallback is `__file__[0:__file__.find("lib")+3] +
"/libsodium.so"`, and `__file__` is
`$MODDIR/lib/python3.13/site-packages/libnacl/__init__.py`, so it resolves
`$MODDIR/lib/libsodium.so` **by absolute path** -- measured working with
`LD_LIBRARY_PATH` unset entirely. Worth writing down that this works because
the prefix happens to contain the string `lib`; a prefix without it would
break silently. Adding `$MODDIR/lib` to the path also works and is the
belt-and-braces option, but it would put our libraries in front of everything
for every mod process, which is the kind of entry this file's own history
argues against.
It computes as well as loads: `sodium_version_string()` is 1.0.20, its
BLAKE2b-256 matches CPython's own `_blake2` on the same kernel, and
`libnacl.sign.Signer/Verifier` -- the exact ed25519 pair `authorization.py`
imports -- signs, verifies and rejects a tampered message. With it,
`/usr/prog/libsodium/lib` can come off `ANVIL_LIBS`, though only as part of the
same one-line `FF_PYTHON` switch, since 3.8's libnacl still needs it.
## What is left before FF_PYTHON can switch
1. **Boot Moonraker on 3.13 and hit the HTTP API.** Every component imports
and the entry point loads, but importing is not serving: nothing has bound
:7125 on 3.13. `case-moonraker.sh` already does this against 3.8; pointing
it at 3.13 is the next gate and the honest remaining unknown.
2. Decide libsodium, above.
3. Wire into `bin/patch.sh`. Note the build needs the **untrimmed** interpreter
stage -- `include/` and `config-3.13-*`, which section 5c's trim currently
deletes -- and the pins want moving into `versions.env` with sha256s.
4. klippy is **not** blocked by packages: cffi, greenlet, jinja2, markupsafe
Capability is settled -- Moonraker serves, the database works, libsodium is
ours. What is left is packaging and integration:
1. **Wire into `bin/patch.sh`.** The blocker is that section 5c's trim deletes
`include/` and `lib/python3.13/config-3.13-*` and then removes the staging
tree, and the package build needs both. Cleanest fix is to run the package
build inside 5c against the untrimmed stage BEFORE the trim, rather than
caching a second tree. Add a step for libsodium, a step to stage the 19
packages into `lib/python3.13/site-packages/`, and move every pin into
`versions.env` with sha256s.
2. **Extend the ABI gate's reach.** It walks `$PY_BUILD/bin` and
`$PY_BUILD/lib` -- the build cache, not the payload -- so libsodium and the
twelve extension `.so` files staged straight into `$MOD_PAYLOAD` would
bypass it. No logic change needed, the two-word `DYN -> 0x70001407` rule
already covers them; only coverage.
3. **Run it through `S62moonraker` under s6 on 3.13.** The entry point was
driven directly on purpose, so the s6 readiness and restart machinery is
still gated only against 3.8.
4. Then the switch itself: `FF_PYTHON` in `anvil-env.sh`, and
`/usr/prog/libsodium/lib` off `ANVIL_LIBS` in the same commit.
5. klippy is **not** blocked by packages: cffi, greenlet, jinja2, markupsafe
and pyserial all build, and `c_helper.so` binds. `numpy` is the only gap,
wanted solely by `extras/stepper_resonance_tester.py`, which Klipper loads
on demand. numpy on 3.13 means numpy >= 2.1 and a Meson cross-file, and
tools/python-packages/build-libsodium.sh
Executable
+147
View File
@@ -0,0 +1,147 @@
#!/bin/bash
# ---------------------------------------------------------------------------
# Cross-build libsodium for the Creator 5 Pro (Ingenic mipsel) into the mod's
# prefix root, so that libnacl -- and therefore Moonraker's `authorization`
# component -- stops needing /usr/prog/libsodium/lib.
#
# This is the LAST /usr/prog string in the phase-6 picture. Everything else
# the 3.13 interpreter and its extensions need is either static inside the
# interpreter or built by tools/python-packages/build.sh.
#
# Unlike the interpreter's seven static dependencies, this one MUST be shared:
# libnacl is pure Python and reaches it through ctypes.cdll.LoadLibrary, which
# is dlopen. So it ships as $MODDIR/lib/libsodium.so.26.2.0 with its soname
# symlink and the `libsodium.so` development symlink -- and that last one is
# not decoration, it is the FIRST name libnacl asks dlopen for.
#
# Same gcc-wrapper trick as tools/python/build.sh: -EL -mnan=2008 baked into
# the driver, so libsodium's libtool link lines cannot lose them. Gated at
# e_flags=0x70001407 (a shared object, so EF_MIPS_PIC is set and it reads
# ...07 where the interpreter's executable reads ...05) before anything ships.
#
# ./build-libsodium.sh # in docker
# ./build-libsodium.sh --in-container # the actual build
# ---------------------------------------------------------------------------
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
REPO="${REPO:-/home/shish/firmware/.claude/worktrees/s6-vs-runit}"
TOOLCHAIN_HOST="$REPO/work/.mips-toolchain/mips-gcc720-glibc229"
SODIUM_VER="${SODIUM_VER:-1.0.20}"
SODIUM_SHA256=ebb65ef6ca439333c2bb41a0c1990587288da07f6c7fd07cb3a18cc18d30ce19
PREFIX=/usr/data/anvil
if [ "${1:-}" != "--in-container" ]; then
[ -x "$TOOLCHAIN_HOST/bin/mips-linux-gnu-gcc" ] || {
echo "!! no toolchain at $TOOLCHAIN_HOST" >&2; exit 1; }
exec docker run --rm \
-v "$TOOLCHAIN_HOST":/toolchain:ro \
-v "$HERE":/work \
-e SODIUM_VER="$SODIUM_VER" -e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
-w /work debian:bookworm bash /work/build-libsodium.sh --in-container
fi
# ======================================================= in the container ===
START=$(date +%s)
export DEBIAN_FRONTEND=noninteractive
TC=/toolchain; W=/work; SRC=$W/src; B=$W/b-sodium; OUT=$W/out
rm -rf "$B"; mkdir -p "$SRC" "$B" "$OUT"
log() { echo; echo "===== $* ====="; }
log "apt"
apt-get update -qq
apt-get install -y -qq --no-install-recommends \
build-essential curl ca-certificates file xz-utils >/dev/null
# ---- the wrappers: -EL -mnan=2008 on every compile AND every link ----------
log "toolchain wrappers"
mkdir -p /opt/xw/bin
for t in gcc g++ cpp; do
cat > /opt/xw/bin/mips-linux-gnu-$t <<EOF
#!/bin/sh
exec $TC/bin/mips-linux-gnu-$t -EL -mnan=2008 "\$@"
EOF
chmod +x /opt/xw/bin/mips-linux-gnu-$t
done
for t in ar as ld nm objcopy objdump ranlib readelf strip strings size; do
ln -sf $TC/bin/mips-linux-gnu-$t /opt/xw/bin/mips-linux-gnu-$t
done
export PATH=/opt/xw/bin:$PATH
# ---- gate the wrapper before building anything on top of it ---------------
eflags() { mips-linux-gnu-readelf -h "$1" | sed -n 's/.*Flags: *\(0x[0-9a-f]*\).*/\1/p'; }
echo 'int main(void){return 0;}' > /tmp/t.c
mips-linux-gnu-gcc /tmp/t.c -o /tmp/t
[ "$(eflags /tmp/t)" = "0x70001405" ] || {
echo "!! wrapper produces $(eflags /tmp/t), not 0x70001405" >&2; exit 1; }
echo "wrapper gate ok: $(eflags /tmp/t)"
# ---- fetch ----------------------------------------------------------------
log "libsodium $SODIUM_VER"
TAR="$SRC/libsodium-$SODIUM_VER.tar.gz"
[ -f "$TAR" ] || curl -fsSL -o "$TAR" \
"https://github.com/jedisct1/libsodium/releases/download/$SODIUM_VER-RELEASE/libsodium-$SODIUM_VER.tar.gz"
echo "$SODIUM_SHA256 $TAR" | sha256sum -c - || {
echo "!! sha256 mismatch: $(sha256sum "$TAR")" >&2; exit 1; }
tar -xzf "$TAR" -C "$B"
cd "$B/libsodium-$SODIUM_VER"
# ---- configure ------------------------------------------------------------
# --disable-static: nothing links it statically and a .a would only be
# trimmed away again.
# --host is what makes autoconf reach for the mips-linux-gnu- prefixed tools
# in the wrapper directory, which is the whole point of the wrappers.
# libsodium's runtime feature probes are AC_RUN_IFELSE with cross defaults
# supplied, so nothing here needs qemu.
log "configure"
./configure \
--host=mips-linux-gnu \
--prefix="$PREFIX" \
--disable-static \
--enable-shared \
--with-pic \
CFLAGS="-O2 -fPIC" \
> "$W/sodium-configure.log" 2>&1 || { tail -40 "$W/sodium-configure.log"; exit 1; }
log "make"
make -j"$(nproc)" > "$W/sodium-make.log" 2>&1 || { tail -60 "$W/sodium-make.log"; exit 1; }
STAGE=$W/stage-sodium
rm -rf "$STAGE"; mkdir -p "$STAGE"
make install DESTDIR="$STAGE" >> "$W/sodium-make.log" 2>&1
# ---- the ABI gate, on the object that ships -------------------------------
log "ABI gate"
SO=$(readlink -f "$STAGE$PREFIX/lib/libsodium.so")
file "$SO"
mips-linux-gnu-readelf -h "$SO" | sed -n '/Class\|Data\|Type\|Flags/p'
F=$(eflags "$SO")
[ "$F" = "0x70001407" ] || { echo "!! e_flags $F, expected 0x70001407" >&2; exit 1; }
# The words that matter are decoded by readelf out of e_flags itself, on the
# header's Flags line -- `.MIPS.abiflags` (readelf -A) reports the ISA and the
# FP ABI but not the NaN encoding, so the header is where nan2008 is read.
FLAGLINE=$(mips-linux-gnu-readelf -h "$SO" | sed -n 's/.*Flags: *//p')
echo " decoded: $FLAGLINE"
for w in nan2008 o32 mips32r2; do
case ",$FLAGLINE," in
*"$w"*) ;;
*) echo "!! e_flags does not decode to $w: $FLAGLINE" >&2; exit 1 ;;
esac
done
mips-linux-gnu-readelf -A "$SO" | sed -n '/ISA\|FP ABI/p'
echo "-- SONAME / NEEDED"
mips-linux-gnu-readelf -d "$SO" | sed -n '/SONAME\|NEEDED/p'
mips-linux-gnu-strip --strip-unneeded "$SO"
# ---- what ships: lib/ only. include/ and pkgconfig are build-time. --------
log "stage"
TREE=$W/sodium-tree
rm -rf "$TREE"; mkdir -p "$TREE/lib"
cp -a "$STAGE$PREFIX"/lib/libsodium.so* "$TREE/lib/"
rm -f "$TREE"/lib/*.la
ls -l "$TREE/lib"
tar -czf "$OUT/sodium.tgz" -C "$TREE" lib
chown -R "${HOST_UID:-0}:${HOST_GID:-0}" "$OUT" "$W"/sodium-*.log "$SRC" 2>/dev/null || true
echo
echo "sodium.tgz: $(du -h "$OUT/sodium.tgz" | cut -f1) ($(($(date +%s)-START))s)"