mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-21 21:02:16 -05:00
The build makes the packages Moonraker runs on, from hashed sources
Section 5c grows a second half and section 5d appears. The 18 packages and libsodium that two spikes proved are now built by bin/patch.sh from pins in versions.env, and land in the payload at lib/python3.13/site-packages and lib/libsodium.so. WHERE THE PACKAGE BUILD HAD TO GO, and it is not obvious: inside 5c, against the UNTRIMMED interpreter stage, before the trim runs. The headers INCLUDEPY names, the static dependency libraries pillow and cffi link against, and the x86-64 build-python all exist only at that moment -- the trim deletes include/ and config-3.13-* and then removes the staging tree entirely. The alternative was caching a second untrimmed ~200MB tree. The price of not doing that is that either stamp going stale rebuilds both halves, which is written down at the stamp rather than left to be discovered. The build no longer touches PyPI. pip comes from --with-ensurepip=install on the build-python, out of the same pinned CPython tarball, and every pip wheel runs --no-index --no-build-isolation --no-binary :all: against hashed sdists in vendor/. That last flag is not caution: during the spike pip silently downloaded x86-64 manylinux wheels for cffi, greenlet and tornado, three wrong-architecture objects sailed through the build, and only the ABI gate caught them. Which is why the gate now reaches the STAGED PAYLOAD rather than the build cache. It walked $PY_BUILD/bin and $PY_BUILD/lib, so libsodium and the twelve extension modules staged straight into $MOD_PAYLOAD would have gone unexamined. 69 ELF objects now, plus libsodium's resolved .so. Deliberately not all of $MOD_PAYLOAD: s6 is musl-built and correctly reads e_flags=0x1007, mips1 and legacy-NaN, which is right for a static binary that does no floating point and is measured working on the printer. It has its own gate in 5b. That distinction is recorded in patch.sh, because a sweep that assumes every object must be nan2008 flags all thirteen s6 binaries and looks like a disaster. Two pins carry reasons that cost someone an afternoon: pillow is 11.0.0 because 10.3.0 cannot be built by ANY 3.13 -- its setup.py execs and then reads locals()["__version__"], which PEP 667 broke -- and streaming-form-data is 1.19.1 because Moonraker's own pin ships pregenerated Cython calling the four-argument _PyLong_AsByteArray, which grew a fifth parameter in 3.13. docker/Dockerfile.build needs zlib1g-dev and patch, and the image must be rebuilt. The first is a genuinely nasty one: a build-python without zlib cannot unpack a wheel, or install pip, and ensurepip fails INSIDE a `make install` that reports success. patch.sh now asserts zlib and pip on the build-python rather than failing later and further away. The install manifest goes from 1872 paths to 2940 and case-upgrade.sh still passes, which is the check that the deletion loop scales. Note that the "70 to 728" in the commit that landed the interpreter was wrong -- 728 predates the interpreter itself; HEAD's own patch.sh produces 1872 here, measured. anvil.tar.xz grows 1.6MB packed, about 15MB on the printer. FF_PYTHON still points at FlashForge's 3.8.2 and /usr/prog/libsodium/lib is still in ANVIL_LIBS. Both comments now say what changed -- the packages exist -- and what has not: nothing has run through S62moonraker under s6 on 3.13. The two lines move together, in the commit that has that gate behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Oleksandr ShyshatskyiandClaude Opus 5 committed
1 parent
6a750f6bcc
commit
a8a46e21d2
8 files changed
+994
-61
No files matched your search
@@ -107,6 +107,49 @@ PY_BUILD="${PY_BUILD:-$ROOT/work/.py313}"
|
||||
export PY_TGZ OPENSSL_TGZ SQLITE_TGZ ZLIB_TGZ LIBFFI_TGZ XZ_TGZ BZIP2_TGZ
|
||||
export EXPAT_TGZ PY_BUILD
|
||||
|
||||
# The third-party python packages that become the interpreter's
|
||||
# site-packages, and libsodium, which libnacl dlopens out of $MODDIR/lib.
|
||||
#
|
||||
# There are eighteen of the former and the list moves, so they are NOT one
|
||||
# variable per tarball the way everything above is: versions.env carries
|
||||
# PYPKG_LIST plus a FILE/SHA256 pair per entry, and the three helpers below
|
||||
# are how both bin/fetch-assets.sh and bin/patch.sh read that table. They live
|
||||
# here rather than being written twice because a package the fetcher and the
|
||||
# builder disagree about is a build that downloads one file and compiles
|
||||
# another -- and because the CACHE KEY has to be computed identically in both
|
||||
# places or the ~203MB Ingenic toolchain is skipped by the fetcher on exactly
|
||||
# the build that needs it.
|
||||
#
|
||||
# pypkg_var takes the list entry (`streaming-form-data`) and a suffix and
|
||||
# returns the variable versions.env spells for it
|
||||
# (PYPKG_STREAMING_FORM_DATA_FILE). Bash indirection, which is why this file
|
||||
# is only ever sourced by bin/*.sh -- payload/* is busybox ash and has none.
|
||||
pypkg_var() {
|
||||
local _n _v
|
||||
_n=$(printf '%s' "$1" | tr 'a-z' 'A-Z' | tr '-' '_')
|
||||
_v="PYPKG_${_n}_$2"
|
||||
printf '%s' "${!_v-}"
|
||||
}
|
||||
# The sdist as bin/fetch-assets.sh leaves it in vendor/.
|
||||
pypkg_tgz() { printf '%s/vendor/%s' "$ROOT" "$(pypkg_var "$1" FILE)"; }
|
||||
# The cache key for the whole package set: every file name (which carries the
|
||||
# version) beside its hash, so a bumped pin AND a re-released upstream
|
||||
# tarball both invalidate. Compared as a whole string, not hashed, so
|
||||
# work/.py313/.pkg-version stays readable -- a diff of it is the changelog.
|
||||
pypkg_stamp() {
|
||||
local _p
|
||||
for _p in $PYPKG_LIST $PYPKG_HOST_LIST; do
|
||||
printf '%s %s\n' "$(pypkg_var "$_p" FILE)" "$(pypkg_var "$_p" SHA256)"
|
||||
done
|
||||
}
|
||||
# libsodium is one pin and one tarball, so it keeps the ordinary shape. Its
|
||||
# cache is separate from work/.py313 because it is a separate build with a
|
||||
# separate stamp: 24 seconds, but 24 seconds that would otherwise drag the
|
||||
# 203MB toolchain download along behind it on every build.
|
||||
SODIUM_TGZ="${SODIUM_TGZ:-$ROOT/vendor/libsodium-${SODIUM_VERSION:-unpinned}.tar.gz}"
|
||||
SODIUM_BUILD="${SODIUM_BUILD:-$ROOT/work/.sodium}"
|
||||
export SODIUM_TGZ SODIUM_BUILD
|
||||
|
||||
# Replica-only settings: the factory image and the partition sizes. They exist
|
||||
# for the tests and never reach a printer, so they live in their own file --
|
||||
# see test.env.example. Values left in config.env keep working.
|
||||
|
||||
+47
-1
@@ -144,8 +144,54 @@ get "https://sourceware.org/pub/bzip2/bzip2-$BZIP2_VERSION.tar.gz" \
|
||||
# file expat-2.6.4, so both spellings appear in the one URL.
|
||||
get "https://github.com/libexpat/libexpat/releases/download/$EXPAT_TAG/expat-$EXPAT_VERSION.tar.gz" \
|
||||
"$EXPAT_TGZ" "$EXPAT_SHA256"
|
||||
|
||||
# The third-party packages that become the interpreter's site-packages, and
|
||||
# libsodium. Same rule as the eight tarballs above and for the same reason:
|
||||
# every package ships them, so there is no BUILD_ flag and no condition --
|
||||
# they come down once and the sha256 cache means never again. 53MB, of which
|
||||
# 46MB is pillow's sdist alone (it carries its own test images; there is no
|
||||
# smaller sdist to have).
|
||||
#
|
||||
# pypi <project> <file> <sha256>
|
||||
#
|
||||
# files.pythonhosted.org publishes every sdist at TWO urls: the hashed
|
||||
# /packages/<a>/<b>/<64 hex>/<file> one that a browser copies, and this
|
||||
# /packages/source/<initial>/<project>/<file> one. Only the second can be
|
||||
# composed from a pin, because the first embeds a digest of the file itself --
|
||||
# so bumping a version there means pasting a URL nobody can check by reading.
|
||||
# The project name is the one thing not in versions.env, because it is a
|
||||
# property of the URL and not of the version: `smart-open` publishes
|
||||
# smart_open-6.4.0.tar.gz, `markupsafe` publishes MarkupSafe-2.1.5.tar.gz.
|
||||
# Same division of labour as EXPAT_TAG above.
|
||||
pypi() {
|
||||
proj="$1"; file="$2"; sha="$3"
|
||||
get "https://files.pythonhosted.org/packages/source/$(printf '%.1s' "$proj")/$proj/$file" \
|
||||
"$ROOT/vendor/$file" "$sha"
|
||||
}
|
||||
# PYPKG_HOST_LIST too: the PEP 517 backends never reach a printer, but they
|
||||
# are what RUNS on the build machine to produce the objects that do, so they
|
||||
# are fetched and checked exactly like the rest.
|
||||
for p in $PYPKG_LIST $PYPKG_HOST_LIST; do
|
||||
pypi "$p" "$(pypkg_var "$p" FILE)" "$(pypkg_var "$p" SHA256)"
|
||||
done
|
||||
get "https://github.com/jedisct1/libsodium/releases/download/$SODIUM_VERSION-RELEASE/libsodium-$SODIUM_VERSION.tar.gz" \
|
||||
"$SODIUM_TGZ" "$SODIUM_SHA256"
|
||||
|
||||
# The Ingenic toolchain, on the condition that decides whether patch.sh has to
|
||||
# compile at all. THREE stamps and not one, because three separate things are
|
||||
# built with this one compiler and any of them going stale means the download
|
||||
# is needed: the interpreter (work/.py313/.version), the packages that go into
|
||||
# its site-packages (work/.py313/.pkg-version -- same cache directory, its own
|
||||
# key, because a package bump forces a full interpreter rebuild; see the
|
||||
# comment in bin/patch.sh) and libsodium (work/.sodium/.version).
|
||||
#
|
||||
# Getting this wrong is not a slow build, it is a stopped one: patch.sh would
|
||||
# get 53MB of sdists and then halt for want of a compiler, halfway through,
|
||||
# rather than here where the fix is one command.
|
||||
if [ "$ALL" = 1 ] \
|
||||
|| [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ]; then
|
||||
|| [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ] \
|
||||
|| [ "$(cat "$PY_BUILD/.pkg-version" 2>/dev/null || true)" != "$(pypkg_stamp)" ] \
|
||||
|| [ "$(cat "$SODIUM_BUILD/.version" 2>/dev/null || true)" != "$SODIUM_VERSION" ]; then
|
||||
get "https://github.com/ballaswag/k1-discovery/releases/download/$MIPS_TOOLCHAIN_VERSION/$MIPS_TOOLCHAIN_FILE" \
|
||||
"$MIPS_TOOLCHAIN_TGZ" "$MIPS_TOOLCHAIN_SHA256"
|
||||
fi
|
||||
+578
-25
@@ -561,8 +561,26 @@ PY_TOOLCHAIN_DIR=work/.mips-toolchain/mips-gcc720-glibc229
|
||||
# a bumped OpenSSL with an unchanged PY_VERSION has to rebuild, and the failure
|
||||
# if it does not is an interpreter linked against a library nobody can name.
|
||||
PY_STAMP="$PY_VERSION $OPENSSL_VERSION $SQLITE_VERSION $ZLIB_VERSION $LIBFFI_VERSION $XZ_VERSION $BZIP2_VERSION $EXPAT_VERSION"
|
||||
# The SECOND cache key, for the third-party packages that go into this
|
||||
# interpreter's site-packages (step 4 below): every sdist file name and hash
|
||||
# from versions.env, as bin/common.sh assembles it.
|
||||
#
|
||||
# TWO STAMPS, ONE CACHE DIRECTORY, AND A REBUILD OF EVERYTHING IF EITHER
|
||||
# MOVES. That is not sloppiness, it is the shape of the dependency. Building a
|
||||
# package for this target needs three things that exist ONLY while this
|
||||
# section is mid-flight: the untrimmed staged tree (the trim below deletes
|
||||
# include/ and lib/python3.13/config-3.13-*, and INCLUDEPY points at the
|
||||
# first), the static C libraries in work/.py-dep (pillow links zlib, cffi
|
||||
# links libffi), and the x86-64 build-python that answers every sysconfig
|
||||
# question for mipsel. Caching those separately means caching a second
|
||||
# ~200MB tree and a second interpreter for the sake of a package bump, which
|
||||
# buys a faster rare path by making the common path bigger. So a package pin
|
||||
# that moves rebuilds CPython too: minutes, on a bump that happens rarely and
|
||||
# deliberately, against a cache that is otherwise never touched.
|
||||
PYPKG_STAMP="$(pypkg_stamp)"
|
||||
|
||||
if [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ]; then
|
||||
if [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ] \
|
||||
|| [ "$(cat "$PY_BUILD/.pkg-version" 2>/dev/null || true)" != "$PYPKG_STAMP" ]; then
|
||||
if [ ! -x "$PY_TOOLCHAIN_DIR/bin/$PY_HOST-gcc" ]; then
|
||||
[ -f "${MIPS_TOOLCHAIN_TGZ:-}" ] || {
|
||||
echo " !! python needs (re)building and there is no toolchain:" >&2
|
||||
@@ -576,6 +594,13 @@ if [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ]; then
|
||||
"${LIBFFI_TGZ:-}" "${XZ_TGZ:-}" "${BZIP2_TGZ:-}" "${EXPAT_TGZ:-}"; do
|
||||
[ -f "$t" ] || { echo " !! no python sources at '$t' -- run ./bin/fetch-assets.sh" >&2; exit 1; }
|
||||
done
|
||||
# And the package sdists, checked here rather than in step 4 -- twenty
|
||||
# minutes into a CPython build is the wrong place to discover that
|
||||
# vendor/ is one tarball short, and the fix is the same one command.
|
||||
for p in $PYPKG_LIST $PYPKG_HOST_LIST; do
|
||||
t="$(pypkg_tgz "$p")"
|
||||
[ -f "$t" ] || { echo " !! no sdist for $p at '$t' -- run ./bin/fetch-assets.sh" >&2; exit 1; }
|
||||
done
|
||||
command -v gcc >/dev/null || {
|
||||
echo " !! no host gcc. Cross-building CPython needs a build-python of" >&2
|
||||
echo " the SAME version, which is compiled here for x86-64 first." >&2
|
||||
@@ -654,10 +679,47 @@ if [ "$(cat "$PY_BUILD/.version" 2>/dev/null || true)" != "$PY_STAMP" ]; then
|
||||
# sources and byte-compile the stdlib, and configure hard-errors when
|
||||
# the version does not match. The image's python3.11 cannot stand in.
|
||||
# x86-64, thrown away at the end of this section, never shipped.
|
||||
#
|
||||
# --with-ensurepip=install, where the CROSS build below has
|
||||
# --without-ensurepip and must keep it. The two are not in tension:
|
||||
# they are opposite answers to opposite questions. On the printer a
|
||||
# pip would need a network and a compiler and has neither. HERE, pip
|
||||
# is what builds the third-party packages in step 4 -- and it comes
|
||||
# out of THIS tarball, whose sha256 is pinned, rather than off
|
||||
# bootstrap.pypa.io the way the spike got it. That is the difference
|
||||
# between a build machine that downloads and runs an unpinned
|
||||
# get-pip.py and one that does not talk to anybody: everything below
|
||||
# this line runs with --no-index against files bin/fetch-assets.sh
|
||||
# already checked.
|
||||
( cd "$SRC/Python-$PY_VERSION"
|
||||
./configure --prefix="$HOSTPY" --without-ensurepip >"$LOG/.py-hostpy.log" 2>&1
|
||||
./configure --prefix="$HOSTPY" --with-ensurepip=install >"$LOG/.py-hostpy.log" 2>&1
|
||||
make -j"$JOBS" >>"$LOG/.py-hostpy.log" 2>&1
|
||||
make install >>"$LOG/.py-hostpy.log" 2>&1 )
|
||||
# zlib, asserted rather than assumed, because CPython's answer to a
|
||||
# missing library is to record the module as absent and CARRY ON --
|
||||
# the same silence that hides a missing _sqlite3 in the cross build
|
||||
# below, here on the host side. Every wheel is a zip and so is pip:
|
||||
# ensurepip installs it out of a bundled .whl, and without zlib that
|
||||
# fails with "can't decompress data" INSIDE a make install that
|
||||
# reports success. The symptom then arrives three steps later as "No
|
||||
# module named pip", nowhere near the cause. zlib1g-dev in
|
||||
# docker/Dockerfile.build is the fix, and this is the sentence that
|
||||
# says so.
|
||||
"$HOSTPY/bin/python$PY_MM" -c 'import zlib' 2>/dev/null || {
|
||||
echo " !! the build-python has no zlib module, so it cannot unpack" >&2
|
||||
echo " a single wheel. Install zlib1g-dev in the build image" >&2
|
||||
echo " (docker/Dockerfile.build has it) and delete work/.py313." >&2
|
||||
exit 1; }
|
||||
# And pip itself, which --with-ensurepip=install above was for. Asked
|
||||
# here and not in step 4 because `make install` runs ensurepip in a
|
||||
# shell fragment that does not stop the install when it fails: without
|
||||
# this line a broken bootstrap is reported by the FIRST PACKAGE to try
|
||||
# to build, twenty lines of C library later.
|
||||
"$HOSTPY/bin/python$PY_MM" -m pip --version >>"$LOG/.py-hostpy.log" 2>&1 || {
|
||||
echo " !! the build-python has no pip, so nothing in step 4 can" >&2
|
||||
echo " be built. ensurepip failed inside 'make install' --" >&2
|
||||
echo " its traceback is in work/.py-hostpy.log." >&2
|
||||
exit 1; }
|
||||
|
||||
# ------------------------------------------- 2. the C libraries, STATIC
|
||||
# All seven go into $DEP as .a with -fPIC, and NONE of them ships. The
|
||||
@@ -805,6 +867,307 @@ EOF
|
||||
echo " there, config.log included." >&2
|
||||
exit 1; }
|
||||
|
||||
# ------------------------------------------- 4. the third-party packages
|
||||
# What runs ON the interpreter: the 18 sdists pinned in versions.env,
|
||||
# cross-built into the staged tree's own lib/python3.13/site-packages.
|
||||
#
|
||||
# WHY IT IS HERE, INSIDE 5c, RATHER THAN IN A SECTION OF ITS OWN. Because
|
||||
# of what it needs, all of which exists only at this exact moment:
|
||||
#
|
||||
# * the UNTRIMMED stage. INCLUDEPY is $PY_PREFIX/include/python3.13 and
|
||||
# every C extension compiles against it -- and the trim thirty lines
|
||||
# below deletes include/ and lib/python3.13/config-3.13-* precisely
|
||||
# because nothing on a PRINTER builds extension modules.
|
||||
# * work/.py-dep, the static -fPIC C libraries. pillow links zlib out of
|
||||
# it and cffi links libffi, and it is deleted with the rest of the
|
||||
# scratch trees at the end of this section.
|
||||
# * work/.py-host, the x86-64 build-python of the SAME 3.13.7. It is the
|
||||
# thing that makes the cross work at all (see below), and it is also
|
||||
# thrown away here.
|
||||
#
|
||||
# The alternative -- cache a second, untrimmed tree so the packages can be
|
||||
# built later and separately -- was considered and declined: it means
|
||||
# keeping ~200MB and a second interpreter warm to make a rare path faster,
|
||||
# and it means two caches that can disagree about which interpreter the
|
||||
# extension modules were compiled against. That disagreement is not
|
||||
# theoretical: an .so built against 3.13 headers and loaded by a 3.14
|
||||
# interpreter is an import-time crash on the printer and nowhere else.
|
||||
# One cache, two stamps, everything rebuilt together (see PYPKG_STAMP).
|
||||
#
|
||||
# THE CROSS TRICK, and there is no crossenv in it. The staged interpreter
|
||||
# carries _sysconfigdata__linux_mipsel-linux-gnu.py, which records the
|
||||
# cross CC, LDSHARED, EXT_SUFFIX (.cpython-313-mipsel-linux-gnu.so) and
|
||||
# INCLUDEPY for the TARGET. Point _PYTHON_SYSCONFIGDATA_NAME at it from an
|
||||
# x86-64 CPython of the same version and sysconfig -- and therefore
|
||||
# setuptools' build_ext -- answers every question about mipsel while
|
||||
# running on x86-64. The one thing the spike needed a container for was
|
||||
# making INCLUDEPY resolve: it bind-mounted the stage at /usr/data/anvil.
|
||||
# There is no daemon to ask for that here (the build lane deliberately has
|
||||
# no docker socket), so instead a COPY of that module is written with
|
||||
# every $PY_PREFIX path rewritten to where the stage actually is. It goes
|
||||
# first on PYTHONPATH, is read only by the build, and never ships.
|
||||
#
|
||||
# NOTHING HERE TALKS TO A NETWORK. pip runs with --no-index against the
|
||||
# sdists bin/fetch-assets.sh already checked the sha256 of, which is the
|
||||
# strong form of the rule that --no-binary :all: below is the weak form
|
||||
# of: pip cannot download a wheel it cannot reach. Both are kept, because
|
||||
# the day someone adds an index URL for one awkward package the second one
|
||||
# is what stops three x86-64 .so files from sailing into the tree -- which
|
||||
# is not a hypothetical, it is what happened on the spike's first run.
|
||||
say "python: cross-building $(echo $PYPKG_LIST | wc -w) third-party packages"
|
||||
(
|
||||
set -e
|
||||
DEP="$PWD/work/.py-dep"
|
||||
STAGE="$PWD/work/.py-stage"
|
||||
XW="$PWD/work/.py-xw"
|
||||
LOG="$PWD/work"
|
||||
HOSTPY="$PWD/work/.py-host/bin/python$PY_MM"
|
||||
PKGSRC="$PWD/work/.py-pkgsrc" # unpacked sdists
|
||||
WHEELS="$PWD/work/.py-wheels" # the wheels they produce
|
||||
XSYS="$PWD/work/.py-xsysconfig" # the rewritten sysconfigdata
|
||||
TREE="$STAGE$PY_PREFIX" # the untrimmed staged interpreter
|
||||
SP="$TREE/lib/python$PY_MM/site-packages"
|
||||
SYSCFG=_sysconfigdata__linux_mipsel-linux-gnu
|
||||
rm -rf "$PKGSRC" "$WHEELS" "$XSYS"
|
||||
mkdir -p "$PKGSRC" "$WHEELS" "$XSYS" "$SP"
|
||||
# The same gcc wrappers step 3 built, still on disk: -EL -mnan=2008
|
||||
# baked into the driver so that setuptools -- which does NOT forward
|
||||
# CFLAGS to its link lines -- cannot lose them on the way to ld.
|
||||
export PATH="$XW/bin:$PATH"
|
||||
|
||||
# -------------------------------------------------- the PEP 517 side
|
||||
# Three backends into the build-python: setuptools (most of the list),
|
||||
# flit_core (jinja2) and poetry_core (libnacl). All three sdists
|
||||
# bootstrap themselves through backend-path, so this needs nothing
|
||||
# that is not already in vendor/, and --no-index proves it.
|
||||
#
|
||||
# Every failure in this subshell is caught BY HAND, with an explicit
|
||||
# `|| { ...; exit 1; }`, and the `set -e` at the top is not what is
|
||||
# doing the work: a compound command on the left of `||` -- which is
|
||||
# what this whole subshell is -- runs with errexit suppressed
|
||||
# THROUGHOUT, so a bare failing command in here would be shrugged off
|
||||
# and the build would carry on to produce a tree with one package
|
||||
# quietly missing from it.
|
||||
for p in setuptools $PYPKG_HOST_LIST; do
|
||||
"$HOSTPY" -m pip install --quiet --no-index --no-cache-dir \
|
||||
--no-build-isolation --no-deps "$PWD/vendor/$(pypkg_var "$p" FILE)" \
|
||||
>>"$LOG/.py-pkg-backends.log" 2>&1 \
|
||||
|| { echo " !! could not install the build backend $p --" \
|
||||
"work/.py-pkg-backends.log" >&2
|
||||
tail -15 "$LOG/.py-pkg-backends.log" | sed 's/^/ /' >&2
|
||||
exit 1; }
|
||||
done
|
||||
|
||||
# ------------------------------------------- sysconfig, for the TARGET
|
||||
# The rewrite described above. It is a blanket substitution of
|
||||
# $PY_PREFIX for the staging path over every string value rather than
|
||||
# a list of the variables that matter, because the list of variables
|
||||
# that matter is exactly the thing nobody gets right by hand:
|
||||
# INCLUDEPY is the one that fails loudly, LIBDIR and LIBPL fail
|
||||
# quietly, and a future setuptools is free to ask for another.
|
||||
"$HOSTPY" - "$TREE/lib/python$PY_MM/$SYSCFG.py" "$XSYS/$SYSCFG.py" \
|
||||
"$PY_PREFIX" "$TREE" <<'PYEOF' || exit 1
|
||||
import sys
|
||||
src, dst, prefix, stage = sys.argv[1:5]
|
||||
ns = {}
|
||||
exec(compile(open(src).read(), src, "exec"), ns)
|
||||
out = {k: (v.replace(prefix, stage) if isinstance(v, str) else v)
|
||||
for k, v in ns["build_time_vars"].items()}
|
||||
with open(dst, "w") as fh:
|
||||
fh.write("# Generated by bin/patch.sh: %s with the build-time prefix\n"
|
||||
"# rewritten to the staging tree. Build-time only; never ships.\n"
|
||||
"build_time_vars = %r\n" % (src, out))
|
||||
PYEOF
|
||||
[ -s "$XSYS/$SYSCFG.py" ] || {
|
||||
echo " !! could not rewrite $SYSCFG for the staging tree" >&2; exit 1; }
|
||||
export _PYTHON_SYSCONFIGDATA_NAME="$SYSCFG"
|
||||
# ONLY the rewritten module -- deliberately NOT the target's stdlib as
|
||||
# well, which is what the spike put here. They are the same 3.13.7, so
|
||||
# it did no harm, but it also means the host interpreter importing
|
||||
# the TARGET's os.py and sysconfig.py by accident is one path-ordering
|
||||
# mistake away, and the only thing that has to be importable is this.
|
||||
export PYTHONPATH="$XSYS"
|
||||
export PYTHONDONTWRITEBYTECODE=1
|
||||
export PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
export CC="$PY_HOST-gcc" CXX="$PY_HOST-g++"
|
||||
export AR="$PY_HOST-ar" RANLIB="$PY_HOST-ranlib" STRIP="$PY_HOST-strip"
|
||||
export LDSHARED="$PY_HOST-gcc -shared -L$DEP/lib"
|
||||
export CFLAGS="-O2 -fPIC -D_FILE_OFFSET_BITS=64 -I$DEP/include -I$TREE/include/python$PY_MM"
|
||||
export CXXFLAGS="$CFLAGS"
|
||||
export LDFLAGS="-L$DEP/lib"
|
||||
# pkg-config must see ONLY our sysroot, for the same reason as step 2:
|
||||
# left alone it answers out of the build image's /usr/lib/pkgconfig and
|
||||
# a package links an x86-64 library into a mipsel extension module.
|
||||
export PKG_CONFIG_LIBDIR="$DEP/lib/pkgconfig"
|
||||
export PKG_CONFIG_PATH="$DEP/lib/pkgconfig"
|
||||
|
||||
# Gate the trick before building 18 packages on top of it. If
|
||||
# _PYTHON_SYSCONFIGDATA_NAME has not taken, every question sysconfig
|
||||
# is asked below is answered for x86-64 -- and the answer is a tree of
|
||||
# host objects that builds perfectly and imports nowhere.
|
||||
"$HOSTPY" - "$TREE/include/python$PY_MM" <<'PYEOF' || exit 1
|
||||
import sys, sysconfig
|
||||
suffix = sysconfig.get_config_var("EXT_SUFFIX")
|
||||
inc = sysconfig.get_config_var("INCLUDEPY")
|
||||
print(" sysconfig answers for %s, headers %s"
|
||||
% (suffix, "found" if __import__("os").path.isdir(inc) else "MISSING"))
|
||||
if suffix != ".cpython-313-mipsel-linux-gnu.so":
|
||||
raise SystemExit(" !! sysconfig is answering for the HOST (EXT_SUFFIX=%s)"
|
||||
" -- _PYTHON_SYSCONFIGDATA_NAME did not take" % suffix)
|
||||
if inc != sys.argv[1]:
|
||||
raise SystemExit(" !! INCLUDEPY is %s, expected the staged %s"
|
||||
% (inc, sys.argv[1]))
|
||||
PYEOF
|
||||
|
||||
# ------------------------------------------------------- the helpers
|
||||
# pywheel <sdist-path> [env=val ...] -- build one wheel into $WHEELS.
|
||||
#
|
||||
# --no-binary :all: is not caution and not decoration; see the top of
|
||||
# this step and versions.env. --no-build-isolation is what keeps
|
||||
# _PYTHON_SYSCONFIGDATA_NAME reaching setup.py: with isolation ON pip
|
||||
# rewrites PYTHONPATH for the build subprocess and the module above
|
||||
# stops being importable.
|
||||
pywheel() {
|
||||
local what=$1; shift
|
||||
env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \
|
||||
--no-index --no-binary :all: --no-cache-dir \
|
||||
-w "$WHEELS" "$what" >"$LOG/.py-pkg-$(basename "$what").log" 2>&1 \
|
||||
|| { echo " !! $(basename "$what") failed to build --" \
|
||||
"work/.py-pkg-$(basename "$what").log" >&2
|
||||
tail -25 "$LOG/.py-pkg-$(basename "$what").log" | sed 's/^/ /' >&2
|
||||
exit 1; }
|
||||
}
|
||||
# unpack <list-entry> -- untar its sdist and echo the source directory.
|
||||
# Every one of these tarballs unpacks to its own basename; a release
|
||||
# that did not would fail on the [ -d ] rather than build the wrong
|
||||
# tree.
|
||||
unpack() {
|
||||
local f d
|
||||
f="$(pypkg_var "$1" FILE)"
|
||||
d="$PKGSRC/${f%.tar.gz}"
|
||||
tar -xzf "$PWD/vendor/$f" -C "$PKGSRC"
|
||||
[ -d "$d" ] || { echo " !! $f does not unpack to $(basename "$d")" >&2; exit 1; }
|
||||
printf '%s' "$d"
|
||||
}
|
||||
|
||||
# ------------------------------------------------------- the packages
|
||||
# PYPKG_LIST is the order, so the order lives in versions.env beside
|
||||
# the pins. Four entries need more than "wheel the sdist", and each
|
||||
# `case` arm says why.
|
||||
for p in $PYPKG_LIST; do
|
||||
SDIST="$PWD/vendor/$(pypkg_var "$p" FILE)"
|
||||
case "$p" in
|
||||
preprocess-cancellation)
|
||||
# NOT BUILT: copied. This sdist declares no [build-system] and
|
||||
# ships no setup.py, so there is no backend to call -- pip's
|
||||
# legacy fallback would auto-discover a flat layout and emit a
|
||||
# wheel named UNKNOWN-0.0.0 containing the one module, which
|
||||
# is the right file arrived at by an accident that a future
|
||||
# setuptools is free to change. The package IS one module, so
|
||||
# the honest operation is a copy. Moonraker imports it in the
|
||||
# metadata SUBPROCESS (gcode object cancellation), lazily, so
|
||||
# a printer without it loses that feature and nothing else.
|
||||
d=$(unpack "$p")
|
||||
cp -f "$d/preprocess_cancellation.py" "$SP/"
|
||||
;;
|
||||
greenlet)
|
||||
# gcc 7.2 refuses greenlet 3.x's C++ designated initializers
|
||||
# ("sorry, unimplemented: non-trivial designated initializers
|
||||
# not supported") wherever they skip a field. The rule was
|
||||
# probed, not assumed: in-order contiguous is fine, a TRAILING
|
||||
# gap is fine, an interior or leading gap is refused.
|
||||
# fill-designators.py writes the skipped fields back as
|
||||
# explicit zeros, taking field order from the target's own
|
||||
# headers and refusing to guess about a designator that is not
|
||||
# in them. These are objects of static storage duration, so
|
||||
# every field it inserts was already zero: it changes the
|
||||
# spelling, not the program. 55 fields across two files.
|
||||
d=$(unpack "$p")
|
||||
# Deliberately unquoted: the find is a LIST of sources to
|
||||
# patch, one argument each, and none of greenlet's file names
|
||||
# has ever contained a space.
|
||||
# shellcheck disable=SC2046
|
||||
"$HOSTPY" tools/python-packages/fill-designators.py \
|
||||
"$TREE/include/python$PY_MM" \
|
||||
$(find "$d/src/greenlet" -name '*.cpp' -o -name '*.hpp' | sort) \
|
||||
>"$LOG/.py-pkg-greenlet-designators.log" 2>&1 \
|
||||
|| { echo " !! fill-designators.py failed --" \
|
||||
"work/.py-pkg-greenlet-designators.log" >&2
|
||||
tail -20 "$LOG/.py-pkg-greenlet-designators.log" | sed 's/^/ /' >&2
|
||||
exit 1; }
|
||||
pywheel "$d"
|
||||
;;
|
||||
lmdb)
|
||||
# LMDB_FORCE_CPYTHON=1 is load-bearing. lmdb's setup.py has two
|
||||
# backends: a real CPython extension and a cffi one that ships
|
||||
# mdb.c and COMPILES IT AT IMPORT TIME. Left to itself here it
|
||||
# picks cffi and produces a py3-none-any wheel with mdb.c
|
||||
# inside -- i.e. a Moonraker that tries to invoke a compiler on
|
||||
# a printer that has none. (Do not "fix" this with
|
||||
# LMDB_FORCE_CFFI=0: setup.py tests the variable for PRESENCE,
|
||||
# so the string "0" selects cffi.)
|
||||
pywheel "$SDIST" LMDB_FORCE_CPYTHON=1
|
||||
;;
|
||||
pillow)
|
||||
# ZLIB ONLY, and pip cannot drive it: the --disable-* flags are
|
||||
# build_ext options and PEP 517 offers no way to pass them, so
|
||||
# setup.py is called directly. Gcode thumbnails are base64 PNG,
|
||||
# so zlib is the only codec on that path; jpeg, tiff, webp,
|
||||
# jpeg2000, lcms, freetype and imagequant are libraries nobody
|
||||
# has cross-built here, and pillow probes for them by trying to
|
||||
# LINK against the HOST's copies unless told not to -- which is
|
||||
# what --disable-platform-guessing is for.
|
||||
d=$(unpack "$p")
|
||||
( cd "$d" && "$HOSTPY" setup.py build_ext \
|
||||
--disable-jpeg --disable-tiff --disable-webp --disable-jpeg2000 \
|
||||
--disable-imagequant --disable-lcms --disable-freetype \
|
||||
--disable-xcb --disable-platform-guessing --enable-zlib \
|
||||
-I"$DEP/include" -L"$DEP/lib" bdist_wheel ) \
|
||||
>"$LOG/.py-pkg-pillow.log" 2>&1 \
|
||||
|| { echo " !! pillow failed to build -- work/.py-pkg-pillow.log" >&2
|
||||
tail -25 "$LOG/.py-pkg-pillow.log" | sed 's/^/ /' >&2
|
||||
exit 1; }
|
||||
cp -a "$d"/dist/*.whl "$WHEELS/" \
|
||||
|| { echo " !! pillow built no wheel -- work/.py-pkg-pillow.log" >&2
|
||||
exit 1; }
|
||||
;;
|
||||
*)
|
||||
pywheel "$SDIST"
|
||||
;;
|
||||
esac
|
||||
printf ' %-24s %s\n' "$p" "$(pypkg_var "$p" FILE)"
|
||||
done
|
||||
|
||||
# ----------------------------------------------------- into the tree
|
||||
# Unzipped rather than pip-installed, because `pip install` insists on
|
||||
# installing FOR the interpreter running it -- and that one is x86-64.
|
||||
# A wheel is a zip whose layout is already the layout of
|
||||
# site-packages, so unzipping it is the whole of what an install does
|
||||
# here.
|
||||
#
|
||||
# .dist-info goes: nothing on the printer resolves a dependency, asks
|
||||
# for a version or runs an entry point, and it is 200KB of metadata
|
||||
# describing a build machine to nobody. The .data directories and the
|
||||
# scripts in them go for a stronger reason -- their console-script
|
||||
# shebangs name the BUILD-PYTHON's path, so shipping them would put
|
||||
# files on the printer that reference /home/somebody/work/.py-host.
|
||||
for w in "$WHEELS"/*.whl; do
|
||||
"$HOSTPY" -m zipfile -e "$w" "$SP" \
|
||||
|| { echo " !! could not unpack $(basename "$w") into site-packages" >&2
|
||||
exit 1; }
|
||||
done
|
||||
find "$SP" -name '__pycache__' -prune -exec rm -rf {} + 2>/dev/null || true
|
||||
find "$SP" -maxdepth 1 -name '*.dist-info' -prune -exec rm -rf {} + 2>/dev/null || true
|
||||
rm -rf "${SP:?}"/bin "${SP:?}"/*.data
|
||||
# Stripped here rather than in the trim below, because the trim's
|
||||
# find only knows about lib-dynload -- and these are the .so files
|
||||
# that are NOT stdlib. 12 of them, ~1.5MB of symbols.
|
||||
find "$SP" -name '*.so' -exec "$PY_HOST-strip" {} + 2>/dev/null || true
|
||||
) || { echo " !! the third-party package build failed. Its logs are" >&2
|
||||
echo " work/.py-pkg-*.log, one per package, and the unpacked" >&2
|
||||
echo " sources are still in work/.py-pkgsrc." >&2
|
||||
exit 1; }
|
||||
|
||||
# --------------------------------------------------------------- trim
|
||||
# 183MB staged, 30MB shipped. What goes: the test package and idlelib
|
||||
# (nothing on a printer runs either), tkinter (there is no X11), the
|
||||
@@ -849,12 +1212,23 @@ EOF
|
||||
# and for the same reason: unpacking it at $MODDIR puts every file back
|
||||
# where it was compiled to live. The .version stamp sits BESIDE them rather
|
||||
# than inside, so it can never be mistaken for something that ships.
|
||||
#
|
||||
# site-packages rides along inside lib/python$PY_MM/ and needs no line of
|
||||
# its own here, which is the whole reason step 4 installs into the staged
|
||||
# tree rather than somewhere beside it: one copy, one cache, one tarball
|
||||
# for the replica gates (test/ffsim/gates.py packs bin/ + lib/ out of this
|
||||
# directory), and the ABI gate below already walks all of it.
|
||||
mkdir -p "$PY_BUILD"
|
||||
cp -a "$PY_TRIM/bin" "$PY_TRIM/lib" "$PY_BUILD/"
|
||||
rm -rf work/.py-src work/.py-dep work/.py-host work/.py-stage work/.py-xw
|
||||
rm -rf work/.py-src work/.py-dep work/.py-host work/.py-stage work/.py-xw \
|
||||
work/.py-pkgsrc work/.py-wheels work/.py-xsysconfig
|
||||
echo "$PY_STAMP" > "$PY_BUILD/.version"
|
||||
# Written LAST and separately, so that an interrupted build leaves a cache
|
||||
# that is stale rather than one that lies: both files have to be right
|
||||
# before the next run is allowed to skip this section.
|
||||
printf '%s\n' "$PYPKG_STAMP" > "$PY_BUILD/.pkg-version"
|
||||
else
|
||||
skip "python: work/.py313 already holds CPython $PY_VERSION"
|
||||
skip "python: work/.py313 already holds CPython $PY_VERSION and its packages"
|
||||
fi
|
||||
|
||||
# The gates, asked of the TREE THAT SHIPS rather than of the build -- so a
|
||||
@@ -879,33 +1253,82 @@ PY_SQLITE=$(ls "$PY_BUILD/lib/python$PY_MM/lib-dynload/"_sqlite3*.so 2>/dev/null
|
||||
|| { echo " !! python: NO _sqlite3 MODULE. configure's link probe failed" >&2
|
||||
echo " silently -- check LIBS/LIBSQLITE3_LIBS and work/.py-configure.log" >&2
|
||||
exit 1; }
|
||||
# ABI, over every ELF in the tree: interpreter, 56 extension modules and
|
||||
# whatever else `make install` put there. The printer's kernel wants
|
||||
# nan2008/o32/mips32r2 and says ENOEXEC to anything else, and a cross-build
|
||||
# that quietly emitted one host object -- or one legacy-NaN object, because a
|
||||
# flag did not reach one link line -- looks like a clean build here.
|
||||
# And the packages, asked of the cache for the same reason: this runs whether
|
||||
# step 4 built them a minute ago or a previous build did. Two claims, because
|
||||
# they fail differently. A site-packages with no EXTENSION MODULES in it is a
|
||||
# build where every native package quietly fell back to something pure or was
|
||||
# skipped -- it imports on the build host and loses lmdb, cffi and greenlet on
|
||||
# the printer. A site-packages missing a NAMED package is a list that changed
|
||||
# without anyone noticing; the three named here are the three whose absence
|
||||
# takes Moonraker or klippy down completely rather than costing a feature.
|
||||
PY_SP="$PY_BUILD/lib/python$PY_MM/site-packages"
|
||||
for m in lmdb tornado cffi; do
|
||||
[ -e "$PY_SP/$m" ] || [ -e "$PY_SP/$m.py" ] \
|
||||
|| { echo " !! python: no '$m' in site-packages -- the package build" >&2
|
||||
echo " did not produce it. Delete work/.py313 and rebuild." >&2
|
||||
exit 1; }
|
||||
done
|
||||
PY_EXT=$(find "$PY_SP" -name '*.so' | wc -l)
|
||||
[ "$PY_EXT" -ge 12 ] \
|
||||
|| { echo " !! python: only $PY_EXT extension modules in site-packages," >&2
|
||||
echo " expected at least 12. A native package fell back to a" >&2
|
||||
echo " pure-python or py3-none-any build; check work/.py-pkg-*.log." >&2
|
||||
exit 1; }
|
||||
|
||||
# ABI, over every ELF the Ingenic toolchain produced: the interpreter, 56
|
||||
# stdlib extension modules, the 12 in site-packages and libsodium in 5d below.
|
||||
# The printer's kernel wants nan2008/o32/mips32r2 and says ENOEXEC to anything
|
||||
# else, and a cross-build that quietly emitted one host object -- or one
|
||||
# legacy-NaN object, because a flag did not reach one link line -- looks like a
|
||||
# clean build here.
|
||||
#
|
||||
# TWO expected words, not one. 0x70001405 is the measured value for an
|
||||
# EXECUTABLE; a shared object additionally carries EF_MIPS_PIC (0x2) and so
|
||||
# reads 0x70001407. That is correct and unavoidable for a DYN -- klippy's own
|
||||
# c_helper.so has it too -- so a gate that pinned one word would fail on every
|
||||
# extension module in the tree.
|
||||
PY_ELF=0
|
||||
while IFS= read -r f; do
|
||||
[ "$(head -c 4 "$f" 2>/dev/null)" = "$(printf '\177ELF')" ] || continue
|
||||
hdr=$(readelf -h "$f" 2>/dev/null || true)
|
||||
case "$hdr" in
|
||||
*nan2008*o32*mips32r2*) ;;
|
||||
*) echo " !! python: $f is not nan2008/o32/mips32r2" >&2
|
||||
readelf -h "$f" 2>&1 | sed 's/^/ /' >&2; exit 1 ;;
|
||||
esac
|
||||
flags=$(awk '/Flags:/{print $2}' <<<"$hdr" | tr -d ,)
|
||||
want=0x70001405; grep -q '^ Type:.*DYN' <<<"$hdr" && want=0x70001407
|
||||
[ "$flags" = "$want" ] || {
|
||||
echo " !! python: $f has e_flags=$flags, want $want" >&2; exit 1; }
|
||||
PY_ELF=$((PY_ELF + 1))
|
||||
done < <(find "$PY_BUILD/bin" "$PY_BUILD/lib" -type f)
|
||||
say "python: $PY_ELF ELF objects, all nan2008/o32/mips32r2; $(basename "$PY_SQLITE") present"
|
||||
#
|
||||
# A FUNCTION, AND POINTED AT THE PAYLOAD. It used to walk $PY_BUILD/bin and
|
||||
# $PY_BUILD/lib -- the build cache. That was the whole tree while the
|
||||
# interpreter was the only thing this toolchain produced, and it stopped being
|
||||
# so the moment site-packages and libsodium arrived: a .so staged into
|
||||
# $MOD_PAYLOAD by a path the gate did not know about ships ungated, and the
|
||||
# first machine to notice is a printer. So the rule did not change (it already
|
||||
# covers a DYN correctly) -- the REACH did, to the staged payload, which is
|
||||
# the only tree that is by definition everything that ships.
|
||||
#
|
||||
# It is NOT pointed at all of $MOD_PAYLOAD, and that is deliberate rather than
|
||||
# lazy: s6 in 5b is built by the musl toolchain and reads e_flags=0x1007
|
||||
# (mips1, legacy NaN, PIE) -- correct for it, measured working on the printer,
|
||||
# and it has its own gate up there. HelixScreen and the stock rootfs binaries
|
||||
# are somebody else's builds again. This gate is about the objects WE compiled
|
||||
# with the Ingenic toolchain, and it names them.
|
||||
mips_abi_gate() {
|
||||
local n=0 f hdr flags want
|
||||
while IFS= read -r f; do
|
||||
# readelf itself is the ELF test, rather than comparing the first four
|
||||
# bytes to \177ELF: that comparison was a command substitution over
|
||||
# arbitrary binary content, and every data file in site-packages that
|
||||
# happens to start with a NUL made bash print "warning: command
|
||||
# substitution: ignored null byte in input" -- eight lines of noise
|
||||
# across a clean build, from the gate that is supposed to be the quiet
|
||||
# one. readelf exits non-zero on anything that is not an ELF, which is
|
||||
# the same question asked of the tool that has to answer it anyway.
|
||||
hdr=$(readelf -h "$f" 2>/dev/null) || continue
|
||||
case "$hdr" in
|
||||
*nan2008*o32*mips32r2*) ;;
|
||||
*) echo " !! $f is not nan2008/o32/mips32r2" >&2
|
||||
readelf -h "$f" 2>&1 | sed 's/^/ /' >&2; return 1 ;;
|
||||
esac
|
||||
flags=$(awk '/Flags:/{print $2}' <<<"$hdr" | tr -d ,)
|
||||
want=0x70001405; grep -q '^ Type:.*DYN' <<<"$hdr" && want=0x70001407
|
||||
[ "$flags" = "$want" ] || {
|
||||
echo " !! $f has e_flags=$flags, want $want" >&2; return 1; }
|
||||
n=$((n + 1))
|
||||
done < <(find "$@" -type f 2>/dev/null)
|
||||
printf '%s' "$n"
|
||||
}
|
||||
|
||||
# Staged into the SAME bin/ and lib/ as everything else in this prefix root,
|
||||
# which is why this is a copy of the contents and not of the directories: s6's
|
||||
# binaries are already in $MOD_PAYLOAD/bin and must stay there.
|
||||
@@ -913,7 +1336,137 @@ cp -a "$PY_BUILD/bin/." "$MOD_PAYLOAD/bin/"
|
||||
mkdir -p "$MOD_PAYLOAD/lib"
|
||||
cp -a "$PY_BUILD/lib/." "$MOD_PAYLOAD/lib/"
|
||||
chmod +x "$MOD_PAYLOAD/bin/python$PY_MM"
|
||||
PY_ELF=$(mips_abi_gate "$MOD_PAYLOAD/bin/python$PY_MM" "$MOD_PAYLOAD/lib/python$PY_MM") || exit 1
|
||||
say "python: $PY_ELF ELF objects staged, all nan2008/o32/mips32r2;" \
|
||||
"$(basename "$PY_SQLITE") present; $PY_EXT extension modules in site-packages"
|
||||
du -sh "$PY_BUILD/lib/python$PY_MM" | awk '{print " "$1"\tlib/python'"$PY_MM"'/"}'
|
||||
du -sh "$PY_SP" | awk '{print " "$1"\tlib/python'"$PY_MM"'/site-packages/"}'
|
||||
|
||||
# ---------------------------------------------------- 5d. libsodium (shipped)
|
||||
# The one library of ours that ships as a .so, and the LAST /usr/prog string
|
||||
# in the phase-6 picture.
|
||||
#
|
||||
# WHY IT CANNOT BE STATIC, when the interpreter's seven dependencies all are.
|
||||
# libnacl is pure python: it reaches libsodium through
|
||||
# ctypes.cdll.LoadLibrary, which is dlopen, and you cannot dlopen an archive.
|
||||
# Moonraker's `authorization` component signs its JWTs with the ed25519 pair
|
||||
# libnacl exposes, so this is on the startup path of the web UI rather than
|
||||
# beside it.
|
||||
#
|
||||
# WHY IT COSTS anvil-env.sh NOTHING, which is worth stating because it looks
|
||||
# like it should need a library path entry. libnacl's third fallback is
|
||||
# __file__[0:__file__.find("lib")+3] + "/libsodium.so", and __file__ here is
|
||||
# $MODDIR/lib/python3.13/site-packages/libnacl/__init__.py -- so it resolves
|
||||
# $MODDIR/lib/libsodium.so BY ABSOLUTE PATH, measured working in the replica
|
||||
# with LD_LIBRARY_PATH unset entirely. That works because this prefix happens
|
||||
# to contain the string "lib"; a prefix without it would break silently, which
|
||||
# is a thing to remember if $MODDIR ever moves. It is also why the plain
|
||||
# `libsodium.so` symlink is not a development leftover to be trimmed: it is
|
||||
# the FIRST name libnacl asks for, and the only one that fallback constructs.
|
||||
#
|
||||
# CACHED ON THE VERSION, like s6 and unlike c_helper.so. 24 seconds is not the
|
||||
# reason -- the reason is bin/fetch-assets.sh: an uncached build of this drags
|
||||
# the ~203MB Ingenic toolchain download along behind it on every build of a
|
||||
# checkout that has nothing else to compile. The stamp is what lets the
|
||||
# fetcher skip it, so the stamp is what makes it free.
|
||||
SODIUM_XW=work/.sodium-xw
|
||||
if [ "$(cat "$SODIUM_BUILD/.version" 2>/dev/null || true)" != "$SODIUM_VERSION" ]; then
|
||||
if [ ! -x "$PY_TOOLCHAIN_DIR/bin/$PY_HOST-gcc" ]; then
|
||||
[ -f "${MIPS_TOOLCHAIN_TGZ:-}" ] || {
|
||||
echo " !! libsodium needs (re)building and there is no toolchain:" >&2
|
||||
echo " $MIPS_TOOLCHAIN_TGZ is missing. Run ./bin/fetch-assets.sh." >&2
|
||||
exit 1; }
|
||||
say "libsodium: unpacking the Ingenic MIPS toolchain"
|
||||
mkdir -p work/.mips-toolchain
|
||||
tar -xzf "$MIPS_TOOLCHAIN_TGZ" -C work/.mips-toolchain
|
||||
fi
|
||||
[ -f "${SODIUM_TGZ:-}" ] || {
|
||||
echo " !! no libsodium source at '$SODIUM_TGZ' -- run ./bin/fetch-assets.sh" >&2
|
||||
exit 1; }
|
||||
say "libsodium: cross-compiling $SODIUM_VERSION for $MODDIR/lib"
|
||||
rm -rf work/.sodium-src work/.sodium-stage "$SODIUM_XW" "$SODIUM_BUILD"
|
||||
mkdir -p work/.sodium-src "$SODIUM_XW/bin"
|
||||
tar -xzf "$SODIUM_TGZ" -C work/.sodium-src
|
||||
(
|
||||
set -e
|
||||
# A subshell, as in 5b and 5c, so the cross-compiler cannot leak.
|
||||
TC="$PWD/$PY_TOOLCHAIN_DIR"
|
||||
XW="$PWD/$SODIUM_XW"
|
||||
STAGE="$PWD/work/.sodium-stage"
|
||||
LOG="$PWD/work"
|
||||
# The same wrapper trick, rebuilt here rather than shared with 5c:
|
||||
# 5c deletes work/.py-xw when its build succeeds, and this step has to
|
||||
# work on a run where 5c did nothing at all because its cache was
|
||||
# warm. -EL -mnan=2008 in the driver, where libsodium's libtool link
|
||||
# lines cannot drop them.
|
||||
for t in gcc g++ cpp; do
|
||||
printf '#!/bin/sh\nexec %s/bin/%s-%s -EL -mnan=2008 "$@"\n' \
|
||||
"$TC" "$PY_HOST" "$t" > "$XW/bin/$PY_HOST-$t"
|
||||
chmod +x "$XW/bin/$PY_HOST-$t"
|
||||
done
|
||||
for t in ar as ld nm objcopy objdump ranlib readelf strip strings size; do
|
||||
ln -sf "$TC/bin/$PY_HOST-$t" "$XW/bin/$PY_HOST-$t"
|
||||
done
|
||||
export PATH="$XW/bin:$PATH"
|
||||
cd "$PWD/work/.sodium-src/libsodium-$SODIUM_VERSION"
|
||||
# --disable-static: nothing links this statically and a .a would only
|
||||
# be deleted again below.
|
||||
# --host is what makes autoconf reach for the mips-linux-gnu- prefixed
|
||||
# tools in the wrapper directory, which is the entire point of them.
|
||||
# libsodium's runtime feature probes are AC_RUN_IFELSE with cross
|
||||
# defaults supplied, so nothing here needs qemu.
|
||||
./configure --host="$PY_HOST" --prefix="$MODDIR" \
|
||||
--disable-static --enable-shared --with-pic \
|
||||
CFLAGS="-O2 -fPIC" >"$LOG/.sodium-configure.log" 2>&1
|
||||
make -j"$(nproc 2>/dev/null || echo 4)" >"$LOG/.sodium-make.log" 2>&1
|
||||
make install DESTDIR="$STAGE" >>"$LOG/.sodium-make.log" 2>&1
|
||||
) || { echo " !! the libsodium cross-build failed -- work/.sodium-configure.log" >&2
|
||||
echo " and work/.sodium-make.log; the source tree is still in" >&2
|
||||
echo " work/.sodium-src." >&2
|
||||
exit 1; }
|
||||
# lib/ ONLY. include/ and lib/pkgconfig exist to BUILD against libsodium,
|
||||
# which happens on a developer's machine and not on a printer -- and
|
||||
# pkgconfig would otherwise drop a .pc file describing this build into the
|
||||
# prefix's shared lib/, next to the interpreter's stdlib. The .la file goes
|
||||
# for the same reason plus one more: it names absolute build-machine paths.
|
||||
mkdir -p "$SODIUM_BUILD/lib"
|
||||
cp -a "work/.sodium-stage$MODDIR/lib/"libsodium.so* "$SODIUM_BUILD/lib/"
|
||||
rm -f "$SODIUM_BUILD/lib/"*.la
|
||||
find "$SODIUM_BUILD/lib" -type f -name 'libsodium.so*' \
|
||||
-exec "$PY_TOOLCHAIN_DIR/bin/$PY_HOST-strip" --strip-unneeded {} +
|
||||
rm -rf work/.sodium-src work/.sodium-stage "$SODIUM_XW"
|
||||
echo "$SODIUM_VERSION" > "$SODIUM_BUILD/.version"
|
||||
else
|
||||
skip "libsodium: work/.sodium already holds $SODIUM_VERSION"
|
||||
fi
|
||||
|
||||
# Staged into the prefix's shared lib/, beside lib/python3.13 -- one prefix,
|
||||
# one library directory. cp -a and not cp: two of these three names are
|
||||
# SYMLINKS (libsodium.so -> libsodium.so.26 -> libsodium.so.26.2.0), the first
|
||||
# is the name libnacl constructs and the second is the SONAME the loader then
|
||||
# asks for, so a copy that dereferenced them would ship three identical 406KB
|
||||
# files and still work -- until someone wondered why the payload had grown.
|
||||
cp -a "$SODIUM_BUILD/lib/"libsodium.so* "$MOD_PAYLOAD/lib/"
|
||||
[ -L "$MOD_PAYLOAD/lib/libsodium.so" ] \
|
||||
|| { echo " !! libsodium: lib/libsodium.so is not a symlink -- libnacl's" >&2
|
||||
echo " dlopen fallback asks for that exact name." >&2; exit 1; }
|
||||
# The gate, over the payload, with the same rule and the same function 5c
|
||||
# uses: this is a DYN, so 0x70001407, and a legacy-NaN or big-endian libsodium
|
||||
# would import perfectly on a build host and fail only inside libnacl's
|
||||
# ctypes call on the printer -- which surfaces as Moonraker's authorization
|
||||
# component failing to load, three layers away from anything that says MIPS.
|
||||
#
|
||||
# Aimed at the resolved object rather than at $MOD_PAYLOAD/lib, which would
|
||||
# be the tidier-looking line: lib/ is also where the interpreter's whole
|
||||
# stdlib lives and 5c has just walked all of it, and re-walking two thousand
|
||||
# files to reach one is a gate that gets deleted the first time somebody times
|
||||
# a build.
|
||||
SODIUM_SO=$(readlink -f "$MOD_PAYLOAD/lib/libsodium.so")
|
||||
SODIUM_ELF=$(mips_abi_gate "$SODIUM_SO") || exit 1
|
||||
say "libsodium: $SODIUM_VERSION staged into lib/ --" \
|
||||
"$SODIUM_ELF object, nan2008/o32/mips32r2," \
|
||||
"$(readelf -d "$SODIUM_SO" | awk '/SONAME/{gsub(/[][]/,"",$5); print $5}')," \
|
||||
"$(du -h "$SODIUM_SO" | cut -f1)"
|
||||
|
||||
# ------------------------------------------------------------------- 6. SSH
|
||||
# Nothing to install. The stock rootfs (kernel-*.tar.xz -> rootfs.squashfs)
|
||||
|
||||
@@ -188,6 +188,37 @@ if [ -f "$WORK/anvil.tar.xz" ]; then
|
||||
grep -qE '(^|/)bin/python3$' <<<"$LIST" \
|
||||
&& bad "payload ships bin/python3 -- it would shadow FlashForge's interpreter on PATH" \
|
||||
|| ok "no bin/python3 symlink (nothing shadows FlashForge's python3 on PATH)"
|
||||
# The half of the 3.13 story that is not the interpreter: what runs ON it.
|
||||
# Same shape of check and the same reason -- nothing on the printer uses
|
||||
# any of this yet, so nothing on the printer would notice it going missing,
|
||||
# and the first symptom would be the release AFTER the FF_PYTHON switch.
|
||||
#
|
||||
# Two of the twelve extension modules by name rather than a count, and
|
||||
# these two: lmdb is Moonraker's DATABASE at the pinned commit (no lmdb,
|
||||
# no Moonraker at all, not a degraded one) and _cffi_backend is what
|
||||
# klippy dlopens c_helper.so through, which is the entire reason this
|
||||
# interpreter had to be glibc rather than musl. They are also the two that
|
||||
# a "cross build" quietly resolving to a manylinux wheel gets WRONG rather
|
||||
# than missing -- and the .so suffix is what says which: a mipsel build
|
||||
# spells itself cpython-313-mipsel-linux-gnu.so, an x86-64 wheel spells
|
||||
# itself cpython-313-x86_64-linux-gnu.so, so the name is the architecture.
|
||||
grep -q 'site-packages/lmdb/cpython.cpython-313-mipsel-linux-gnu\.so' <<<"$LIST" \
|
||||
&& ok "lmdb's mipsel CPython extension present (Moonraker's database at this pin)" \
|
||||
|| bad "no mipsel lmdb extension in site-packages -- Moonraker cannot open its database; check bin/patch.sh section 5c step 4"
|
||||
grep -q 'site-packages/_cffi_backend.cpython-313-mipsel-linux-gnu\.so' <<<"$LIST" \
|
||||
&& ok "cffi's mipsel extension present (klippy's route to c_helper.so)" \
|
||||
|| bad "no mipsel _cffi_backend in site-packages -- klippy could not load chelper on 3.13"
|
||||
# libsodium is the one library of ours that ships as a .so, because libnacl
|
||||
# dlopens it -- and the bare `libsodium.so` name is not decoration, it is
|
||||
# the exact name libnacl's path fallback constructs. A payload with the
|
||||
# versioned file and no symlink would pass a naive "is libsodium there"
|
||||
# check and fail at Moonraker's authorization component.
|
||||
grep -qE 'lib/libsodium\.so\.[0-9]' <<<"$LIST" \
|
||||
&& ok "libsodium present in lib/ (libnacl's ed25519, no /usr/prog needed)" \
|
||||
|| bad "no lib/libsodium.so.* in the payload -- bin/patch.sh section 5d did not stage the cross-build"
|
||||
grep -qE 'lib/libsodium\.so$' <<<"$LIST" \
|
||||
&& ok "lib/libsodium.so link present (the name libnacl's dlopen fallback builds)" \
|
||||
|| bad "lib/libsodium.so link missing -- libnacl looks for that exact name and would fall through to /usr/prog"
|
||||
grep -q 'mainsail/index.html' <<<"$LIST" && ok "Mainsail present" || warn "no Mainsail in payload"
|
||||
grep -q 'helixscreen/bin/helix-screen' <<<"$LIST" && ok "HelixScreen present" || warn "no HelixScreen in payload"
|
||||
# We deliberately ship no dropbear: the stock rootfs already has one, with
|
||||
|
||||
+19
-2
@@ -13,6 +13,23 @@
|
||||
# pkg-config and perl are for the same section -- CPython's configure probes
|
||||
# libffi through pkg-config, and OpenSSL's Configure is a perl program.
|
||||
#
|
||||
# zlib1g-dev is for THE HOST HALF of that build and nothing else, which is why
|
||||
# it looks out of place beside two cross-toolchains: the build-python needs a
|
||||
# working zlib module. Every wheel the package step produces is a zip, and so
|
||||
# is the pip that produces them -- ensurepip installs pip out of a bundled
|
||||
# .whl, and a build-python without zlib cannot even unpack it ("can't
|
||||
# decompress data; zlib not available"). CPython records the module as missing
|
||||
# and carries on, so the symptom lands three steps later as "No module named
|
||||
# pip". The TARGET's zlib is nothing to do with this: that one is the static
|
||||
# -fPIC libz.a bin/patch.sh cross-builds from the pinned tarball.
|
||||
#
|
||||
# `patch` is in the list above for one package: lmdb's setup.py shells out to
|
||||
# /usr/bin/patch BY ABSOLUTE PATH to apply its own env-copy-txn.patch to the
|
||||
# bundled liblmdb before compiling it, and raises "Applying patch failed" when
|
||||
# that is not there -- which is a build failure with nothing in it to say that
|
||||
# a two-hundred-kilobyte utility is what is missing. lmdb is Moonraker's
|
||||
# database at the pinned commit, so this is not an optional package.
|
||||
#
|
||||
# The alternative was a container of its own, the way tools/supervisor and
|
||||
# tools/python build for measurement. The build lane deliberately gets no
|
||||
# docker socket (only the test lane does), so there is no daemon to ask.
|
||||
@@ -23,9 +40,9 @@ RUN echo 'Acquire::Retries "3";' > /etc/apt/apt.conf.d/80-retries \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends \
|
||||
bash coreutils tar xz-utils gzip openssl unzip curl ca-certificates \
|
||||
python3 python3-jinja2 python3-pytest python3-pyflakes \
|
||||
binutils file findutils grep sed gawk make git diffutils procps \
|
||||
binutils file findutils grep sed gawk make git diffutils patch procps \
|
||||
squashfs-tools qemu-user-static docker.io shellcheck \
|
||||
gcc g++ libc6-dev pkg-config perl \
|
||||
gcc g++ libc6-dev zlib1g-dev pkg-config perl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
+39
-14
@@ -73,6 +73,15 @@
|
||||
# rather than ImportError when it cannot, which is what made
|
||||
# the authorization outage read as a component crash rather
|
||||
# than a missing library.
|
||||
# THIS ONE IS ON BORROWED TIME, and stays only because
|
||||
# FF_PYTHON below still points at 3.8.2. The payload now
|
||||
# ships a libsodium 1.0.20 of our own at $MODDIR/lib
|
||||
# (bin/patch.sh section 5d), which our 3.13's libnacl finds
|
||||
# by absolute path with no library path entry at all. It
|
||||
# cannot come off this list yet: 3.8's libnacl, in
|
||||
# FlashForge's site-packages, still asks the loader for
|
||||
# libsodium.so.18 and the rootfs has none. The two lines move
|
||||
# together -- see FF_PYTHON.
|
||||
#
|
||||
# NOT /usr/prog/mjpg-streamer: that one is a plugin directory rather than a
|
||||
# library package, it carries its own libjpeg.so.9, and putting it in front of
|
||||
@@ -105,22 +114,38 @@ export LD_LIBRARY_PATH
|
||||
# at the point it is missing.
|
||||
#
|
||||
# IT IS STILL FLASHFORGE'S 3.8.2, AND THAT IS A STATEMENT ABOUT TIMING, NOT A
|
||||
# PREFERENCE. Since phase 6 the payload carries a complete CPython 3.13 of our
|
||||
# own, installed into this same prefix root -- $MODDIR/bin/python3.13, with its
|
||||
# stdlib in $MODDIR/lib/python3.13 -- cross-built by bin/patch.sh section 5c,
|
||||
# with the working sqlite3 this firmware's 3.8.2 has not got. Nothing runs on
|
||||
# it yet, because klippy, Moonraker and bin/ff-startup.py import third-party C
|
||||
# extensions -- tornado, lmdb, cffi, greenlet, pillow, libnacl -- that exist on
|
||||
# this printer only as mipsel .so files built against 3.8, in FlashForge's
|
||||
# site-packages. None has been cross-built for 3.13. Point FF_PYTHON at ours
|
||||
# before they exist and the printer boots to a dark screen with an ImportError
|
||||
# in a log nobody is reading.
|
||||
# PREFERENCE. The payload carries a complete CPython 3.13 of our own, installed
|
||||
# into this same prefix root -- $MODDIR/bin/python3.13, with its stdlib in
|
||||
# $MODDIR/lib/python3.13 -- cross-built by bin/patch.sh section 5c, with the
|
||||
# working sqlite3 this firmware's 3.8.2 has not got.
|
||||
#
|
||||
# WHAT CHANGED, AND WHY THIS LINE STILL HAS NOT. The reason written here used
|
||||
# to be that klippy, Moonraker and bin/ff-startup.py import third-party C
|
||||
# extensions -- tornado, lmdb, cffi, greenlet, pillow, libnacl -- which existed
|
||||
# on this printer only as mipsel .so files built against 3.8, in FlashForge's
|
||||
# site-packages, and that none had been cross-built for 3.13. That is no longer
|
||||
# true: all of them now ship beside the interpreter in
|
||||
# $MODDIR/lib/python3.13/site-packages, cross-built by the same section, and
|
||||
# libsodium with them in $MODDIR/lib. Moonraker has been measured SERVING on
|
||||
# that interpreter in the replica -- :7125 bound, 23 components loaded and 0
|
||||
# failed, its lmdb database written and read back, nothing under /usr/prog
|
||||
# mapped by the running process.
|
||||
#
|
||||
# What is left is not capability, it is the boot path. Nothing has yet run
|
||||
# through init.d/S62moonraker and s6 on 3.13: the readiness handshake, the
|
||||
# restart machinery and the shutdown ordering are all still gated only against
|
||||
# 3.8, and those are exactly the parts that fail at 3am on a printer rather
|
||||
# than in a case script. klippy's numpy gap is separate and smaller (it costs
|
||||
# input-shaper calibration, not printing). So the switch is deferred one more
|
||||
# commit ON PURPOSE, exactly as s6 was shipped-but-unstarted for a release:
|
||||
# this line and the /usr/prog/libsodium entry above move TOGETHER, in a commit
|
||||
# that changes nothing else, with `make test-python`, `make test-moonraker` and
|
||||
# `make test-services` green on the replica first. Flipping it early costs a
|
||||
# dark screen and an ImportError in a log nobody is reading.
|
||||
#
|
||||
# FlashForge's tree is not touched either way: nothing here writes to
|
||||
# /usr/prog, and the 3.13 interpreter lives entirely under /usr/data/anvil like
|
||||
# every other thing this mod installs. The switch, when the extensions are
|
||||
# built, is this one line -- and `make test-python` and `make test-moonraker`
|
||||
# are what have to be green first.
|
||||
# /usr/prog, and everything of ours lives under /usr/data/anvil like every
|
||||
# other thing this mod installs.
|
||||
#
|
||||
# Note what is NOT on PATH below: $MODDIR/bin is prepended (s6 needs it), and
|
||||
# the interpreter is in there, but it is called python3.13 and only that.
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
# The Python packages Moonraker and klippy need, cross-built for mipsel
|
||||
|
||||
The second half of phase 6. `tools/python/` builds the interpreter;
|
||||
this builds what runs on it. **Not yet wired into `bin/patch.sh`** -- see
|
||||
"What is left" at the bottom. `build.sh` is the measurement harness, preserved
|
||||
here because it is where the findings live, exactly as `tools/supervisor/`
|
||||
is for s6.
|
||||
this builds what runs on it. **Now wired into `bin/patch.sh` section 5c,
|
||||
step 4** -- see "What is left" at the bottom for what that did and did not
|
||||
settle. `build.sh` is the measurement harness, preserved here because it is
|
||||
where the findings live, exactly as `tools/supervisor/` is for s6; it is not
|
||||
what a release runs any more, and the two have drifted on purpose (patch.sh
|
||||
builds offline from `vendor/` and needs no docker, because the build lane has
|
||||
no docker socket). `fill-designators.py` is the exception: it is a real build
|
||||
input, called by `patch.sh` for greenlet.
|
||||
|
||||
Everything below was run in the printer replica on the real
|
||||
`rootfs.squashfs`, not read off a project page.
|
||||
@@ -12,8 +16,10 @@ Everything below was run in the printer replica on the real
|
||||
|
||||
## What builds
|
||||
|
||||
19 packages, 12 mipsel extension modules, from a 56-second clean build in one
|
||||
throwaway `debian:bookworm`. Every `.so` is gated at `e_flags=0x70001407`,
|
||||
18 sdists, 12 mipsel extension modules, from a 56-second clean build in one
|
||||
throwaway `debian:bookworm`. (This said 19 for a while; the list below is 18,
|
||||
and `versions.env` -- which is now the list that matters -- pins 18.) Every
|
||||
`.so` is gated at `e_flags=0x70001407`,
|
||||
`nan2008/o32/mips32r2` -- shared objects carry `EF_MIPS_PIC`, so they read
|
||||
`…07` where an executable reads `…05`.
|
||||
|
||||
@@ -166,18 +172,28 @@ same one-line `FF_PYTHON` switch, since 3.8's libnacl still needs it.
|
||||
Capability is settled -- Moonraker serves, the database works, libsodium is
|
||||
ours. What is left is packaging and integration:
|
||||
|
||||
1. **Wire into `bin/patch.sh`.** The blocker is that section 5c's trim deletes
|
||||
`include/` and `lib/python3.13/config-3.13-*` and then removes the staging
|
||||
tree, and the package build needs both. Cleanest fix is to run the package
|
||||
build inside 5c against the untrimmed stage BEFORE the trim, rather than
|
||||
caching a second tree. Add a step for libsodium, a step to stage the 19
|
||||
packages into `lib/python3.13/site-packages/`, and move every pin into
|
||||
`versions.env` with sha256s.
|
||||
2. **Extend the ABI gate's reach.** It walks `$PY_BUILD/bin` and
|
||||
`$PY_BUILD/lib` -- the build cache, not the payload -- so libsodium and the
|
||||
twelve extension `.so` files staged straight into `$MOD_PAYLOAD` would
|
||||
bypass it. No logic change needed, the two-word `DYN -> 0x70001407` rule
|
||||
already covers them; only coverage.
|
||||
1. ~~**Wire into `bin/patch.sh`.**~~ **Done.** The package build runs inside
|
||||
5c against the untrimmed stage, BEFORE the trim, which was the
|
||||
recommendation here and is still the only place all three things it needs
|
||||
exist at once: the headers `INCLUDEPY` points at, the static C libraries in
|
||||
`work/.py-dep` (pillow's zlib, cffi's libffi) and the x86-64 build-python.
|
||||
The consequence is that `work/.py313` has **two stamps** -- `.version` for
|
||||
the interpreter and `.pkg-version` for the packages -- and a package bump
|
||||
rebuilds CPython too, which is the price of not caching a second untrimmed
|
||||
tree. libsodium is its own section, 5d, cached separately in
|
||||
`work/.sodium`. Every pin is in `versions.env` with a sha256 and
|
||||
`bin/fetch-assets.sh` fetches them, so **patch.sh no longer talks to a
|
||||
network**: no `get-pip.py`, no PyPI, `pip --no-index` against hashed
|
||||
sdists. The build-python gets its pip from `--with-ensurepip=install` out
|
||||
of the same pinned CPython tarball. Two build-image packages were needed
|
||||
and are in `docker/Dockerfile.build` with the reason beside them:
|
||||
`zlib1g-dev` (a build-python without zlib cannot unpack a wheel, or pip
|
||||
itself) and `patch` (lmdb's `setup.py` shells out to `/usr/bin/patch`).
|
||||
2. ~~**Extend the ABI gate's reach.**~~ **Done**, as described: same rule,
|
||||
pointed at the staged payload instead of the build cache -- the
|
||||
interpreter, `lib/python3.13` (stdlib *and* site-packages) and libsodium's
|
||||
resolved `.so`. Not all of `$MOD_PAYLOAD`, because s6 is built by the musl
|
||||
toolchain and correctly reads `e_flags=0x1007`; it has its own gate in 5b.
|
||||
3. **Run it through `S62moonraker` under s6 on 3.13.** The entry point was
|
||||
driven directly on purpose, so the s6 readiness and restart machinery is
|
||||
still gated only against 3.8.
|
||||
|
||||
+203
-1
@@ -5,7 +5,9 @@
|
||||
# a version here and the next build picks it up; the old file stays cached.
|
||||
#
|
||||
# To bump: edit the version -- for HelixScreen that means HELIX_FILE too,
|
||||
# which embeds it -- set the matching sha256 to SKIP, then run
|
||||
# which embeds it, and for the PYPKG_* python packages the FILE is
|
||||
# the only place the version appears -- set the matching sha256 to
|
||||
# SKIP, then run
|
||||
# `make vendor`. It prints the real hash to paste back over SKIP.
|
||||
# With a stale hash left in place fetch-assets.sh exits 1 instead of
|
||||
# printing anything.
|
||||
@@ -204,3 +206,203 @@ BZIP2_SHA256="ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269"
|
||||
EXPAT_VERSION="2.6.4"
|
||||
EXPAT_TAG="R_2_6_4"
|
||||
EXPAT_SHA256="fd03b7172b3bd7427a3e7a812063f74754f24542429b634e0db6511b53fb2278"
|
||||
|
||||
# ------------------------------------------- CPython's third-party packages
|
||||
# The 18 sdists that become $MODDIR/lib/python3.13/site-packages -- what
|
||||
# Moonraker and klippy import, cross-built for mipsel by bin/patch.sh section
|
||||
# 5c against the interpreter pinned above. SHIPPED BUT NOT USED, exactly like
|
||||
# the interpreter itself: payload/anvil-env.sh still points FF_PYTHON at
|
||||
# FlashForge's 3.8.2 and the switch is a later commit.
|
||||
#
|
||||
# THE LIST IS NOT moonraker-requirements.txt, and that is a decision rather
|
||||
# than an omission. That file installs every OPTIONAL component's
|
||||
# dependencies -- apprise, ldap3, paho-mqtt, zeroconf, python-periphery. None
|
||||
# of those components is enabled in assets/moonraker.conf, so none of them is
|
||||
# built. What is here is the closure of what the ENABLED set imports
|
||||
# (moonraker.conf's sections plus server.CORE_COMPONENTS) plus klippy's, and
|
||||
# nothing else. msgspec and uvloop are absent on purpose: Moonraker guards
|
||||
# both with contextlib.suppress(ImportError) and falls back to the stdlib,
|
||||
# which was measured rather than assumed.
|
||||
#
|
||||
# Every one is an SDIST, and bin/patch.sh builds them with `--no-binary :all:`
|
||||
# for a reason that is recorded here because it cost real time to find: pip
|
||||
# will HELPFULLY download a prebuilt manylinux wheel when one exists, and on
|
||||
# the first run of the spike it did exactly that for cffi, greenlet and
|
||||
# tornado. Three x86-64 .so files went into the tree, built cleanly, imported
|
||||
# cleanly on the build host, and would have failed at import on the printer
|
||||
# with nothing but ENOEXEC to explain themselves. Only the ABI gate caught
|
||||
# them. A wheel is not a build; it is somebody else's build.
|
||||
#
|
||||
# WHERE THE URLS COME FROM. bin/fetch-assets.sh composes
|
||||
# files.pythonhosted.org/packages/source/<initial>/<project>/<file>, the
|
||||
# stable path form, rather than the hashed /packages/<a>/<b>/<64 hex>/ one a
|
||||
# browser copies out of PyPI -- that path embeds a digest of the file and
|
||||
# changes shape with no version bump, which is a URL nobody can maintain by
|
||||
# hand. So the FILE is what is pinned here: it carries the version, the way
|
||||
# HELIX_FILE does, and its SPELLING is upstream's business (MarkupSafe,
|
||||
# dbus_next and streaming_form_data are not the names their projects use).
|
||||
#
|
||||
# PYPKG_LIST is the build order and the ship list at once; bin/patch.sh
|
||||
# iterates it and bin/fetch-assets.sh fetches it, so a package added here and
|
||||
# nowhere else is fetched, built, gated and shipped. The variable names are
|
||||
# the list entry uppercased with - as _, which is how both scripts find them.
|
||||
PYPKG_LIST="distro inotify-simple libnacl dbus-next preprocess-cancellation
|
||||
pyserial pyserial-asyncio pycparser smart-open jinja2 setuptools
|
||||
tornado markupsafe cffi greenlet lmdb streaming-form-data pillow"
|
||||
|
||||
# ---- pure python. No compiler is invoked; they are here to be PRESENT. ----
|
||||
PYPKG_DISTRO_FILE="distro-1.9.0.tar.gz"
|
||||
PYPKG_DISTRO_SHA256="2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed"
|
||||
PYPKG_INOTIFY_SIMPLE_FILE="inotify_simple-1.3.5.tar.gz"
|
||||
PYPKG_INOTIFY_SIMPLE_SHA256="8440ffe49c4ae81a8df57c1ae1eb4b6bfa7acb830099bfb3e305b383005cc128"
|
||||
# libnacl is pure python and reaches libsodium through ctypes -- which is why
|
||||
# SODIUM_VERSION at the bottom of this file exists at all, and why a bump to
|
||||
# either one is a reason to re-run the ed25519 half of the moonraker gate.
|
||||
PYPKG_LIBNACL_FILE="libnacl-2.1.0.tar.gz"
|
||||
PYPKG_LIBNACL_SHA256="f3418da7df29e6d9b11fd7d990289d16397dc1020e4e35192e11aee826922860"
|
||||
PYPKG_DBUS_NEXT_FILE="dbus_next-0.2.3.tar.gz"
|
||||
PYPKG_DBUS_NEXT_SHA256="f4eae26909332ada528c0a3549dda8d4f088f9b365153952a408e28023a626a5"
|
||||
# preprocess-cancellation 0.2.1 is a poetry project whose sdist declares NO
|
||||
# [build-system] table at all and ships no setup.py, so there is no backend to
|
||||
# call and pip's legacy fallback would have to guess. It is also ONE FILE, so
|
||||
# bin/patch.sh copies preprocess_cancellation.py into site-packages rather
|
||||
# than building it -- see the comment there. (The same release also declares
|
||||
# version 0.0.0 in its own metadata, a broken upstream release, so a version
|
||||
# specifier would be refused by pip even if there were something to build.)
|
||||
PYPKG_PREPROCESS_CANCELLATION_FILE="preprocess_cancellation-0.2.1.tar.gz"
|
||||
PYPKG_PREPROCESS_CANCELLATION_SHA256="e2f1224e1ba1603bdfdbf6937caaf91082dc849e5122e80a5328aa999433ce79"
|
||||
PYPKG_PYSERIAL_FILE="pyserial-3.4.tar.gz"
|
||||
PYPKG_PYSERIAL_SHA256="6e2d401fdee0eab996cf734e67773a0143b932772ca8b42451440cfed942c627"
|
||||
PYPKG_PYSERIAL_ASYNCIO_FILE="pyserial-asyncio-0.6.tar.gz"
|
||||
PYPKG_PYSERIAL_ASYNCIO_SHA256="b6032923e05e9d75ec17a5af9a98429c46d2839adfaf80604d52e0faacd7a32f"
|
||||
PYPKG_PYCPARSER_FILE="pycparser-2.22.tar.gz" # cffi's only runtime dep
|
||||
PYPKG_PYCPARSER_SHA256="491c8be9c040f5390f5bf44a5b07752bd07f56edf992381b05c701439eec10f6"
|
||||
# smart_open is not optional, whatever the name suggests, and it is not
|
||||
# Moonraker that wants it: streaming-form-data's targets.py imports it at
|
||||
# MODULE SCOPE, so `import streaming_form_data` raises ModuleNotFoundError
|
||||
# without it. Measured in the replica, where every other package imported and
|
||||
# that one did not. moonraker-requirements.txt pins it for the same reason.
|
||||
PYPKG_SMART_OPEN_FILE="smart_open-6.4.0.tar.gz"
|
||||
PYPKG_SMART_OPEN_SHA256="be3c92c246fbe80ebce8fbacb180494a481a77fcdcb7c1aadb2ea5b9c2bee8b9"
|
||||
PYPKG_JINJA2_FILE="jinja2-3.1.4.tar.gz"
|
||||
PYPKG_JINJA2_SHA256="4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369"
|
||||
# setuptools/pkg_resources are a runtime dependency of nothing in this list.
|
||||
# They are the fix for a MEASURED failure: a tree without them made lmdb fall
|
||||
# back to its cffi path and try to invoke mips-linux-gnu-gcc ON THE PRINTER at
|
||||
# Moonraker startup, and there is no compiler on a printer. (Installing lmdb
|
||||
# as a wheel rather than as an egg removes that path as well, so this is now
|
||||
# belt and braces -- 4.9MB of it, 36% of site-packages, and the one obvious
|
||||
# thing to drop if the data partition ever gets tight.)
|
||||
#
|
||||
# It is ALSO the host-side build backend for most of this list: patch.sh
|
||||
# installs this same sdist into the throwaway x86-64 build-python. One pin,
|
||||
# both jobs -- so the setuptools that generates the shipped .so files is the
|
||||
# setuptools whose hash is checked here.
|
||||
PYPKG_SETUPTOOLS_FILE="setuptools-75.6.0.tar.gz"
|
||||
PYPKG_SETUPTOOLS_SHA256="8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6"
|
||||
|
||||
# ---- native: twelve mipsel extension modules come out of these six -------
|
||||
# tornado's C speedup is optional and its published wheel carries an x86-64
|
||||
# copy of it, which is exactly what --no-binary :all: above is for.
|
||||
PYPKG_TORNADO_FILE="tornado-6.4.tar.gz"
|
||||
PYPKG_TORNADO_SHA256="72291fa6e6bc84e626589f1c29d90a5a6d593ef5ae68052ee2ef000dfd273dee"
|
||||
# markupsafe -- jinja2's escape accelerator, on the path of every gcode macro
|
||||
# klippy renders. A pure-python fallback (_native.py) exists if it ever fights.
|
||||
PYPKG_MARKUPSAFE_FILE="MarkupSafe-2.1.5.tar.gz"
|
||||
PYPKG_MARKUPSAFE_SHA256="d283d37a890ba4c1ae73ffadf8046435c76e7bc2247bbb63c00bd1a709c6544b"
|
||||
# cffi -- klippy's chelper, and the package that justifies the whole glibc
|
||||
# decision: this is what dlopens FlashForge's own c_helper.so, measured. Wants
|
||||
# libffi, and gets the static -fPIC one patch.sh already built for _ctypes.
|
||||
PYPKG_CFFI_FILE="cffi-1.17.1.tar.gz"
|
||||
PYPKG_CFFI_SHA256="1c39c6016c32bc48dd54561950ebd6836e1670f2ae46128f67cf49e789c52824"
|
||||
# greenlet -- klippy's reactor. The MIPS stack-switching assembly was never
|
||||
# the problem; C++ designated initializers were. greenlet 3.x writes its
|
||||
# PyTypeObjects as designated initializers, which gcc 7.2 -- the only compiler
|
||||
# that produces this printer's ABI -- accepts ONLY in declaration order with
|
||||
# no gaps, and refuses anything else with "sorry, unimplemented: non-trivial
|
||||
# designated initializers not supported". patch.sh runs
|
||||
# tools/python-packages/fill-designators.py over the sources first. Bumping
|
||||
# this pin means re-running that, and it fails loudly rather than quietly: the
|
||||
# script refuses to guess when a designator is not in the target's own headers.
|
||||
PYPKG_GREENLET_FILE="greenlet-3.1.1.tar.gz"
|
||||
PYPKG_GREENLET_SHA256="4ce3ac6cdb6adf7946475d7ef31777c26d94bccc377e070a7986bd2d5c515467"
|
||||
# lmdb -- Moonraker's database AT THE PIN ABOVE, and the package that
|
||||
# disappears the moment MOONRAKER_VERSION moves past 80c7620 (lmdb -> sqlite),
|
||||
# which is the whole reason for having built an interpreter with _sqlite3.
|
||||
# patch.sh sets LMDB_FORCE_CPYTHON=1; do not "fix" that with LMDB_FORCE_CFFI=0
|
||||
# -- setup.py tests the variable for PRESENCE, so the string "0" selects cffi.
|
||||
PYPKG_LMDB_FILE="lmdb-1.4.1.tar.gz"
|
||||
PYPKG_LMDB_SHA256="1f4c76af24e907593487c904ef5eba1993beb38ed385af82adb25a858f2d658d"
|
||||
# streaming-form-data -- Moonraker's upload parser, Cython.
|
||||
#
|
||||
# NOT 1.13.0, which is what moonraker-requirements.txt pins. That release
|
||||
# ships a PREGENERATED _parser.c from a Cython that predates 3.13: it calls
|
||||
# the four-argument _PyLong_AsByteArray, which grew a fifth parameter in
|
||||
# 3.13, so every integer conversion in the file fails to compile. 1.19.1
|
||||
# ships C that is already regenerated, which is also why no Cython is pinned
|
||||
# among the host backends below. Moonraker asks five names of it
|
||||
# (StreamingFormDataParser, ParseFailedException, FileTarget, ValueTarget,
|
||||
# SHA256Target), all present throughout the 1.x line -- and this pin is why
|
||||
# smart_open above is in the list at all.
|
||||
PYPKG_STREAMING_FORM_DATA_FILE="streaming_form_data-1.19.1.tar.gz"
|
||||
PYPKG_STREAMING_FORM_DATA_SHA256="8172dd509a42ac0d1033deb1d13acb68f0cbae7e1492257d13af0922bdc4dd9b"
|
||||
# pillow -- Moonraker's ONLY use of it is components/file_manager/metadata.py,
|
||||
# which runs as a SUBPROCESS to pull thumbnails out of gcode, so it is not on
|
||||
# the startup path: without it Moonraker runs and Mainsail shows no thumbnails.
|
||||
#
|
||||
# NOT 10.3.0, which is what moonraker-requirements.txt pins, and not for any
|
||||
# reason to do with this target: 10.3.0 CANNOT BE BUILT BY ANY 3.13, cross or
|
||||
# native. Its setup.py reads its own version with exec(...) followed by
|
||||
# locals()["__version__"], and PEP 667 (new in 3.13) made a function's
|
||||
# locals() a snapshot -- so the name exec() defined is not there any more and
|
||||
# setup.py dies with KeyError: '__version__' before it compiles a single file.
|
||||
# 11.0.0 is the first line that supports 3.13. Moonraker asks PIL for
|
||||
# Image.open, resize and save, all unchanged across that gap.
|
||||
PYPKG_PILLOW_FILE="pillow-11.0.0.tar.gz"
|
||||
PYPKG_PILLOW_SHA256="72bacbaf24ac003fea9bff9837d1eedb6088758d41e100c1552930151f677739"
|
||||
|
||||
# ---- host-side build backends. THESE NEVER SHIP. -------------------------
|
||||
# Two of the sdists above name a PEP 517 backend that is not setuptools, and
|
||||
# bin/patch.sh builds everything with --no-build-isolation -- which is not a
|
||||
# preference either: with isolation ON, pip rewrites PYTHONPATH for the build
|
||||
# subprocess, the cross _sysconfigdata module stops being importable, and the
|
||||
# build either dies outright ("No module named
|
||||
# '_sysconfigdata__linux_mipsel-linux-gnu'") or, far worse, silently answers
|
||||
# every sysconfig question for x86-64 and produces host objects. So every
|
||||
# backend has to be present in advance, and every one is pinned by hash for
|
||||
# the same reason the compilers are: this is code that runs on the build
|
||||
# machine and decides what the printer executes.
|
||||
#
|
||||
# Both bootstrap themselves (their sdists declare backend-path), so no third
|
||||
# package is needed to install them, and neither is on PYPKG_LIST -- they go
|
||||
# into the throwaway x86-64 build-python and are thrown away with it.
|
||||
# setuptools, the third backend, is on the ship list above.
|
||||
PYPKG_HOST_LIST="flit-core poetry-core"
|
||||
PYPKG_FLIT_CORE_FILE="flit_core-3.10.1.tar.gz" # jinja2's backend
|
||||
PYPKG_FLIT_CORE_SHA256="66e5b87874a0d6e39691f0e22f09306736b633548670ad3c09ec9db03c5662f7"
|
||||
PYPKG_POETRY_CORE_FILE="poetry_core-1.9.1.tar.gz" # libnacl's backend
|
||||
PYPKG_POETRY_CORE_SHA256="7a2d49214bf58b4f17f99d6891d947a9836c9899a67a5069f52d7b67217f61b8"
|
||||
|
||||
# --------------------------------------------------------------- libsodium
|
||||
# The LAST /usr/prog string in the picture. libnacl -- and with it Moonraker's
|
||||
# `authorization` component, which signs its JWTs with ed25519 -- is pure
|
||||
# python and loads libsodium through ctypes.cdll.LoadLibrary, i.e. dlopen. So
|
||||
# unlike the interpreter's seven static dependencies this one MUST be shared,
|
||||
# and it is the only library of ours that lands in $MODDIR/lib as a .so.
|
||||
#
|
||||
# It costs anvil-env.sh nothing, which is the pleasant part and worth writing
|
||||
# down because it is an accident: libnacl's third fallback is
|
||||
# __file__[0:__file__.find("lib")+3] + "/libsodium.so", and __file__ is
|
||||
# $MODDIR/lib/python3.13/site-packages/libnacl/__init__.py -- so it resolves
|
||||
# $MODDIR/lib/libsodium.so BY ABSOLUTE PATH, measured working with
|
||||
# LD_LIBRARY_PATH unset entirely. That works because the prefix happens to
|
||||
# contain the string "lib"; a prefix without it would break silently. Adding
|
||||
# $MODDIR/lib to ANVIL_LIBS also works and is the belt-and-braces option, but
|
||||
# it would put our libraries in front of everything for every mod process,
|
||||
# which is the kind of entry this file's own history argues against.
|
||||
#
|
||||
# 1.0.20 is the current stable release and there is nothing exotic about the
|
||||
# pin. It cross-builds in 24 seconds and is cached on this version string in
|
||||
# work/.sodium, the same shape as work/.s6 and work/.py313.
|
||||
SODIUM_VERSION="1.0.20"
|
||||
SODIUM_SHA256="ebb65ef6ca439333c2bb41a0c1990587288da07f6c7fd07cb3a18cc18d30ce19"
|
||||
Reference in new issue
Block a user