From d898d105e0b314088cc314695b58fb2ab3254dde Mon Sep 17 00:00:00 2001 From: Oleksandr Shyshatskyi Date: Sun, 30 Aug 2026 15:07:35 +0300 Subject: [PATCH] The Moonraker pin was frozen by a missing _sqlite3 we now build ourselves MOONRAKER_VERSION was 9d0d09d, the last commit before 80c7620 moved the database from lmdb to sqlite, because FlashForge's python 3.8.2 has no _sqlite3 and every release after it died loading the database component. This repo builds and ships its own CPython 3.13 now, and it has one. The pin is v0.11.0 -- a release tag, not a sha, so PKG_VERSION is just 0.11.0 and no longer the Debian-style 0.8.0+git snapshot string. GitHub strips the leading "v" from the directory inside /archive/.tar.gz, which is what ${MOONRAKER_VERSION#v} in the recipe is for. The dependency set is the part that actually moved. Read against 0.11.0's CORE_COMPONENTS plus the sections moonraker.conf enables: dbus-next became dbus-fast. dbus_manager is a core component and imports it at module scope, so this is not optional whatever [machine] provider says. Its sdist cythonises nine modules; SKIP_CYTHON=1 ships the python they were generated from and keeps the package ARCH all. The wheel still comes out platform-tagged -- poetry-core marks the distribution impure for declaring a build script at all -- but it holds 29 .py and no .so, and the content is what ships. importlib-metadata is new, imported at module scope by utils/source_info.py, and brings zipp. Both build through setuptools_scm, so PYPKG_HOST_LIST grew packaging and setuptools-scm. That list is now install order: pkg_buildpython pip-installs each with --no-deps against no index, flit-core builds packaging, setuptools_scm imports it. pyserial-asyncio is gone. Nothing in 0.11.0 imports serial_asyncio; utils/async_serial.py drives pyserial over asyncio itself. pyserial stays -- klippy needs it and declares it. LMDB STAYS, for a reason that is not the old one. 0.11.0 converts an existing data.mdb to sqlite on first start and imports lmdb to read it. With the module missing it tries to pip install one, and a printer has no index to install from -- the old database would be dropped in silence. It is the upgrade path now, not the database, and the comment on WANTED_MODULES says so. inotify-simple STAYS AT 1.3.5 though 0.11.0 pins ==2.0.1: 2.0.1 declares its licence the PEP 639 way and needs setuptools>=77 to build, against the 75.6.0 pinned here for lmdb's sake. It fails at metadata generation, not at import, which is the kind of thing that reads as a broken recipe later. 1.3.5 carries every name 0.11.0 asks of it, and moonraker.conf sets file_system_observer: none, so the import is all that is asked. jinja2 goes to 3.1.6 to match. Not taken: paho-mqtt, zeroconf, apprise, ldap3 and python-periphery, whose components this printer does not enable, and msgspec and uvloop, which Moonraker imports under contextlib.suppress(ImportError). Verified: every changed sha256 downloads and matches at the URL fetch-assets.sh builds; all five affected wheels build offline (--no-index --no-build-isolation) on a real 3.13 with the host backends installed in the declared order; qa/static 122 passed, including the gates that cross-check PYPKG_LIST against the recipes and every PKG_DEPENDS against the feed. The mipsel cross-build and `make qa-replica` have not run yet, and versions.env still says a bump is gated on the latter. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012oRdgKM7scrEsr6waAByaY --- bin/common.sh | 2 +- docs/how-it-works.md | 35 ++++++---- docs/versions.md | 2 +- pkgs/3rdparty/python-dbus-fast/build.sh | 20 ++++++ pkgs/3rdparty/python-dbus-fast/pkg.conf | 34 +++++++++ pkgs/3rdparty/python-dbus-next/build.sh | 15 ---- pkgs/3rdparty/python-dbus-next/pkg.conf | 22 ------ .../python-importlib-metadata/build.sh | 20 ++++++ .../python-importlib-metadata/pkg.conf | 26 +++++++ .../3rdparty/python-pyserial-asyncio/build.sh | 15 ---- .../3rdparty/python-pyserial-asyncio/pkg.conf | 22 ------ pkgs/3rdparty/python-zipp/build.sh | 18 +++++ pkgs/3rdparty/python-zipp/pkg.conf | 23 +++++++ pkgs/moonraker/build.sh | 6 +- pkgs/moonraker/pkg.conf | 31 +++++---- qa/replica/test_what_ships.py | 5 +- tools/python-packages/README.md | 5 ++ versions.env | 69 ++++++++++++++----- 18 files changed, 249 insertions(+), 121 deletions(-) create mode 100755 pkgs/3rdparty/python-dbus-fast/build.sh create mode 100644 pkgs/3rdparty/python-dbus-fast/pkg.conf delete mode 100755 pkgs/3rdparty/python-dbus-next/build.sh delete mode 100644 pkgs/3rdparty/python-dbus-next/pkg.conf create mode 100755 pkgs/3rdparty/python-importlib-metadata/build.sh create mode 100644 pkgs/3rdparty/python-importlib-metadata/pkg.conf delete mode 100755 pkgs/3rdparty/python-pyserial-asyncio/build.sh delete mode 100644 pkgs/3rdparty/python-pyserial-asyncio/pkg.conf create mode 100755 pkgs/3rdparty/python-zipp/build.sh create mode 100644 pkgs/3rdparty/python-zipp/pkg.conf diff --git a/bin/common.sh b/bin/common.sh index dfdf65c..8674e68 100644 --- a/bin/common.sh +++ b/bin/common.sh @@ -83,7 +83,7 @@ pypkg_var() { # The sdist as bin/fetch-assets.sh leaves it in vendor/. pypkg_tgz() { printf '%s/vendor/%s' "$ROOT" "$(pypkg_var "$1" FILE)"; } # Version taken from the pinned file name, after the LAST dash -- which is what -# makes inotify_simple-1.3.5 and pyserial-asyncio-0.6 both come out right. +# makes importlib_metadata-8.7.0 and dbus_fast-5.0.22 both come out right. pypkg_version() { local _f _f="$(pypkg_var "$1" FILE)" diff --git a/docs/how-it-works.md b/docs/how-it-works.md index ad50508..7f410bc 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -136,7 +136,7 @@ never invoked — the moonraker service starts the server itself. **FlashForge's python 3.8.2 is not what this runs on any more.** `FF_PYTHON` in `anvil-env.sh` names a CPython 3.13 of our own, cross-built for mipsel by `pkgs/3rdparty/python`, with every third-party C extension Moonraker needs beside it in -`$MODDIR/lib/python3.13/site-packages` — eighteen packages, one `pkgs/3rdparty/python-*` +`$MODDIR/lib/python3.13/site-packages` — nineteen packages, one `pkgs/3rdparty/python-*` recipe and one `.ipk` each — and libsodium in `$MODDIR/lib`; none of it borrowed from `/usr/prog`. Measured through the real boot path (the scandir, the moonraker service, readiness gating on `:7125` @@ -144,19 +144,26 @@ actually listening, a `kill -9` respawn, a stop that stays stopped). klippy is o too: the `klipper` service execs `$FF_PYTHON` against the klippy tree in `$MODDIR/klipper`, so both halves are ours. -**The Moonraker commit pin is still a 2023 one, though the reason it exists -has changed.** FlashForge built python 3.8.2 without the `_sqlite3` module, -and Moonraker moved its database from lmdb to sqlite in v0.9.0, so on 3.8.2 -every release from there on got as far as loading the database component and -died with `ModuleNotFoundError: No module named '_sqlite3'`. Our 3.13 has a -working `_sqlite3` (measured: create, insert, select, reopen -- and exercised -for real every run, since Moonraker itself comes up on it in -`qa/replica/test_web.py`), which is what unpins this -- but the pin has not moved yet: nobody -has taken a newer Moonraker through this build and its replica gates, and a -version bump is exactly the kind of change that wants its own measurement, -not a side effect of an interpreter switch. Until it does, the printer's lmdb -store is used in place -- nothing is converted, and reverting to stock is a -clean round trip. +**Moonraker is a release now, `v0.11.0`, and no longer a 2023 commit.** +FlashForge built python 3.8.2 without the `_sqlite3` module, and Moonraker +moved its database from lmdb to sqlite in v0.9.0, so on 3.8.2 every release +from there on got as far as loading the database component and died with +`ModuleNotFoundError: No module named '_sqlite3'`. Our 3.13 has a working +`_sqlite3` (measured: create, insert, select, reopen -- and exercised for real +every run, since Moonraker itself comes up on it in `qa/replica/test_web.py`), +which is what unpinned it. + +Upgrading a printer that has run an older release **converts** its database: +Moonraker reads the existing `data.mdb` once and writes an sqlite store beside +it, which is why `anvil-python-lmdb` still ships even though nothing imports +lmdb at runtime any more. Without that module Moonraker tries to `pip install` +one, and a printer has no index to install from -- the old database would be +dropped in silence. Reverting to stock is still a clean round trip: the lmdb +store is read, not deleted. + +The dependency set moved with it. `dbus-next` became `dbus-fast`, +`importlib-metadata` (and its `zipp`) is new, and `pyserial-asyncio` is gone, +imported by nothing in 0.11.0. This was not worked out in advance; v0.9.3 was built, shipped and tried on the printer first, and that is what it said. What came out of it: when bumping the diff --git a/docs/versions.md b/docs/versions.md index 9c2b299..abe6ba9 100644 --- a/docs/versions.md +++ b/docs/versions.md @@ -9,7 +9,7 @@ whatever was current when it was built. | Stock FlashForge base | `1.9.7-1.2.9-20260810`, fetched from [ghzserg/FF](https://github.com/ghzserg/FF) at build time | | [Klipper](https://github.com/Klipper4FlashForge/klipper/tree/creator5) | a current fork (branch `creator5`) with the `ff_*` toolchanger extras | | [Mainsail](https://github.com/mainsail-crew/mainsail) | [`v2.18.2`](https://github.com/mainsail-crew/mainsail/releases/tag/v2.18.2) | -| [Moonraker](https://github.com/Arksine/moonraker) | commit [`9d0d09d`](https://github.com/Arksine/moonraker/commit/9d0d09de8063922696359c1b88c86a86d6fdb296) — a commit, not a release, for a hard reason: [how-it-works.md](how-it-works.md#moonraker) | +| [Moonraker](https://github.com/Arksine/moonraker) | [`v0.11.0`](https://github.com/Arksine/moonraker/releases/tag/v0.11.0) | | [HelixScreen](https://github.com/Klipper4FlashForge/helixscreen) | [`v0.99.115-creator5.1`](https://github.com/Klipper4FlashForge/helixscreen/releases/tag/v0.99.115-creator5.1) | How the pieces fit together on the printer — which of them is a service, what diff --git a/pkgs/3rdparty/python-dbus-fast/build.sh b/pkgs/3rdparty/python-dbus-fast/build.sh new file mode 100755 index 0000000..42e254f --- /dev/null +++ b/pkgs/3rdparty/python-dbus-fast/build.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# dbus_fast -- built for the printer's CPython, into its site-packages. +# +# SKIP_CYTHON=1 is read by the sdist's build_ext.py, which returns before it +# imports Cython. That is the supported way to ask for the pure-python build, +# and it is also the only one available here: pkg_pywheel builds with +# --no-build-isolation, so Cython is not in the build-python to import. +set -euo pipefail +. ./bin/common.sh +. pkgs/lib.sh + +pkg_begin python-dbus-fast || exit 0 +pkg_toolchain +pkg_deps +pkg_buildpython +pkg_pytarget +pkg_unpack "$(pypkg_tgz dbus-fast)" +pkg_pywheel dbus-fast SKIP_CYTHON=1 +pkg_ship "lib/python$PY_MM/site-packages" +pkg_end diff --git a/pkgs/3rdparty/python-dbus-fast/pkg.conf b/pkgs/3rdparty/python-dbus-fast/pkg.conf new file mode 100644 index 0000000..29e20d5 --- /dev/null +++ b/pkgs/3rdparty/python-dbus-fast/pkg.conf @@ -0,0 +1,34 @@ +# anvil-python-dbus-fast -- for the CPython this feed builds. +# +# Moonraker's dbus_manager component, which is a CORE component and imports +# this at module scope: without it the server does not start, whatever +# [machine] provider is configured. Replaces anvil-python-dbus-next, which +# Moonraker 0.11.0 stopped using. +# +# ARCHITECTURE all. The sdist cythonises nine of its modules for speed, and +# build.sh sets SKIP_CYTHON=1 so it ships the python those were generated +# from -- the same code, running anywhere. Compiling them would make this +# package mipsel and buy speed on a path that carries a handful of polkit +# calls per boot. +# +# ARCH all EVEN THOUGH THE WHEEL IS PLATFORM-TAGGED. poetry-core marks the +# distribution impure because pyproject declares a build script at all, so +# SKIP_CYTHON=1 still produces a cp313-...-x86_64 file name. What that file +# CONTAINS is 29 .py and not one .so, and the content is what ships -- which +# is also why this recipe does not call pkg_pynative. +# +# THE PIN IS IN versions.env, as PYPKG_DBUS_FAST_FILE, beside its sha256 and +# beside the others. PKG_VERSION is READ OUT OF that file name rather than +# written again here, so there is one string to bump and no way for the two +# to disagree. +PKG_NAME=anvil-python-dbus-fast +PKG_VERSION="$(pypkg_version dbus-fast)" +PKG_RELEASE=1 +PKG_SECTION=python +PKG_ARCH=all + +PKG_EXCLUDE=".version" +PKG_DEPENDS="anvil-python" +PKG_BUILD_DEPENDS="python" + +PKG_DESCRIPTION="dbus_fast -- a DBus library, shipped as its pure-python build. Moonraker's dbus_manager uses it." diff --git a/pkgs/3rdparty/python-dbus-next/build.sh b/pkgs/3rdparty/python-dbus-next/build.sh deleted file mode 100755 index e90d353..0000000 --- a/pkgs/3rdparty/python-dbus-next/build.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/env bash -# dbus_next -- built for the printer's CPython, into its site-packages. -set -euo pipefail -. ./bin/common.sh -. pkgs/lib.sh - -pkg_begin python-dbus-next || exit 0 -pkg_toolchain -pkg_deps -pkg_buildpython -pkg_pytarget -pkg_unpack "$(pypkg_tgz dbus-next)" -pkg_pywheel dbus-next -pkg_ship "lib/python$PY_MM/site-packages" -pkg_end diff --git a/pkgs/3rdparty/python-dbus-next/pkg.conf b/pkgs/3rdparty/python-dbus-next/pkg.conf deleted file mode 100644 index 169238b..0000000 --- a/pkgs/3rdparty/python-dbus-next/pkg.conf +++ /dev/null @@ -1,22 +0,0 @@ -# anvil-python-dbus-next -- for the CPython this feed builds. -# -# Moonraker's dbus_manager component. -# -# ARCHITECTURE all. Nothing in this package is compiled; it is python source -# and it runs on any interpreter that can import it. -# -# THE PIN IS IN versions.env, as PYPKG_DBUS_NEXT_FILE, beside its sha256 and -# beside the other seventeen. PKG_VERSION is READ OUT OF that file name rather -# than written again here, so there is one string to bump and no way for the -# two to disagree. -PKG_NAME=anvil-python-dbus-next -PKG_VERSION="$(pypkg_version dbus-next)" -PKG_RELEASE=1 -PKG_SECTION=python -PKG_ARCH=all - -PKG_EXCLUDE=".version" -PKG_DEPENDS="anvil-python" -PKG_BUILD_DEPENDS="python" - -PKG_DESCRIPTION="dbus_next -- a pure-python DBus library. Moonraker's dbus_manager uses it." diff --git a/pkgs/3rdparty/python-importlib-metadata/build.sh b/pkgs/3rdparty/python-importlib-metadata/build.sh new file mode 100755 index 0000000..c55fde6 --- /dev/null +++ b/pkgs/3rdparty/python-importlib-metadata/build.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# importlib_metadata -- built for the printer's CPython, into its +# site-packages. +# +# Its backend is setuptools_scm, which normally reads the version out of a git +# checkout. There is none here, and there does not need to be: an sdist +# carries PKG-INFO and setuptools_scm takes the version from it. +set -euo pipefail +. ./bin/common.sh +. pkgs/lib.sh + +pkg_begin python-importlib-metadata || exit 0 +pkg_toolchain +pkg_deps +pkg_buildpython +pkg_pytarget +pkg_unpack "$(pypkg_tgz importlib-metadata)" +pkg_pywheel importlib-metadata +pkg_ship "lib/python$PY_MM/site-packages" +pkg_end diff --git a/pkgs/3rdparty/python-importlib-metadata/pkg.conf b/pkgs/3rdparty/python-importlib-metadata/pkg.conf new file mode 100644 index 0000000..360f5f1 --- /dev/null +++ b/pkgs/3rdparty/python-importlib-metadata/pkg.conf @@ -0,0 +1,26 @@ +# anvil-python-importlib-metadata -- for the CPython this feed builds. +# +# New with Moonraker 0.11.0: moonraker/utils/source_info.py imports it at +# module scope. The BACKPORT and not the 3.13 stdlib importlib.metadata, +# because that is the name upstream writes, and a package whose imports we +# rewrite is a package we maintain. +# +# ARCHITECTURE all. Nothing in this package is compiled; it is python source +# and it runs on any interpreter that can import it. +# +# THE PIN IS IN versions.env, as PYPKG_IMPORTLIB_METADATA_FILE, beside its +# sha256 and beside the others. PKG_VERSION is READ OUT OF that file name +# rather than written again here, so there is one string to bump and no way +# for the two to disagree. +PKG_NAME=anvil-python-importlib-metadata +PKG_VERSION="$(pypkg_version importlib-metadata)" +PKG_RELEASE=1 +PKG_SECTION=python +PKG_ARCH=all + +PKG_EXCLUDE=".version" +# zipp is its one runtime dependency, and it is imported at module scope. +PKG_DEPENDS="anvil-python, anvil-python-zipp" +PKG_BUILD_DEPENDS="python" + +PKG_DESCRIPTION="importlib_metadata -- the backport of the stdlib module. Moonraker reads its own version through it." diff --git a/pkgs/3rdparty/python-pyserial-asyncio/build.sh b/pkgs/3rdparty/python-pyserial-asyncio/build.sh deleted file mode 100755 index 83ec775..0000000 --- a/pkgs/3rdparty/python-pyserial-asyncio/build.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/env bash -# pyserial-asyncio -- built for the printer's CPython, into its site-packages. -set -euo pipefail -. ./bin/common.sh -. pkgs/lib.sh - -pkg_begin python-pyserial-asyncio || exit 0 -pkg_toolchain -pkg_deps -pkg_buildpython -pkg_pytarget -pkg_unpack "$(pypkg_tgz pyserial-asyncio)" -pkg_pywheel pyserial-asyncio -pkg_ship "lib/python$PY_MM/site-packages" -pkg_end diff --git a/pkgs/3rdparty/python-pyserial-asyncio/pkg.conf b/pkgs/3rdparty/python-pyserial-asyncio/pkg.conf deleted file mode 100644 index 11a3ec7..0000000 --- a/pkgs/3rdparty/python-pyserial-asyncio/pkg.conf +++ /dev/null @@ -1,22 +0,0 @@ -# anvil-python-pyserial-asyncio -- for the CPython this feed builds. -# -# It is a transport layer over pyserial and imports it at module scope. -# -# ARCHITECTURE all. Nothing in this package is compiled; it is python source -# and it runs on any interpreter that can import it. -# -# THE PIN IS IN versions.env, as PYPKG_PYSERIAL_ASYNCIO_FILE, beside its sha256 and -# beside the other seventeen. PKG_VERSION is READ OUT OF that file name rather -# than written again here, so there is one string to bump and no way for the -# two to disagree. -PKG_NAME=anvil-python-pyserial-asyncio -PKG_VERSION="$(pypkg_version pyserial-asyncio)" -PKG_RELEASE=1 -PKG_SECTION=python -PKG_ARCH=all - -PKG_EXCLUDE=".version" -PKG_DEPENDS="anvil-python, anvil-python-pyserial" -PKG_BUILD_DEPENDS="python" - -PKG_DESCRIPTION="pyserial-asyncio -- asyncio transport over pyserial." diff --git a/pkgs/3rdparty/python-zipp/build.sh b/pkgs/3rdparty/python-zipp/build.sh new file mode 100755 index 0000000..b6d232d --- /dev/null +++ b/pkgs/3rdparty/python-zipp/build.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# zipp -- built for the printer's CPython, into its site-packages. +# +# setuptools_scm is its backend; see python-importlib-metadata/build.sh for +# why that needs no git checkout. +set -euo pipefail +. ./bin/common.sh +. pkgs/lib.sh + +pkg_begin python-zipp || exit 0 +pkg_toolchain +pkg_deps +pkg_buildpython +pkg_pytarget +pkg_unpack "$(pypkg_tgz zipp)" +pkg_pywheel zipp +pkg_ship "lib/python$PY_MM/site-packages" +pkg_end diff --git a/pkgs/3rdparty/python-zipp/pkg.conf b/pkgs/3rdparty/python-zipp/pkg.conf new file mode 100644 index 0000000..f40b67b --- /dev/null +++ b/pkgs/3rdparty/python-zipp/pkg.conf @@ -0,0 +1,23 @@ +# anvil-python-zipp -- for the CPython this feed builds. +# +# Here for one reason: importlib_metadata imports it. Nothing else in the feed +# does, and if that dependency ever goes away this package goes with it. +# +# ARCHITECTURE all. Nothing in this package is compiled; it is python source +# and it runs on any interpreter that can import it. +# +# THE PIN IS IN versions.env, as PYPKG_ZIPP_FILE, beside its sha256 and beside +# the others. PKG_VERSION is READ OUT OF that file name rather than written +# again here, so there is one string to bump and no way for the two to +# disagree. +PKG_NAME=anvil-python-zipp +PKG_VERSION="$(pypkg_version zipp)" +PKG_RELEASE=1 +PKG_SECTION=python +PKG_ARCH=all + +PKG_EXCLUDE=".version" +PKG_DEPENDS="anvil-python" +PKG_BUILD_DEPENDS="python" + +PKG_DESCRIPTION="zipp -- pathlib-compatible zipfile.Path. importlib_metadata's one runtime dependency." diff --git a/pkgs/moonraker/build.sh b/pkgs/moonraker/build.sh index b78a852..a892f52 100755 --- a/pkgs/moonraker/build.sh +++ b/pkgs/moonraker/build.sh @@ -1,8 +1,10 @@ #!/usr/bin/env bash # Moonraker -- stage the pinned source tree into the prefix. # -# GitHub's generated tarball wraps everything in moonraker-/, and what we +# GitHub's generated tarball wraps everything in moonraker-/, and what we # want is the moonraker/ package directory inside it, not the repository root. +# The pin is a tag and GitHub strips its leading "v" from that directory name, +# which ${MOONRAKER_VERSION#v} undoes. # The unpack keeps the wrapper and the stage reaches through it, rather than # --strip-components hiding the archive's shape from whoever reads this. # @@ -15,7 +17,7 @@ set -euo pipefail pkg_begin moonraker || exit 0 pkg_unpack "$MOONRAKER_TGZ" -_src="$PKG_WORK/src/moonraker-$MOONRAKER_VERSION/moonraker" +_src="$PKG_WORK/src/moonraker-${MOONRAKER_VERSION#v}/moonraker" # The guard bin/payload.sh has always had, kept: a tarball whose shape changed # under us would otherwise stage nothing, which looks like a clean build and a # dead web UI. diff --git a/pkgs/moonraker/pkg.conf b/pkgs/moonraker/pkg.conf index 1db24ec..fa0b93b 100644 --- a/pkgs/moonraker/pkg.conf +++ b/pkgs/moonraker/pkg.conf @@ -7,19 +7,18 @@ PKG_NAME=anvil-moonraker PKG_ARCH=all -# THE PIN IS A COMMIT, NOT A RELEASE, and versions.env explains why: no -# release has both lmdb and the webcam "enabled" flag Mainsail filters on. A -# 40-character sha is not a package version anybody can compare, so the field -# is composed the way Debian composes one for a snapshot -- the last release -# this descends from, then the commit -- which opkg orders after 0.8.0 and -# before 0.9.0. -PKG_VERSION="0.8.0+git$(printf '%.8s' "$MOONRAKER_VERSION")" +# THE PIN IS A RELEASE TAG now, so the version is just the release. It used to +# be a commit -- 0.8.0+git -- because no release had both lmdb and the +# webcam "enabled" flag Mainsail filters on, and this repo could not ship the +# sqlite the releases after it wanted. It builds its own CPython now, so it +# can. opkg orders 0.11.0 after every 0.8.0+git string that shipped. +PKG_VERSION="${MOONRAKER_VERSION#v}" PKG_RELEASE=1 PKG_SECTION=net PKG_EXCLUDE=".version" -# Moonraker runs on the CPython 3.13 this repo builds and imports thirteen +# Moonraker runs on the CPython 3.13 this repo builds and imports twelve # third-party packages, each its own .ipk. # # THE LIST IS moonraker-requirements.txt MINUS WHAT MOONRAKER DOES NOT IMPORT. @@ -28,12 +27,20 @@ PKG_EXCLUDE=".version" # anvil-python-preprocess-cancellation, imported lazily in that same # subprocess. Both are Recommends in spirit, and opkg has no such field. # anvil-libsodium is not here either: libnacl declares it, which is where a -# ctypes dependency belongs. +# ctypes dependency belongs. Neither are the requirements 0.11.0 grew for +# components this printer does not enable -- paho-mqtt, zeroconf, apprise, +# ldap3, python-periphery -- nor its msgspec and uvloop speedups, both of +# which Moonraker imports under contextlib.suppress(ImportError). +# +# WHAT 0.11.0 CHANGED HERE: dbus-next became dbus-fast, importlib-metadata +# (and its zipp) is new, and pyserial-asyncio is gone -- nothing imports it +# any more. anvil-python-lmdb STAYS, and versions.env says why: it is what +# reads the old database during the one-time conversion to sqlite. PKG_DEPENDS="anvil-python, anvil-python-tornado, anvil-python-lmdb, anvil-python-streaming-form-data, anvil-python-distro, anvil-python-inotify-simple, anvil-python-libnacl, - anvil-python-dbus-next, anvil-python-jinja2, - anvil-python-pyserial, anvil-python-pyserial-asyncio, + anvil-python-dbus-fast, anvil-python-jinja2, + anvil-python-pyserial, anvil-python-importlib-metadata, anvil-python-smart-open, anvil-python-setuptools" PKG_BUILD_DEPENDS="" @@ -44,4 +51,4 @@ PKG_STAMP_EXTRA="$(pkg_payload_hash)" PKG_WHEN='[ "${BUILD_MOONRAKER:-1}" = "1" ]' -PKG_DESCRIPTION="Moonraker, the Klipper API server, pinned to the newest commit that still uses lmdb. Runs on the interpreter this mod ships, from /usr/data/anvil/moonraker." +PKG_DESCRIPTION="Moonraker, the Klipper API server. Runs on the interpreter this mod ships, from /usr/data/anvil/moonraker." diff --git a/qa/replica/test_what_ships.py b/qa/replica/test_what_ships.py index 6a8c9e2..ff1f127 100644 --- a/qa/replica/test_what_ships.py +++ b/qa/replica/test_what_ships.py @@ -177,7 +177,10 @@ def _host_files(): # configure's link probe fail, and CPython then records the # module as missing and builds a perfectly working # interpreter without it. -# lmdb Moonraker's database at the pinned commit. +# lmdb NOT Moonraker's database any more -- 0.11.0 keeps it in +# sqlite. It is what reads the OLD database during the +# one-time conversion on a printer upgrading from a release +# that used it, so it still has to import. # _cffi_backend klippy's route to c_helper.so, and the reason this # interpreter had to be glibc rather than musl. # diff --git a/tools/python-packages/README.md b/tools/python-packages/README.md index eb23d78..b1cca28 100644 --- a/tools/python-packages/README.md +++ b/tools/python-packages/README.md @@ -30,6 +30,11 @@ speedups, pillow. Pure: jinja2, distro, inotify-simple, libnacl, dbus-next, preprocess-cancellation, pyserial, pyserial-asyncio, pycparser, smart_open, setuptools. +That is the list as this harness measured it. The feed's has since moved with +Moonraker's: `v0.11.0` swapped dbus-next for dbus-fast, added +importlib-metadata and zipp, and dropped pyserial-asyncio. `versions.env` is +the list that matters. + **The list is deliberately not `moonraker-requirements.txt`**, which installs every optional component's dependencies. It is the closure of what the *enabled* component set imports -- `moonraker.conf`'s sections plus diff --git a/versions.env b/versions.env index 103ac42..5bd2b7b 100644 --- a/versions.env +++ b/versions.env @@ -45,11 +45,23 @@ HELIX_VERSION="v0.99.115-creator5.2" HELIX_FILE="helixscreen-creator5-v0.99.115.tar.gz" HELIX_SHA256="dd66dd9b1bdd73f77d7c7f7d76003b14121c5cf102a755083ef7506fbdb22d7d" -# Moonraker: newest commit with the webcam "enabled" flag that still uses lmdb -# -- the one after it, 80c7620, migrates the database to sqlite. Nothing on the -# printer falls back to a stock build, so gate a bump on `make qa-replica`. -MOONRAKER_VERSION="9d0d09de8063922696359c1b88c86a86d6fdb296" -MOONRAKER_SHA256="6ddcba08a1a41f173fc6ee393a24e41f27b25f079e7c49921b87273ecff595a8" +# Moonraker: a RELEASE TAG, not a commit. The old pin was the last commit +# before 80c7620 moved the database from lmdb to sqlite, because 3.8.2 has no +# _sqlite3 -- and this repo now builds and ships its own CPython, which does. +# +# /archive/ takes the tag, but GitHub strips the leading "v" from the +# directory inside the tarball, which is why pkgs/moonraker/build.sh reaches +# for moonraker-${MOONRAKER_VERSION#v}. +# +# LMDB STILL SHIPS. Moonraker converts an existing data.mdb to sqlite on first +# start and imports lmdb to read it; with the module missing it tries to pip +# install one, which on a printer with no index means the old database is +# silently dropped. It is the upgrade path, not a runtime dependency. +# +# Nothing on the printer falls back to a stock build, so gate a bump on +# `make qa-replica`. +MOONRAKER_VERSION="v0.11.0" +MOONRAKER_SHA256="bba9042e9fc904b30cd46690320fec2bffe27c4458b539db65560582c6318ab3" # s6: a matched set that will not build across generations, so all four bump # together. Three load-bearing build flags -- tools/supervisor/README.md. @@ -94,37 +106,55 @@ EXPAT_SHA256="fd03b7172b3bd7427a3e7a812063f74754f24542429b634e0db6511b53fb2278" # site-packages: the closure of what the ENABLED Moonraker components and # klippy import, not moonraker-requirements.txt. Sdists only -- a manylinux # wheel is somebody else's x86-64 build. The FILE carries the version. -PYPKG_LIST="distro inotify-simple libnacl dbus-next preprocess-cancellation - pyserial pyserial-asyncio pycparser smart-open jinja2 setuptools - tornado markupsafe cffi greenlet lmdb streaming-form-data pillow" +PYPKG_LIST="distro inotify-simple libnacl dbus-fast preprocess-cancellation + pyserial pycparser smart-open jinja2 setuptools zipp + importlib-metadata tornado markupsafe cffi greenlet lmdb + streaming-form-data pillow" PYPKG_DISTRO_FILE="distro-1.9.0.tar.gz" PYPKG_DISTRO_SHA256="2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed" +# STAYS AT 1.3.5 though Moonraker 0.11.0 pins ==2.0.1: 2.0.1 declares its +# licence the PEP 639 way and needs setuptools>=77 to build, and the +# setuptools below is 75.6.0 and pinned for its own reasons. 1.3.5 carries +# every name 0.11.0 imports -- INotify(nonblocking=), fileno, add_watch, +# rm_watch, read(timeout=), Event, and each flag in WATCH_FLAGS. Nothing on +# this printer even reaches them: moonraker.conf sets +# `file_system_observer: none`, so the import is all that is asked of it. PYPKG_INOTIFY_SIMPLE_FILE="inotify_simple-1.3.5.tar.gz" PYPKG_INOTIFY_SIMPLE_SHA256="8440ffe49c4ae81a8df57c1ae1eb4b6bfa7acb830099bfb3e305b383005cc128" # reaches libsodium through ctypes, so it is tied to SODIUM_VERSION below PYPKG_LIBNACL_FILE="libnacl-2.1.0.tar.gz" PYPKG_LIBNACL_SHA256="f3418da7df29e6d9b11fd7d990289d16397dc1020e4e35192e11aee826922860" -PYPKG_DBUS_NEXT_FILE="dbus_next-0.2.3.tar.gz" -PYPKG_DBUS_NEXT_SHA256="f4eae26909332ada528c0a3549dda8d4f088f9b365153952a408e28023a626a5" +# replaced dbus-next in Moonraker 0.11.0. The recipe sets SKIP_CYTHON=1: the +# sdist would otherwise cythonise nine of its modules for speed, and the +# python those were generated from ships instead, which keeps this ARCH all +PYPKG_DBUS_FAST_FILE="dbus_fast-5.0.22.tar.gz" +PYPKG_DBUS_FAST_SHA256="34dc67d7d21a12399828dd13e63b352750580beea54ea7c729e708f2d2905fef" # no [build-system] and no setup.py; pkg/ copies its one file into site-packages PYPKG_PREPROCESS_CANCELLATION_FILE="preprocess_cancellation-0.2.1.tar.gz" PYPKG_PREPROCESS_CANCELLATION_SHA256="e2f1224e1ba1603bdfdbf6937caaf91082dc849e5122e80a5328aa999433ce79" PYPKG_PYSERIAL_FILE="pyserial-3.4.tar.gz" PYPKG_PYSERIAL_SHA256="6e2d401fdee0eab996cf734e67773a0143b932772ca8b42451440cfed942c627" -PYPKG_PYSERIAL_ASYNCIO_FILE="pyserial-asyncio-0.6.tar.gz" -PYPKG_PYSERIAL_ASYNCIO_SHA256="b6032923e05e9d75ec17a5af9a98429c46d2839adfaf80604d52e0faacd7a32f" PYPKG_PYCPARSER_FILE="pycparser-2.22.tar.gz" # cffi's only runtime dep PYPKG_PYCPARSER_SHA256="491c8be9c040f5390f5bf44a5b07752bd07f56edf992381b05c701439eec10f6" # imported at module scope by streaming-form-data; not optional PYPKG_SMART_OPEN_FILE="smart_open-6.4.0.tar.gz" PYPKG_SMART_OPEN_SHA256="be3c92c246fbe80ebce8fbacb180494a481a77fcdcb7c1aadb2ea5b9c2bee8b9" -PYPKG_JINJA2_FILE="jinja2-3.1.4.tar.gz" -PYPKG_JINJA2_SHA256="4a3aee7acbbe7303aede8e9648d13b8bf88a429282aa6122a993f0ac800cb369" +PYPKG_JINJA2_FILE="jinja2-3.1.6.tar.gz" +PYPKG_JINJA2_SHA256="0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d" # keeps lmdb off its cffi path; also the backend pkg_buildpython installs PYPKG_SETUPTOOLS_FILE="setuptools-75.6.0.tar.gz" PYPKG_SETUPTOOLS_SHA256="8199222558df7c86216af4f84c30e9b34a61d8ba19366cc914424cdbd28252f6" +# moonraker/utils/source_info.py imports it at module scope. The backport and +# not the 3.13 stdlib importlib.metadata, because that is what upstream writes; +# zipp is its one runtime dependency. Both build through setuptools_scm, which +# is why PYPKG_HOST_LIST below grew two entries. +PYPKG_IMPORTLIB_METADATA_FILE="importlib_metadata-8.7.0.tar.gz" +PYPKG_IMPORTLIB_METADATA_SHA256="d13b81ad223b890aa16c5471f2ac3056cf76c5f10f82d6f9292f0b415f389000" +PYPKG_ZIPP_FILE="zipp-3.21.0.tar.gz" +PYPKG_ZIPP_SHA256="2c9958f6430a2040341a52eb608ed6dd93ef4392e02ffe219417c1b28b5dd1f4" + PYPKG_TORNADO_FILE="tornado-6.4.tar.gz" PYPKG_TORNADO_SHA256="72291fa6e6bc84e626589f1c29d90a5a6d593ef5ae68052ee2ef000dfd273dee" PYPKG_MARKUPSAFE_FILE="MarkupSafe-2.1.5.tar.gz" @@ -146,11 +176,18 @@ PYPKG_PILLOW_SHA256="72bacbaf24ac003fea9bff9837d1eedb6088758d41e100c1552930151f6 # Host-side PEP 517 backends. THESE NEVER SHIP: pkg_pywheel builds with # --no-build-isolation, so each must already be in work/.py-host. -PYPKG_HOST_LIST="flit-core poetry-core" +# ORDER IS INSTALL ORDER: pkg_buildpython pip-installs each with --no-deps +# against no index, so a backend must come after whatever builds it. flit-core +# builds packaging; setuptools_scm imports packaging at build time. +PYPKG_HOST_LIST="flit-core poetry-core packaging setuptools-scm" PYPKG_FLIT_CORE_FILE="flit_core-3.10.1.tar.gz" # jinja2's backend PYPKG_FLIT_CORE_SHA256="66e5b87874a0d6e39691f0e22f09306736b633548670ad3c09ec9db03c5662f7" -PYPKG_POETRY_CORE_FILE="poetry_core-1.9.1.tar.gz" # libnacl's backend +PYPKG_POETRY_CORE_FILE="poetry_core-1.9.1.tar.gz" # libnacl's, dbus-fast's PYPKG_POETRY_CORE_SHA256="7a2d49214bf58b4f17f99d6891d947a9836c9899a67a5069f52d7b67217f61b8" +PYPKG_PACKAGING_FILE="packaging-24.2.tar.gz" # setuptools_scm's dep +PYPKG_PACKAGING_SHA256="c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f" +PYPKG_SETUPTOOLS_SCM_FILE="setuptools_scm-8.1.0.tar.gz" # zipp's, importlib's +PYPKG_SETUPTOOLS_SCM_SHA256="42dea1b65771cba93b7a515d65a65d8246e560768a66b9106a592c8e7f26c8a7" # dlopened by libnacl, so it must be shared -- our one .so in $MODDIR/lib. SODIUM_VERSION="1.0.20"