# A Python for the Creator 5 Pro that has sqlite3 An adoption, not an investigation -- but it started as one, and this is what the investigation measured. It answers one question: can we put an interpreter on this printer that is not FlashForge's, and does it have the module whose absence has been shaping this repo's decisions for two releases? Everything below was measured on the printer replica (real `rootfs.squashfs` under qemu-mipsel) or read out of the objects with `readelf`, not off a project page. `build.sh` beside this file rebuilds the whole thing in a throwaway container and is where the numbers came from; `pkg/python` is the same build wired into the real build lane, and `qa/replica/` is what proves the result works on the printer's own kernel -- `test_install.py` over the installed interpreter, `test_boot_screen.py` running our own code on it, and `test_abi.py` over every ELF it ships. ## Why: one missing module, two releases of consequences FlashForge built `/usr/prog/Python-3.8.2` **without `_sqlite3`**, and shipped no `libsqlite3` anywhere on the rootfs. Measured on the replica, with the library path the interpreter needs to start at all: ModuleNotFoundError: No module named '_sqlite3' That single omission is why `MOONRAKER_VERSION` in `versions.env` is a 2023 commit rather than a release: every Moonraker from v0.9.0 onward keeps its database in sqlite, and v0.9.0 is where it moved off lmdb. The pin is the last commit that still uses lmdb *and* has the webcam fix Mainsail needs. No release has both. That is not a Moonraker problem; it is this problem, seen from downstream. The alternatives were: build a `_sqlite3` for 3.8.2 (a module against an interpreter we did not build, whose ABI we would be guessing at), or build a whole interpreter. A from-source cross-build costs the same either way, so it is a whole interpreter -- and a current one, which also ends 3.8's end-of-life status as a thing anyone has to think about. ## What it is CPython **3.13.7**, `--disable-shared`, cross-compiled with the repo's pinned Ingenic gcc 7.2 / glibc 2.29 toolchain, against seven static C libraries built alongside it (OpenSSL 3.0.15, sqlite 3.46.1, zlib 1.3.1, libffi 3.4.6, xz 5.4.7, bzip2 1.0.8, expat 2.6.4). Everything is pinned by sha256 in `versions.env`. `lib-dynload` comes to 60 extension modules, including the ones this printer could never have had: `_sqlite3`, `_ssl`, `_hashlib`, `_ctypes`, `_lzma`, `_bz2`, `zlib`, `pyexpat`, `_asyncio`, `_decimal`, `_multiprocessing`. ## The ABI, which is the whole game The printer's kernel loads **little-endian, NAN2008, o32, mips32r2** and nothing else. It refuses everything else with `ENOEXEC` -- "cannot execute binary file" -- and explains nothing. e_flags = 0x70001405 an EXECUTABLE e_flags = 0x70001407 a SHARED OBJECT -- 0x2 more, EF_MIPS_PIC loader = /lib/ld-linux-mipsn8.so.1 rootfs glibc 2.33 Both words are correct and any ABI gate must accept both, or it fails on every one of the 56 extension modules in the tree. `qa/replica/test_abi.py` -- the only ABI gate there is now -- masks the low three bits off for exactly this reason. `c_helper.so` reads `0x70001407` too, for the same reason. **`-EL -mnan=2008` are mandatory in CFLAGS *and* in LDFLAGS.** This toolchain defaults to big-endian legacy-NaN; the flags are what move it. Passing them in `CFLAGS` is not enough, because several of the eight projects here do not forward `CFLAGS` to their link line, and an object linked without them is legacy-NaN however it was compiled. So the build does not pass the flags at all. It installs **PATH wrapper scripts that bake them into the gcc driver**: #!/bin/sh exec /toolchain/bin/mips-linux-gnu-gcc -EL -mnan=2008 "$@" and no build system gets a vote. That is the single most important trick in either copy of this build. The wrapper is itself gated -- compile `int main(void){return 0;}`, read `e_flags`, refuse to continue unless it is `0x70001405` -- before 300MB of anything is built on top of it. ## musl is forbidden here s6 is built against musl in this same repo, and s6 is 73MB with static glibc versus 3.6MB with musl. None of that applies to the interpreter: klippy `dlopen`s `c_helper.so`, which is a **glibc** shared object, and a musl-linked process cannot dlopen it. So this uses the Ingenic glibc toolchain -- the same one that builds `c_helper.so`, which is also what makes them a matched pair. glibc 2.29 (toolchain) against 2.33 (rootfs) is forward compatible, and the measured `NEEDED` list is six ordinary sonames: libpthread.so.0 libdl.so.2 libatomic.so.1 libm.so.6 libutil.so.1 libc.so.6 `libatomic.so.1` is the one worth naming. 64-bit atomics on mips32 are out-of-line calls into libatomic and CPython 3.13's `_Py_atomic_*` needs them. It **is** on the rootfs -- `/lib/libatomic.so.1.2.0`, measured -- but it is the only one of the six that is not obviously part of a base system, so the gate says so out loud. ## Static dependencies, and a static libpython All seven C libraries are built `-fPIC` and linked in as `.a`. Nothing of ours ships as a `.so`, and `--disable-shared` puts libpython inside the executable rather than beside it. The reason is not size, it is **isolation**. A shared build needs an `LD_LIBRARY_PATH`, and on this printer that path is a minefield: `/usr/prog` carries `libffi.so.8` while the rootfs carries `libffi.so.7`, FlashForge's openssl is 1.0.2d, and a stock OTA can replace any of it. Statically linked, the interpreter maps *nothing* under `/usr/prog` -- which `case-python.sh` section 3 proved by reading `/proc/self/maps` of a process that has imported `ssl`, `ctypes`, `sqlite3`, `lzma`, `zlib` and `hashlib` first. Zero mappings. It also means it starts with `LD_LIBRARY_PATH` empty, which FlashForge's cannot. The cost is a few MB of duplicated libcrypto between `_ssl.so` and `_hashlib.so`. At 30MB that is a good trade. A `--disable-shared` interpreter is not automatically usable by third-party extension modules -- they resolve Python's symbols out of the running executable, which only works if it was linked `-export-dynamic`. That was checked, not assumed: an out-of-tree extension module was cross-compiled against the staged `include/` and imported successfully on the replica. ## The trap that costs a day: `_sqlite3` and `-lm` **The one module this entire build exists for goes missing silently.** A shared `libsqlite3.so` carries its own `DT_NEEDED` on libm. A `libsqlite3.a` does not. So CPython's checking for sqlite3_bind_double in -lsqlite3 link probe fails with a wall of `undefined reference to floor/log/pow/...`, and CPython 3.13 **records `_sqlite3` as "missing" and carries on**. It is a probe failure, not a compile failure. Nothing in 400 lines of build output says why. You get a clean build, a working interpreter, and no sqlite3. The fix is two exports: LIBS="-latomic -lm" LIBSQLITE3_LIBS="-L$DEP/lib -lsqlite3 -lm" the second stating the link line outright so pkg-config cannot reorder `-lm` out from under the probe. And because a silent failure fixed by an easily-deleted export is a silent failure waiting to come back, **both copies of this build hard-fail when `_sqlite3` is absent**, `bin/verify.sh` checks the packaged tree for it, and `case-python.sh` did not merely import it: it created a database, inserted rows, closed it, and reopened it from disk **in a second process**. An import proves the `.so` loaded. Only the second process proves there is a database. ## Two OpenSSL traps Both hit for real on 3.0.15: * **`no-docs` does not exist before OpenSSL 3.1.** On 3.0.x it is an "Unsupported options" *hard error*, not a warning. * **The `linux-mips32` target hardcodes `-mips2`** into its cflags. This toolchain defaults to `-mfp64`, and gcc rejects the combination below mips32r2: *"'-mgp32' and '-mfp64' can only be combined if the target supports the mfhc1 and mthc1 instructions"*. User cflags land *after* the target's on the command line, so appending `-mips32r2` puts the ISA back where the printer is. If a future OpenSSL orders them the other way, `linux-generic32` (portable C, no mips assembly) is the fallback -- and both copies of the build take it automatically rather than leaving it as a note, because the failure is a screenful of assembler errors that says nothing about ISA levels. This is why `OPENSSL_VERSION` is pinned to the 3.0 LTS branch and not bumped casually. ## The gap: there are no CA certificates `--openssldir` points at `/usr/data/anvil/ssl`, which is where the interpreter looks for a trust store **on the printer**. Nothing installs one there, and the rootfs has none to borrow. So: `import ssl` works, TLS works, and **verifying a certificate chain does not**. `ssl.create_default_context()` builds a context with an empty trust store and every verified connection fails. Anything that fetches over https on this interpreter has to be handed a `cafile` of its own until somebody ships a `ca-certificates` bundle into that directory -- which is ~200KB and an open piece of work, not a decision. `case-python.sh` printed the trust-store paths and said which of the two states the printer was in, rather than asserting either, so the day someone shipped a bundle the gate would report it instead of failing. ## Sizes and time | | files | on disk | |---|---|---| | `make install` output | 2909 | 183 MB | | trimmed -- what ships | 608 | 30 MB | | trimmed, gzipped | | 10.4 MB | The trim drops the CPython test package and `idlelib` (nothing on a printer runs either), `tkinter` (there is no X11), `include/` and `config-*/` (they exist to *build* extension modules, which happens on a developer's machine), every `.a`, and every `__pycache__` -- 12MB of `.pyc` for modules that will be imported once if ever, and which the interpreter regenerates on demand anyway. The interpreter and every `lib-dynload/*.so` are stripped. It also empties `bin/` of everything except `python3.13` itself -- the `python3` symlink, `idle3`, `pydoc3` and the `*-config` scripts -- and drops `lib/pkgconfig`. Those are all build-time or convenience files, and in a *shared* prefix root they are not neutral: `bin/` and `lib/` here also hold s6's binaries and, through `anvil-env.sh`, sit on every mod process's PATH. See "Where it lands" above. The build asserts `bin/` holds exactly one name afterwards, so a future CPython that installs one more launcher fails the build rather than quietly adding a name to PATH. **Cold build: 142 seconds** (`tools/python/build.sh`, 16 cores), of which about 90 is the x86-64 build-python. In the build lane it is ~100s and it happens **once**: `bin/payload.sh` caches the cross-built tree in `work/.py313` under a stamp of all eight versions, so every later build copies rather than compiles. The stamp is all eight and not just CPython's, because a bumped OpenSSL under an unchanged `PY_VERSION` has to rebuild. Cross-building CPython needs a build-python of the **same** version -- the Makefile runs it to freeze modules, generate the deepfreeze sources and byte-compile the stdlib, and `configure` hard-errors when the version does not match. That is why `docker/Dockerfile.build` carries a host C compiler, and why `PY_VERSION` moves both halves of the build at once. ## Where it lands `$MODDIR` -- `/usr/data/anvil` -- is a `--prefix` root. That is the whole reason s6 was configured for it, and it is why the mod's tree has `bin/`, `lib/`, `libexec/` and `etc/` directly inside it rather than a directory per package. The interpreter is configured `--prefix=/usr/data/anvil` like everything else, so it installs as `bin/python3.13` beside `bin/s6-svscan`, with its stdlib in `lib/python3.13/`. One prefix, one place to look. That path is **compiled in** -- `sys.prefix` and the stdlib search that follows from it -- so moving the directory on the printer breaks the interpreter the way moving s6 breaks its waiting verbs. It has to be rebuilt, not renamed; which is itself an argument for keeping it in the one prefix the mod already guarantees rather than inventing a second one. **One thing does not ship: the `python3` symlink.** `make install` puts it beside `python3.13`, along with `idle3`, `pydoc3` and the `*-config` scripts. `anvil-env.sh` prepends `$MODDIR/bin` to `PATH` -- it has to, because `s6-svscan` execs `s6-supervise` by name -- so a `bin/python3` of ours would sit *in front of* FlashForge's for every process that sources it, and anything saying `python3` rather than `"$FF_PYTHON"` would change interpreter without anyone deciding to. The build deletes it before staging and asserts that `bin/` contains `python3.13` and nothing else; `bin/verify.sh` checks the packaged tree for it; and `case-python.sh` section 6 asked the *shell*, with the mod's own PATH in force, what `python3` resolved to. The answer is still FlashForge's binary. ## It ships. Nothing runs it. *[Superseded: `FF_PYTHON` is `$MODDIR/bin/python3.13` now, and Moonraker, klippy, `ff-startup.py`, `ffscreen.py` and the MCU bring-up all run on it. What follows is the state this investigation ended in.]* `payload/anvil-env.sh` still points `FF_PYTHON` at `/usr/prog/Python-3.8.2/bin/python3`. That is timing, not preference: klippy, Moonraker and `ff-startup.py` import third-party C extensions -- tornado, lmdb, cffi, greenlet, pillow, libnacl -- which exist on this printer only as mipsel `.so` files built against 3.8. None has been cross-built for 3.13. Switching before they exist is a dark screen and an ImportError in a log nobody reads. FlashForge's tree is left alone in both directions: nothing writes to `/usr/prog`, and our interpreter lives entirely under `/usr/data/anvil` like everything else this mod installs. This is deliberately the same shape as phase 2, which shipped s6 without starting it -- the artefact reaches real printers where it can be measured, and the switch stays a one-line change instead of a build change and a boot change arriving together. Cross-building the extensions is the actual work in adopting this; not the interpreter. ## Reproducing ./bin/fetch-assets.sh --all # pins + the Ingenic toolchain ./tools/python/build.sh # 142s, artefacts in tools/python/out/ That is how the numbers above were measured, and it is kept here for the record and for trying a version bump before it becomes a pin. It is not how the shipped interpreter is built: `pkg/python` compiles it inside the repo's own build image, from the sha256-pinned tarballs, against the seven static libraries the feed builds as their own `-dev` packages, and caches it in `work/pkg/python`. `bin/payload.sh` section 5c runs that recipe and stages what it produced; it no longer contains a compiler invocation of its own. The gate, against the tree the build actually produced: make build && make qa-replica `case-python.sh`, `printer-exec.py` and the `make test-python` target that drove them are gone. The replica lane installs the real package through the printer's own installer and asks its questions of that, which is strictly more than hand-feeding a tarball into a case script could.