diff --git a/docker/Dockerfile.build b/docker/Dockerfile.build index ff1a1b3..ac9e8dd 100644 --- a/docker/Dockerfile.build +++ b/docker/Dockerfile.build @@ -28,6 +28,18 @@ # this container; the pin-and-hash rule governs what reaches the printer, and # the printer's opkg is cross-compiled against the libarchive OUR feed built. # Same call already made for gcc. +# +# ninja-build IS DEBIAN'S ON THE SAME TERMS, and MESON IS NOT -- the split is +# the point. numpy builds with meson-python, so both are needed; but meson is +# a Python module our build-python imports, and bookworm's 1.0.1 is below the +# >=1.2.99 numpy's meson.build demands, so it is pinned in versions.env with +# the rest of PYPKG_HOST_LIST. ninja is a program meson merely execs, no pin of +# ours would change what it emits, and nothing it produces reaches a printer +# except through the cross compiler that is pinned. +# +# IT IS NEEDED EARLIER THAN NUMPY, which is the part that surprises: meson-python +# is self-hosting (backend-path = ['.']), so pkg_buildpython runs meson and +# ninja just to install the backend itself. FROM debian:bookworm-slim ENV DEBIAN_FRONTEND=noninteractive @@ -39,7 +51,7 @@ RUN echo 'Acquire::Retries "3";' > /etc/apt/apt.conf.d/80-retries \ binutils file findutils grep sed gawk make git diffutils patch procps \ squashfs-tools qemu-user-static docker.io shellcheck \ gcc g++ libc6-dev zlib1g-dev pkg-config perl \ - libarchive-dev \ + libarchive-dev ninja-build \ && rm -rf /var/lib/apt/lists/* WORKDIR /src diff --git a/docs/notes/00-overview.md b/docs/notes/00-overview.md index 10b6db9..392100e 100644 --- a/docs/notes/00-overview.md +++ b/docs/notes/00-overview.md @@ -71,7 +71,7 @@ Files ending `-recovered` are reconstructions of the stock app's behaviour; the - `25-app-vs-klipper-ownership.md` — which side owns which behaviour, and what is ported - `30-toolchange.md` — dock/grab/release sequences, sensors, mounted-tool state - `40-offsets.md` — where per-unit numbers live; per-tool offsets; the absolute print Z offset -- `44-vfa-calibration.md` — VFA (motor torque-ripple) compensation: the stock chain, and the numpy gap that blocks it +- `44-vfa-calibration.md` — VFA (motor torque-ripple) compensation: the stock chain, what applies it, and the numpy gap that stopped klippy booting - `45-tool-offset-calibration.md` — the Klipper port: storage layout, commands, guards - `46-offset-calibration-recovered.md` — the app's calibration sequence it was ported from - `47-filament-load-recovered.md` / `48-filament-load-port.md` — load/unload/purge diff --git a/docs/notes/44-vfa-calibration.md b/docs/notes/44-vfa-calibration.md index b8e311e..18e4b02 100644 --- a/docs/notes/44-vfa-calibration.md +++ b/docs/notes/44-vfa-calibration.md @@ -1,4 +1,4 @@ -# VFA calibration — what the stock app automates, and what it would take here +# VFA calibration — what the stock app automates, and how it was brought back "VFA" in FlashForge's UI is **Vertical Fine Artifacts**: the fine vertical banding a stepper's torque ripple prints onto a wall. The stock firmware calls it *VFA vertical @@ -181,83 +181,123 @@ so the printer runs without it and loses resonance testing" in `anvil-env.sh`, a "both are import-guarded" in `pkgs/klipper/pkg.conf`. Both were wrong; both are corrected in the commit that adds this note. -**This needs confirming against a real `printer.log` before anything else here is -built.** If printers do boot today, then something is supplying numpy and that is worth -knowing exactly as much. +That is the failure this note was opened for, and it is closed below — but the +diagnosis was never confirmed against a real `printer.log`. Every gate listed at the +end of this note passes, which says a printer built from this tree starts; it does not +prove that one built before it did not. If you have a machine that was running the +numpy-less build, its `printer.log` is still worth reading. -## Migration options +## How it was closed -### A. Package numpy (restores VFA *and* `SHAPER_CALIBRATE`) +numpy is packaged. `pkgs/3rdparty/python-numpy/` builds numpy 2.1.3 for +`mipsel_xburst2`, `anvil-klipper` depends on it, and the calibration is reachable +again — along with `SHAPER_CALIBRATE` and `TEST_RESONANCES`, which had been dead for +exactly the same reason. -The one option that fixes the startup failure and input-shaper calibration together. +Two alternatives were considered and dropped. Dropping numpy from +`stepper_resonance_tester.py` was feasible — every measurement asks for the amplitude +at **one** known frequency, so the rFFT collapses to three Goertzel evaluations — but it +leaves `SHAPER_CALIBRATE` broken and is a second DSP implementation to carry, the kind +`consolidate-dont-carry-two` argues against. Guarding the import stops the boot failure +and restores nothing; it remains the right emergency patch if a release is ever due +before a numpy build lands. -- `pkgs/3rdparty/python-numpy/`, following `python-pillow` — it is the existing proof - that a non-trivial C extension cross-compiles here (`pkg_buildpython`, `pkg_pytarget`, - `pkg_pywheel`, `pkg_pynative`). -- **The catch:** numpy ≥ 2.1 is required for CPython 3.13, and numpy 2.x builds with - meson-python, not `setup.py`. `pkg_pywheel`'s non-`PKG_PY_SETUP_ARGS` path runs - `pip wheel --no-build-isolation`, which drives a PEP 517 backend fine, but there is no - way to pass `--config-settings=setup-args=--cross-file=…`. That is a small, contained - addition to `pkgs/lib.sh` (a `PKG_PY_PIP_ARGS`), plus a meson cross file for - `mipsel_xburst2` and meson/ninja/Cython in the build python. -- Cost: build-system work that is new to this repo, and ~15-30 MB installed. +### What it took, and what was not obvious -### B. Drop numpy from `stepper_resonance_tester.py` (VFA only) +**numpy 2.x builds with meson-python, not setuptools**, so the +`_PYTHON_SYSCONFIGDATA_NAME` trick every other `python-*` recipe relies on is not enough +on its own — meson has never heard of that variable. `pkg_mesoncross` in `pkgs/lib.sh` +writes a real cross file instead, and `pkg_pywheel` grew a `PKG_PY_PIP_ARGS` branch to +hand it over as `--config-settings`. -A fork-local patch in `pkgs/klipper/payload/klipper/klippy/extras/`, which already -overlays five `ff_*` extras. +**The `python =` line in that cross file is `$HOSTPY`, and that is not a mistake.** +Meson introspects an interpreter by *running* it and asking `sysconfig`, and +`pkg_pytarget` has already made the host interpreter's sysconfig answer for mipsel. So +the existing cross trick does extend to meson — just not by the variable's own +mechanism. -The measurement path the calibration actually walks is narrow: `mean`, `hanning`, -`rfft`/`rfftfreq`, `abs`, `searchsorted`, `argmax`, plus a 3-parameter `linalg.lstsq` -(normal equations) and a `polyfit(deg=2)` over 3 points (closed form). Every -measurement asks for the amplitude at **one** known frequency, so the rFFT can collapse -to three Goertzel evaluations — ~3k float ops against a move that takes about a second. -Cheap, and it stays pure Python. +**numpy's `meson.build` hard-errors on GCC below 8.4 and ours is 7.2.** The gate is +conservative rather than load-bearing: it comes from the SciPy Toolchain Roadmap, which +says what the project supports, not what the code needs. Probed before it was lowered — +this toolchain compiles `if constexpr`, structured bindings, `std::optional`, +`std::string_view`, and `pocketfft_hdronly.h`, 3635 lines of the heaviest template C++ +in the tree and the half `np.fft.rfft` needs. The recipe lowers the floor to 7.2 behind +a guard that fails loudly if upstream moves the check. -It leaves `SHAPER_CALIBRATE` broken, and it is a patch we own against a file that is -currently byte-identical to upstream's. +**meson-python is self-hosting** (`backend-path = ['.']`), so `pkg_buildpython` runs +meson and ninja just to *install the backend* — which is why `ninja-build` is in +`docker/Dockerfile.build` and why the four new `PYPKG_HOST_LIST` entries are in +dependency order rather than alphabetical. -### C. Guard the import (stops the bleeding, restores nothing) +**pip puts `meson` and `cython` in `work/.py-host/bin`, which nothing had ever looked +in.** Without that directory on PATH, meson-python fails to install with +`meson executable "meson" not found` while `pip list` shows meson installed and +importable. `pkg_buildpython` prepends it. -One `try/except ImportError` at the top of the module and an error at command time. -Worth doing on its own merits regardless of A or B, and it is the whole fix if the -startup failure is confirmed and a release is due before either lands. +Result: 19 extension modules, every one ELF32 MIPS32r2 little-endian at `e_flags` +`0x70001405`/`0x70001407`, 14 MB shipped after the recipe prunes `tests/`, `f2py/`, +`.pyi` stubs and the static libs. -**Recommendation: C now, then A.** C is a three-line patch against a confirmed -inability to boot; A is the only one that also gives back input-shaper calibration, -and B's hand-rolled DSP is a second implementation to carry — the kind -`consolidate-dont-carry-two` argues against. +## Using it -## Once numpy is there +The commands are FlashForge's and already on the printer; nothing here is a wizard. +From the Mainsail console: -Very little is left, because the macro already exists on the printer. +``` +STEPPER_RESONANCE_FACTORY_CALIBRATE +SAVE_CONFIG +``` -- **Wrap it like the other resonance commands.** `ff-toolchange.cfg:180-215` already - wraps `SHAPER_CALIBRATE` / `TEST_RESONANCES` / `MEASURE_AXES_NOISE` in - `_FF_SHAPER_PREP`, which homes and grabs `shaper_tool` when nothing is mounted. - The VFA test moves the carriage and measures moving mass the same way, so it wants - the same prep: +**`SAVE_CONFIG` is required and is deliberately not in any macro.** The search applies +its result live with `MCLIB_SET_RESONANCE_DAMP` and stages it with `configfile.set`, +which only `SAVE_CONFIG` writes — so without one the calibration is lost on the next +restart. The stock app had the same split (see above), and `SAVE_CONFIG` restarts +klippy, which is not something a macro should do to somebody mid-session. - ``` - [gcode_macro CALIBRATE_VFA] - gcode: - _FF_SHAPER_PREP - G1 Z100 F1200 - M400 - STEPPER_RESONANCE_FACTORY_CALIBRATE - SAVE_CONFIG ; the app's step, not the macro's — see above - ``` +`ff-toolchange.cfg` wraps `STEPPER_RESONANCE_SEARCH_CALIBRATE` and +`STEPPER_RESONANCE_REFINE_CALIBRATE` in `_FF_SHAPER_PREP`, the same prep +`SHAPER_CALIBRATE` already gets: home if needed, grab `shaper_tool` if nothing is +mounted, because an empty carriage measures the wrong moving mass. - Note `SAVE_CONFIG` restarts klippy, so it must be last and the caller must expect it. -- **Check the move geometry against our limits.** The stock centres - (`stepper_x_center_xy: 40,220`, `stepper_y_center_xy: 220,220`) with - `move_distance` 100 at ±45° sweep X 4.6→75.4 and Y 184.6→255.4. Our - `position_max` is X 310 / Y 260 — it fits, but Y clears by 4.6 mm, and - `x_range`/`y_range` in the config are sanity-checked at load and then **never used to - clamp the move**. On a toolchanger with docks at X≈297 this is worth a dry run before - it is offered to owners. -- **Decide where it is exposed.** HelixScreen has no VFA tile; the macro is reachable - from Mainsail's console either way. +`STEPPER_RESONANCE_FACTORY_CALIBRATE` is **not** wrapped, and the obvious edit there is +wrong. It is a `[gcode_macro]` in FlashForge's `printer.vibration.cfg`, not a +Python-registered command, and Klipper parses with `RawConfigParser(strict=False)` — so +a second section of the same name *merges* with the first rather than wrapping it, and +`rename_existing` would point the macro at itself. It needs no wrapper anyway: its whole +body is two `STEPPER_RESONANCE_SEARCH_CALIBRATE` calls, and those are wrapped. + +### Still worth a dry run + +The move geometry is FlashForge's and has not been checked on a real Creator 5 Pro by +anyone here. The stock centres (`stepper_x_center_xy: 40,220`, +`stepper_y_center_xy: 220,220`) with `move_distance` 100 at ±45° sweep X 4.6→75.4 and +Y 184.6→255.4, against our `position_max` of X 310 / Y 260. It fits, but Y clears by +4.6 mm, and `x_range`/`y_range` in the config are sanity-checked at load and then +**never used to clamp the move**. On a toolchanger with docks at X≈297 that is worth +watching once before it is offered to owners. + +Nothing in the replica can tell you whether the calibration produces *good numbers* — +that needs a printer, an accelerometer and real motion. What the gates below prove is +that klippy starts and that numpy computes correctly, which is a different and smaller +claim. + +## The gates that would have caught this + +- `qa/replica/test_klippy_extras_import.py` — walks the real `/usr/data/config` include + graph, maps every section to `extras/.py`, and imports each on the printer's own + interpreter. Needs no MCU: config load happens in `_read_config`, before + `klippy:mcu_identify`. This is the test whose absence let two wrong comments stand. +- `qa/replica/test_numpy.py` — asks for numeric answers rather than a module object. + `np.linalg.lstsq` is in there specifically because `-Dallow-noblas=true` selects the + bundled `lapack_lite`, and `_r2_sinusoidal_fit` calls `lstsq` on every calibration + round. +- `qa/replica/test_install.py::test_every_include_resolves_not_just_ours` — the old + check read only `ff-*.cfg`, so FlashForge's four includes, `printer.vibration.cfg` + among them, were never asked about at all. +- `qa/static/test_recipe_layout.py::test_klipper_still_depends_on_numpy` — the cheapest + gate on the most expensive mistake. +- `qa/replica/test_abi.py` needed no change: it sweeps every ELF object on the + filesystem, so numpy's 19 extensions are covered the moment they ship. ## Address-verified symbols diff --git a/pkgs/3rdparty/python-numpy/build.sh b/pkgs/3rdparty/python-numpy/build.sh new file mode 100755 index 0000000..8369334 --- /dev/null +++ b/pkgs/3rdparty/python-numpy/build.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# numpy -- built for the printer's CPython, into its site-packages. +# +# THE ONLY meson-python RECIPE IN THE FEED. Every other python-* package here +# builds with setuptools, where _PYTHON_SYSCONFIGDATA_NAME alone is enough to +# make the build answer for mipsel. Meson has never heard of that variable, so +# this one hands it a cross file instead -- see pkg_mesoncross in pkgs/lib.sh +# for what is in it and why each line is there. +set -euo pipefail +. ./bin/common.sh +. pkgs/lib.sh + +pkg_begin python-numpy || exit 0 +pkg_toolchain +pkg_deps +pkg_buildpython +pkg_pytarget +pkg_unpack "$(pypkg_tgz numpy)" + +_np=$(pkg_pysrc numpy) + +# numpy's meson.build refuses gcc below 8.4, and ours is 7.2. THE GATE IS +# CONSERVATIVE RATHER THAN LOAD-BEARING, and that was probed before it was +# lowered: numpy 2.1's requirement is a C++17 compiler, and this toolchain +# compiles the features it actually uses -- if constexpr, structured bindings, +# std::optional, std::string_view -- as well as pocketfft_hdronly.h, 3635 lines +# of the heaviest template C++ in the tree and the half np.fft.rfft needs. +# The upstream number comes from the SciPy Toolchain Roadmap, which is a +# statement about what the project supports, not about what the code requires. +# +# The floor is LOWERED, not removed: a genuinely ancient gcc should still be +# refused here rather than three screens into a compile log. If this recipe +# ever fails with a C++ error rather than a version error, that is the gate +# turning out to be real after all, and the fix is a newer cross toolchain -- +# not a smaller number. +if ! grep -q "version_compare('>=8.4')" "$_np/meson.build"; then + pkg_die "python-numpy: the gcc version gate in meson.build is not where + this recipe patches it -- numpy $(pypkg_version numpy) has moved + it, and the comment above needs re-checking against the new one" +fi +sed -i "s/version_compare('>=8.4')/version_compare('>=7.2')/" "$_np/meson.build" + +# -Dallow-noblas: no BLAS/LAPACK in this feed, so take the bundled lapack_lite. +# Without this numpy 2.x fails configure rather than falling back. +# -Ddisable-optimization: numpy's SIMD dispatch machinery generates and builds +# a matrix of per-feature objects. There is no MIPS path in it, so every one +# of them is a baseline build anyway -- this drops the generation step and, +# with it, the largest surface for a gcc 7.2 to refuse something. +PKG_PY_PIP_ARGS="--config-settings=setup-args=--cross-file=$(pkg_mesoncross) + --config-settings=setup-args=-Dallow-noblas=true + --config-settings=setup-args=-Ddisable-optimization=true" + +pkg_pywheel numpy + +# PRUNE BEFORE SHIP, and only things nothing on a printer can reach. +# tests/ ~8MB of pytest files; pytest is not on this machine. +# f2py/ a Fortran wrapper generator, and there is no Fortran here. +# *.pyi, py.typed type stubs, read by type checkers and never at runtime. +# _core/lib/*.a static libs for BUILDING against numpy's C API, which +# nothing on the printer does. +# The .so files are not touched here: pkg_ship runs the cross strip over +# everything it copies. +find "$PKG_PYSP/numpy" -type d -name tests -prune -exec rm -rf {} + 2>/dev/null || true +rm -rf "$PKG_PYSP/numpy/f2py" +find "$PKG_PYSP/numpy" -name '*.pyi' -delete 2>/dev/null || true +find "$PKG_PYSP/numpy" -name 'py.typed' -delete 2>/dev/null || true +find "$PKG_PYSP/numpy" -name '*.a' -delete 2>/dev/null || true + +pkg_ship "lib/python$PY_MM/site-packages" + +# A COUNT, AND A HIGH ONE ON PURPOSE. numpy has a pure-python fallback for +# nothing -- if the cross build quietly produced no extensions there would be +# no import at all, but a partial build that lost _multiarray_umath while +# keeping the small ones would import far enough to look installed. Eight is +# comfortably below what a whole numpy ships and far above any partial one. +pkg_pynative 8 +pkg_end diff --git a/pkgs/3rdparty/python-numpy/pkg.conf b/pkgs/3rdparty/python-numpy/pkg.conf new file mode 100644 index 0000000..9d31e70 --- /dev/null +++ b/pkgs/3rdparty/python-numpy/pkg.conf @@ -0,0 +1,36 @@ +# anvil-python-numpy -- numpy for the printer's CPython, into its +# site-packages. +# +# THIS IS NOT AN OPTIONAL SCIENTIFIC EXTRA. It is the package klippy cannot +# start without. extras/stepper_resonance_tester.py opens with a bare +# `import numpy as np`; klippy.py:122 walks EVERY config section through a +# load_object that does not catch ImportError; and printer.base.cfg includes +# FlashForge's printer.vibration.cfg, which declares [stepper_resonance_tester]. +# So on a numpy-less install klippy never reaches ready -- it does not merely +# lose a feature. anvil-klipper names this package in its Depends for that +# reason. docs/notes/44-vfa-calibration.md has the whole chain. +# +# WHAT IT BUYS BACK, beyond the boot: the VFA (motor torque-ripple) calibration +# behind STEPPER_RESONANCE_FACTORY_CALIBRATE, and SHAPER_CALIBRATE / +# TEST_RESONANCES, which have been dead for exactly the same reason since +# klippy moved to our own CPython 3.13. +PKG_NAME=anvil-python-numpy + +# ARCHITECTURE IS THE DEFAULT, mipsel_xburst2, and it is not a formality here +# any more than it is for anvil-klipper: this package is mostly compiled code. +# An `all` would let it install on a machine whose kernel refuses every .so in +# it, and the failure would arrive as an ImportError halfway through a boot. +PKG_VERSION="$(pypkg_version numpy)" +PKG_RELEASE=1 +PKG_SECTION=python +PKG_EXCLUDE=".version" + +PKG_DEPENDS="anvil-python" + +# python ALONE, and no BLAS: -Dallow-noblas=true below picks numpy's bundled +# lapack_lite instead. That is not a corner cut for size -- there is no BLAS in +# this feed to link against, and lapack_lite is enough for the one linalg call +# the calibration makes (np.linalg.lstsq, in _r2_sinusoidal_fit). +PKG_BUILD_DEPENDS="python" + +PKG_DESCRIPTION="numpy for klippy -- the FFT, least-squares and polynomial fitting that extras/stepper_resonance_tester.py and shaper_calibrate.py are written against. Built with meson-python against a cross file, without BLAS." diff --git a/pkgs/anvil-core/payload/anvil-env.sh b/pkgs/anvil-core/payload/anvil-env.sh index 35ecc31..455ba90 100644 --- a/pkgs/anvil-core/payload/anvil-env.sh +++ b/pkgs/anvil-core/payload/anvil-env.sh @@ -66,19 +66,23 @@ export LD_LIBRARY_PATH # klippy -- which reaches c_helper.so through _cffi_backend, the reason this # interpreter had to be a glibc build. # -# klippy's numpy gap (extras/stepper_resonance_tester.py) is this switch's -# problem and it is NOT a smaller item. This comment used to say the module -# guards its own import, so the printer runs without it and loses resonance -# testing. Both halves are wrong: line 1 of that file is a bare +# THE NUMPY GAP WAS THIS SWITCH'S, and it was not the small item this comment +# used to call it. The old text said extras/stepper_resonance_tester.py guards +# its own import, so the printer runs without numpy and merely loses resonance +# testing. Both halves were wrong: line 1 of that file is a bare # `import numpy as np`, and klippy.py:122 loads EVERY config section through a # load_object that does not catch ImportError. printer.base.cfg includes # FlashForge's printer.vibration.cfg, which declares -# [stepper_resonance_tester] -- so on this interpreter klippy does not come up -# at all, rather than coming up without one feature. +# [stepper_resonance_tester] -- so moving klippy to an interpreter without +# numpy did not cost a feature, it stopped klippy reaching ready at all. # -# See docs/notes/44-vfa-calibration.md for the chain and the options. Nothing -# here is fixed by editing this line; it takes either a packaged numpy or a -# guarded import in the fork. +# CLOSED BY pkgs/3rdparty/python-numpy, which anvil-klipper now names in its +# Depends. FlashForge's 3.8.2 had numpy in its rootfs and a 3.8 .so is not +# importable from 3.13, so the interpreter switch had to bring its own -- and +# it is the reason ninja-build is in docker/Dockerfile.build and +# pkgs/lib.sh grew pkg_mesoncross. docs/notes/44-vfa-calibration.md has the +# chain, and qa/replica/test_klippy_extras_import.py is the gate that would +# have caught it. # # Note what is NOT on PATH below: $MODDIR/bin is prepended and the interpreter # is in there, but it is called python3.13 and only that. bin/payload.sh diff --git a/pkgs/klipper-config/payload/config/ff-toolchange.cfg b/pkgs/klipper-config/payload/config/ff-toolchange.cfg index 6e20f80..55fa4b9 100644 --- a/pkgs/klipper-config/payload/config/ff-toolchange.cfg +++ b/pkgs/klipper-config/payload/config/ff-toolchange.cfg @@ -209,3 +209,50 @@ rename_existing: _FF_MEASURE_AXES_NOISE_BASE gcode: _FF_SHAPER_PREP _FF_MEASURE_AXES_NOISE_BASE {rawparams} + +# VFA (vertical fine artifacts) -- the motor torque-ripple compensation, which +# is a DIFFERENT mechanism from the input shaping above and shares only the +# accelerometer. The commands come from klippy/extras/stepper_resonance_tester.py; +# these wrappers are ours, and exist for the reason the three above do: both +# sweep the carriage and read [lis2dw], so an empty carriage measures the wrong +# moving mass. +# +# STEPPER_RESONANCE_FACTORY_CALIBRATE IS DELIBERATELY NOT WRAPPED, and the +# reason is worth stating because the obvious edit is wrong. It is not a +# Python-registered command like the two below -- it is a [gcode_macro] in +# FlashForge's printer.vibration.cfg. Klipper parses with +# RawConfigParser(strict=False), so a second section of the same name MERGES +# with the first rather than sitting beside it: a +# `[gcode_macro STEPPER_RESONANCE_FACTORY_CALIBRATE]` here would not wrap that +# macro, it would BE that macro, with a rename_existing pointing at a command +# that does not exist yet. +# +# It needs no wrapper anyway. Its whole body is two +# STEPPER_RESONANCE_SEARCH_CALIBRATE calls, and those are wrapped here -- so +# the prep happens either way, once, from whichever command is reached first +# (_FF_SHAPER_PREP homes only if unhomed and grabs a tool only if none is +# mounted, so the second call is a no-op). +# +# NO SAVE_CONFIG HERE EITHER. The search applies its result live with +# MCLIB_SET_RESONANCE_DAMP and stages it with configfile.set, which only +# SAVE_CONFIG writes -- so without one the calibration is lost on the next +# restart. The stock app had exactly this split: the macro's own SAVE_CONFIG is +# commented out, and the app issued one separately at the end of the whole +# calibration run. It stays the operator's step because SAVE_CONFIG restarts +# klippy, which is not a thing a macro should do to somebody mid-session: +# +# STEPPER_RESONANCE_FACTORY_CALIBRATE +# SAVE_CONFIG +# +# See docs/notes/44-vfa-calibration.md. +[gcode_macro STEPPER_RESONANCE_SEARCH_CALIBRATE] +rename_existing: _FF_STEPPER_RESONANCE_SEARCH_CALIBRATE_BASE +gcode: + _FF_SHAPER_PREP + _FF_STEPPER_RESONANCE_SEARCH_CALIBRATE_BASE {rawparams} + +[gcode_macro STEPPER_RESONANCE_REFINE_CALIBRATE] +rename_existing: _FF_STEPPER_RESONANCE_REFINE_CALIBRATE_BASE +gcode: + _FF_SHAPER_PREP + _FF_STEPPER_RESONANCE_REFINE_CALIBRATE_BASE {rawparams} diff --git a/pkgs/klipper/pkg.conf b/pkgs/klipper/pkg.conf index 80de9d7..8d908d3 100644 --- a/pkgs/klipper/pkg.conf +++ b/pkgs/klipper/pkg.conf @@ -24,14 +24,15 @@ PKG_EXCLUDE=".version" # HERE. python-can is not packaged (no CAN transport on this machine) and it # is import-guarded, so its absence costs nothing. # -# NUMPY IS NOT IN THAT CLASS and this comment used to claim it was. -# extras/stepper_resonance_tester.py opens with a bare `import numpy as np`, -# and klippy.py:122 walks every config section through a load_object that does -# not catch ImportError -- so with printer.base.cfg including FlashForge's -# printer.vibration.cfg ([stepper_resonance_tester]), a numpy-less install is -# a klippy that never reaches ready. It is a Depends this package cannot yet -# declare because no anvil-python-numpy exists to name; -# docs/notes/44-vfa-calibration.md has the chain and what packaging it takes. +# NUMPY IS NOT IN THAT CLASS, and it is the one Depends here whose absence is +# not a missing feature but a dead printer. extras/stepper_resonance_tester.py +# opens with a bare `import numpy as np`, and klippy.py:122 walks every config +# section through a load_object that does not catch ImportError -- so with +# printer.base.cfg including FlashForge's printer.vibration.cfg +# ([stepper_resonance_tester]), a numpy-less install is a klippy that never +# reaches ready. Read the rest of this list as "klippy imports it"; read this +# one as "klippy does not start". docs/notes/44-vfa-calibration.md has the +# chain. # # markupsafe and setuptools are not named # because anvil-python-jinja2 declares the first and nothing needs the second. @@ -46,7 +47,7 @@ PKG_EXCLUDE=".version" # printer.cfg includes and every other config hangs off it. PKG_DEPENDS="anvil-klipper-config, anvil-python, anvil-python-cffi, anvil-python-greenlet, anvil-python-pyserial, - anvil-python-jinja2" + anvil-python-jinja2, anvil-python-numpy" PKG_BUILD_DEPENDS="" # NO PKG_WHEN. The fork is the only Klipper, and nothing here is optional -- diff --git a/pkgs/lib.sh b/pkgs/lib.sh index d2f346f..32f4f8d 100644 --- a/pkgs/lib.sh +++ b/pkgs/lib.sh @@ -197,7 +197,7 @@ pkg_begin() { # its project. See pkg_build. PKG_CONFIGURE='./configure'; PKG_CONFIGURE_AUTO=1 PKG_MAKE_TARGET=''; PKG_INSTALL_TARGET='install'; PKG_MAKE_ARGS='' - PKG_PY_SETUP_ARGS=''; PKG_CC_SHARED='' + PKG_PY_SETUP_ARGS=''; PKG_PY_PIP_ARGS=''; PKG_CC_SHARED='' return 0 } @@ -285,6 +285,25 @@ pkg_buildpython() { HOSTPY="$PKG_HOSTPY_ROOT/bin/python$PY_MM" export HOSTPY + # THE BUILD-PYTHON'S SCRIPT DIRECTORY GOES ON PATH, because two of the + # backends installed below are programs another backend EXECS rather than + # modules it imports. meson-python looks for a `meson` executable and + # numpy's meson.build does find_program('cython'); pip puts both in + # $PKG_HOSTPY_ROOT/bin, which nothing has ever had a reason to look in. The + # symptom without this is meson-python failing to install with + # `meson executable "meson" not found` while `pip list` shows meson right + # there -- installed, importable and unreachable. + # + # It also puts this python on PATH as python3, which is why it is done HERE + # and not in bin/common.sh: build-packages.sh runs each recipe as a child + # process, so an export in a recipe cannot reach the one bare `python3` in + # the build lane (build-packages.sh's opkg-index call) and cannot change + # what that resolves to. + case ":$PATH:" in + *":$PKG_HOSTPY_ROOT/bin:"*) ;; + *) PATH="$PKG_HOSTPY_ROOT/bin:$PATH"; export PATH ;; + esac + _hpstamp="$PY_VERSION" for _p in setuptools $PYPKG_HOST_LIST; do _hpstamp="$_hpstamp $(pypkg_var "$_p" FILE)" @@ -602,6 +621,74 @@ PYEOF pkg_say "$PKG_ID: sysconfig answers for mipsel, headers at $PKG_PYINC" } +# pkg_mesoncross -- write a meson cross file for this target and echo its path. +# After pkg_toolchain and pkg_pytarget. For meson-python packages only; a +# setuptools package needs none of this. +# +# WHY IT EXISTS AT ALL: _PYTHON_SYSCONFIGDATA_NAME is a distutils/setuptools +# mechanism. Meson has never heard of it and will not cross-compile without +# being told, in this file, what the host machine is. +# +# THE COMPILERS ARE THE WRAPPERS, NOT THE TOOLCHAIN'S OWN. $PKG_XW/bin holds +# three-line shims that add $PKG_CC_FLAGS -- `-EL -mnan=2008` -- and the raw +# mips-linux-gnu-gcc beside them does not. A cross file naming the raw binary +# produces a tree that builds perfectly, passes every test that only reads +# symbols, and is refused at exec() on the printer, which is the failure +# pkg_toolchain's own e_flags gate exists to prevent. Reaching past the wrapper +# here would walk straight around that gate. +# +# python = $HOSTPY IS NOT A MISTAKE. Meson's python module introspects an +# interpreter by RUNNING it and asking sysconfig, and pkg_pytarget has already +# made $HOSTPY's sysconfig answer for mipsel. So the host interpreter reports +# the target's EXT_SUFFIX and INCLUDEPY, which is exactly what meson needs and +# what pkg_pytarget's closing gate has already proved before we get here. +# +# longdouble_format IS REQUIRED AND HAS NO DEFAULT. numpy determines it by +# compiling and RUNNING a probe, which a cross build cannot do; without this +# property the configure step dies with "Cannot determine long double format". +# o32 MIPS has a 64-bit long double -- the same representation as double -- +# which is IEEE_DOUBLE_LE on a little-endian target. +pkg_mesoncross() { + [ -n "${PKG_XW:-}" ] || pkg_die \ + "$PKG_ID: pkg_mesoncross needs the toolchain -- call pkg_toolchain first" + [ -n "${PKG_PYSP:-}" ] || pkg_die \ + "$PKG_ID: pkg_mesoncross needs the target environment -- call pkg_pytarget first" + + _xf="$PWD/$PKG_WORK/meson-cross.ini" + cat > "$_xf" <&2 BECAUSE THIS FUNCTION'S STDOUT IS ITS RETURN VALUE. pkg_say prints to + # stdout like every other reporter here, and a recipe reads this as + # $(pkg_mesoncross) -- so an unredirected line would be captured into the + # path and handed to meson as part of a filename. + pkg_say "$PKG_ID: meson cross file at $_xf" >&2 + printf '%s' "$_xf" +} + # pkg_pysrc -- where this package's sdist unpacked to: the file # name with .tar.gz removed, under $PKG_WORK/src. pkg_pysrc() { @@ -615,6 +702,15 @@ pkg_pysrc() { # PKG_PY_SETUP_ARGS drive setup.py build_ext with these and then # bdist_wheel, instead of pip. PEP 517 offers no way to # pass build_ext options, and pillow's --disable-* are. +# PKG_PY_PIP_ARGS extra arguments for the pip path, unquoted as a list. +# For --config-settings, which is the ONE thing PEP 517 +# does standardise -- and the only way to hand +# meson-python a --cross-file. numpy's are. +# +# THE TWO OPTIONS ARE NOT ALTERNATIVES TO EACH OTHER. PKG_PY_SETUP_ARGS leaves +# the pip path entirely; PKG_PY_PIP_ARGS stays on it. Setting both is a recipe +# asking for setuptools and configuring meson, so it is refused below rather +# than silently resolved in favour of whichever branch is written first. # # NOTHING HERE TALKS TO A NETWORK: pip runs --no-index against a tree whose # sha256 bin/fetch-assets.sh checked, and --no-binary :all: guards against @@ -630,6 +726,9 @@ pkg_pywheel() { "$PKG_ID: $(pypkg_var "$_pw" FILE) did not unpack to $(basename "$_wdir")" [ -n "${PKG_PYSP:-}" ] || pkg_die \ "$PKG_ID: pkg_pywheel needs the target environment -- call pkg_pytarget first" + [ -z "${PKG_PY_SETUP_ARGS:-}" ] || [ -z "${PKG_PY_PIP_ARGS:-}" ] || pkg_die \ + "$PKG_ID: PKG_PY_SETUP_ARGS and PKG_PY_PIP_ARGS are both set, and they + drive different build paths -- setup.py or pip, not both" _wheels="$PWD/$PKG_WORK/wheels" rm -rf "$_wheels"; mkdir -p "$_wheels" _wlog="$PKG_LOG/wheel-$_pw.log" @@ -648,8 +747,12 @@ pkg_pywheel() { cp -a "$_wdir"/dist/*.whl "$_wheels/" \ || pkg_die "$PKG_ID: $_pw's setup.py produced no wheel -- see $_wlog" else + # Deliberately unquoted: PKG_PY_PIP_ARGS is a list of options, empty + # for every recipe that does not set it. + # shellcheck disable=SC2086 env "$@" "$HOSTPY" -m pip wheel --no-deps --no-build-isolation \ - --no-index --no-binary :all: --no-cache-dir -w "$_wheels" "$_wdir" \ + --no-index --no-binary :all: --no-cache-dir \ + ${PKG_PY_PIP_ARGS:-} -w "$_wheels" "$_wdir" \ > "$_wlog" 2>&1 || { printf ' !! %s: %s failed to build -- see %s\n' "$PKG_ID" "$_pw" "$_wlog" >&2 tail -25 "$_wlog" | sed 's/^/ /' >&2 diff --git a/qa/replica/test_install.py b/qa/replica/test_install.py index 3e692b5..82902b6 100644 --- a/qa/replica/test_install.py +++ b/qa/replica/test_install.py @@ -287,6 +287,52 @@ def test_the_config_include_set_is_wired_up(box): assert not missing, "included but not installed: %s" % missing +def test_every_include_resolves_not_just_ours(box): + """The other four. + + The test above deliberately reads only `ff-*.cfg`, which is the mod's own + set -- and for a long time that was the whole check, so the FOUR + FlashForge includes at the top of printer.base.cfg (printer.filament.cfg, + printer.probe.cfg, printer.mesh.cfg, printer.vibration.cfg) were never + asked about at all. They are not ours and never will be: they arrive from + the stock firmware and we merely include them. + + That is exactly why they need a gate. Klipper treats an [include] that + matches no file as FATAL, so a missing one is not a degraded printer, it is + a printer that does not start -- and it would be missing for a reason + nothing else here would notice, because no package of ours ships it and no + recipe of ours could. + + printer.vibration.cfg is the one with teeth: it declares + [stepper_resonance_tester], which is why anvil-klipper depends on + anvil-python-numpy at all. See qa/replica/test_klippy_extras_import.py and + docs/notes/44-vfa-calibration.md. + """ + base = box.file("/usr/data/config/printer.base.cfg") + assert base.exists, "no printer.base.cfg -- the mod's config is not wired up" + + included = set(re.findall(r"\[include\s+([^\]]+)\]", base.text)) + included = {name.strip() for name in included} + assert len(included) > 4, ( + "printer.base.cfg declares only %d includes (%s) -- that is fewer than " + "the four FlashForge ones alone, so this file is not the one the " + "printer reads" % (len(included), ", ".join(sorted(included)))) + + # `ls` rather than File.exists, because Klipper accepts a glob here and one + # of ours is a directory of per-model chamber configs. + missing = [] + for name in sorted(included): + listing = box.sh("ls -1 /usr/data/config/%s 2>/dev/null" % name) + if not listing.out.strip(): + missing.append(name) + assert not missing, ( + "printer.base.cfg includes %s, and no such file is in " + "/usr/data/config. Klipper treats a missing include as fatal, so this " + "printer would not start. If these are FlashForge's own configs, the " + "replica's /usr/data was seeded without them -- see " + "tools/replica/printer/seed-prog.sh." % missing) + + def test_the_user_printer_cfg_was_not_clobbered(box): """The one file on the machine that is the owner's, not ours.""" cfg = box.file("/usr/data/config/printer.cfg") diff --git a/qa/replica/test_klippy_extras_import.py b/qa/replica/test_klippy_extras_import.py new file mode 100644 index 0000000..6d783cc --- /dev/null +++ b/qa/replica/test_klippy_extras_import.py @@ -0,0 +1,206 @@ +"""Every klippy extra the shipped config names, imported on the interpreter that will import it. + +THE FAILURE THIS EXISTS FOR + +klippy moved from FlashForge's CPython 3.8.2 to our own 3.13, and 3.8.2's +rootfs had numpy where ours did not. `extras/stepper_resonance_tester.py` +opens with a bare `import numpy as np`; `printer.base.cfg` includes +FlashForge's `printer.vibration.cfg`, which declares +`[stepper_resonance_tester]`; and `klippy.py:122` walks EVERY config section +through a `load_object` whose `importlib.import_module` is not wrapped in +`except ImportError`. So the result was not a printer missing resonance +testing. It was a klippy that never reached ready. + +Two repo comments asserted the opposite -- that the module guarded its own +import and the printer simply carried on without it -- and both survived +review, because nothing anywhere asked the question this file asks. The +package list, the ipk contents and the ABI sweep all passed: the missing +dependency was not a broken file, it was a file that was never there. + +WHY THE REPLICA AND NOT qa/static + +The config that broke us is FlashForge's. `printer.vibration.cfg` is not in +this repo and never will be -- it arrives on the printer from the stock +firmware, and `printer.base.cfg:19` merely includes it. A static test could +only check the includes we ship, which are exactly the ones that were never +the problem. The real config graph exists on a real machine, so the question +gets asked there. + +WHY IT NEEDS NO MCU + +Importing is what fails here, and importing happens in `_read_config`, which +klippy runs BEFORE `klippy:mcu_identify`. The replica has no /dev/ttyS4,5,7 +and klippy can never reach ready on it (see test_s6rc.py) -- but it does not +need to. This walks the same graph klippy walks and imports the same modules +klippy imports, one step ahead of the hardware it does not have. + +WHAT IS DELIBERATELY NOT ASSERTED + +That a section maps to an extra AT ALL. Most do not: `[stepper_x]`, +`[printer]`, `[board_pins]` and every `[gcode_macro ...]` have no file under +extras/, and `load_object` returns None for them rather than failing. This +mirrors that -- a section with no module is skipped, exactly as klippy skips +it. What is asserted is the other half: where a module DOES exist, it must +import, because klippy has no path that tolerates one that does not. +""" +import re + +import pytest + +pytestmark = pytest.mark.replica + +MODDIR = "/usr/data/anvil" + +# Ours, not FlashForge's 3.8.2 -- anvil-env.sh exports this as FF_PYTHON, and +# etc/s6-rc/source/klipper/run execs it against the tree below. The whole point +# of this file is to ask about THIS interpreter's importable set. +PY = MODDIR + "/bin/python3.13" +ENV = MODDIR + "/anvil-env.sh" +KLIPPY = MODDIR + "/klipper/klippy" + +# What klippy is actually given on the command line. Not printer.base.cfg: +# printer.cfg is the user's file and the one that includes it, so starting +# anywhere else would walk a graph the printer does not have. +PRINTER_CFG = "/usr/data/config/printer.cfg" + +CONFIG_DIR = "/usr/data/config" + +_INCLUDE = re.compile(r"^\s*\[include\s+([^\]]+)\]\s*$", re.M) +_SECTION = re.compile(r"^\s*\[([^\]]+)\]\s*$", re.M) + + +def _read_graph(box, path, seen): + """Text of `path` and everything it includes, depth first. + + Klipper resolves an [include] relative to the directory of the file that + wrote it and accepts a glob. Every config on this machine sits directly in + /usr/data/config, so the directory is constant and the glob is expanded by + the shell below rather than reimplemented here. + """ + if path in seen: + return [] + seen.add(path) + f = box.file(path) + if not f.exists: + pytest.fail( + "%s does not exist, so klippy's config graph cannot resolve and " + "the printer would not start. An [include] naming a missing file " + "is fatal to Klipper." % path) + text = f.text + out = [(path, text)] + for spec in _INCLUDE.findall(text): + spec = spec.strip() + listing = box.sh("ls -1 %s/%s 2>/dev/null" % (CONFIG_DIR, spec)) + names = [ln.strip() for ln in listing.out.splitlines() if ln.strip()] + if not names: + pytest.fail( + "[include %s] in %s matches no file in %s -- Klipper treats " + "that as fatal, so this printer would not start." + % (spec, path, CONFIG_DIR)) + for name in names: + out.extend(_read_graph(box, name, seen)) + return out + + +@pytest.fixture(scope="module") +def box(printer): + """The installed machine, with the interpreter and the klippy tree.""" + for path in (PY, KLIPPY): + if not printer.file(path).exists: + pytest.fail( + "there is no %s, so there is nothing to ask this of -- " + "`make build` first." % path) + return printer + + +@pytest.fixture(scope="module") +def sections(box): + """Every section name in the real config graph, in first-word form. + + `[mcu eboard]` and `[gcode_macro FOO]` are prefix sections: klippy takes + the module name from the FIRST word and the rest is the instance name, so + that is what is collected here. + """ + names = set() + for _path, text in _read_graph(box, PRINTER_CFG, set()): + for raw in _SECTION.findall(text): + head = raw.split()[0].strip() + if head: + names.add(head) + assert names, ( + "no sections found anywhere under %s -- the graph walk read nothing, " + "so every assertion below would be vacuous" % PRINTER_CFG) + return sorted(names) + + +def test_the_config_graph_is_not_trivial(sections): + """Anti-vacuity. A walk that resolved one file and stopped would let the + real test pass while asking almost nothing.""" + assert len(sections) > 20, ( + "only %d config sections found (%s) -- printer.cfg's includes did not " + "resolve, so the import test below is not covering the machine" + % (len(sections), ", ".join(sections))) + + +def test_the_vibration_config_is_in_the_graph(sections): + """The specific section this file was written for. + + Pinned by name rather than left to the sweep, because it is the one whose + absence would make everything else here pass for the wrong reason: if + FlashForge's printer.vibration.cfg ever stops being included, the numpy + dependency stops being load-bearing and nobody would notice this test had + gone quiet. + """ + assert "stepper_resonance_tester" in sections, ( + "[stepper_resonance_tester] is not in the config graph. Either " + "printer.base.cfg no longer includes FlashForge's " + "printer.vibration.cfg, or that file has changed -- and if so, " + "anvil-klipper's Depends on anvil-python-numpy and " + "docs/notes/44-vfa-calibration.md both need re-reading.") + + +def test_every_named_extra_imports(box, sections): + """The gate. + + One interpreter start for the whole set, not one per module: each is a + qemu-mipsel start and the sweep is dozens of modules. The script reports + per-module rather than dying on the first failure, because "which ones are + broken" is the question a maintainer has when this goes red. + """ + script = r""" +. %(env)s +cd %(klippy)s +exec %(py)s - <<'PYEOF' +import importlib, os, sys +sys.path.insert(0, ".") +names = %(names)r +for name in names: + base = os.path.join("extras", name) + if not os.path.exists(base + ".py") and \ + not os.path.exists(os.path.join(base, "__init__.py")): + print("SKIP %%s (no module -- klippy's load_object returns None)" %% name) + continue + try: + importlib.import_module("extras." + name) + except BaseException as exc: + print("FAIL %%s %%s: %%s" %% (name, type(exc).__name__, exc)) + else: + print("OK %%s" %% name) +PYEOF +""" % {"env": ENV, "klippy": KLIPPY, "py": PY, "names": list(sections)} + + res = box.sh(script, timeout=600) + lines = [ln.strip() for ln in res.out.splitlines() if ln.strip()] + failed = [ln for ln in lines if ln.startswith("FAIL ")] + imported = [ln for ln in lines if ln.startswith("OK ")] + + assert imported, ( + "not one extra imported, so the interpreter or the tree is wrong " + "rather than any single module being broken.\nexit=%s\n%s\n%s" + % (res.code, res.out[-2000:], res.err[-2000:])) + assert not failed, ( + "%d klippy extra(s) named by the shipped config do not import on %s. " + "klippy.py:103 does not catch ImportError and klippy.py:122 loads " + "every section, so each of these is a printer that never reaches " + "ready -- not a missing feature:\n %s" + % (len(failed), PY, "\n ".join(failed))) diff --git a/qa/replica/test_numpy.py b/qa/replica/test_numpy.py new file mode 100644 index 0000000..5b45fe6 --- /dev/null +++ b/qa/replica/test_numpy.py @@ -0,0 +1,135 @@ +"""numpy on the printer's interpreter: does it import, and do the four things klippy asks of it work. + +WHY A SMOKE TEST AND NOT JUST AN IMPORT + +anvil-python-numpy is the one package in this feed built with meson-python +rather than setuptools, against a cross file this repo writes itself +(pkg_mesoncross), with `-Dallow-noblas=true` and `-Ddisable-optimization=true`. +Each of those is a place where a build can succeed, install cleanly, import +fine, and be wrong underneath: + + * allow-noblas selects numpy's bundled `lapack_lite` instead of a real + LAPACK. `np.linalg.lstsq` is the call that goes through it, and it is not + hypothetical -- `_r2_sinusoidal_fit` in extras/stepper_resonance_tester.py + makes it on every VFA calibration round. + * a cross build that silently produced host objects would fail at import, + which test_klippy_extras_import.py already covers -- but one that lost an + accelerated path while keeping the module would not. + +So this asks for numeric answers, not for a module object. The four calls +below are exactly the ones the calibration path makes: rfft and hanning in +`harmonic_extract_using_fft`, polyfit in `parabolic_interpolation`, lstsq in +`_r2_sinusoidal_fit`. + +WHY NOT THE ABI + +test_abi.py already sweeps every ELF object on the filesystem, so numpy's +extensions are covered there the moment they ship, and duplicating it here +would only make that sweep's failures arrive twice. This is the other half: +the ABI test proves the kernel will load them, this proves they compute. +""" +import pytest + +pytestmark = pytest.mark.replica + +MODDIR = "/usr/data/anvil" + +# Ours, not FlashForge's 3.8.2. FlashForge's rootfs HAD a numpy; that one is +# built for 3.8 and is not importable here, which is the whole reason this +# package exists. +PY = MODDIR + "/bin/python3.13" +ENV = MODDIR + "/anvil-env.sh" + + +@pytest.fixture(scope="module") +def box(printer): + """The installed machine, with the interpreter this asks about.""" + if not printer.file(PY).exists: + pytest.fail( + "there is no %s, so there is no interpreter to ask -- " + "`make build` first." % PY) + return printer + + +def _py(box, body, timeout=300): + return box.sh(". %s\nexec %s - <<'PYEOF'\n%s\nPYEOF\n" % (ENV, PY, body), + timeout=timeout) + + +def test_numpy_imports_at_all(box): + """The dependency anvil-klipper declares, present on the machine. + + Separate from the maths below so a missing package reads as a missing + package rather than as four failed calculations. + """ + res = _py(box, "import numpy; print(numpy.__version__)") + assert res.ok, ( + "numpy does not import on %s, so klippy cannot start: klippy.py:122 " + "loads [stepper_resonance_tester] and klippy.py:103 does not catch " + "ImportError.\nexit=%s\n%s\n%s" + % (PY, res.code, res.out[-2000:], res.err[-2000:])) + assert res.out.strip(), "numpy imported but reported no version" + + +def test_the_four_calls_the_calibration_makes(box): + """rfft, hanning, polyfit, lstsq -- with answers checked, not just no traceback. + + The values are the ones a correct numpy must produce, computed against + closed forms rather than against a recorded run of this same code, so a + subtly wrong build cannot agree with its own output. + """ + body = r''' +import numpy as np + +fails = [] + +def check(name, got, want, tol=1e-6): + if abs(got - want) > tol: + fails.append("%s: got %r, want %r" % (name, got, want)) + +# rfft on a pure tone: a 16-sample signal at exactly bin 3 must put all its +# energy in that bin and (near) none elsewhere. +n = 16 +sig = np.cos(2 * np.pi * 3 * np.arange(n) / n) +spec = np.abs(np.fft.rfft(sig)) +check("rfft peak bin", float(np.argmax(spec)), 3.0) +check("rfft peak amplitude", float(spec[3]), n / 2.0, 1e-9) + +# hanning: a symmetric window, zero at both ends, 1.0 in the middle for odd N. +w = np.hanning(9) +check("hanning[0]", float(w[0]), 0.0, 1e-12) +check("hanning[-1]", float(w[-1]), 0.0, 1e-12) +check("hanning mid", float(w[4]), 1.0, 1e-12) + +# polyfit deg 2 on points taken FROM a known parabola: 2x^2 - 3x + 1. +x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0]) +y = 2 * x ** 2 - 3 * x + 1 +c = np.polyfit(x, y, 2) +check("polyfit a", float(c[0]), 2.0, 1e-6) +check("polyfit b", float(c[1]), -3.0, 1e-6) +check("polyfit c", float(c[2]), 1.0, 1e-6) + +# lstsq: THE lapack_lite CALL. Overdetermined exact-fit system, y = 2a + 3b. +A = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) +rhs = np.array([2.0, 3.0, 5.0]) +sol, _res, _rank, _sv = np.linalg.lstsq(A, rhs, rcond=None) +check("lstsq x0", float(sol[0]), 2.0, 1e-6) +check("lstsq x1", float(sol[1]), 3.0, 1e-6) + +if fails: + print("FAILURES") + for f in fails: + print(" " + f) +else: + print("ALL OK") +''' + res = _py(box, body) + assert res.ok, ( + "the numpy smoke test did not run to completion on %s.\nexit=%s\n%s\n%s" + % (PY, res.code, res.out[-3000:], res.err[-3000:])) + assert "ALL OK" in res.out, ( + "numpy is installed and imports, but computes wrong answers -- so the " + "cross build produced something that loads and does not work. lstsq " + "failing on its own points at -Dallow-noblas and the bundled " + "lapack_lite; the rest points at the cross file.\n%s" + % res.out[-3000:]) diff --git a/qa/static/test_recipe_layout.py b/qa/static/test_recipe_layout.py index a2e8eb1..fc61345 100644 --- a/qa/static/test_recipe_layout.py +++ b/qa/static/test_recipe_layout.py @@ -86,3 +86,43 @@ def test_recipe_names_are_unique_across_the_two_levels(): assert not dupes, ( "recipe name at both levels: %s -- pkg_dir would silently pick one" % ", ".join(dupes)) + + +def test_klipper_still_depends_on_numpy(): + """The one Depends in this repo whose absence is a printer that will not boot. + + Every other name in anvil-klipper's Depends means "klippy imports this". + numpy means "klippy does not start": extras/stepper_resonance_tester.py + opens with a bare `import numpy as np`, klippy.py:122 walks EVERY config + section through a load_object, and klippy.py:103 does not catch + ImportError -- so with printer.base.cfg including FlashForge's + printer.vibration.cfg ([stepper_resonance_tester]), dropping this line does + not lose a feature, it bricks the boot. + + It is asserted HERE, statically, because the replica gates that would catch + it (qa/replica/test_klippy_extras_import.py, test_numpy.py) need an image + baked and a container up, and this one is a grep. It is the cheapest gate + on the most expensive mistake, and the mistake has already been made once: + two comments in this repo asserted the module guarded its own import, and + the dependency did not exist at all until the note that disproved them. + + See docs/notes/44-vfa-calibration.md. + """ + conf = os.path.join(ROOT, "pkgs", "klipper", "pkg.conf") + assert os.path.isfile(conf), "pkgs/klipper/pkg.conf is gone" + text = open(conf, encoding="utf-8").read() + + # The recipe it must name has to be a recipe, or the Depends resolves to + # nothing at install time and opkg refuses the package. + assert any(n == "python-numpy" for n, _ in recipes()), \ + "no python-numpy recipe, so anvil-klipper's Depends names a package " \ + "no feed of ours builds" + + depends = "".join( + line for line in text.splitlines(keepends=True) + if not line.lstrip().startswith("#")) + assert "anvil-python-numpy" in depends, ( + "anvil-klipper no longer depends on anvil-python-numpy. That is not a " + "lost feature -- klippy loads [stepper_resonance_tester] from " + "FlashForge's printer.vibration.cfg and dies on the ImportError, so " + "the printer never reaches ready.") diff --git a/versions.env b/versions.env index 12981cc..751e072 100644 --- a/versions.env +++ b/versions.env @@ -109,7 +109,7 @@ EXPAT_SHA256="fd03b7172b3bd7427a3e7a812063f74754f24542429b634e0db6511b53fb2278" PYPKG_LIST="distro inotify-simple libnacl dbus-fast preprocess-cancellation pyserial pycparser smart-open jinja2 setuptools zipp importlib-metadata tornado markupsafe cffi greenlet lmdb - streaming-form-data pillow" + streaming-form-data pillow numpy" PYPKG_DISTRO_FILE="distro-1.9.0.tar.gz" PYPKG_DISTRO_SHA256="2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed" @@ -174,12 +174,32 @@ PYPKG_STREAMING_FORM_DATA_SHA256="8172dd509a42ac0d1033deb1d13acb68f0cbae7e149225 PYPKG_PILLOW_FILE="pillow-11.0.0.tar.gz" PYPKG_PILLOW_SHA256="72bacbaf24ac003fea9bff9837d1eedb6088758d41e100c1552930151f677739" +# THE ONE PACKAGE HERE KLIPPY CANNOT START WITHOUT, which is not what a reader +# expects of numpy. extras/stepper_resonance_tester.py opens with a bare +# `import numpy as np`, klippy.py:122 walks EVERY config section through a +# load_object that does not catch ImportError, and printer.base.cfg includes +# FlashForge's printer.vibration.cfg, which declares [stepper_resonance_tester]. +# Without this package that is a klippy which never reaches ready -- not a +# missing feature. See docs/notes/44-vfa-calibration.md. +# +# 2.1.3 AND NOT THE NEWEST. 2.1 is the first release with CPython 3.13 support, +# so there is no going below it; staying at the bottom of that range is +# deliberate, because every release since adds SIMD dispatch machinery that has +# no MIPS path and only more ways for a gcc 7.2 to refuse. Cython below is held +# at 3.0.x for the matching reason. +# +# It also builds with meson-python rather than setuptools, which is why +# PYPKG_HOST_LIST grew four entries and pkgs/lib.sh grew pkg_mesoncross. +PYPKG_NUMPY_FILE="numpy-2.1.3.tar.gz" +PYPKG_NUMPY_SHA256="aa08e04e08aaf974d4458def539dece0d28146d866a39da5639596f4921fd761" + # Host-side PEP 517 backends. THESE NEVER SHIP: pkg_pywheel builds with # --no-build-isolation, so each must already be in work/.py-host. # ORDER IS INSTALL ORDER: pkg_buildpython pip-installs each with --no-deps # against no index, so a backend must come after whatever builds it. flit-core # builds packaging; setuptools_scm imports packaging at build time. -PYPKG_HOST_LIST="flit-core poetry-core packaging setuptools-scm" +PYPKG_HOST_LIST="flit-core poetry-core packaging setuptools-scm + meson pyproject-metadata meson-python cython" PYPKG_FLIT_CORE_FILE="flit_core-3.10.1.tar.gz" # jinja2's backend PYPKG_FLIT_CORE_SHA256="66e5b87874a0d6e39691f0e22f09306736b633548670ad3c09ec9db03c5662f7" PYPKG_POETRY_CORE_FILE="poetry_core-1.9.1.tar.gz" # libnacl's, dbus-fast's @@ -189,6 +209,33 @@ PYPKG_PACKAGING_SHA256="c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260 PYPKG_SETUPTOOLS_SCM_FILE="setuptools_scm-8.1.0.tar.gz" # zipp's, importlib's PYPKG_SETUPTOOLS_SCM_SHA256="42dea1b65771cba93b7a515d65a65d8246e560768a66b9106a592c8e7f26c8a7" +# --- numpy's four, and the order among them is not alphabetical by accident. +# +# meson-python IS SELF-HOSTING: its pyproject declares backend-path = ['.'] and +# builds itself with the mesonpy in its own tree, so installing it RUNS meson +# and ninja. That is why ninja-build is in docker/Dockerfile.build and why +# these three must precede it -- meson because mesonpy execs it, +# pyproject-metadata and packaging because mesonpy imports them. Reorder this +# and pkg_buildpython fails on a backend that cannot build itself, which reads +# as a network problem and is not one. +# +# meson FROM PyPI AND NOT FROM apt, unlike ninja. Debian bookworm ships 1.0.1 +# and numpy's meson.build asks for >=1.2.99, so the distro's is simply too old +# -- and a vendored sdist is a pinned hash where an apt package is a moving +# target. ninja earns the exception the way libarchive-dev does: it is a +# program meson execs, not a module anything imports, and no pin of ours would +# change what it emits. +PYPKG_MESON_FILE="meson-1.7.0.tar.gz" +PYPKG_MESON_SHA256="08efbe84803eed07f863b05092d653a9d348f7038761d900412fddf56deb0284" +PYPKG_PYPROJECT_METADATA_FILE="pyproject_metadata-0.9.0.tar.gz" +PYPKG_PYPROJECT_METADATA_SHA256="8511c00a4cad96686af6a6b4143433298beb96105a9379afdc9b0328f4f260c9" +PYPKG_MESON_PYTHON_FILE="meson_python-0.17.1.tar.gz" +PYPKG_MESON_PYTHON_SHA256="efb91f69f2e19eef7bc9a471ed2a4e730088cc6b39eacaf3e49fc4f930eb5f83" +# HELD AT 3.0.x DELIBERATELY: Cython 3.1 does not build numpy below 2.2, and +# numpy is pinned at 2.1.3 above. numpy's meson.build wants >= 3.0.6. +PYPKG_CYTHON_FILE="cython-3.0.11.tar.gz" +PYPKG_CYTHON_SHA256="7146dd2af8682b4ca61331851e6aebce9fe5158e75300343f80c07ca80b1faff" + # dlopened by libnacl, so it must be shared -- our one .so in $MODDIR/lib. SODIUM_VERSION="1.0.20" SODIUM_SHA256="ebb65ef6ca439333c2bb41a0c1990587288da07f6c7fd07cb3a18cc18d30ce19"