mirror of
https://github.com/Klipper4FlashForge/firmware.git
synced 2026-10-18 19:39:45 -05:00
master
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleksandr Shyshatskyi
|
80a4d64ffd |
Prepare next firmware release
Add Fluidd and custom boot scripts, migrate DC24V control, update the web UIs and filament macros, improve startup status reporting, and expand upgrade and replica coverage. |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f8b925e48f |
Replace opkg with apk, and sign the feed
docs/notes/85-packaging.md chose opkg in 2026-08 on one argument: every package
is cross-built here anyway, so the winner is whichever format is cheapest to
PRODUCE -- and .ipk had an upstream tool where .apk meant abuild, a keypair and
an Alpine host. Two lines of that comparison were wrong.
apk mkpkg is an applet of apk itself: -I key:value metadata, -F directory, no
keypair, no abuild, no Alpine host. The "making a package" row described
apk-tools 2. And the on-device client was counted a wash when it is better than
one -- openssl and zlib were already cross-built for CPython, and libarchive
existed in this tree only for opkg, so adopting apk REMOVES a cross-build.
What made it possible at all is that apk resolves its database relative to
--root and uses the same root as the file destination, while / here is a
read-only squashfs -- measured: neither /lib/apk nor /etc/apk can be created.
pkgs/3rdparty/apk-tools/prefix.patch splits the two. Upstream already carries
root_fd and dest_fd separately and merely aliases them in context.c, so the
patch defaults the database root to $MODDIR and leaves files extracting at /.
That is opkg's compile-time VARDIR plus `dest root /`, in apk's spelling.
THE FEED IS SIGNED NOW, which it never was: opkg signed the index and
bin/build-packages.sh signed nothing, so nothing was verified on a printer.
apk verifies with raw PEM public keys, not X.509 -- measured on the machine,
where every certificate directory is empty. ./bin/apk-keygen.sh makes an RSA
pair, APK_SIGN_KEY in config.env points at the private half, and the public
half both seeds the USB stick (so the first install can verify) and ships in
anvil-core (so the printer keeps trusting). An empty APK_SIGN_KEY still builds
an unsigned feed, deliberately: `make packages` has to run on a bare checkout
or the packaging lane stops being a CI gate.
WHAT IT COST is a carried patch, against an internal with one consumer, on a
branch mid v2->v3 transition, on an architecture with no upstream apk port to
share the rebasing. That is the standing risk and it does not go away.
Four bugs this shook out, all silent, all found by a gate rather than by
reading:
* The prune deleted seventeen packages it had just built. The all -> noarch
mapping went into the emitter and not into pkg_pkgfile, so the prune looked
for a name nothing wrote. `make packages` exited 0.
* Whitespace-separated Depends dropped nineteen packages. opkg's pkg_parse.c
splits that field on ',' and nothing else, so the whole list became one
dependency whose name contained spaces and resolved to nothing -- and opkg
neither warned nor failed. `make build` shipped a payload with no Tornado
in it. test_the_payload_is_dependency_closed is what catches that now.
* Ownership was recorded as `nobody`. apk resolves an owner to a NAME against
<--root>/etc/passwd and the build lane runs --user with no passwd entry, so
every file in every package would have installed as uid 65534, with no
error at build or install time. build-packages.sh writes a two-line
synthetic passwd naming the build user root and points --root at it.
* Four recipes carried versions apk cannot parse -- two spelled a commit
+git<sha> where the grammar's hash slot is ~, x264 had a snapshot name
whose -2245 reads as a malformed release, and helixscreen had
0.99.115.creator5.3 where apk allows one trailing letter and then only a
suffix from a closed set of nine words. All four were valid opkg versions,
which is why nothing had noticed. pkg_version_ok gates every recipe now.
A v3 .apk begins with ADBd magic and is not a tar stream, so there is no apk
equivalent of the `ar -p | gunzip | tar` that hand-unpacked anvil-opkg: you
cannot open a package without a package manager. bin/build-payload.py puts the
cross-built binary on the stick at /mnt/apk instead. It is the same build
anvil-apk-tools contains, that package installs normally, and the bootstrap
copy never lands.
Two gates retired rather than moved. apk's installed database is plain text
(F: sets a directory, R: names a file in it), so the payload-ownership check
reads one file where it used to glob info/*.list. And it carries no clock at
all -- apk records no install time and mkpkg sets no build-time -- so the
Installed-Time normalisation case-build-payload.sh did on every build is gone,
and the test asserts the ABSENCE of a t: line instead. Clock-free by
construction rather than by repair.
Also removed: PKG_SECTION, which 45 recipes declared and nothing read once
apk's schema had no field for it, and the `make apk-host` target -- the
packager is built on demand by bin/build-packages.sh and stamped on the pin,
the way pkg_toolchain and pkg_buildpython already work.
Verified from a cold work/host and work/packages: 47 packages built and signed,
`make build` installs 35 and verifies them against the shipped public key,
`make qa-static` 195 passed, `make qa-replica` 135 passed -- the printer
installing off a simulated USB stick and booting. The installed set was checked
against the declared closure of the eleven roots: 35 expected, 35 installed,
nothing extra and nothing missing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013rbKEyLXrzSE6wm3iNApVp
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
9ef21c1fb8 |
wpa_supplicant is a supervised longrun, not a oneshot that forks
The wifi oneshot backgrounded a subshell and returned at once, so that the 15s wait for wlan0 and the 20s wait to associate did not stall the boot transition firmwareExe blocks on. Its comment was explicit that nothing could depend on it, which was true and was not the problem. The problem is the subshell it left behind: a child of s6rc-oneshot-runner with its stdout on a pipe s6-rc closes the moment the up script returns, so a sleep in the wlan0 wait followed by an echo is a write to a closed pipe. OBSERVED on a printer: a healthy supervision tree, s6-rc reporting wifi up, no wlan0 configured, no lease, and not one line anywhere saying why -- HelixScreen calling it "wifi system is not ready" because there was no supplicant holding a control socket. Getting in at all needed a repacked .tgz that brought wifi up ahead of s6. The subshell was never caught in the act and could not be: once its pipe is gone it cannot report its own death. Which is the argument, whatever the exact signal was. A oneshot that forks and returns cannot be relied on and cannot be debugged when it isn't. So: type longrun, no -B, exec wpa_supplicant in the foreground. Blocking IS the job now, s6-supervise holds the process, and a supplicant that dies is restarted rather than mourned. up/down are gone -- a longrun's down is silently dropped anyway. wifi-dhcp is new and is not optional. wpa_supplicant only associates; it never asks for an address, and running udhcpc was the oneshot's last act. bin/wifi-action.sh has shipped since the s6 migration for exactly this hook and was invoked by nothing at all -- `wpa_cli -a` is what invokes it. That also fixes what the oneshot never covered: it ran udhcpc ONCE after boot, so an owner joining a network from the HelixScreen panel an hour later was associated with no address, which the panel reports as a connection timeout. Its run script re-runs the action script when wpa_state is already COMPLETED. wpa_cli fires on CHANGES, and it is not established that it emits CONNECTED on attach to an already-connected supplicant, so without this the lease that finish drops would not come back on a restart. The action script replaces any previous client first, so it is idempotent with a real event. udhcpc keeps -b inside the action script. Different flag, different program: wpa_cli blocks until the script returns, so a foreground udhcpc would wedge event processing. ntp/run reasoned about "the wifi oneshot" and about nothing being allowed to depend on it; it still takes no wifi dependency, now because wifi being up means a supplicant is running rather than that anything is routable. The service tables in how-it-works.md and building.md, and test_s6rc.py -- where wifi sat in HW_SET as "the oneshot above" and test_execline_shipped_too used it as its example -- follow. Both wifi and wifi-dhcp stay out of BOOT_SET: they want a real wlan0, which the replica has not got. qa-static passes. The database is NOT compiled here -- s6-rc-compile is a MIPS binary that runs in the replica -- so make qa-replica is still owed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QPQqLMitZzrRcaehiYLHdx |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
d898d105e0 |
The Moonraker pin was frozen by a missing _sqlite3 we now build ourselves
MOONRAKER_VERSION was 9d0d09d, the last commit before 80c7620 moved the
database from lmdb to sqlite, because FlashForge's python 3.8.2 has no
_sqlite3 and every release after it died loading the database component.
This repo builds and ships its own CPython 3.13 now, and it has one. The pin
is v0.11.0 -- a release tag, not a sha, so PKG_VERSION is just 0.11.0 and no
longer the Debian-style 0.8.0+git<sha> snapshot string. GitHub strips the
leading "v" from the directory inside /archive/<tag>.tar.gz, which is what
${MOONRAKER_VERSION#v} in the recipe is for.
The dependency set is the part that actually moved. Read against 0.11.0's
CORE_COMPONENTS plus the sections moonraker.conf enables:
dbus-next became dbus-fast. dbus_manager is a core component and imports
it at module scope, so this is not optional whatever [machine] provider
says. Its sdist cythonises nine modules; SKIP_CYTHON=1 ships the python
they were generated from and keeps the package ARCH all. The wheel still
comes out platform-tagged -- poetry-core marks the distribution impure
for declaring a build script at all -- but it holds 29 .py and no .so,
and the content is what ships.
importlib-metadata is new, imported at module scope by
utils/source_info.py, and brings zipp. Both build through setuptools_scm,
so PYPKG_HOST_LIST grew packaging and setuptools-scm. That list is now
install order: pkg_buildpython pip-installs each with --no-deps against
no index, flit-core builds packaging, setuptools_scm imports it.
pyserial-asyncio is gone. Nothing in 0.11.0 imports serial_asyncio;
utils/async_serial.py drives pyserial over asyncio itself. pyserial stays
-- klippy needs it and declares it.
LMDB STAYS, for a reason that is not the old one. 0.11.0 converts an
existing data.mdb to sqlite on first start and imports lmdb to read it. With
the module missing it tries to pip install one, and a printer has no index
to install from -- the old database would be dropped in silence. It is the
upgrade path now, not the database, and the comment on WANTED_MODULES says
so.
inotify-simple STAYS AT 1.3.5 though 0.11.0 pins ==2.0.1: 2.0.1 declares its
licence the PEP 639 way and needs setuptools>=77 to build, against the 75.6.0
pinned here for lmdb's sake. It fails at metadata generation, not at import,
which is the kind of thing that reads as a broken recipe later. 1.3.5 carries
every name 0.11.0 asks of it, and moonraker.conf sets file_system_observer:
none, so the import is all that is asked. jinja2 goes to 3.1.6 to match.
Not taken: paho-mqtt, zeroconf, apprise, ldap3 and python-periphery, whose
components this printer does not enable, and msgspec and uvloop, which
Moonraker imports under contextlib.suppress(ImportError).
Verified: every changed sha256 downloads and matches at the URL
fetch-assets.sh builds; all five affected wheels build offline
(--no-index --no-build-isolation) on a real 3.13 with the host backends
installed in the declared order; qa/static 122 passed, including the gates
that cross-check PYPKG_LIST against the recipes and every PKG_DEPENDS
against the feed. The mipsel cross-build and `make qa-replica` have not run
yet, and versions.env still says a bump is gated on the latter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012oRdgKM7scrEsr6waAByaY
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
bf7d0f7e06 |
The printer has no clock and nothing was setting it
There is no RTC and no battery, so every boot starts at whatever the kernel was built with and stays there: job timestamps, log ordering and certificate validity are all wrong on a machine that has been unplugged. Nothing on the stock rootfs could fix it -- FlashForge's busybox is v1.31.1 built without the ntpd applet, and its rdate speaks RFC 868. anvil-sntpd is troglobit's sntpd, a fork of Doolittle's ntpclient: one small binary, servers and interval on the command line, so there is no config file, no privsep user and no chroot to place on a read-only rootfs. It ships $MODDIR/sbin/sntpd and nothing else. A LONGRUN, NOT A HOOK ON THE WIFI ONESHOT. That oneshot runs once, at boot, and returns before wlan0 has associated -- so a sync hung off its end would fire too early, never retry, and never fire at all for the owner who joins a network from the HelixScreen panel an hour later. And with no RTC a clock is not set once but maintained. sntpd keeps its schedule with the network down instead of exiting, which is what lets this service need no wifi dependency -- just as well, since nothing may depend on that oneshot. The poll interval is 300 rather than upstream's 600 because it is also the ceiling on the down-network retry backoff, and so bounds how long a printer that joined wifi late keeps the wrong time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0149y4iyLpEwYKh2PS4fmf4u |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b98552386b |
No FlashForge component ships; patch.sh is payload.sh
The software component existed to give the mod one exec hook. It is not needed for that any more, so a release now carries our installer and our payload and nothing else: 26MB smaller, 21MB of which was FlashForge's own firmwareExe travelling back to the partition it was already on, along with their klippy tree, their wifi module and their configs. WHAT IT GIVES BACK. /usr/prog/PROGRAM/software is no longer rewritten, so the version directory there stays FlashForge's with their genuine firmwareExe inside it -- and app_startup.sh's five-second watchdog, which copies one out of that directory when nothing by that name is running, has something to copy again. A printer whose payload fails to install now comes up on the stock UI instead of a blank screen. That recovery has been broken since we started deleting their binary out of a component we shipped; docs/how-it-works.md claimed it worked throughout. THE ROOT PASSWORD moves with it, because it used to ride in the component's shadow file. bin/pack.sh bakes ROOT_PW_HASH into the installer, single-quoted -- an unquoted $6$... would have the printer's shell expand $6 as a positional parameter and ship a truncated hash -- and the installer writes it to /usr/prog/etc/shadow. The random-password path asks one question instead of four: FlashForge ship the same root hash on every machine, so a live shadow still carrying it means nobody has set a password and it is ours to set. The .prev-root-hash recording, the shipped-vs-live comparison and the restore-it- afterwards step were all machinery for surviving a file that nothing overwrites any more, and are gone. pack.sh checks the hardcoded constant against the stock package it built from, because getting it wrong is silent and total: every printer would read as "already set" and none would ever get a password. bin/patch.sh is bin/payload.sh. It patched nothing any more -- no splice, no shadow edit, no firmwareExe deletion -- and what is left is the payload: the feed installed by the printer's own opkg, the seeded config, the manifest. Its sections are renumbered 1-5 with no gaps, and the five tombstone comments marking where removed sections used to be are gone with them. test_recipe_layout.py loses six of its nine tests. Each compared the tree against a description of the tree -- no fourth directory kind, each kind still used, 3rdparty carries no files of ours, installer/ holds exactly two named files -- and none could go red for a reason a printer would notice, while all six went red the moment the layout legitimately changed. The two that bite are kept: a payload/ path escaping $MODDIR, and a recipe name at both levels, which makes pkg_dir silently build the wrong package. SW_VER is deleted from config.env.example and both workflows: it stamped a component that is not built. Gates: 112 static tests pass. The generated installer is verified after substitution -- all four lines present, and `sh -n` over the result, because a hash with a stray character in it would otherwise reach a printer as a syntax error whose only symptom is a package that installs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
2a6331a831 |
We own runFirmwareExe.sh; the install is no longer spliced into run.sh
app_startup.sh runs whatever it finds under that name in the package it just decrypted. That is the entire contract -- plus the exit code, where 0 means "flashed, power-cycle me" and non-zero means "carry on booting" -- so owning the name is all it takes to own the install. We were not using it. We shipped FlashForge's copy verbatim and reached the printer through the one exec hook their file offers: the software component's run.sh, into which bin/patch.sh spliced two marked blocks with 40 lines of regex. installer/runFirmwareExe.sh is that file now. run-pre.sh and run-append.sh are folded into it unchanged in substance -- the manifest delete, the HelixScreen settings rescue, the s6 scandir sweep, the config merge, the password dance are all the same code -- and the two orderings that mattered are kept explicitly rather than by where a regex put them: the backups run BEFORE any component is installed, because the software component ships /usr/prog's shadow and passwd and its run.sh copies them over the live ones. The four component installers become one function. Their semantics are kept exactly, including that a component failing md5sum.list is discarded rather than half-installed, and that every previous version directory is wiped before the new one lands. What is NOT kept is the third of that file which never ran: create_key(), defined and never called, and built its paths without a slash so it would have written /usr/progprivate.pem; SSID_NAME/SSID_PASS, assigned and never read; the commented-out kernel version gate; the screwflag factory bypass of the model check, which we ship no screwflag for. bin/pack.sh stages it, rewriting MACHINE=, PID= and MOD_PW_AUTO= and then GREPPING for all three -- a package whose gate still said Creator5Pro because a sed missed would install on the wrong machine. MOD_PW_AUTO is 0 in the checkout, which is the safe default and not an arbitrary one: test_upgrade.py runs the unsubstituted file, and 1 would have it reroll the replica's root password and drop a password file on /mnt every run. The software component still ships, unchanged, and its run.sh is left exactly as FlashForge wrote it. Dropping it is the next commit; this one moves the vehicle without changing what reaches /usr/prog. Gates: qa/static/test_shell_syntax.py already covered installer/*.sh under both bash -n and the printer's ash plus shellcheck, so the new file was checked the day it landed -- 91 pass. test_upgrade.py runs the real installer over two synthetic payloads and is repointed at it. Two doc claims corrected while their sections were open, both false before this change rather than because of it: how-it-works.md said app_startup.sh's firmwareExe recovery "still works" -- it cannot, because it copies out of the version directory and ours has had firmwareExe deleted from it -- and it described a Moonraker preflight-and-rollback that went with the /usr/prog copy; moonraker-preflight.py exists nowhere in the tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CH7Yb4wuPhnppHAB25UX7e |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
fbb3182e68 |
bin/verify.sh is retired; four of its forty checks move to qa/replica
verify.sh described itself as "simulate every check the printer performs,
against a built package". The word carrying that sentence is SIMULATE, and it
is why thirty-six of the forty are deleted rather than moved.
THE REPLICA LANE DOES NOT SIMULATE THE INSTALL. install-package.sh puts the
package on a genuine FAT filesystem at /dev/sda1 and runs the machine's own
/usr/prog/app_startup.sh over it, verbatim, under qemu. So these are
PRECONDITIONS of the lane now rather than assertions in it -- break any and
installed_image() raises with the install log and the whole lane goes red:
* the decrypt with the firmware key (the printer's unTar does it)
* runFirmwareExe.sh present, components plain tar not xz (bare `tar -xvf`)
* md5sum.list verifying (the installer aborts and deletes the payload)
* the MACHINE= gate (runFirmwareExe.sh refuses a foreign package)
* the /mnt/<Model>-*.tgz glob (app_startup.sh finds the file by it)
A host-side re-reading of those rules is a second implementation, and a second
implementation can agree with itself while disagreeing with the printer. That
is the defect qa/replica/conftest.py records this repo learning twice about
hand-placed payloads.
A second block was asking what the suite already answers, of an installed
filesystem rather than of a tar listing: the klippy tree and c_helper.so
(test_install, test_abi), the s6-rc database and the oneshot runner's execline
(test_s6rc), s6 and s6-ftrigrd (test_supervisor), the .install-manifest
(test_upgrade), libsodium's bare .so symlink and the chamber configs
(static/test_ipk), and every shipped script's syntax
(static/test_shell_syntax).
THE FOUR THAT MOVED are the ones nothing else asks, and each is stronger for
the move -- a `tar -t` listing cannot tell you that a shared object loads, or
what a name resolves to on a PATH:
* the interpreter IMPORTS sqlite3, lmdb and _cffi_backend, rather than a
grep finding three .so files. This is what catches a cross build that
resolved to an x86_64 manylinux wheel: present on disk, raises on import.
* nothing in $MODDIR/bin shadows FlashForge's python3 -- asked with
`command -v` after sourcing anvil-env.sh, which is the file that PREPENDS
$MODDIR/bin to PATH and so creates the hazard in the first place.
* CPython's dev half (headers, pkgconfig, config-3.13-*) did not ship.
* the ship boundary: no file on the installed printer is byte-identical to
one under bin/ or docker/. Asked of the machine rather than of the .tgz,
so it also sees whatever a maintainer script wrote as root at install
time. The sweep filters on exact size before hashing, which is what keeps
a walk of /usr/prog through qemu cheap.
WHAT GENUINELY LEFT, recorded in docs/qa-migration.md rather than only here:
there is no longer a check you can run on the build host, in seconds and
without docker, against a package you are about to flash. Nothing warns you
before you copy the file to a stick that it was built from the wrong model's
stock package. The printer still refuses it and `make qa-replica` still
catches it, but the fast host-side warning the hardware checklist used is
gone. `make verify` is removed and `make release` no longer calls it.
VERIFIED. All four ran against a real replica (the cached installed image,
PRINTER_IMAGE=monstrofil/creator5-printer:latest): 4 passed in 17s. Then each
was DRIVEN RED on purpose with a throwaway module, which is this repo's own
standard from
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
214a4e58b9 |
anvil-core owns firmwareExe and start.sh; kill the 3.8.2 myth
Three things, all downstream of klippy having moved to $MODDIR and our own
CPython 3.13.
SECTIONS 8 AND 9 ARE GONE. Both files install at $MODDIR/prog/ and
anvil-link-prog.sh points the stock paths at them -- from run-append.sh on a
flash and from anvil-core's postinst on `opkg upgrade anvil-core`. Nothing
execs either between run.sh distributing the component and link-prog running,
so the component copy bought no ordering. It cost, though: our firmwareExe
with no payload behind it logs "no anvil-env.sh", finds no s6-svscan and then
`while true; do sleep 3600; done` -- a printer that boots to a blank screen for
ever and never reaches app_startup.sh's own recovery. Absent, that recovery
restores a firmwareExe from a version directory. So patch.sh now REMOVES both
from the component: the stock component carries them, and deleting the two cp
lines alone would have shipped FlashForge's originals.
SECTION 5c IS GONE. It did no work -- two du prints and a comment whose first
line was the falsehood below.
THE FALSEHOOD. "klippy runs on the stock 3.8.2, started by
/usr/prog/klipper/start.sh" had spread well past patch.sh, and it had reached
things people follow:
- docs/toolchange.md told owners to scp the ff_*.py extras to
/usr/prog/klipper/klippy/extras/ -- the wrong partition -- and warned that
an OTA would delete them there. Same wrong path in three ff-*.cfg headers
and in ff_toolchange.py.
- versions.env labelled our CPython 3.13 and its eighteen packages
"SHIPPED BUT NOT USED".
- docs/how-it-works.md said Moonraker runs on python3.8, contradicting line
126 of the same file.
All corrected. Historical notes keep their history and get a [Superseded: ...]
marker instead -- including 80-s6-migration.md's "s6-rc-compile is not
shipped", which pkgs/3rdparty/s6-rc/build.sh contradicts deliberately.
Found while reviewing: every docs/ reference to the ff-*.cfg pointed at
pkgs/anvil-core/payload/config/, which does not exist -- they are
anvil-klipper-config's. 28 references across 9 files, several of them broken
markdown links on the published site. All 24 relative links under docs/ now
resolve.
Gates: verify.sh syntax-checks prog/start.sh and prog/firmwareExe out of the
payload (firmwareExe was never syntax-checked before) and fails if the
component carries either. A new replica test asserts both stock paths are
symlinks into $MODDIR/prog and do not dangle. One existing replica test stubbed
firmwareExe by redirecting into $FE, which now writes THROUGH the symlink into
the packaged wrapper; it moves the link aside instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La5MaDKqTW675k5fsXsP6S
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
6aa622d2ff |
6,280 lines of case scripts reported 13 bits. They are 118 tests now.
test/run-tests.py is gone, and with it the bespoke harness that wrapped
pytest, re-parsed its JUnit XML, and drove thirteen replica case scripts to
get one boolean back from each. Every gate it ran is either a module under
qa/replica/ or deliberately retired.
PORTED. case-install.sh (579 lines, 1 bit) is test_install.py, 22 tests, asked
of the machine the printer's own app_startup.sh built rather than one the
harness assembled. case-upgrade.sh is 21, case-supervisor.sh 31,
case-boot-screen.sh 17, case-mcu-bringup.sh 8, and case-nginx.sh plus
case-moonraker313-s6.sh are test_web.py's 7.
RETIRED, subject deleted by the s6-rc switch: case-services.sh, case-nginx.sh,
case-moonraker.sh, case-moonraker313-s6.sh, case-camera.sh and
case-priority.sh drove $MODDIR/init.d/S* or sourced anvil-service.sh, neither
of which exists. Decided against real output -- `make test-services` failed
with "the payload ships no anvil-service.sh". case-pyext.sh and
case-moonraker313.sh went too: no gate, no target, no CI step, never run once.
RETIRED as a coverage decision: case-python.sh (test_web.py runs real Moonraker
on that interpreter, which is harder), case-libpath.sh, and case-recovery.sh.
The last is the brick-safety rollback and has no replacement; it is recorded in
docs/qa-migration.md rather than left to be discovered.
THREE ASSERTIONS HAD STOPPED MEANING ANYTHING, and each had been green for
months:
* case-mcu-bringup.sh's negative control had INVERTED. It asserted the
bring-up fails without LD_LIBRARY_PATH; CPython is --disable-shared now and
needs no library path, so the case's own skip branch was the one firing.
* case-upgrade.sh's headline assertion was VACUOUS. run-append.sh:150 does
`rm -rf $MODDIR/init.d` unconditionally, outside the manifest branch, so
"the update removed the stale S60web" passed regardless of the manifest
logic it was written to test.
* case-boot-screen.sh CONFIRMED ITS OWN ARITHMETIC, computing expected pixel
packing by calling ffscreen._pixel inside the replica -- a swapped R/B
matched itself.
case-install.sh's boot-log grep is not reproduced for the same reason: the log
line names every service in the boot set including the ones that then fail, so
grepping it cannot go red. The compiled database is asked instead.
TWO REAL BUGS FOUND ON THE WAY.
assemble.sh attached /stick.img with `losetup` and wrote the device to
/stick.loop, which NOTHING HAS EVER READ. Loop devices are the host's, not the
container's, so every USB_STICK=1 run leaked one permanently and a WSL2 kernel
has thirteen. Once they were gone every replica run failed with "failed to
setup loop device", which looks like a docker fault and is not. Detached now in
entrypoint.sh on the way out, and in qa's stop() for the held-open containers
that never take that path.
test_startup.py was 43 tests red: it built its argparse.Namespace by hand and
had never been taught --only-bringup / --no-bringup, and its FakeKlipper still
modelled start.sh plus a separate stop when klipper is restarted with one
`s6-svc -wr -t`.
CI AND RELEASE. The replica job was getting its package as a side effect of
run-tests.py running unpack/patch/pack; that is an explicit step now, as the
comment there predicted it would have to be. release.yml tested the exact files
it ships through sim-install.py, whose exit status the loop was discarding --
it is `REAL_PKG=<pkg> pytest ./qa/replica` per model now, 22 named results each.
WHAT IS NOT TESTED ANY MORE: bin/unpack.sh, patch.sh, pack.sh and verify.sh.
The packaging build on a synthetic stock package went with the runner and has
no counterpart in qa/. That is the path producing the .tgz a user flashes.
make-stock-fixture.sh is kept as the input a replacement needs.
Measured: qa/static 246 passed, qa/replica 110 passed and 8 blocked on hardware
the replica has not got (no /dev/ttyS*, no /dev/video*, so ok-all is
unreachable), test/ pytest 185 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013BTAXgjtF8GdhqSxvdXCFW
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
abd4a01d73 |
Adopt the s6-rc boot graph -- DOES NOT BOOT YET, see below
Merge of qa-suite-bridgehead into the packaging line. The seven init.d/S*
scripts and the hand-written etc/s6 servicedirs are gone; anvil-core ships
etc/s6-rc/source/ and bin/patch.sh compiles it into etc/s6-rc/compiled/ after
the payload is installed. klipperDaemon joins firmwareExe and start.sh in
payload/prog/, all three linked into place by anvil-link-prog.sh.
WHAT IS KNOWN BROKEN. qa/replica/test_s6rc.py fails: the tree never comes up,
because s6-rc-init dies with
unable to supervise service directories in /run/s6-rc/servicedirs:
Value too large for defined data type
That is EOVERFLOW out of the readdir() loop in s6rc_servicedir_manage.c, which
checks errno after every entry. Measured, and these are ruled out:
* not the live directory's filesystem -- fails identically with -l on
/usr/data as with the default /run, and the inodes there are 3 and
1358496, nowhere near 32 bits
* not the database -- s6-rc-db lists all eleven services out of it, and a
database recompiled inside the replica by the TARGET s6-rc-compile fails
exactly like the one compiled natively on x86
* not a missing -D_FILE_OFFSET_BITS=64 -- both s6-svscan and s6-rc-init
reference readdir64 and never plain readdir
What is left is the toolchain. qa-suite-bridgehead built s6, s6-rc, execline
and skalibs with the Bootlin musl toolchain, where readdir has no EOVERFLOW
path at all; the packaging line builds them as recipes against the Ingenic
glibc, where it does. s6-svscan looked healthy only because it was started on
an EMPTY scandir, which never reaches the loop.
Everything else is green: 262 static tests, and bin/verify.sh passes on the
73MB package, including the graph's own "compiled s6-rc database" check.
Resolution notes: the five MOD_* switches stay deleted -- firmwareExe defaults
every gate to 1, so nothing changes -- but MOD_CAM is now worth having by hand
on a cameraless printer, because the camera is a longrun that waits out
timeout-up. anvil-service.sh went with init.d; nothing sourced it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpUmBb4xtmnmtTosonqeNi
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5b463b889a |
Five switches that were only ever set to 1
MOD_WEB, MOD_CAM, MOD_UI, MOD_SSH and MOD_WIFI all defaulted to on, and what
each one really offered was a half-installed printer: Mainsail with no
moonraker behind it, a camera that would not stream, a screen dark on purpose,
a root password deliberately not set on a machine whose entire recovery story
is "ssh in". Nobody wants those states. What someone who sets one actually
wants is a stock printer, and that is one FlashForge package away.
What they cost was a second answer to every question. Each init script carried
a gate, each `down` file argued its existence from a flag rather than from the
ordering that was the real reason, and anvil.conf had five lines that told a
user they had a choice.
MOD_SSH was the clearest case: it never turned ssh off. dropbear is the
rootfs's own and listens either way, so all the flag did was decline to set a
password.
Each gate is replaced by the fact it was standing in front of, not by a
constant:
S60nginx, S62moonraker nothing -- the web stack is what this mod is
S65camera there is no video device, which the script already
finds out and reports
S50wifi there is no $WIFI_CONF, likewise
S80ui HelixScreen is not installed, which was already the
other half of the condition
To leave a piece out, leave it out: BUILD_MAINSAIL, BUILD_MOONRAKER and
BUILD_HELIX decide what goes into a build, and `opkg remove` decides what stays
on a printer. What survives in anvil.conf is what has a range rather than a
state -- MOD_CAM_RES/FPS/PORT and the NICE_* priorities -- plus MOD_SPLASH,
MOD_STARTUP, MOD_IMPORT and MOD_S6, which are untouched here.
The `down` files stay. They were argued from MOD_WEB and MOD_CAM, but the
argument never depended on them: s6-svscan starts a service the instant the
directory appears, which is before any init script has checked a prerequisite.
Their text now says so.
Measured: `make build` produces a payload with no live read of any of the five,
and qa-static is 265 passed with shellcheck over the edited init scripts. The
test suite still exercises the removed flags and has not been updated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gKRUCqj18TVF1Vz1oMpmC
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
f67e7a269f |
ls pkg/ was thirty-eight entries and four of them mattered
pkg/ -> pkgs/, and the thirty-four recipes that build a pinned tarball and
carry no files of this repo move down a level into pkgs/3rdparty/. What is
left at the top is the four components a person actually edits: anvil-core,
klipper, moonraker, helixscreen.
THE LINE IS MECHANICAL, NOT EDITORIAL, and that is why it can be a directory
rather than a field in pkg.conf. "Do we actively modify it" drifts -- the day
somebody patches zlib nobody would agree which side it belongs on, and the
answer would be a rename touching CI and docs decided by taste. "Does it carry
files of ours" is a fact about the tree: it is exactly whether the recipe has
a payload/, prog/ or seed/, checkable in one assertion, and
test_recipe_layout.py now demands the move in both directions -- a 3rdparty
recipe that grows one of the three must come up, and a top-level recipe with
none of them must go down, or the top level slowly refills and the split stops
meaning anything.
ONE FUNCTION KNOWS THE TWO DEPTHS. pkgs/lib.sh's pkg_dir resolves a recipe id
to its directory and nothing else spells either path -- not a recipe, not
bin/build-packages.sh, not a test. Names stay unique across both levels
because pkg_out, pkg_stamp and the .ipk filename are all keyed by the bare
name; a duplicate would resolve to whichever level is searched first and
build the wrong thing silently, so that is a test too.
Recipes now source `. ./bin/common.sh` instead of counting `..` from $0.
Depth-counting would have needed two spellings for two levels, and it was
never buying anything: `. pkg/lib.sh` on the next line already required the
working directory to be the repo root.
A SECOND SILENT-SHRINK BUG, and this one was live. qa/static's _files()
expanded a list of globs and returned whatever matched, so "pkgs/*/build.sh"
quietly stopped covering thirty-four recipes the moment they moved -- the lane
stayed green, and the only visible sign was a test count dropping by 34 in a
run nobody was counting. _files() now refuses a pattern that matches nothing.
That is the same failure the ash-compatibility check had a guard for and the
syntax check did not, which is why the guard reads the way it does.
AND ONE PACKAGE WAS SHIPPING PYTHON BYTECODE. anvil-core stages a directory of
.py helpers wholesale, so any test that imported one left a __pycache__ beside
it -- gitignored, invisible in a diff, copied by cp -a like anything else. The
package's contents depended on whether pytest had been run on the machine that
built it: two builds of one commit, different sha256, cause nowhere in the
tree. What it shipped was worthless as well as unstable, compiled by the build
image's python 3.11 for a printer running the 3.13 this repo cross-builds. The
sweep is in pkg_ship now, once, and the two hand-written copies in
pkgs/klipper and pkgs/moonraker are deleted -- the trap is set for any recipe
that stages a tree of .py files, and the two recipes that had hit it had each
solved it locally while the third, which had not, was the one shipping.
Measured:
* the rename changes no package bytes. Same cached build trees, feed built
with the diff stashed and unstashed: all 41 .ipk match.
* with the bytecode sweep, anvil-core returns to exactly the bytes the
previous commit produced.
* no package in the feed contains a .pyc or a __pycache__.
* make qa-static: 247 passed (236 before this work + 11 layout tests).
* make test: 8 passed, 0 failed, including unpack/patch/pack/verify on the
synthetic stock package. make test-py: 181 passed.
Not explained: anvil-python-dev produced two different sha256 from an
UNCHANGED work/pkg/python at two points today, under code that was identical
both times -- so it is not reproducible from a cached tree, and it is not this
commit's doing (the stashed and unstashed builds above agree on it). Worth a
look; the dev split in bin/build-packages.sh is where to start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8cc92e8266 |
A wheel could only be built while three scratch trees happened to coexist
That is why bin/patch.sh section 5c was 825 lines and why a Pillow bump
rebuilt CPython. Cross-building an extension module needs the untrimmed
staging tree (INCLUDEPY points into it), the private sysroot of static C
libraries (pillow links zlib, cffi links libffi) and an x86-64 build-python
of the same version -- and all three existed only mid-section, so the wheels
had to be built there too, on a cache with two stamps that invalidated each
other.
Each of those three is now a durable thing: anvil-python-dev carries the
headers, pkg_deps fills each recipe's own sysroot from the feed, and
pkg_buildpython caches one build-python for all nineteen recipes. So the
coupling goes away with them -- `make packages PKG=python-pillow` builds
Pillow.
pkg/python -> anvil-python + anvil-python-dev
pkg/python-<name> x18 -> anvil-python-distro, anvil-python-pillow, ...
Three verbs in pkg/lib.sh, shared by all nineteen rather than one per awkward
project: pkg_buildpython provides the build-python, pkg_pytarget points its
sysconfig at the target headers in the sysroot, pkg_pywheel builds one wheel
and unpacks it into the staging site-packages. greenlet's gcc 7.2 designated
initializers, lmdb's LMDB_FORCE_CPYTHON and pillow's build_ext options are
recipe lines and a PKG_PY_SETUP_ARGS knob, not three more verbs.
Eighteen versions that were invisible are now in the feed index, anvil-moonraker
declares the thirteen packages it imports, and a printer that does not want
gcode thumbnails can leave Pillow uninstalled.
PY_STAMP, PYPKG_STAMP and PY_BUILD are gone: pkg_stamp derives the key from
PKG_BUILD_DEPENDS, and bin/fetch-assets.sh asks pkg_needs instead of
re-deriving a stamp bin/patch.sh also derived. Two of its four toolchain
conditions were that second spelling.
Measured:
* 40 packages; 959 site-packages files, none owned by two packages
* 12 mipsel extension modules -- the count the old gate demanded
* every Depends resolves inside the feed (24 declare one, 0 unsatisfiable)
* two cold builds byte-identical, with work/pkg, work/.py-host and the
unpacked toolchain deleted in between
* 233 static tests, 4 new: the PYPKG_LIST/recipe agreement both ways, no
version pinned twice, patch.sh builds every package, and no Depends
naming a package no recipe produces
Not verified here: the replica suite. It needs the stock FlashForge package,
which the packaging lane deliberately does without.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
057a3a11d7 |
The boot was seven scripts in filename order. It is one compiled graph now.
firmwareExe ran $MODDIR/init.d/S* one at a time in the foreground, and that
loop WAS the boot order. Three of the seven were already thin wrappers over a
supervisor that was already running, so what the loop still carried was not
supervision, it was sequencing -- expressed as filenames, in a directory
nothing but the loop read.
This ships s6-rc and moves the sequencing into a database compiled at build
time. firmwareExe now starts the scanner as its own child, initialises s6-rc,
asks for one machine state, and blocks on `wait "$SCANNER_PID"`.
WHAT THE GRAPH ACTUALLY SAYS
mcu-bringup -> klipper -> ff-startup -> ui
^
moonraker
ok-all = wifi nginx moonraker camera klipper ff-startup ui
Those are the first real edges the plan has ever had; it predicted they would
not arrive until phase 7. bin/ff-startup.py became two services, which is what
made them possible: the MCU bring-up has to precede klipper and the readiness
wait has to follow it, and both facts now live where a program can read them.
Deliberately still absent: nginx before moonraker. S60nginx's header wants it,
but dependencies.d/nginx under moonraker means `s6-rc -d change nginx` stops
moonraker too -- the exact failure splitting S60web into two scripts prevented.
The cost of not writing it is a second where a reloading browser gets a refused
connection instead of a 502.
AND WHAT THEY DO NOT SAY. klipper has no notification-fd, so s6-rc calls it up
when it is forked, not when klippy is usable. ff-startup keeps its own poll of
moonraker's API. moonraker and camera DO carry one, so their readiness is real.
The asymmetry is written down at each site rather than glossed.
init.d/ IS DELETED, REVERSING PHASE 4
Phase 4 promised the S* scripts would survive as thin wrappers, because
`S60nginx restart` is in people's hands. They did not survive, and the reason
is the argument nginx/run already made: a service that only works when a shell
script happened to run first is not supervised, it is lucky. Every prerequisite
a wrapper did belongs in the run script anyway, because s6 restarts a service
without going near a shell. Once they moved, the wrappers were verb translation
in a directory whose S<NN> numbers meant nothing. s6-rc is the interface now.
anvil-service.sh goes with them -- 625 lines whose last caller was firmwareExe,
and a shared library for one consumer is the redundancy the wrappers were.
So do the fallbacks. S60nginx and S62moonraker each carried a second,
unsupervised implementation reachable only via MOD_S6=0; that switch is gone,
and with it the argument for a code path exercised only on broken printers. The
cost is stated plainly in anvil.conf: a printer whose s6 fails now has nothing.
klipperDaemon is replaced with a no-op `start`. It is FlashForge's script and
off-repo callers remain -- stock start.sh, the UI's restart path, habits over
ssh -- and each of them would fork a second klippy beside the supervised one.
WHAT SUPERVISES THE SUPERVISOR, ANSWERED
Phase 3 left this open and guessed a dead scanner was a dead printer either
way. It is not. firmwareExe's tail used to be `while true; do sleep 3600; done`
purely to satisfy app_startup.sh's watchdog, which greps ps for our name 5s in.
It waits on the scanner instead, so exiting IS the recovery: app_startup.sh --
FlashForge's own init, which we never patch -- re-execs us and the tree comes
back. Not a respawn loop in ash, which phase 3 rightly refused. It restarts
everything rather than the scanner alone, and that is written down.
MEASURED, NOT READ OFF THE MANUAL
Host- and target-compiled databases are byte-identical (db md5
80f5261437191fa591744ada6087c6f7 both ways), so the database is built here and
shipped. /run is tmpfs and writable, so the live directory needs no fallback.
Both were BLOCKING UNKNOWNs in the note and are now settled sections.
Three traps the replica found that reading could not:
* s6-svscan resolves s6-supervise through PATH. Without $MODDIR/bin on it the
scanner answers s6-svscanctl -a perfectly and then fails every spawn with
ENOENT for ever, and s6-rc-init HANGS rather than failing.
* s6-rc-init creates one symlink per service in the scandir and dies with
"File exists" if the name is taken. So the cutover is not additive, and
run-append.sh sweeps.
* s6-rc-compile does not reject a `down` file. It accepts it and SILENTLY
DROPS it -- the servicedir is byte-identical. "Down by default" can only
mean absent from the bundle, which is why mcu-bringup is the one service
outside ok-all and arrives as klipper's dependency.
execline turned out not to be optional: s6-rc GENERATES two servicedirs whose
run scripts are execline, and every oneshot's `up` goes through one of them.
The binary list was read off a real compile, not guessed -- seven execline
binaries of 53, plus six s6 ones we built and did not ship.
Two build bugs fell out. fetch-assets.sh compared against a stamp patch.sh
never wrote, so every build pulled the ~100MB toolchain the skip exists to
avoid; S6_STAMP is one string in common.sh now. And `chmod +x .../bin/s6-*`
would have missed all seven execline binaries.
qa: 83 static. The suite gained the s6-rc run scripts and klipperDaemon, which
no glob had ever covered, and lost a dead payload/init.d/S* glob -- exactly the
silently-retired check that file's own docstring warns about. The oneshots'
up/down are NOT in it: they are execline carrying shell inside a /bin/sh -c
argument, so both checkers would parse the wrapper and pass.
Not verified here: no config.env and no stock package on this machine, so the
mipsel cross-build of execline/s6-rc and the ABI gate over 33 binaries have
never run. The native compile and the database are proven; that half is not.
The replica gate is the next commit, and it still owes an answer on whether
helix-screen runs under s6-supervise at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
c870d8417d |
FF_PYTHON moves to our own 3.13, and CI stops rehearsing around it
The gate this switch was waiting on now exists and passes:
case-moonraker313-s6.sh drives Moonraker through the real boot path --
S40s6's scandir, S62moonraker, readiness gated on :7125 actually
listening, a kill -9 respawn, a stop that stays stopped -- on the
interpreter this commit ships. anvil-env.sh now points FF_PYTHON at
$MODDIR/bin/python3.13 and drops /usr/prog/libsodium/lib from
ANVIL_LIBS, since nothing left running under FF_PYTHON imports libnacl
from there any more (Moonraker carries its own libsodium; klippy is not
on this list at all -- it stays on FlashForge's 3.8.2, started
independently by /usr/prog/klipper/start.sh).
Three real bugs surfaced chasing this through the replica, none of them
about the switch itself:
* The Phase-1 manifest-based install can only delete files it once
tracked, so an init.d/S60web surviving from a release that predates
the manifest system would never be cleaned -- caught by
case-install.sh, which had never run against real stock packages
before today. run-append.sh now clears $MODDIR/init.d unconditionally
on every update: those scripts are 100% mod-owned and get executed,
so a diff against tracked history is not a strong enough guarantee.
* case-moonraker313-s6.sh's own readiness-ordering check flaked once:
its bind-time sampler polled at 1-second resolution while s6-svwait
is event-driven and returns within a fraction of a second, so
`date +%s` truncation could show readiness before the bind though the
real events were in order. Tightened to 0.2s.
* case-mcu-bringup.sh and case-boot-screen.sh source anvil-env.sh but
named their own directory variable $MOD, not $MODDIR -- harmless
while FF_PYTHON was a literal path, and silently wrong the moment it
started reading $MODDIR. Aliased.
Every other test/integration file touched here is the same story:
FF_PYTHON's callers were hardcoded to FlashForge's 3.8.2 throughout the
suite, and each one either now stages our interpreter as a tarball
(gates.py's mcu_bringup/boot_screen/moonraker, matching the pattern
already used for python()/moonraker313_s6) or asserts against the new
target. case-libpath.sh's premise changes with it: the three ANVIL_LIBS
entries that remain are read as what klippy's process tree still needs,
not what every mod process needs -- Moonraker maps zero /usr/prog
libraries now, measured.
payload/anvil-service.sh's SVC_S6_SCANDIR line tripped test_paths.py's
absolute-path regex, which reads a closing brace immediately followed by
a suffix as a stock rootfs path in its own right; split onto its own
line so a word character precedes it instead, which the regex already
excludes.
bin/fetch-assets.sh had a real latent bug of its own, independent of
this switch: the Ingenic toolchain fetch was gated on BUILD_KLIPPER=fork
alone, a condition that predates CPython 3.13's unconditional need for
the same compiler. A BUILD_KLIPPER=stock build from a clean vendor/
would reach the Python cross-build with sources and no compiler. Also
added curl --retry -- musl.cc flaked mid-download in CI while this was
being tracked down.
.github/workflows/ci.yml's fixture-only `test` job now actually
cross-builds s6 and CPython 3.13 instead of never exercising that path:
it fetches just those two build inputs (Mainsail/HelixScreen/Klipper
stay stubbed, unchanged) and caches vendor/, work/.s6, work/.py313 and
work/.sodium on versions.env.
Full suite, replica required, against real stock packages fetched for
this run: 23 passed, 0 failed, 0 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
8994b461b0 |
Every service starts the same way, and Moonraker stays on /usr/data
Moonraker is no longer copied to /usr/prog. The payload already extracts
that exact tree to /usr/data/anvil/moonraker, so the installer was making a
second identical copy onto the small firmware partition -- the one install
step that could fail on space, whose failure was "this printer has no
working web UI". Worse, /usr/prog is what a stock flash overwrites while
/usr/data survives, so a firmware update silently reverted Moonraker to
FlashForge's 2022 build and left the rest of the mod in place: exactly the
"which Moonraker is this printer running?" ambiguity the previous commit set
out to end. Neither of the two things thought to pin it there holds -- the
moonraker-env virtualenv is not on sys.path (imports resolve from
Python-3.8.2/lib/python3.8/site-packages, checked on the real image) and
moonrakerDaemon is never invoked. FlashForge's tree is left alone, unused.
S60web became S60nginx and S62moonraker. nginx and moonraker fail for
different reasons and are now stopped, started and diagnosed separately;
MOD_WEB still governs both.
payload/anvil-service.sh is the shape all six services now share. They had
been written one at a time and had arrived at four different answers to "is
it running" and two to "restart", none of them decisions. Two real bugs were
living in that drift, and both are fixed here:
* busybox start-stop-daemon -K returns before the process is dead and
never removes the pidfile, so restart raced its own start -- stop
returned, start found the pidfile still held, and the service ended up
stopped rather than restarted. The `sleep 2` in front of restart was
standing in for that wait; svc_stop_daemon does it properly and the
sleep is gone.
* klippy does not unlink /tmp/uds on exit. supervise() knew that; stop()
did not, so status reported a dead klippy as running and restart
declined to do anything at all.
run-append.sh now clears $MODDIR/init.d before extracting. Without it the
rename above would leave a stale S60web beside the two new scripts on any
printer that already has the mod, and firmwareExe runs every executable
S* in filename order -- starting nginx and moonraker twice, from scripts
disagreeing about where moonraker lives.
bin/patch.sh stages anvil-service.sh. It is sourced, so no chmod +x.
test/integration/printer/case-services.sh is a new replica gate: it runs
every service on the printer's own busybox and checks the contract they
share rather than grepping them for how they are spelled. case-moonraker.sh
now asserts moonraker runs from /usr/data, that nothing under /usr/prog is
executed, that restart yields a genuinely new pid, and that stopping
moonraker leaves nginx up -- the point of the split.
The .modold rollback recovery in docs/how-it-works.md described a code path
removed in the previous commit and is rewritten for what happens now.
Verified in the replica (real rootfs.squashfs, qemu-mipsel): case-moonraker,
case-services, case-mcu-bringup and case-boot-screen all pass; pytest 181
passed, 5 skipped. case-install.sh and case-recovery.sh are syntax-checked
only -- they need a stock FlashForge package, which is not in this checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
ac558f6686 |
Document the Moonraker swap honestly, and finish its doc coverage
The reversibility claim was the strongest promise in the README and
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
d05565284d |
Sections E, F, G: wrong values, stale claims, overclaims
E -- values and counts. The START_PRINT prologue has no M400 and an idle timeout of 864000, not 1800000. The nozzle-clean temperature comes from the material table FIRST, so NOZZLE=240 with no per-tool temp cleans at 220 -- documented backwards before. TOOLS is the paired 0:220,2:240 form. The wipe Z is the raw eddy frame, not the homed one. The runout M117 strings are "T<n> clog" / "T<n> out of filament", not E-codes. Grab-verify raises E0051+tool, not E0143. Docks are at X~297; 250/280 are staging waypoints. The grab sequence is documented in the order _grab actually emits it. ff_print exposes seven status fields, not nine, and never emits TOOLS=. Package sizes reconciled against the real artifacts (62MB built, ~93MB stock, ~370MB of CI assets, ~60s not 90s). testing.md's sample output block matched no header the code prints, so it is replaced with the real ones; its pytest row listed two of six suites. F -- stale claims. _FF_PREFLIGHT is not "missing". SDCARD_SET_CHANNEL is not still issued ( |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
b2c651b11f |
Fix the runout insert path and TOOL=ALL staging; drop SAFE-MODE for MOD_UI
A2: _FF_INSERT's body was entirely inside {% if kind == 'switch' %}, yet
insert_gcode is wired on all four fm_ex* with KIND=motion. A motion insert --
which is what clearing a clog looks like, filament moving again -- produced no
output at all, on exactly the path where the operator most needs the next
step. Adds the motion branch the comment already described.
A3: TOOL_OFFSET_CALIBRATE staged each tool with set_nozzle() inside the loop,
so a gap-check failure on a later tool left earlier tools pending for
SAVE_CONFIG -- while raising an error that says "nothing saved". True for
TOOL=<n>, false for TOOL=ALL, and told to the operator at the moment they
decide whether to save. Staging now happens after every tool has passed.
A8: SAFE-MODE is removed rather than repaired. It never worked: the settle
check cleared the counter 90s after every boot on a condition
(`[ -s $CHOICE ]`) that cannot be false, since $CHOICE is written two lines
earlier and "none" is not empty. Nothing checked whether helix-screen was
alive, so the latch only fired on three reboots inside 90 seconds -- not the
crash-loop it was written for.
In its place, MOD_UI in anvil.conf, alongside MOD_WEB/MOD_CAM/MOD_SSH/MOD_WIFI
and templated by patch.sh the same way. It stops the UI being started; it does
not pretend to repair one. The docs said `touch SAFE-MODE` was how you recover
a blank screen. Nothing recovers a blank screen on the machine: HelixScreen is
the only UI, and getting one back means flashing a package. Said plainly now,
in the README, the recovery section and the troubleshooting table -- along with
what is genuinely unaffected, which is ssh, Mainsail, Klipper and printing.
48 passed, 5 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
e5bd171cf8 |
Correct the docs the code had drifted away from
Applies the audit's sections B, C and D, plus two decisions. Behaviour (A1): fm_ex1..3 carried stock's `extruder: extruder`, so a clog on T1..T3 was measured against T0's filament movement. Each motion sensor now watches its own extruder. Gate count (A6): "four gates" was the number of make test-* targets, not of gates run -- run-tests.py emits twelve. The number is dropped rather than restated, in docs/testing.md and the Makefile help. B, install instructions that failed as written: STOCK_TGZ_CREATOR5 missing its .tgz; a pointer to a ./build.sh that does not exist; klippy-extras/ paths that moved to payload/klipper/extras/ in 0ef9d6b; a calibration recipe both commands refuse without PLATE_REMOVED=1; a storage table that put dock_x/dock_y in an include, which SAVE_CONFIG refuses; four files telling the user to add includes printer.base.cfg already ships; ./pack.sh style paths that do not resolve from where the scripts leave you. The MAINSAIL_ZIP / HELIX_TGZ override was documented in three places as "used as-is, never checksummed". It is checksummed, and fetch-assets.sh overwrites the user's file with the pinned release -- SKIP re-downloads too. Corrected, with the way that actually works (match the hash in versions.env). C, files that contradicted themselves: the -5/-3 ESTOP target, the t<n>_offset_* storage names, the chamber heater's model split, the LED premise, the include order comment in test_includes.py, gear_stepper as a "feed hub" when it is the tool lock, an object list ff-toolchange.cfg says does not exist, the eddy probe vs the under-bed cylinder, "installs them without clobbering" against a gate that enforces the clobber, "keeps two versions" when the installer keeps one, "two changes from stock" when there are three, and ff_legacy's "nothing here runs during normal operation" when auto_import runs on every ready. D, dangling references: OKF/* and firmwareExe-decompiled/* repointed at docs/notes/*, plus case-boot.sh, sim-image.sh, printer-exec.sh, the test/printer build context, wait_for_banner, ff_toolchanger_compat and a nonexistent orca snippet. Not touched, pending a decision: sections E-H, and the printer.base.cfg "verbatim" claim (A7), which depends on how the unpack drift check is fixed. 48 passed, 5 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
4460488ea9 |
Moonraker: pin the last lmdb build -- this python has no sqlite3
v0.9.3 does not start on the printer. It was built, flashed by hand and
tried, and it died on:
ModuleNotFoundError: No module named '_sqlite3'
FlashForge built python 3.8.2 without that module -- there is no
_sqlite3*.so in lib-dynload and no libsqlite3 anywhere on the image,
only the pure-python sqlite3/ wrapper that cannot work without it.
Moonraker moved its database from lmdb to sqlite in v0.9.0, so every
release from there on gets as far as loading the database component and
exits. The last release still on lmdb is v0.8.0, which predates the
webcam "enabled" flag this whole change exists to get.
No release has both, so the pin is now the newest COMMIT that does:
9d0d09d, 2023-12-05, the one before "database: migrate from lmdb to
sqlite". It has the webcam field and reads the printer's existing lmdb
store in place, so nothing is converted and going back to stock is a
clean round trip. Verified on python 3.8 against the printer's exact
library versions with _sqlite3 removed from the interpreter to match.
Two things follow from having shipped a build that could not start.
run-append.sh now asks the printer's OWN python to import the tree
before it moves anything, and keeps the stock server if it cannot. A bad
pin refuses to install instead of leaving the machine with no web UI --
confirmed on the replica, where the v0.9.3 payload is now declined and
the stock Moonraker keeps running.
And the test that passed this build is fixed. It asked only whether a
moonraker.py process ever appeared, which a build that dies on an import
satisfies for a second or two on its way down. It now requires the
process to still be there after a settle, and the log to be free of
import errors. Re-pinned to v0.9.3 it fails, as it should have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
a40bc861dd |
Ship our own Moonraker: the stock 2022 build hides the camera
Mainsail has shown no webcam panel on this printer no matter how the stream is configured, and the stream was never the problem: nginx serves /webcam/ correctly and mjpg-streamer answers it. FlashForge's Moonraker is a 2022 build (API 1.0.5). Moonraker only grew the webcam "enabled" flag in April 2023, and Mainsail filters its webcam list on exactly that field -- so every [webcam] entry is discarded and the panel is removed from the dashboard entirely. Nothing on either side of that can be configured around; the server has to be newer. So the mod now ships Moonraker the way it ships Mainsail: pinned in versions.env, fetched to vendor/ against a sha256, installed over the stock tree. v0.9.3 is the newest release that runs on the printer's existing python environment untouched -- v0.10.0 wants dbus-fast, a compiled module that is not there and would have to be cross-compiled for mipsel. Only the python package is replaced; the interpreter, the moonraker-env beside it and moonrakerDaemon stay FlashForge's. It rides in the mod payload rather than the software component because the stock run.sh does not extract that component over /usr/prog -- it copies a hand-written list of paths out of it, and a moonraker/ dropped in beside them would never be looked at. run-append.sh installs it, moving the old tree aside and putting it back if the copy fails: this is the one thing the mod writes to /usr/prog that the stock installer does not, and a printer whose Moonraker did not survive has no web UI at all. The database migrates itself -- v0.9.x moved from lmdb to sqlite, lmdb 1.3.0 is installed, so Mainsail's settings survive the upgrade. Covered end to end on the printer replica: that the swap happened, that the installed webcam.py has the enabled field, and that the shipped tree really runs on the printer's own python3.8 under qemu-mipsel. With BUILD_MOONRAKER=0 those skip and the stock server is left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
91604c2dfb |
Cut the suite to four gates, and rewrite the Python half as pytest
Twelve gates, most of them the weaker copy of a gate that survives. What is
left: pytest for the Klipper config, and three replica gates -- the end-to-end
install, MCU bring-up, and stock recovery.
Fix first, because the cut depends on it: a skip counted as a pass. Every
launcher exits 0 when it decides it cannot run, and sub() reported that as
`ok`. That is how test-abi.sh sat in the suite printing green while checking
nothing -- run-tests.sh deleted work/modpayload immediately before it ran and
CI set KLIPPER_FORK="", so it had no targets on any run, ever. Skips are now
counted apart, the summary says so, and the suite exits non-zero unless
ALLOW_SKIP=1 says the gap is deliberate. With four gates instead of twelve, a
gate that silently does not run is the failure mode that matters most.
Dropped, each because something else already does the job properly:
test-abi.sh bin/patch.sh refuses to build a non-nan2008
c_helper.so, bin/verify.sh re-checks the packaged
one, and case-install.sh reads the ELF header of
the file that actually landed on the machine.
test-ash-conformance.sh case-install.sh runs `sh -n` over every installed
script with the same qemu'd busybox and then greps
the boot log for "Syntax error".
test-model-gate.sh verify.sh SS8b/SS9 checks the same invariant and
pack.sh exits on MODEL MISMATCH before a package
exists. Its header claimed it proved the two models
ship different files; no such check was in it.
test-macros.py It used jinja2.Environment() -- default '{{ }}'.
The configs have no '{{' at all, only single-brace
expressions, so it checked '{% %}' block structure
and never looked inside an expression. Folded into
the chamber tests, which had the delimiters right.
test-base-cfg.py It read work/software/klipper/config/printer.base.cfg
as its stock reference -- the file bin/patch.sh
overwrites with ours. It was diffing our file
against itself: green on a cold tree, red on a
second run, blind to real drift either way. The
comparison moves to bin/unpack.sh, where a pristine
tree exists, as a warning rather than a gate.
lint-danger.sh Its structural greps were satisfiable by comments:
firmwareExe:44 and S80ui:16 *mention* helix and
SAFE-MODE, so they passed with the logic deleted.
case-ui.sh Goes with the helix-only decision.
applets.allow Ten entries, none of which the extractor had asked
about in a long time.
The command-word half of test-applets went too. It extracted only the first
word of a simple unprefixed command, so `if timeout 5 foo; then` yielded
['if','echo','fi'] -- the one failure its docstring named was invisible to it.
The absolute-path half survives: S50wifi's binaries sit behind a
[ -d /sys/class/net/wlan0 ] guard no simulation ever reaches, so nothing else
checks they exist.
The Python half is pytest now. Real assertions instead of hand-rolled
PASS/FAIL counters, underscored filenames so the modules import normally
(ffcfg keeps only the parser; the dash in test-macros.py was why one test used
to reach the other's parser through importlib), and a `rootfs` marker that
splits the tests needing the proprietary package from those that do not --
deselected in the PR lane, run in the replica half, so neither lane reports
skips it did not expect. test_default_delimiters_are_blind pins the jinja2 bug
so it cannot come back.
The config gate also moves out of the replica half into the half that runs on
a plain pull request. It needs python3, jinja2 and files in the repo, and it
is the only safety-relevant check in the suite -- a chamber heater declared on
a Creator 5 that has no element for it -- so it had no business being gated
behind the proprietary firmware.
Verified: pytest 21 passed / 5 rootfs-marked; the fixture pipeline
(unpack, patch, pack, verify) passes end to end; injecting a broken
single-brace expression fails test_every_macro_parses with the exact line, and
parses clean under the old defaults; the skip path reports SKIP and exits 1,
and ALLOW_SKIP=1 accepts it.
Also drops the CI `abi` job, which ran on a bare checkout with no binaries and
so was permanently green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
607e6595fb |
Start the camera ourselves: mjpg-streamer was on the disk all along
Correcting the previous commit, which said the :8080 stream could not be
restored without cross-compiling a streamer. Both halves are actually true:
* firmwareExe really did serve the stream itself -- its own accept loop,
`GET /?action=stream`, boundarydonotcross, even announcing
`Server: MJPG-Streamer/0.2` -- so killing it does take the camera down.
* ...but /usr/prog/mjpg-streamer/ ships the genuine article: mjpg_streamer,
input_uvc.so, output_http.so, its own libjpeg.so.9 and www/.
app_startup.sh:159 exports LD_LIBRARY_PATH for it and then never launches
it. Nothing needs building; the binary just needs starting.
S65camera does that, on the S50wifi pattern -- the wait for /dev/video0 and the
respawn loop run in a background subshell, so a printer with no camera plugged
in never holds up Klipper or the UI. stop() kills the supervisor before the
streamer, or it would come back three seconds later.
Two details are load-bearing and easy to lose: libjpeg.so.9 lives in the
mjpg-streamer directory, so without LD_LIBRARY_PATH the binary dies on a
missing shared object (the same trap moonrakerDaemon has in S60web); and the
plugin .so names resolve against the working directory, hence the cd.
nginx proxies it at /webcam/ with buffering off and no read timeout, since
MJPEG is one endless response, and moonraker.conf's [webcam] entry uses
relative URLs through that proxy -- so one forwarded port carries the stream,
tunnels included. MOD_CAM=0 turns it off; MOD_CAM_RES/FPS/PORT tune it.
The earlier claim is corrected in docs/notes/25 and in the README, which no
longer lists the camera as something you give up.
Verified: parses under the printer's own busybox ash, lint-danger clean. The
full suite was not re-run to completion -- the working tree is being edited
concurrently and a build lane raced it (tar: file changed as we read it).
bin/patch.sh's MOD_CAM line is deliberately left out of this commit for the
same reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
5bdf78109a |
Retire firmwareExe: own the web stack, and say what goes with it
The mod replaces FlashForge's application outright, so the reasoning that still framed the web stack as coexisting with it was wrong on its own terms. * nginx and moonraker are documented as the only web stack on the printer. MOD_WEB=0 is a debugging switch that leaves ssh and the touchscreen, not a way back to stock -- that is a stock FlashForge package. The own-prefix rule stays, for the reason that is actually true: /usr/prog is the firmware partition and every FlashForge package reinstalls it, so a config written there is undone by the next stock flash. * Configs the mod ships now update when you have not edited them. A live file still byte-identical to the one the last package wrote is replaced; one you changed is kept and the new version lands beside it as .mod-new. $MODDIR/config-installed holds that last-written copy. Without this a config we own could only ever be written once: it exists on the second flash, so it would land as .mod-new forever and updates would never reach the printer. * Dropped the three dead /server/files/ alias blocks from nginx.conf. The regex location above them matches /server/ too and nginx gives regex locations precedence, so they never fired -- and had they fired, serving those paths from disk would 405 the DELETE and POST that Mainsail sends to the very same URLs. * Recorded what dies with firmwareExe, in docs/notes/25 and in the README: the :8898 REST API (FlashPrint, Orca's FlashForge profile, the mobile app) and the :8080 camera stream. The camera is the one that looks restorable and is not -- the MJPEG server is compiled into firmwareExe (its own accept loop, `GET /?action=stream`, boundarydonotcross) rather than being mjpg-streamer, so there is no daemon left to point a [webcam] at. assets/moonraker.conf carries the block, commented, for when a streamer is built for this board. Also in this change: the MOD_UI switch and the stock-UI path are gone, the chamber-heater model gate arrives with ff-model.cfg/ff-chamber.cfg, and the tests follow. make test: 32 passed, 1 failed. The failure is the probe end-to-end, which dies at "mount: failed to setup loop device for /stick.img" -- leaked loop devices on this build host, not the package. Probe installs nothing, and the default profile's install -> re-install -> boot passes in the same run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Oleksandr ShyshatskyiandClaude Opus 5
|
a5300bde12 |
Split the repo into ship / build / test lanes, and add the printer replica
Two strands, both landing here because they had grown into each other in the
working tree.
The printer replica: the printer's real rootfs.squashfs chrooted under
qemu-mipsel, with /usr/prog and /usr/data from the factory image, so a package
is installed by FlashForge's own updater running on the printer's own busybox,
tar, md5sum and unTar. There are no stubs -- if the real prog partition is
missing the harness refuses to run rather than test against mocks. A release
workflow is blocked on it.
The lanes: a file's directory now says whether it can ever reach a printer.
payload/, assets/ ship
bin/, profiles/ build the package
test/, docs/ neither, ever
* the replica's own infrastructure moved out of bin/ and docker/ and in
beside the harness it belongs to (test/extract-rootfs.sh,
test/build-printer-image.sh, test/printer/Dockerfile{,.full})
* the Makefile has two runners: only RUNSIM gets the docker socket and the
replica's variables, so a build cannot reach the docker daemon at all
* replica settings moved to test.env, which nothing in a package reads
* build-time flags renamed BUILD_* (APPLY, REPORT, KLIPPER, TOOLCHANGE,
MAINSAIL, HELIX). MOD_* now means only the runtime switches written into
anvil.conf and re-read by the printer at every boot
* verify.sh fails if any file in a package is byte-identical to one in
bin/, test/, docker/ or profiles/ -- by content, because a name blacklist
would trip over whatever Mainsail happens to call common.sh
Fixed while mapping it:
* BUILD_REPORT was read only inside the BUILD_APPLY=0 branch, so four
profiles declared it and got nothing. It is honoured on both paths now,
and off outside probe: report.sh copies /etc, passwd and shadow onto the
USB stick, which is a diagnostic, not something to hand out.
* three checks in the brick lint had been quietly dead. Two were guarded on
a payload/boot.sh replaced by the firmwareExe wrapper long ago; the third
was the file glob itself, which matched neither firmwareExe nor the
init.d/S* scripts -- the UI wrapper and every service went unscanned. A
missing file is now a failure rather than a silent pass.
* run-tests.sh overwrote config.env and copied it back, leaving the config
you edited one crashed run away from being a fixture. It uses CONFIG_ENV.
* bin/make-uninstall.sh does not exist and never did; README, run-pre.sh and
full.env all pointed at it. Recovery is the stock package, which is what
test-recovery actually proves.
README is now for someone who wants to flash their printer: what it is, which
file, how to flash, what to do when it goes wrong. The rest moved to
docs/how-it-works.md, docs/building.md and docs/testing.md.
make test: 89 passed, 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|