From a4f80e63a2ceabdb088d41c55f92f7f1fe9fb1b3 Mon Sep 17 00:00:00 2001 From: Mathew Beane Date: Thu, 1 Oct 2026 14:44:09 -0400 Subject: [PATCH] docs(adr): ADR-379 OTA first-boot health check, with hardware evidence Records the decision behind the two preceding firmware commits: why an OTA'd image has to confirm itself, the health signals (STA IP plus one CSI frame accepted by the stream sender), the 30 s minimum uptime and 120 s deadline, the FORCE_FAIL test hook, ota_state in /ota/status, and the httpd stack sizing. MEASURED 2026-09-29: - ESP32-S3 (16 MB, rollback bootloader): OTA push to ota_1, HTTP 200; "httpd stack after POST /ota: 4900 of 8192 bytes never used"; pending_verify then valid; still valid on ota_1 after a hard reset. A FORCE_FAIL build pushed to ota_0 rolled back to ota_1 after ~30 s. - ESP32-C6 (4 MB layout, rollback bootloader): push to ota_1, HTTP 200; pending_verify then valid ~28 s later; still valid after a hard reset. Not yet verified: the 120 s no-IP timeout path, a true power cycle (an RTS hard reset was used), and forced rollback on the C6. Numbered 379: main has up to ADR-378 and open PRs claim nothing higher (2026-10-02). Co-Authored-By: claude-flow --- .../ADR-379-ota-first-boot-health-check.md | 195 ++++++++++++++++++ docs/adr/README.md | 2 + 2 files changed, 197 insertions(+) create mode 100644 docs/adr/ADR-379-ota-first-boot-health-check.md diff --git a/docs/adr/ADR-379-ota-first-boot-health-check.md b/docs/adr/ADR-379-ota-first-boot-health-check.md new file mode 100644 index 000000000..cc7144076 --- /dev/null +++ b/docs/adr/ADR-379-ota-first-boot-health-check.md @@ -0,0 +1,195 @@ +# ADR-379: An OTA'd image confirms itself with a bounded health check, or rolls back + +**Status:** Proposed. Hardware-verified on 2026-09-29 on an ESP32-S3 (node 4) and an ESP32-C6 (node 5). +**Date:** 2026-09-29 +**Numbering:** On 2026-10-02 `main` had up to ADR-378 and open PRs claimed +nothing higher. If another branch also takes 379, renumber whichever merges +second. + +## Context + +Every 16 MB node was flashed with `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` +(`sdkconfig.defaults.16mb`). With that option the bootloader starts a freshly +OTA'd image in `ESP_OTA_IMG_PENDING_VERIFY`. The app must then call +`esp_ota_mark_app_valid_cancel_rollback()`. If it doesn't, the bootloader marks +the image `ABORTED` at the next reset and boots the previous slot. + +Nothing in `firmware/esp32-csi-node/main` made that call, and `nm` on the ELF +confirmed the symbol was not linked. RUNBOOK.md said an +`ota_rollback_boot_check()` "sits in the app", but no such function existed +anywhere in the tree. The consequences: + +- An image delivered by `POST /ota` ran until the next reset, then reverted + silently. OTA updates were not durable. +- While the image was pending, `esp_ota_begin()` refused a second OTA with + `ESP_ERR_OTA_ROLLBACK_INVALID_STATE`. +- Nothing checked health, so a bad image was never rolled back on purpose. It + only reverted if it happened to crash or be power-cycled. + +A second defect meant none of this was ever reached: **OTA never succeeded +on this firmware before this change.** MEASURED on 2026-09-29, on node 4 (an +ESP32-S3 running the v0.8.12 16 MB build, with the OTA key provisioned): + +- A `POST /ota` of a valid 1,228,672 B image logged `OTA update started`, then + `stack overflow in task httpd`, then `rst:0xc RTC_SW_CPU_RST`. The node came + back on its old slot. Both attempts failed the same way. +- The cause: the server used `HTTPD_DEFAULT_CONFIG()`, which gives a 4096 B + task stack. The handler held a 1 KB receive buffer on that stack, while + calling `esp_ota_*` and formatting a float progress log. + +## Decision + +At boot, `ota_health_start()` (in `main/ota_health.c`, called early in +`app_main`) reads the running partition's state: + +| state | meaning here | action | +|---|---|---| +| `PENDING_VERIFY` | just OTA'd, rollback armed | run the health check | +| `VALID` | USB-flashed with `ota_data_initial.bin` (the bootloader writes VALID), or already confirmed | nothing | +| `UNDEFINED` | selected by a build without rollback; boots unconditionally | nothing | +| `NEW` | bootloader lacks rollback, so it never promoted the image | nothing | +| error (`NOT_FOUND`/`NOT_SUPPORTED`) | no otadata entry, or factory partition | nothing | + +For this table I read the ESP-IDF v5.5 source: `bootloader_utility.c` +(`set_actual_ota_seq`, the NEW to PENDING_VERIFY to ABORTED transitions) and +`esp_ota_ops.c`. + +**The health check.** A low-priority task polls once a second and feeds +`ota_health_step()`, which is a pure function in `ota_health.h`: + +- **Signals, both latched:** the STA got an IP (`IP_EVENT_STA_GOT_IP`, via + `ota_health_note_got_ip()` in `main.c`'s handler), and the stream sender + accepted at least one CSI frame (`csi_collector_get_send_ok_count() > 0`). + The second signal covers driver callback, serialization and `sendto()` + together. +- **Minimum uptime** `CONFIG_OTA_HEALTH_MIN_UPTIME_S` (default 30 s): the image + is not confirmed before this point even with both signals present. An image + that crashes soon after starting its pipelines still reverts, because a crash + while pending is a rollback. +- **Deadline** `CONFIG_OTA_HEALTH_TIMEOUT_S` (default 120 s, measured from + boot). Both signals are required by then. The minimum uptime is clamped to + the deadline. +- **Pass:** `esp_ota_mark_app_valid_cancel_rollback()`, which logs + `marked valid after in ms`. +- **Fail:** logs `health check failed: , rolling back`, then calls + `esp_ota_mark_app_invalid_rollback_and_reboot()`. If no other bootable slot + exists, that call returns an error. The node then logs and stays up rather + than reboot-looping. +- **Terminal verdict:** once the check has confirmed or rolled back, it never + changes. + +**Negative-test hook.** `CONFIG_OTA_HEALTH_FORCE_FAIL` (default n, test only) +makes the check roll back at the point where it would have confirmed. Use it to +prove rollback end to end. + +**Remote visibility.** `GET /ota/status` now includes `ota_state` (`valid`, +`pending_verify`, `new`, `undefined`, `other`, or `none`). An operator can +confirm durability without a serial console. + +**The httpd stack.** The OTA/WASM HTTP server now sets `config.stack_size = +CONFIG_OTA_HTTPD_STACK_SIZE`, which defaults to 8192. Three smaller changes go +with it: + +- The OTA receive buffer is static rather than on the stack. Handlers run one + at a time on the single httpd task, so one buffer is enough. +- The progress log prints an integer percent instead of a float. +- `GET /wasm/list` keeps its 2 KB JSON buffer on the heap, not the stack. + +`recv_wait_timeout` stays at 30 s. Each `POST /ota` and `POST /wasm/upload` now +logs `httpd stack after : N of M bytes never used`, so the size can +be re-tuned from device measurements. + +Sizing: this is an estimate from static data, not a device measurement. It +comes from a `-fstack-usage` build of the S3 16 MB configuration with IDF v5.5. + +- **Frame sizes along the OTA path** (bytes): + + | function | bytes | + |---|---| + | `httpd_thread` | 144 | + | `httpd_sess_process` | 32 | + | `httpd_req_new` | 176 | + | `httpd_uri` | 64 | + | `ota_upload_handler` | 192 (1,216 with the old 1 KB buffer) | + | `esp_ota_end` | 48 | + | `image_validate` | 304 | + | `esp_image_verify` | 32 | + | `image_load` | 80 | + | `process_segments` | 112 | + | SHA-256 | about 110 | + | mmap/flash read | about 190 | + +- That chain comes to about 1.5 KB now, and about 2.5 KB with the old buffer. +- Two things are not in the static data: + - libc's `vfprintf` behind every `ESP_LOG`. This build uses full newlib, + not nano, and the old progress line formatted a float. + - The flash-chip driver, which is reached through function pointers. + + Together they plausibly add 1 to 1.5 KB. That explains an overflow at 4096 + with the old handler. +- The WASM upload path adds `wasm_upload_process` (336) and + `rvf_verify_signature` (352), plus a wasm3 module load that wasn't measured. +- 8192 leaves roughly 4 to 5 KB above the estimated worst case, and costs + 4 KB of internal RAM. The high-water-mark log is the check on that estimate. + +When the build has no `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE` (the 8 MB S3 and +4 MB C6 CI lanes), both entry points compile to no-ops. + +## Consequences + +- OTA updates become durable on rollback-enabled nodes once the image passes + the check. A second OTA is accepted after confirmation, which can come as + early as `MIN_UPTIME_S`. +- A bad image now reverts instead of limping, but "bad" only means it failed + to reach the network or deliver CSI. An image that passes and then + misbehaves later is not caught. +- If the AP is down when a node boots an OTA'd image, a good image rolls back. + That is the safe direction: push it again later. +- Mock/QEMU builds that skip Wi-Fi can never pass. They must not enable + bootloader rollback, and none do today. +- Bootloader capability is still invisible remotely (RUNBOOK §2). A node + whose bootloader lacks rollback reports `ota_state: "new"` after an OTA. That + is the first remote hint of which bootloader a board has. + +## Evidence + +- Host test `firmware/esp32-csi-node/test/test_ota_health.c` runs in + `make host_tests`, which CI runs. It covers pass, each timeout reason, the + terminal verdict, force-fail, the clamp, and zero soak. SYNTHETIC: it + exercises the decision function, not a device. +- ESP-IDF v5.5 S3 and C6 builds with rollback enabled link + `esp_ota_mark_app_valid_cancel_rollback` (verified with `nm`). A build is not + hardware evidence. +- The httpd overflow is MEASURED on hardware (node 4, above). +- Hardware verification, MEASURED on 2026-09-29 on node 4 (ESP32-S3, + 16 MB, rollback bootloader, OTA key provisioned). The + build from this branch was first written over USB. The pushes were made from + a Pi 5 with the `sensor-ota-push` cog. + 1. Positive. An image built from this branch was pushed to `ota_1`: HTTP 200, + 1,230,848 B in 22 s. + - The node logged "httpd stack after POST /ota: 4900 of 8192 bytes never + used", so the upload's peak stack use is about 3.3 KB. With the old 1 KB + on-stack buffer that is over 4 KB, which is consistent with the + overflow. + - It rebooted and logged "OTA image pending verify on ota_1". + - `/ota/status` then read `ota_state: valid`. + - After a hard reset it came back on the same build in `ota_1`, still + `valid`, so the update is durable. + 2. Negative. A `CONFIG_OTA_HEALTH_FORCE_FAIL=y` build was pushed to `ota_0`: + HTTP 200. + - `/ota/status` showed it running as `pending_verify` for about 30 s. + - The node then rebooted and came back on the previous build in `ota_1`, + `valid`, so rollback works. + 3. One push attempt failed mid-upload when a serial capture was opened on the + node's USB-Serial-JTAG port at the same moment. The node reset and was + unharmed. Don't hold the USB console open while pushing OTA to a + USB-attached node. +- C6, MEASURED on 2026-09-29 on node 5 (ESP32-C6, 8 MB + flash, 4 MB layout, rollback bootloader). + - The branch build was first written over USB and the OTA key provisioned. + - A second build was pushed from the Pi 5 cog to `ota_1`: HTTP 200 in 11 s. + - `/ota/status` showed `pending_verify`, then `valid` about 28 s later. + - After a hard reset it stayed on the new build, `valid`. +- Not yet verified: the 120 s no-IP timeout path; a real power-cycle (a hard + reset via RTS was used instead); the rollback test on the C6 (run on the S3 + only). diff --git a/docs/adr/README.md b/docs/adr/README.md index c5051feaa..ccd6576ab 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -2,6 +2,7 @@ Latest proposed decisions: +- [ADR-379: An OTA'd image confirms itself with a bounded health check, or rolls back](ADR-379-ota-first-boot-health-check.md) - [ADR-378: `ruview-live` showcase views — CSI waterfall, radar fan, animated cells](ADR-378-ruview-live-showcase-views.md) - [ADR-377: `ruview-live` — a Claude Code mod shipped inside `@ruvnet/ruview`](ADR-377-ruview-live-claude-code-mod.md) - [ADR-376: `ruview` — one npm install for every RuView component](ADR-376-ruview-umbrella-npm-package.md) @@ -74,6 +75,7 @@ Statuses: **Proposed** (under discussion), **Accepted** (approved and/or impleme | [ADR-040](ADR-040-wasm-programmable-sensing.md) | WASM Programmable Sensing (Tier 3) | Accepted | | [ADR-041](ADR-041-wasm-module-collection.md) | WASM Module Collection (65 edge modules) | Accepted (hardware-validated) | | [ADR-044](ADR-044-provisioning-tool-enhancements.md) | Provisioning Tool Enhancements | Proposed | +| [ADR-379](ADR-379-ota-first-boot-health-check.md) | OTA'd image confirms itself with a bounded health check (IP + CSI frame) or rolls back; OTA httpd stack sized from device data | Proposed (host-tested; hardware-verified on one S3 and one C6) | | [ADR-110](ADR-110-esp32-c6-firmware-extension.md) | ESP32-C6 firmware extension — Wi-Fi 6 / 802.15.4 / TWT / LP-core | Accepted, P1-P10 complete, firmware-side substrate closed at **[v0.7.0-esp32](https://github.com/ruvnet/RuView/releases/tag/v0.7.0-esp32)**. Companion docs: [`WITNESS-LOG-110`](../WITNESS-LOG-110.md) (13 §A0.x entries · 99.56 % cross-board RX · **104.1 µs smoothed sync stdev** · ≤100 µs target met), [`ADR-110-REVIEW-GUIDE`](../ADR-110-REVIEW-GUIDE.md) (one-page reviewer tour), [`ADR-110-BRANCH-STATE`](../ADR-110-BRANCH-STATE.md) (coordination map vs `feat/adr-115-ha-mqtt-matter`). Host decoders + tests: Python `SyncPacketParser` (10) + Rust `wifi_densepose_hardware::SyncPacket` (15), cross-language hex pin gates drift. | ### Signal processing and sensing