fix(firmware): confirm OTA'd images with a bounded health check

Rollback-enabled nodes (CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y) boot an
OTA'd image as PENDING_VERIFY, but nothing called
esp_ota_mark_app_valid_cancel_rollback(), so every OTA reverted at the next
reset, a second OTA was refused until then, and a bad image was never
deliberately rolled back.

- main/ota_health.{c,h}: at boot, if the running image is PENDING_VERIFY,
  poll for STA got-IP and >=1 CSI frame accepted by the stream sender.
  Confirm after CONFIG_OTA_HEALTH_MIN_UPTIME_S (30 s) once both are in;
  roll back via esp_ota_mark_app_invalid_rollback_and_reboot() if they are
  not in by CONFIG_OTA_HEALTH_TIMEOUT_S (120 s from boot). VALID, UNDEFINED,
  NEW and no-otadata/factory images are left alone. The decision is a pure
  header function, host-tested in test/test_ota_health.c (make host_tests).
- CONFIG_OTA_HEALTH_FORCE_FAIL (test only) rolls back where it would have
  confirmed, for proving rollback on hardware.
- GET /ota/status reports ota_state so durability is visible remotely.
- RUNBOOK: replace the nonexistent ota_rollback_boot_check() claim and
  document the behaviour and the hardware check (2.1).

Hardware-verified on an ESP32-S3 (push, confirm, hard reset, forced
rollback) and an ESP32-C6 (push, confirm, hard reset); see ADR-379, next in
this series. Host tests: make host_tests passes, including run_ota_health.

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
Mathew Beaneandclaude-flow committed 2026-10-02 03:11:07 -04:00
1 parent d25fb8192e
commit f6a3a3579f
13 files changed
+547 -12

No files matched your search

+43 -3
View File
@@ -31,9 +31,9 @@ It does **not** pick up `sdkconfig.defaults.16mb`, and that file is where
`CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` lives.
Build without it and you get an image that is correct in every visible respect
and **cannot roll back a bad OTA** — while `ota_rollback_boot_check()` sits in
the app expecting the capability. Silent, and only discoverable by pushing a
deliberately bad image.
and **cannot roll back a bad OTA**, while `main/ota_health.c` sits in the app
expecting the capability (it compiles to a no-op without it). Silent, and only
discoverable by pushing a deliberately bad image. See §2.1.
That is why the command above concatenates all three explicitly rather than
relying on discovery.
@@ -104,6 +104,46 @@ Corollary: "every node reports health" proves the **app** is current and proves
nothing about the bootloader. Do not answer a bootloader question with app
evidence.
### 2.1 An OTA'd image must confirm itself (ADR-379)
On a rollback-capable bootloader an OTA'd image boots **`PENDING_VERIFY`**.
Until it is confirmed, the **next reset reverts to the previous slot** and
`POST /ota` refuses a second update. `main/ota_health.c` does the confirming:
- It runs only when the running image is `PENDING_VERIFY`. USB-flashed images
(the bootloader writes `VALID`), already-confirmed images, `UNDEFINED` and
`NEW` are left alone.
- It confirms once the STA has an IP **and** the stream sender has accepted at
least one CSI frame, but not before `CONFIG_OTA_HEALTH_MIN_UPTIME_S`
(default 30 s). Log: `ota_health: marked valid after ... in <ms> ms`.
- If both signals have not arrived by `CONFIG_OTA_HEALTH_TIMEOUT_S` (default
120 s from boot), it logs `health check failed: <reason>, rolling back` and
reboots into the previous slot. A crash or reset before confirmation is also
a rollback.
- `GET /ota/status` reports `ota_state`. `pending_verify` means the image is
not yet durable. `valid` means it survives a power cycle. `new` after an OTA
means this board's bootloader has **no** rollback support.
Consequence for rollouts: wait for `ota_state: "valid"` before pushing again,
and before power-cycling. If the AP is down when the node boots, a good image
rolls back. That is expected; re-push it.
**Hardware check, one node.** It needs a node with the OTA PSK provisioned and
a rollback bootloader.
1. Note `version` and `running_partition` from `/ota/status`.
2. OTA-push the new image. Watch the log for `OTA image pending verify`, then
`marked valid after ... in <ms> ms`. `/ota/status` must show the new
version, the other partition, and `ota_state: "valid"`.
3. Power-cycle it. It must come back on the same version and partition with
`ota_state: "valid"`. Before ADR-379 it came back on the old one.
**Negative check.** Build with `CONFIG_OTA_HEALTH_FORCE_FAIL=y` and push it.
The log must show `health check failed: forced failure ..., rolling back` at
about `MIN_UPTIME_S`, and the node must come back on the step-2 version and
partition. Then push a normal image to restore it. Never leave a FORCE_FAIL
image in `release_bins`.
---
## 3. Deploying