fix(firmware): confirm OTA'd images with a bounded health check

Rollback-enabled nodes (CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y) boot an
OTA'd image as PENDING_VERIFY, but nothing called
esp_ota_mark_app_valid_cancel_rollback(), so every OTA reverted at the next
reset, a second OTA was refused until then, and a bad image was never
deliberately rolled back.

- main/ota_health.{c,h}: at boot, if the running image is PENDING_VERIFY,
  poll for STA got-IP and >=1 CSI frame accepted by the stream sender.
  Confirm after CONFIG_OTA_HEALTH_MIN_UPTIME_S (30 s) once both are in;
  roll back via esp_ota_mark_app_invalid_rollback_and_reboot() if they are
  not in by CONFIG_OTA_HEALTH_TIMEOUT_S (120 s from boot). VALID, UNDEFINED,
  NEW and no-otadata/factory images are left alone. The decision is a pure
  header function, host-tested in test/test_ota_health.c (make host_tests).
- CONFIG_OTA_HEALTH_FORCE_FAIL (test only) rolls back where it would have
  confirmed, for proving rollback on hardware.
- GET /ota/status reports ota_state so durability is visible remotely.
- RUNBOOK: replace the nonexistent ota_rollback_boot_check() claim and
  document the behaviour and the hardware check (2.1).

Hardware-verified on an ESP32-S3 (push, confirm, hard reset, forced
rollback) and an ESP32-C6 (push, confirm, hard reset); see ADR-379, next in
this series. Host tests: make host_tests passes, including run_ota_health.

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
Mathew Beaneandclaude-flow committed 2026-10-02 03:11:07 -04:00
1 parent d25fb8192e
commit f6a3a3579f
13 files changed
+547 -12

No files matched your search

+7
View File
@@ -98,6 +98,12 @@ For an existing provisioned node, back up its current application and inspect
`http://DEVICE_IP:8032/ota/status` before choosing an application-only update.
Writing only offset `0x20000` is safe only when the status endpoint reports
`running_partition` as `ota_0` and the downloaded image matches the board.
After an OTA, the same endpoint's `ota_state` must read `valid` before you
power-cycle the node or push again. `pending_verify` means the image has not
passed its first-boot health check yet and would revert on reset. See
[RUNBOOK §2.1](RUNBOOK.md) and ADR-379.
The full bundles do not include NVS, so the documented four-offset install
preserves WiFi and node configuration while replacing the boot and application
images.
@@ -632,6 +638,7 @@ cargo build -p wifi-densepose-wasm-edge --target wasm32-unknown-unknown --releas
| `main/nvs_config.c` / `.h` | Runtime configuration: loads Kconfig defaults, overrides from NVS |
| `main/edge_processing.c` / `.h` | Tier 0-2 DSP pipeline: SPSC ring buffer, biquad IIR filters, Welford stats, BPM extraction, presence, fall detection |
| `main/ota_update.c` / `.h` | HTTP OTA firmware update server on port 8032 |
| `main/ota_health.c` / `.h` | First-boot health check that confirms an OTA'd image or rolls it back (ADR-379) |
| `main/power_mgmt.c` / `.h` | Battery-aware light sleep duty cycling |
| `main/wasm_runtime.c` / `.h` | WASM3 interpreter: module slots, host API bindings, budget guard, per-frame dispatch |
| `main/wasm_upload.c` / `.h` | HTTP endpoints for WASM module upload, list, start, stop, delete |
+43 -3
View File
@@ -31,9 +31,9 @@ It does **not** pick up `sdkconfig.defaults.16mb`, and that file is where
`CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` lives.
Build without it and you get an image that is correct in every visible respect
and **cannot roll back a bad OTA** — while `ota_rollback_boot_check()` sits in
the app expecting the capability. Silent, and only discoverable by pushing a
deliberately bad image.
and **cannot roll back a bad OTA**, while `main/ota_health.c` sits in the app
expecting the capability (it compiles to a no-op without it). Silent, and only
discoverable by pushing a deliberately bad image. See §2.1.
That is why the command above concatenates all three explicitly rather than
relying on discovery.
@@ -104,6 +104,46 @@ Corollary: "every node reports health" proves the **app** is current and proves
nothing about the bootloader. Do not answer a bootloader question with app
evidence.
### 2.1 An OTA'd image must confirm itself (ADR-379)
On a rollback-capable bootloader an OTA'd image boots **`PENDING_VERIFY`**.
Until it is confirmed, the **next reset reverts to the previous slot** and
`POST /ota` refuses a second update. `main/ota_health.c` does the confirming:
- It runs only when the running image is `PENDING_VERIFY`. USB-flashed images
(the bootloader writes `VALID`), already-confirmed images, `UNDEFINED` and
`NEW` are left alone.
- It confirms once the STA has an IP **and** the stream sender has accepted at
least one CSI frame, but not before `CONFIG_OTA_HEALTH_MIN_UPTIME_S`
(default 30 s). Log: `ota_health: marked valid after ... in <ms> ms`.
- If both signals have not arrived by `CONFIG_OTA_HEALTH_TIMEOUT_S` (default
120 s from boot), it logs `health check failed: <reason>, rolling back` and
reboots into the previous slot. A crash or reset before confirmation is also
a rollback.
- `GET /ota/status` reports `ota_state`. `pending_verify` means the image is
not yet durable. `valid` means it survives a power cycle. `new` after an OTA
means this board's bootloader has **no** rollback support.
Consequence for rollouts: wait for `ota_state: "valid"` before pushing again,
and before power-cycling. If the AP is down when the node boots, a good image
rolls back. That is expected; re-push it.
**Hardware check, one node.** It needs a node with the OTA PSK provisioned and
a rollback bootloader.
1. Note `version` and `running_partition` from `/ota/status`.
2. OTA-push the new image. Watch the log for `OTA image pending verify`, then
`marked valid after ... in <ms> ms`. `/ota/status` must show the new
version, the other partition, and `ota_state: "valid"`.
3. Power-cycle it. It must come back on the same version and partition with
`ota_state: "valid"`. Before ADR-379 it came back on the old one.
**Negative check.** Build with `CONFIG_OTA_HEALTH_FORCE_FAIL=y` and push it.
The log must show `health check failed: forced failure ..., rolling back` at
about `MIN_UPTIME_S`, and the node must come back on the step-2 version and
partition. Then push a normal image to restore it. Never leave a FORCE_FAIL
image in `release_bins`.
---
## 3. Deploying
+1 -1
View File
@@ -2,7 +2,7 @@ set(SRCS
"main.c" "csi_collector.c" "node_log.c"
"thermal.c" "stream_sender.c" "nvs_config.c"
"serial_onboarding.c" "serial_onboarding_protocol.c"
"edge_processing.c" "ota_update.c" "power_mgmt.c"
"edge_processing.c" "ota_update.c" "ota_health.c" "power_mgmt.c"
"wasm_runtime.c" "wasm_upload.c" "rvf_parser.c"
"mmwave_sensor.c"
"swarm_bridge.c"
@@ -693,6 +693,50 @@ config UPLINK_WATCHDOG_TIMEOUT_S
endmenu
menu "OTA health check (ADR-379)"
config OTA_HEALTH_TIMEOUT_S
int "Seconds an OTA'd image has to prove itself before rolling back"
depends on BOOTLOADER_APP_ROLLBACK_ENABLE
range 30 900
default 120
help
Only applies when the running image is ESP_OTA_IMG_PENDING_VERIFY, i.e.
it was just delivered by OTA to a node whose bootloader has
CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE. USB-flashed and already-confirmed
images are never checked.
Healthy means: the STA got an IP, and the stream sender accepted at least
one CSI frame. Both must happen within this many seconds of boot, or the
image is marked invalid and the node reboots into the previous slot.
Measured from boot, and the WiFi boot wait alone can take about two
minutes against an absent AP, so an AP outage during a rollout rolls a
good image back. That is the safe direction: re-push it later.
config OTA_HEALTH_MIN_UPTIME_S
int "Minimum uptime before an OTA'd image may be confirmed"
depends on BOOTLOADER_APP_ROLLBACK_ENABLE
range 0 600
default 30
help
Holds confirmation back for this long after boot even once both signals
are in, so an image that crashes shortly after starting its pipelines
still reverts (a crash while PENDING_VERIFY is a rollback). Clamped to
OTA_HEALTH_TIMEOUT_S.
config OTA_HEALTH_FORCE_FAIL
bool "TEST ONLY: fail the OTA health check to exercise rollback"
depends on BOOTLOADER_APP_ROLLBACK_ENABLE
default n
help
Builds an image that rolls itself back at the point it would otherwise
confirm. OTA-push it to a node to prove rollback works end to end: the
node should reboot into the previous image. Has no effect when USB
flashed (such an image is not PENDING_VERIFY). Never ship this.
endmenu
config CSI_GATE_MESH_ALIGNED
bool "Align the CSI rate gate to mesh time (enables cross-node pairing)"
default y
@@ -750,6 +750,11 @@ uint16_t csi_collector_get_send_fail_count(void)
return (f > 0xFFFFu) ? 0xFFFFu : (uint16_t)f;
}
uint32_t csi_collector_get_send_ok_count(void)
{
return s_send_ok;
}
/* The gate here is fixed at compile time (mesh-aligned bucketing, falling
* back to elapsed time when unsynced), so there is no mode or period to
* report. Return the explicit sentinel rather than a number that reads as
@@ -139,6 +139,18 @@ uint16_t csi_collector_get_pkt_yield_per_sec(void);
*/
uint16_t csi_collector_get_send_fail_count(void);
/**
* Get the cumulative count of CSI frames the stream sender accepted since boot.
*
* Nonzero means the whole capture path worked at least once: the driver
* delivered a CSI callback, the frame serialized, and sendto() accepted it.
* Used by the OTA health check (ota_health.c) as its "operational" signal.
*
* @return Number of successful stream_sender_send() calls from the CSI
* callback path.
*/
uint32_t csi_collector_get_send_ok_count(void);
/** Reported by the gate accessors when this build has no selectable gate. */
#define CSI_GATE_NOT_CONFIGURABLE 0xFF
+7
View File
@@ -28,6 +28,7 @@
#include "serial_onboarding.h"
#include "edge_processing.h"
#include "ota_update.h"
#include "ota_health.h"
#include "power_mgmt.h"
#include "wasm_runtime.h"
#include "wasm_upload.h"
@@ -142,6 +143,7 @@ static void event_handler(void *arg, esp_event_base_t event_base,
ip_event_got_ip_t *event = (ip_event_got_ip_t *)event_data;
ESP_LOGI(TAG, "Got IP: " IPSTR, IP2STR(&event->ip_info.ip));
s_retry_num = 0;
ota_health_note_got_ip();
xEventGroupSetBits(s_wifi_event_group, WIFI_CONNECTED_BIT);
}
}
@@ -338,6 +340,11 @@ void app_main(void)
ESP_LOGI(TAG, "%s CSI Node (ADR-018 / ADR-110) — v%s — Node ID: %d",
target_name, app_desc->version, g_nvs_config.node_id);
/* ADR-379: an OTA'd image boots PENDING_VERIFY and reverts at the next
* reset unless it confirms itself. Start the clock before anything that
* can block (WiFi waits up to ~2 min for an AP), so the deadline is real. */
ota_health_start();
/* Apply the opt-in XIAO RF path before WiFi starts. Generic C6 boards keep
* ownership of GPIO3 and GPIO14 because the default implementation is a
* no-op unless CONFIG_C6_XIAO_ANTENNA_SELECT is enabled. */
+139
View File
@@ -0,0 +1,139 @@
/* First-boot health check for an OTA-delivered image. See ota_health.h and
* ADR-379 for why it exists; the decision itself is ota_health_step(). */
#include "ota_health.h"
#include "sdkconfig.h"
#ifdef CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_log.h"
#include "esp_ota_ops.h"
#include "esp_timer.h"
#include "csi_collector.h"
static const char *TAG = "ota_health";
#define OTA_HEALTH_POLL_MS 1000
/* Written from the event loop task, read from the health task. A single
* aligned bool is enough; it only ever goes false -> true. */
static volatile bool s_got_ip;
static const char *state_name(esp_ota_img_states_t st)
{
switch (st) {
case ESP_OTA_IMG_NEW: return "NEW";
case ESP_OTA_IMG_PENDING_VERIFY: return "PENDING_VERIFY";
case ESP_OTA_IMG_VALID: return "VALID";
case ESP_OTA_IMG_INVALID: return "INVALID";
case ESP_OTA_IMG_ABORTED: return "ABORTED";
case ESP_OTA_IMG_UNDEFINED: return "UNDEFINED";
}
return "?";
}
static void ota_health_task(void *arg)
{
(void)arg;
const ota_health_cfg_t cfg = {
.timeout_ms = (uint32_t)CONFIG_OTA_HEALTH_TIMEOUT_S * 1000u,
.min_uptime_ms = (uint32_t)CONFIG_OTA_HEALTH_MIN_UPTIME_S * 1000u,
#ifdef CONFIG_OTA_HEALTH_FORCE_FAIL
.force_fail = true,
#else
.force_fail = false,
#endif
};
ota_health_sm_t sm;
ota_health_sm_init(&sm);
for (;;) {
uint32_t now_ms = (uint32_t)(esp_timer_get_time() / 1000);
ota_health_verdict_t v = ota_health_step(&sm, &cfg, s_got_ip,
csi_collector_get_send_ok_count() > 0,
now_ms);
if (v == OTA_HEALTH_MARK_VALID) {
esp_err_t err = esp_ota_mark_app_valid_cancel_rollback();
if (err == ESP_OK) {
ESP_LOGI(TAG, "marked valid after %s in %lu ms",
sm.reason, (unsigned long)now_ms);
} else {
ESP_LOGE(TAG, "mark valid failed: %s (image stays PENDING_VERIFY "
"and reverts on the next reset)", esp_err_to_name(err));
}
break;
}
if (v == OTA_HEALTH_ROLLBACK) {
ESP_LOGE(TAG, "health check failed: %s, rolling back (%lu ms)",
sm.reason, (unsigned long)now_ms);
/* Returns only on failure: no other bootable slot. Rebooting then
* would just loop, so stay up; the next reset aborts this image. */
esp_err_t err = esp_ota_mark_app_invalid_rollback_and_reboot();
ESP_LOGE(TAG, "rollback not possible: %s -- staying on this image",
esp_err_to_name(err));
break;
}
vTaskDelay(pdMS_TO_TICKS(OTA_HEALTH_POLL_MS));
}
vTaskDelete(NULL);
}
void ota_health_start(void)
{
const esp_partition_t *running = esp_ota_get_running_partition();
esp_ota_img_states_t st;
esp_err_t err = esp_ota_get_state_partition(running, &st);
/* ESP_ERR_NOT_SUPPORTED: running from factory. ESP_ERR_NOT_FOUND: no
* otadata entry for this slot (erased otadata). Neither can be rolled
* back, so there is nothing to confirm. */
if (err != ESP_OK) {
ESP_LOGI(TAG, "running %s has no OTA state (%s): no health check",
running ? running->label : "?", esp_err_to_name(err));
return;
}
/* VALID: USB-flashed with ota_data_initial (the bootloader writes VALID)
* or already confirmed. UNDEFINED: selected by a build without rollback;
* the bootloader boots it unconditionally. NEW: the bootloader has no
* rollback support, so it never promoted the image to PENDING_VERIFY.
* None of these revert on reset. */
if (st != ESP_OTA_IMG_PENDING_VERIFY) {
ESP_LOGI(TAG, "running %s state %s: no health check",
running->label, state_name(st));
return;
}
ESP_LOGW(TAG, "OTA image pending verify on %s: need IP + CSI frame within %d s "
"(min uptime %d s)%s",
running->label, CONFIG_OTA_HEALTH_TIMEOUT_S,
CONFIG_OTA_HEALTH_MIN_UPTIME_S,
#ifdef CONFIG_OTA_HEALTH_FORCE_FAIL
" -- FORCE_FAIL test build, will roll back"
#else
""
#endif
);
if (xTaskCreate(ota_health_task, "ota_health", 4096, NULL, 2, NULL) != pdPASS) {
/* Without the task nothing confirms the image, which is the same as
* failing the check: the next reset reverts it. Say so. */
ESP_LOGE(TAG, "health task create failed; image will revert on next reset");
}
}
void ota_health_note_got_ip(void)
{
s_got_ip = true;
}
#else /* !CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE */
/* Without bootloader rollback no image is ever PENDING_VERIFY. */
void ota_health_start(void) {}
void ota_health_note_got_ip(void) {}
#endif /* CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE */
+110
View File
@@ -0,0 +1,110 @@
/* First-boot health check for an OTA-delivered image (ADR-379).
*
* With CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE the bootloader starts a freshly
* OTA'd image in ESP_OTA_IMG_PENDING_VERIFY. Until the app confirms itself,
* the next reset reverts to the previous slot and esp_ota_begin() refuses a
* second OTA. Nothing confirmed it before this module existed, so every OTA
* on a rollback-enabled node was undone by the next power cycle.
*
* The decision logic is a pure function kept in this header, like
* thermal_next_state(), so the host test exercises the exact code the device
* runs. ota_health.c only feeds it events and a clock and acts on the verdict. */
#pragma once
#include <stdbool.h>
#include <stdint.h>
typedef enum {
OTA_HEALTH_WAIT = 0, /* keep checking */
OTA_HEALTH_MARK_VALID, /* confirm the image: esp_ota_mark_app_valid_cancel_rollback() */
OTA_HEALTH_ROLLBACK, /* esp_ota_mark_app_invalid_rollback_and_reboot() */
} ota_health_verdict_t;
typedef struct {
uint32_t timeout_ms; /* deadline, measured from boot */
uint32_t min_uptime_ms; /* earliest the image may be confirmed; clamped to timeout_ms */
bool force_fail; /* test hook: roll back where it would have confirmed */
} ota_health_cfg_t;
typedef struct {
bool got_ip; /* STA associated and holds an IPv4 address (latched) */
bool csi_delivered; /* stream sender accepted >= 1 CSI frame (latched) */
ota_health_verdict_t verdict;
const char *reason; /* set once verdict leaves WAIT; static string */
} ota_health_sm_t;
static inline void ota_health_sm_init(ota_health_sm_t *sm)
{
sm->got_ip = false;
sm->csi_delivered = false;
sm->verdict = OTA_HEALTH_WAIT;
sm->reason = "waiting";
}
/**
* Advance the health check.
*
* Pure: no globals, no hardware. Both signals latch, because each proves a
* capability the image has, and a later AP outage is not evidence against the
* image. Once the verdict leaves WAIT it is terminal: later calls return it
* unchanged, so a caller that polls after acting cannot act twice.
*
* @param sm State, initialised with ota_health_sm_init().
* @param cfg Timeout, minimum uptime, force-fail hook.
* @param got_ip Event seen since the last step: STA got an IP.
* @param csi_delivered Event seen since the last step: a CSI frame was sent.
* @param now_ms Milliseconds since boot.
* @return The verdict. sm->reason names why, for the log.
*/
static inline ota_health_verdict_t ota_health_step(ota_health_sm_t *sm,
const ota_health_cfg_t *cfg,
bool got_ip,
bool csi_delivered,
uint32_t now_ms)
{
if (sm->verdict != OTA_HEALTH_WAIT) {
return sm->verdict;
}
sm->got_ip = sm->got_ip || got_ip;
sm->csi_delivered = sm->csi_delivered || csi_delivered;
/* A minimum uptime above the timeout could never be satisfied; that would
* turn every OTA into a rollback. Clamp rather than trust the config. */
uint32_t min_up = cfg->min_uptime_ms < cfg->timeout_ms
? cfg->min_uptime_ms : cfg->timeout_ms;
if (sm->got_ip && sm->csi_delivered && now_ms >= min_up) {
if (cfg->force_fail) {
sm->verdict = OTA_HEALTH_ROLLBACK;
sm->reason = "forced failure (CONFIG_OTA_HEALTH_FORCE_FAIL)";
} else {
sm->verdict = OTA_HEALTH_MARK_VALID;
sm->reason = "Wi-Fi got IP and CSI frame delivered";
}
return sm->verdict;
}
if (now_ms >= cfg->timeout_ms) {
sm->verdict = OTA_HEALTH_ROLLBACK;
if (!sm->got_ip && !sm->csi_delivered) {
sm->reason = "timeout: no IP and no CSI frame delivered";
} else if (!sm->got_ip) {
sm->reason = "timeout: no IP";
} else {
sm->reason = "timeout: no CSI frame delivered";
}
return sm->verdict;
}
return OTA_HEALTH_WAIT;
}
/* Device side (ota_health.c). No-ops when the bootloader has no rollback. */
/** Call once, early in app_main. Starts the check only if the running image
* is ESP_OTA_IMG_PENDING_VERIFY; otherwise logs the state and returns. */
void ota_health_start(void);
/** Call from the IP_EVENT_STA_GOT_IP handler. Cheap; safe from any task. */
void ota_health_note_got_ip(void);
+13 -1
View File
@@ -90,14 +90,26 @@ static esp_err_t ota_status_handler(httpd_req_t *req)
const esp_partition_t *running = esp_ota_get_running_partition();
const esp_partition_t *update = esp_ota_get_next_update_partition(NULL);
/* ADR-379: lets an operator confirm remotely that an OTA'd image left
* pending_verify (and so survives a power cycle) without a serial log. */
esp_ota_img_states_t st;
const char *ota_state = "none";
if (running && esp_ota_get_state_partition(running, &st) == ESP_OK) {
ota_state = st == ESP_OTA_IMG_VALID ? "valid" :
st == ESP_OTA_IMG_PENDING_VERIFY ? "pending_verify" :
st == ESP_OTA_IMG_NEW ? "new" :
st == ESP_OTA_IMG_UNDEFINED ? "undefined" : "other";
}
char response[512];
int len = snprintf(response, sizeof(response),
"{\"version\":\"%s\",\"date\":\"%s\",\"time\":\"%s\","
"\"running_partition\":\"%s\",\"next_partition\":\"%s\","
"\"max_size\":%lu}",
"\"ota_state\":\"%s\",\"max_size\":%lu}",
app->version, app->date, app->time,
running ? running->label : "unknown",
update ? update->label : "none",
ota_state,
(unsigned long)(update ? update->size : 0));
httpd_resp_set_type(req, "application/json");
@@ -18,8 +18,8 @@ CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_16mb.csv"
# Firmware rollback: an OTA'd image boots PENDING_VERIFY and must confirm
# itself, or the bootloader reverts on the next boot. See ota_rollback_* in
# ota_update.c for when confirmation happens and why.
# itself, or the bootloader reverts on the next boot. main/ota_health.c does
# the confirming (IP + one CSI frame within OTA_HEALTH_TIMEOUT_S), ADR-379.
CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y
# Coredump to flash. The 64K `coredump` partition has existed in
+16 -5
View File
@@ -56,10 +56,11 @@ FUZZ_JOBS ?= 1
test_vitals run_vitals test_mmwave_detect run_mmwave_detect host_tests \
test_thermal run_thermal test_csi_sanitize run_csi_sanitize \
test_c6_antenna run_c6_antenna test_serial_onboarding run_serial_onboarding \
run_delivery_contract
run_delivery_contract test_ota_health run_ota_health
all: fuzz_serialize fuzz_edge fuzz_nvs test_adr110 test_vitals test_mmwave_detect \
test_thermal test_csi_sanitize test_c6_antenna test_serial_onboarding
test_thermal test_csi_sanitize test_c6_antenna test_serial_onboarding \
test_ota_health
# --- XIAO ESP32-C6 RF switch selection ---
# Host-side truth table for the GPIO14 antenna selector. GPIO3 is always driven
@@ -133,12 +134,21 @@ test_serial_onboarding: test_serial_onboarding_protocol.c $(MAIN_DIR)/serial_onb
run_serial_onboarding: test_serial_onboarding
./test_serial_onboarding
# --- OTA first-boot health check (ADR-379) ---
# Host-side. Exercises ota_health_step() from ../main/ota_health.h directly --
# the decision the device runs to confirm or roll back an OTA'd image.
test_ota_health: test_ota_health.c $(MAIN_DIR)/ota_health.h
cc -std=c99 -Wall -Wextra -Werror -I$(MAIN_DIR) -o $@ $<
run_ota_health: test_ota_health
./test_ota_health
run_delivery_contract:
python3 test_delivery_contract.py
host_tests: run_adr110 run_vitals run_mmwave_detect run_thermal run_csi_sanitize \
run_c6_antenna run_serial_onboarding run_delivery_contract
@echo "Host tests passed (ADR-110 + CSI sanitation + vitals + mmwave + thermal + C6 antenna + USB onboarding + delivery contract)"
run_c6_antenna run_serial_onboarding run_delivery_contract run_ota_health
@echo "Host tests passed (ADR-110 + CSI sanitation + vitals + mmwave + thermal + C6 antenna + USB onboarding + delivery contract + OTA health)"
# --- Serialize fuzzer ---
# Tests csi_serialize_frame() with random wifi_csi_info_t inputs.
@@ -175,5 +185,6 @@ run_all: run_serialize run_edge run_nvs
clean:
rm -f fuzz_serialize fuzz_edge fuzz_nvs test_adr110 test_vitals test_mmwave_detect \
test_thermal test_csi_sanitize test_c6_antenna test_serial_onboarding
test_thermal test_csi_sanitize test_c6_antenna test_serial_onboarding \
test_ota_health
rm -rf corpus_serialize/ corpus_edge/ corpus_nvs/
@@ -0,0 +1,148 @@
/**
* @file test_ota_health.c
* @brief Pins the OTA first-boot health check decision (ADR-379).
*
* Exercises `ota_health_step()` from ../main/ota_health.h directly -- the same
* function the device build runs, not a copy -- so the test and the firmware
* cannot drift apart.
*
* The regression that motivated this: nothing ever called
* esp_ota_mark_app_valid_cancel_rollback(), so on rollback-enabled nodes every
* OTA'd image reverted at the next reset, and a bad image was never caught.
*
* Config under test: timeout 120 s, minimum uptime 30 s.
*/
#include <stdio.h>
#include <string.h>
#include "ota_health.h"
#define TIMEOUT_MS 120000u
#define MIN_UP_MS 30000u
static int g_failures = 0;
static const char *name(ota_health_verdict_t v)
{
switch (v) {
case OTA_HEALTH_WAIT: return "WAIT";
case OTA_HEALTH_MARK_VALID: return "MARK_VALID";
case OTA_HEALTH_ROLLBACK: return "ROLLBACK";
}
return "?";
}
static void expect(ota_health_verdict_t got, ota_health_verdict_t want,
const char *why)
{
if (got != want) {
printf(" FAIL: got %s, expected %s\n %s\n", name(got), name(want), why);
g_failures++;
}
}
static void expect_reason(const ota_health_sm_t *sm, const char *needle,
const char *why)
{
if (strstr(sm->reason, needle) == NULL) {
printf(" FAIL: reason \"%s\" lacks \"%s\"\n %s\n",
sm->reason, needle, why);
g_failures++;
}
}
static const ota_health_cfg_t CFG = { TIMEOUT_MS, MIN_UP_MS, false };
int main(void)
{
ota_health_sm_t sm;
printf("ota health check (timeout %u ms, min uptime %u ms)\n",
TIMEOUT_MS, MIN_UP_MS);
/* Healthy boot: IP at 4 s, first CSI frame at 6 s, confirmed at 30 s. */
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &CFG, false, false, 1000), OTA_HEALTH_WAIT,
"nothing yet, well inside the deadline");
expect(ota_health_step(&sm, &CFG, true, false, 4000), OTA_HEALTH_WAIT,
"IP alone is not enough");
expect(ota_health_step(&sm, &CFG, false, true, 6000), OTA_HEALTH_WAIT,
"both signals in, but below the minimum uptime");
expect(ota_health_step(&sm, &CFG, false, false, 29999), OTA_HEALTH_WAIT,
"one ms short of the minimum uptime");
expect(ota_health_step(&sm, &CFG, false, false, 30000), OTA_HEALTH_MARK_VALID,
"latched IP + CSI at the minimum uptime must confirm the image");
expect_reason(&sm, "IP", "the log names the signals that passed");
/* Terminal: a verdict never changes, even past the deadline. */
expect(ota_health_step(&sm, &CFG, false, false, 500000), OTA_HEALTH_MARK_VALID,
"a confirmed image must not later be rolled back by the same check");
/* Signals arriving after the minimum uptime confirm immediately. */
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &CFG, true, true, 45000), OTA_HEALTH_MARK_VALID,
"both signals after min uptime -> confirm now");
/* No IP by the deadline. */
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &CFG, false, false, 119999), OTA_HEALTH_WAIT,
"one ms before the deadline");
expect(ota_health_step(&sm, &CFG, false, false, 120000), OTA_HEALTH_ROLLBACK,
"no signals at the deadline -> roll back");
expect_reason(&sm, "no IP and no CSI", "both missing");
/* IP but the CSI pipeline never delivers. */
ota_health_sm_init(&sm);
ota_health_step(&sm, &CFG, true, false, 5000);
expect(ota_health_step(&sm, &CFG, false, false, 120000), OTA_HEALTH_ROLLBACK,
"associated but no CSI frame -> roll back");
expect_reason(&sm, "no CSI frame", "names the missing CSI signal");
/* CSI sent (can happen only with IP in practice) but no IP event seen. */
ota_health_sm_init(&sm);
ota_health_step(&sm, &CFG, false, true, 5000);
expect(ota_health_step(&sm, &CFG, false, false, 130000), OTA_HEALTH_ROLLBACK,
"no IP -> roll back");
expect_reason(&sm, "no IP", "names the missing IP signal");
/* Rollback is terminal too. */
expect(ota_health_step(&sm, &CFG, true, true, 131000), OTA_HEALTH_ROLLBACK,
"late signals must not resurrect a rolled-back verdict");
/* Force-fail hook rolls back exactly where it would have confirmed. */
{
const ota_health_cfg_t ff = { TIMEOUT_MS, MIN_UP_MS, true };
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &ff, true, true, 10000), OTA_HEALTH_WAIT,
"force-fail still waits for the minimum uptime");
expect(ota_health_step(&sm, &ff, false, false, 30000), OTA_HEALTH_ROLLBACK,
"force-fail must roll back instead of confirming");
expect_reason(&sm, "FORCE_FAIL", "names the test hook");
}
/* Minimum uptime above the timeout is clamped, not a guaranteed rollback. */
{
const ota_health_cfg_t bad = { 60000u, 600000u, false };
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &bad, true, true, 59999), OTA_HEALTH_WAIT,
"clamped min uptime: still waiting just before it");
expect(ota_health_step(&sm, &bad, false, false, 60000), OTA_HEALTH_MARK_VALID,
"healthy at the deadline must confirm, not roll back");
}
/* Zero minimum uptime confirms on the first step with both signals. */
{
const ota_health_cfg_t zero = { TIMEOUT_MS, 0u, false };
ota_health_sm_init(&sm);
expect(ota_health_step(&sm, &zero, true, true, 0), OTA_HEALTH_MARK_VALID,
"no soak configured -> confirm immediately");
}
if (g_failures) {
printf("FAILED: %d case(s)\n", g_failures);
return 1;
}
printf("OK: all OTA health cases pass\n");
return 0;
}