# Firmware runbook — ESP32-C6 CSI node Operational procedure. `README.md` explains the project; this says exactly what to run, in what order, and what to check. Follow it top to bottom. If you are an agent: **read this file before touching firmware.** Everything below was learned by getting it wrong at least once. --- ## 1. Build **One command. Copy it.** From the **repository root**, not this directory: ```bash MSYS_NO_PATHCONV=1 docker run --rm \ -v "$(pwd)/firmware/esp32-csi-node:/project" -w /project \ espressif/idf:v5.4 bash -c \ "cat sdkconfig.defaults sdkconfig.defaults.16mb sdkconfig.defaults.esp32c6 \ > sdkconfig.defaults.build && \ SDKCONFIG_DEFAULTS='sdkconfig.defaults.build' idf.py set-target esp32c6 && \ idf.py build" ``` Takes ~3 minutes cold, well under a minute incremental. ### The three defaults files are NOT all automatic IDF picks up `sdkconfig.defaults` and `sdkconfig.defaults.` on its own. It does **not** pick up `sdkconfig.defaults.16mb`, and that file is where `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` lives. Build without it and you get an image that is correct in every visible respect and **cannot roll back a bad OTA**, while `main/ota_health.c` sits in the app expecting the capability (it compiles to a no-op without it). Silent, and only discoverable by pushing a deliberately bad image. See §2.1. That is why the command above concatenates all three explicitly rather than relying on discovery. ### Never `rm -rf sdkconfig` without a backup The README's headline example opens with `rm -rf build sdkconfig`. `sdkconfig` is **untracked and gitignored**. `partitions_16mb.csv` was lost to exactly this, and `CLAUDE.md` warns about it *even when a README documents it*. ```bash cp sdkconfig "sdkconfig.backup-$(date +%Y%m%d-%H%M%S)" # then regenerate ``` A stale `sdkconfig` is a real hazard, not a hypothetical: one found on 2026-09-03 said `FLASHSIZE=4MB`, `partitions_4mb.csv`, `DYNAMIC_TX_BUFFER_NUM=64` — against a fleet running 16MB and 128. Anything built from it would silently not be the configuration under test. ### Verify before you flash anything ```bash grep -E "^CONFIG_(IDF_TARGET|ESPTOOLPY_FLASHSIZE|PARTITION_TABLE_CUSTOM_FILENAME|\ ESP_WIFI_DYNAMIC_TX_BUFFER_NUM|BOOTLOADER_APP_ROLLBACK_ENABLE)=" sdkconfig ``` All five must read: | key | expected | |---|---| | `CONFIG_IDF_TARGET` | `"esp32c6"` | | `CONFIG_ESPTOOLPY_FLASHSIZE` | `"16MB"` | | `CONFIG_PARTITION_TABLE_CUSTOM_FILENAME` | `"partitions_16mb.csv"` | | `CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM` | `128` | | `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE` | `y` | **Do not use the native `C:\Espressif\...esp-idf-v5.5.5`.** Wrong version, its venv does not match the default python, and a container-built `build/` refuses a native build outright. Docker only. --- ## 2. What OTA can and cannot reach This is the distinction that decides whether a change needs a cable. | region | offset | reachable by OTA? | |---|---|---| | bootloader | `0x0` | **NO — USB only** | | partition table | `0x8000` | **NO — USB only** | | otadata | `0xF000` | rewritten by OTA | | ota_0 | `0x20000` | yes (4 MB) | | ota_1 | `0x420000` | yes (4 MB) | | coredump | `0x820000` | no | | storage (fat) | `0x830000` | no | | nvs | `0x9000` | never written by a flash — provisioning survives | **OTA replaces the app partition only.** Anything in the bootloader — `CONFIG_BOOTLOADER_*`, notably `APP_ROLLBACK_ENABLE` — or any partition-table change **requires USB on every board**. There is **no bootloader version on the wire**. A node's sync packet reports app health, not bootloader capability, so you cannot determine remotely which boards have a rollback-capable bootloader. Track it per board, or do a USB pass across all of them. Corollary: "every node reports health" proves the **app** is current and proves nothing about the bootloader. Do not answer a bootloader question with app evidence. ### 2.1 An OTA'd image must confirm itself (ADR-379) On a rollback-capable bootloader an OTA'd image boots **`PENDING_VERIFY`**. Until it is confirmed, the **next reset reverts to the previous slot** and `POST /ota` refuses a second update. `main/ota_health.c` does the confirming: - It runs only when the running image is `PENDING_VERIFY`. USB-flashed images (the bootloader writes `VALID`), already-confirmed images, `UNDEFINED` and `NEW` are left alone. - It confirms once the STA has an IP **and** the stream sender has accepted at least one CSI frame, but not before `CONFIG_OTA_HEALTH_MIN_UPTIME_S` (default 30 s). Log: `ota_health: marked valid after ... in ms`. - If both signals have not arrived by `CONFIG_OTA_HEALTH_TIMEOUT_S` (default 120 s from boot), it logs `health check failed: , rolling back` and reboots into the previous slot. A crash or reset before confirmation is also a rollback. - A pass means only "reached the network and sent CSI". It does not show that sensing, calibration, or any other function of the new image works. - `GET /ota/status` reports `ota_state`. `pending_verify` means the image is not yet durable. `valid` means it survives a power cycle. `new` after an OTA means this board's bootloader has **no** rollback support. Consequence for rollouts: wait for `ota_state: "valid"` before pushing again, and before power-cycling. If the AP is down when the node boots, a good image rolls back. That is expected; re-push it. **Hardware check, one node.** It needs a node with the OTA PSK provisioned and a rollback bootloader. 1. Note `version` and `running_partition` from `/ota/status`. 2. OTA-push the new image. Watch the log for `OTA image pending verify`, then `marked valid after ... in ms`. `/ota/status` must show the new version, the other partition, and `ota_state: "valid"`. 3. Power-cycle it. It must come back on the same version and partition with `ota_state: "valid"`. Before ADR-379 it came back on the old one. **Negative check.** Build with `CONFIG_OTA_HEALTH_FORCE_FAIL=y` and push it. The log must show `health check failed: forced failure ..., rolling back` at about `MIN_UPTIME_S`, and the node must come back on the step-2 version and partition. Then push a normal image to restore it. Never leave a FORCE_FAIL image in `release_bins`. --- ## 3. Deploying ### Over the air — app-only changes Works end-to-end and is the default for anything that is not a bootloader or partition change. Roll to **one node**, soak it, then the rest. **Before ADR-379, `POST /ota` never completed on this firmware.** MEASURED 2026-09-29 on node 4 (S3, v0.8.12): it overflowed the 4 KB httpd stack (`stack overflow in task httpd`, `RTC_SW_CPU_RST`), and the node came back on its old slot. The client sees `connection reset`. Any board still on an older build has to be moved to a fixed one **over USB**. The server now runs with `CONFIG_OTA_HTTPD_STACK_SIZE` (8192). Each upload logs `httpd stack after POST /ota: N of 8192 bytes never used`. Record that number when you verify a node, because it is the evidence the size rests on. ### Over USB — bootloader, partition table, or recovery ```bash python -m esptool --chip esp32c6 -b 460800 \ --before default_reset --after hard_reset write_flash \ --flash_mode dio --flash_size 16MB --flash_freq 80m \ 0x0 build/bootloader/bootloader.bin \ 0x8000 build/partition_table/partition-table.bin \ 0xf000 build/ota_data_initial.bin \ 0x20000 build/esp32-csi-node.bin ``` `0x9000` is not in that list, so SSID, password and `node_id` survive. ### Safe single-board diagnostic flash, with a revert path Nodes run from whichever slot `otadata` selects. **Parse it before flashing** — two 32-byte entries at offsets 0 and 0x1000; active slot is `(max ota_seq - 1) % 2`. 1. Back up `otadata` (8 KB) **and** the live app slot with `esptool read-flash`. 2. Flash the diagnostic build — it lands in `ota_0` and rewrites `otadata`. 3. **Revert = write the 8 KB `otadata` backup back.** Production in `ota_1` was never touched, so no rebuild is needed. Validate any backup image before trusting it: byte 0 must be `0xE9`, and the `esp_app_desc_t` magic at offset `0x20` must be `0xABCD5432`. **Never flash the whole fleet from a workstation, and never use the server's OTA endpoint for a diagnostic.** USB, one board, revert path secured first. --- ## 4. Serial CH343 USB-serial at 115200. To read a running board **without resetting it**, open with `DtrEnable=$false; RtsEnable=$false`. esptool asserts RTS and *will* reset the board, which loses the state you were trying to observe. --- ## 5. Changing config without a cable Most parameters do not need a flash at all: ```bash export RUVIEW_OTA_PSK_FILE= # never inline the key python config_push.py --node --get python config_push.py --node --set led_brightness=10 ``` Keys marked `assoc: true` in `config_api.c` (`wifi_ssid`, `wifi_password`, `channel_hop_count`, `dwell_ms`, `csi_channel`) are applied as a **trial**: the node banks its current values, reboots, and keeps the change only if it re-associates. Everything else applies without that risk. `config_push.py` has **no unset**. To restore a default you must set it explicitly. Current defaults: | key | default | source | |---|---|---| | `vital_interval_ms` | 1000 | `Kconfig.projbuild` | | `swarm_heartbeat_sec` | 30 | `nvs_config.c` | | `swarm_ingest_sec` | 5 | `nvs_config.c` | | `beacon_period_ms` | 0 = derive from fleet size | `nvs_config.c` | --- ## 6. Reading a node remotely `GET /api/v1/mesh` returns per-node `health`: `min_heap_kib`, `die_c`, `thermal_state`, `tx_dbm`, `reset_reason`, and from sync proto v3 the TX-path counters `send_fail` / `rate_skip` / `early_drop`. `min_heap_kib` is a **minimum-since-boot watermark**, not an instantaneous reading. It resets on reboot and only ever falls. Sampling it within a minute of a restart tells you nothing — that mistake produced a wrong "no effect" verdict on 2026-09-03; the real answer, once it had settled, was −56 KiB. `/api/v1/nodes` `rssi_dbm` is **not** the AP link — it is whatever link landed most recently and swings 20+ dB with no physical change. Filter `/api/v1/links` by the AP's BSSIDs instead, and read `csi_fps_ema` alongside, because link count alone hides a starved node. --- ## 7. Tunables, and which ones depend on your house `idf.py menuconfig` exposes these. The defaults are what a nine-node ESP32-C6 fleet in one house settled on; several are **not** universal, and the point of listing them here is that you can pick and measure rather than inherit someone else's building. | option | default | depends on your house? | |---|---|---| | `CSI_GATE_MESH_ALIGNED` | y | **yes** | | `CSI_SEQ_DIAG` | n | diagnostic only | | `THERMAL_MONITOR` | y | no | | `THERMAL_THROTTLE` | n | **yes** | | `THERMAL_THROTTLE_C` | 72 | **yes** | | `THERMAL_HYSTERESIS_C` | 8 | no | | `UPLINK_WATCHDOG` | y | no | | `UPLINK_WATCHDOG_TIMEOUT_S` | 1200 | **yes** | ### The ones worth measuring yourself **`CSI_GATE_MESH_ALIGNED`** decides whether nodes keep the *same* frames or merely the same *number* of frames, which is what makes cross-node pairing possible at all. Whether alignment helps depends on how much your nodes actually overhear in common, and that is a property of your walls and your transmitters. In one measured nine-node fleet, aligning windows bought +0.5 pp of pairing while selecting frames by `rx_seq` bought +31.7 pp -- so do not assume the default is doing the heavy lifting in your building. Turn on `CSI_SEQ_DIAG` to measure it. It logs `(source MAC, rx_seq)` per CSI callback, which is what lets you compute the fraction of frames two nodes hold in common instead of guessing. **`THERMAL_THROTTLE` / `THERMAL_THROTTLE_C`** depend on enclosure and placement, not on the chip. A bare bench board never reaches 72 C; the same board sealed in a printed case on a sunny wall can. Read `die_c` from `/api/v1/mesh` for a few days before deciding, because throttling transmit power costs range. **`UPLINK_WATCHDOG_TIMEOUT_S`** is a bet about your AP. 1200 s suits a network where a wedge is rare and a reboot is cheap. If your uplink drops briefly and often, a short timeout turns a recoverable gap into a reboot loop; if it wedges hard, a long one leaves a node dark for twenty minutes. ### A worked example: the same two options, in one house MEASURED on a nine-node ESP32-C6 fleet in a single detached house. **These are our numbers, not yours.** They are here to show the shape of the decision, and how much a building can move it. Four arms as a 2x2 factorial, 20-minute runs, three Latin-square blocks so slow drift landed on every arm equally. Metric is the fraction of transmissions held in common by two or more nodes. | arm | mean | |---|---| | `elapsed` (per-node gate) | 0.2434 | | `mesh` (aligned windows) | 0.2480 | | `seq` (keep `rx_seq` divisible by 8) | **0.5605** | | `mesh` + `seq` | 0.5452 | `seq` roughly doubled pairing, +31.7 pp, about 70x the measured drift floor and reproduced in all three blocks. Mesh alignment bought **+0.5 pp** -- inside noise. Stacking them was very slightly *worse* than `seq` alone. **Why that happened here, and why yours may differ.** These nodes already overheard a great deal in common -- two boards side by side heard the same transmissions 72% of the time -- so there was a large pool of shared transmissions for `seq` to select from, and little left for window alignment to recover. A house with fewer shared transmitters, thicker interior walls, or nodes spread so that few pairs hear the same radios has less for `seq` to work with, and phase alignment may matter more rather than less. We did not measure such a house and do not claim to know. **It is a trade, not a free win.** At period 8 the `seq` arms carried ~75-95k transmissions per 20 minutes against ~120-160k ungated: pairing is bought with raw frame rate. Absolute paired transmissions per minute still rose (2629 vs 2524), because the discarded frames are disproportionately ones only one node would have kept. Whether that trade is right depends on what consumes your stream. On-device edge processing was **unaffected** here -- frames processed stayed ~1900 per node per 4 minutes in both modes, because that pipeline is separately rate-limited -- so the cost fell on the uplink, not on local sensing. If your uplink is the scarce resource, that changes the answer. **The two options differ in kind, not just in effect.** Mesh alignment is an agreement between receivers: it needs a shared clock, so it inherits whatever the sync topology does. Selecting by `rx_seq` needs no agreement at all. The 802.11 sequence control field is assigned by the TRANSMITTER and is identical at every receiver, so it is a coordination-free name for one transmission -- two nodes that have never heard of each other, sit in different sync domains, or run with no mesh whatsoever still keep the same subset of frames. In the code the seq test runs before any sync check and reads only the frame; only the mesh branch calls `c6_sync_espnow_is_valid()`. That asymmetry is the reason `seq` had nothing to degrade in our runs, and it is worth weighing directly: if your fleet's sync is fragile, `seq` is not merely the better performer, it is the one with no dependency that can quietly fail. **A limitation of our own measurement, which cuts against the mesh arm.** The mesh gate uses mesh-time buckets only while `c6_sync_espnow_is_valid()`; a node with no valid sync falls back to the elapsed gate rather than gating on a meaningless epoch. That is the right behaviour, but it means our `mesh` arms were partly `elapsed` arms: three of nine nodes never synchronised, so for those three the setting under test was silently not in effect. Read the +0.5 pp as "mesh where sync held, elapsed where it did not" rather than as a clean fleet-wide measurement of alignment. This is itself a house-dependent property. Time sync here is leader-elected and hub-and-spoke -- one node broadcasts and the rest follow. Whether every node can hear that leader is a question about your building, not about the firmware. In a larger or more divided house the leader may not reach everyone, the fleet fragments into sync domains, and any option that depends on shared time degrades quietly for the nodes that fell out. Check `is_leader` and sync validity in `/api/v1/mesh` before concluding that an alignment setting did or did not help. **A prediction we got wrong, kept because it is the useful part.** Our mesh had split into two leader domains with three nodes never synchronising at all. We expected `seq` to win most on pairs spanning that split. Per-pair yield said otherwise: same-domain 5.82x, cross-domain 6.24x, involving a never-synced node **5.16x** -- the lowest, the opposite of the prediction -- and within-group spread (3.6x-8.1x) dwarfed every between-group difference. So `seq` is **not** a workaround for broken time sync; it wins roughly uniformly, including where sync is healthy. The consequence is negative and worth having: repairing the mesh split would probably not have closed the gap, and that effort would have been spent for nothing. ### Measuring rather than guessing A gate comparison needs a fixed environment, repeated blocks, and every arm run on every node -- otherwise node placement is confounded with the setting. The harness used for the published comparison is in the repository; use it rather than eyeballing a frame rate, which moves for reasons unrelated to the setting.