RuView/firmware/esp32-csi-node/RUNBOOK.md

# Firmware runbook — ESP32-C6 CSI node

Operational procedure. `README.md` explains the project; this says exactly what
to run, in what order, and what to check. Follow it top to bottom.

If you are an agent: **read this file before touching firmware.** Everything
below was learned by getting it wrong at least once.

---

## 1. Build

**One command. Copy it.** From the **repository root**, not this directory:

```bash
MSYS_NO_PATHCONV=1 docker run --rm \
  -v "$(pwd)/firmware/esp32-csi-node:/project" -w /project \
  espressif/idf:v5.4 bash -c \
  "cat sdkconfig.defaults sdkconfig.defaults.16mb sdkconfig.defaults.esp32c6 \
     > sdkconfig.defaults.build && \
   SDKCONFIG_DEFAULTS='sdkconfig.defaults.build' idf.py set-target esp32c6 && \
   idf.py build"
```

Takes ~3 minutes cold, well under a minute incremental.

### The three defaults files are NOT all automatic

IDF picks up `sdkconfig.defaults` and `sdkconfig.defaults.<target>` on its own.
It does **not** pick up `sdkconfig.defaults.16mb`, and that file is where
`CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` lives.

Build without it and you get an image that is correct in every visible respect
and **cannot roll back a bad OTA**, while `main/ota_health.c` sits in the app
expecting the capability (it compiles to a no-op without it). Silent, and only
discoverable by pushing a deliberately bad image. See §2.1.

That is why the command above concatenates all three explicitly rather than
relying on discovery.

### Never `rm -rf sdkconfig` without a backup

The README's headline example opens with `rm -rf build sdkconfig`. `sdkconfig`
is **untracked and gitignored**. `partitions_16mb.csv` was lost to exactly this,
and `CLAUDE.md` warns about it *even when a README documents it*.

```bash
cp sdkconfig "sdkconfig.backup-$(date +%Y%m%d-%H%M%S)"   # then regenerate
```

A stale `sdkconfig` is a real hazard, not a hypothetical: one found on
2026-09-03 said `FLASHSIZE=4MB`, `partitions_4mb.csv`,
`DYNAMIC_TX_BUFFER_NUM=64` — against a fleet running 16MB and 128. Anything
built from it would silently not be the configuration under test.

### Verify before you flash anything

```bash
grep -E "^CONFIG_(IDF_TARGET|ESPTOOLPY_FLASHSIZE|PARTITION_TABLE_CUSTOM_FILENAME|\
ESP_WIFI_DYNAMIC_TX_BUFFER_NUM|BOOTLOADER_APP_ROLLBACK_ENABLE)=" sdkconfig
```

All five must read:

| key | expected |
|---|---|
| `CONFIG_IDF_TARGET` | `"esp32c6"` |
| `CONFIG_ESPTOOLPY_FLASHSIZE` | `"16MB"` |
| `CONFIG_PARTITION_TABLE_CUSTOM_FILENAME` | `"partitions_16mb.csv"` |
| `CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM` | `128` |
| `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE` | `y` |

**Do not use the native `C:\Espressif\...esp-idf-v5.5.5`.** Wrong version, its
venv does not match the default python, and a container-built `build/` refuses
a native build outright. Docker only.

---

## 2. What OTA can and cannot reach

This is the distinction that decides whether a change needs a cable.

| region | offset | reachable by OTA? |
|---|---|---|
| bootloader | `0x0` | **NO — USB only** |
| partition table | `0x8000` | **NO — USB only** |
| otadata | `0xF000` | rewritten by OTA |
| ota_0 | `0x20000` | yes (4 MB) |
| ota_1 | `0x420000` | yes (4 MB) |
| coredump | `0x820000` | no |
| storage (fat) | `0x830000` | no |
| nvs | `0x9000` | never written by a flash — provisioning survives |

**OTA replaces the app partition only.** Anything in the bootloader —
`CONFIG_BOOTLOADER_*`, notably `APP_ROLLBACK_ENABLE` — or any partition-table
change **requires USB on every board**.

There is **no bootloader version on the wire**. A node's sync packet reports app
health, not bootloader capability, so you cannot determine remotely which boards
have a rollback-capable bootloader. Track it per board, or do a USB pass across
all of them.

Corollary: "every node reports health" proves the **app** is current and proves
nothing about the bootloader. Do not answer a bootloader question with app
evidence.

### 2.1 An OTA'd image must confirm itself (ADR-379)

On a rollback-capable bootloader an OTA'd image boots **`PENDING_VERIFY`**.
Until it is confirmed, the **next reset reverts to the previous slot** and
`POST /ota` refuses a second update. `main/ota_health.c` does the confirming:

- It runs only when the running image is `PENDING_VERIFY`. USB-flashed images
  (the bootloader writes `VALID`), already-confirmed images, `UNDEFINED` and
  `NEW` are left alone.
- It confirms once the STA has an IP **and** the stream sender has accepted at
  least one CSI frame, but not before `CONFIG_OTA_HEALTH_MIN_UPTIME_S`
  (default 30 s). Log: `ota_health: marked valid after ... in <ms> ms`.
- If both signals have not arrived by `CONFIG_OTA_HEALTH_TIMEOUT_S` (default
  120 s from boot), it logs `health check failed: <reason>, rolling back` and
  reboots into the previous slot. A crash or reset before confirmation is also
  a rollback.
- A pass means only "reached the network and sent CSI". It does not show that
  sensing, calibration, or any other function of the new image works.
- `GET /ota/status` reports `ota_state`. `pending_verify` means the image is
  not yet durable. `valid` means it survives a power cycle. `new` after an OTA
  means this board's bootloader has **no** rollback support.

Consequence for rollouts: wait for `ota_state: "valid"` before pushing again,
and before power-cycling. If the AP is down when the node boots, a good image
rolls back. That is expected; re-push it.

**Hardware check, one node.** It needs a node with the OTA PSK provisioned and
a rollback bootloader.

1. Note `version` and `running_partition` from `/ota/status`.
2. OTA-push the new image. Watch the log for `OTA image pending verify`, then
   `marked valid after ... in <ms> ms`. `/ota/status` must show the new
   version, the other partition, and `ota_state: "valid"`.
3. Power-cycle it. It must come back on the same version and partition with
   `ota_state: "valid"`. Before ADR-379 it came back on the old one.

**Negative check.** Build with `CONFIG_OTA_HEALTH_FORCE_FAIL=y` and push it.
The log must show `health check failed: forced failure ..., rolling back` at
about `MIN_UPTIME_S`, and the node must come back on the step-2 version and
partition. Then push a normal image to restore it. Never leave a FORCE_FAIL
image in `release_bins`.

---

## 3. Deploying

### Over the air — app-only changes

Works end-to-end and is the default for anything that is not a bootloader or
partition change. Roll to **one node**, soak it, then the rest.

**Before ADR-379, `POST /ota` never completed on this firmware.** MEASURED
2026-09-29 on node 4 (S3, v0.8.12): it overflowed the 4 KB httpd stack
(`stack overflow in task httpd`, `RTC_SW_CPU_RST`), and the node came back on
its old slot. The client sees `connection reset`. Any board still on an older
build has to be moved to a fixed one **over USB**.

The server now runs with `CONFIG_OTA_HTTPD_STACK_SIZE` (8192). Each upload
logs `httpd stack after POST /ota: N of 8192 bytes never used`. Record that
number when you verify a node, because it is the evidence the size rests on.

### Over USB — bootloader, partition table, or recovery

```bash
python -m esptool --chip esp32c6 -b 460800 \
  --before default_reset --after hard_reset write_flash \
  --flash_mode dio --flash_size 16MB --flash_freq 80m \
  0x0     build/bootloader/bootloader.bin \
  0x8000  build/partition_table/partition-table.bin \
  0xf000  build/ota_data_initial.bin \
  0x20000 build/esp32-csi-node.bin
```

`0x9000` is not in that list, so SSID, password and `node_id` survive.

### Safe single-board diagnostic flash, with a revert path

Nodes run from whichever slot `otadata` selects. **Parse it before flashing** —
two 32-byte entries at offsets 0 and 0x1000; active slot is
`(max ota_seq - 1) % 2`.

1. Back up `otadata` (8 KB) **and** the live app slot with `esptool read-flash`.
2. Flash the diagnostic build — it lands in `ota_0` and rewrites `otadata`.
3. **Revert = write the 8 KB `otadata` backup back.** Production in `ota_1` was
   never touched, so no rebuild is needed.

Validate any backup image before trusting it: byte 0 must be `0xE9`, and the
`esp_app_desc_t` magic at offset `0x20` must be `0xABCD5432`.

**Never flash the whole fleet from a workstation, and never use the server's OTA
endpoint for a diagnostic.** USB, one board, revert path secured first.

---

## 4. Serial

CH343 USB-serial at 115200.

To read a running board **without resetting it**, open with
`DtrEnable=$false; RtsEnable=$false`. esptool asserts RTS and *will* reset the
board, which loses the state you were trying to observe.

---

## 5. Changing config without a cable

Most parameters do not need a flash at all:

```bash
export RUVIEW_OTA_PSK_FILE=<path to the psk file>       # never inline the key
python config_push.py --node <node-ip> --get
python config_push.py --node <node-ip> --set led_brightness=10
```

Keys marked `assoc: true` in `config_api.c` (`wifi_ssid`, `wifi_password`,
`channel_hop_count`, `dwell_ms`, `csi_channel`) are applied as a **trial**: the
node banks its current values, reboots, and keeps the change only if it
re-associates. Everything else applies without that risk.

`config_push.py` has **no unset**. To restore a default you must set it
explicitly. Current defaults:

| key | default | source |
|---|---|---|
| `vital_interval_ms` | 1000 | `Kconfig.projbuild` |
| `swarm_heartbeat_sec` | 30 | `nvs_config.c` |
| `swarm_ingest_sec` | 5 | `nvs_config.c` |
| `beacon_period_ms` | 0 = derive from fleet size | `nvs_config.c` |

---

## 6. Reading a node remotely

`GET /api/v1/mesh` returns per-node `health`: `min_heap_kib`, `die_c`,
`thermal_state`, `tx_dbm`, `reset_reason`, and from sync proto v3 the TX-path
counters `send_fail` / `rate_skip` / `early_drop`.

`min_heap_kib` is a **minimum-since-boot watermark**, not an instantaneous
reading. It resets on reboot and only ever falls. Sampling it within a minute of
a restart tells you nothing — that mistake produced a wrong "no effect" verdict
on 2026-09-03; the real answer, once it had settled, was −56 KiB.

`/api/v1/nodes` `rssi_dbm` is **not** the AP link — it is whatever link landed
most recently and swings 20+ dB with no physical change. Filter `/api/v1/links`
by the AP's BSSIDs instead, and read `csi_fps_ema` alongside, because link count
alone hides a starved node.

---

## 7. Tunables, and which ones depend on your house

`idf.py menuconfig` exposes these. The defaults are what a nine-node ESP32-C6
fleet in one house settled on; several are **not** universal, and the point of
listing them here is that you can pick and measure rather than inherit someone
else's building.

| option | default | depends on your house? |
|---|---|---|
| `CSI_GATE_MESH_ALIGNED` | y | **yes** |
| `CSI_SEQ_DIAG` | n | diagnostic only |
| `THERMAL_MONITOR` | y | no |
| `THERMAL_THROTTLE` | n | **yes** |
| `THERMAL_THROTTLE_C` | 72 | **yes** |
| `THERMAL_HYSTERESIS_C` | 8 | no |
| `UPLINK_WATCHDOG` | y | no |
| `UPLINK_WATCHDOG_TIMEOUT_S` | 1200 | **yes** |

### The ones worth measuring yourself

**`CSI_GATE_MESH_ALIGNED`** decides whether nodes keep the *same* frames or
merely the same *number* of frames, which is what makes cross-node pairing
possible at all. Whether alignment helps depends on how much your nodes
actually overhear in common, and that is a property of your walls and your
transmitters. In one measured nine-node fleet, aligning windows bought
+0.5 pp of pairing while selecting frames by `rx_seq` bought +31.7 pp -- so do
not assume the default is doing the heavy lifting in your building.

Turn on `CSI_SEQ_DIAG` to measure it. It logs `(source MAC, rx_seq)` per CSI
callback, which is what lets you compute the fraction of frames two nodes hold
in common instead of guessing.

**`THERMAL_THROTTLE` / `THERMAL_THROTTLE_C`** depend on enclosure and placement,
not on the chip. A bare bench board never reaches 72 C; the same board sealed in
a printed case on a sunny wall can. Read `die_c` from `/api/v1/mesh` for a few
days before deciding, because throttling transmit power costs range.

**`UPLINK_WATCHDOG_TIMEOUT_S`** is a bet about your AP. 1200 s suits a network
where a wedge is rare and a reboot is cheap. If your uplink drops briefly and
often, a short timeout turns a recoverable gap into a reboot loop; if it wedges
hard, a long one leaves a node dark for twenty minutes.

### A worked example: the same two options, in one house

MEASURED on a nine-node ESP32-C6 fleet in a single detached house. **These are
our numbers, not yours.** They are here to show the shape of the decision, and
how much a building can move it.

Four arms as a 2x2 factorial, 20-minute runs, three Latin-square blocks so slow
drift landed on every arm equally. Metric is the fraction of transmissions held
in common by two or more nodes.

| arm | mean |
|---|---|
| `elapsed` (per-node gate) | 0.2434 |
| `mesh` (aligned windows) | 0.2480 |
| `seq` (keep `rx_seq` divisible by 8) | **0.5605** |
| `mesh` + `seq` | 0.5452 |

`seq` roughly doubled pairing, +31.7 pp, about 70x the measured drift floor and
reproduced in all three blocks. Mesh alignment bought **+0.5 pp** -- inside
noise. Stacking them was very slightly *worse* than `seq` alone.

**Why that happened here, and why yours may differ.** These nodes already
overheard a great deal in common -- two boards side by side heard the same
transmissions 72% of the time -- so there was a large pool of shared
transmissions for `seq` to select from, and little left for window alignment to
recover. A house with fewer shared transmitters, thicker interior walls, or
nodes spread so that few pairs hear the same radios has less for `seq` to work
with, and phase alignment may matter more rather than less. We did not measure
such a house and do not claim to know.

**It is a trade, not a free win.** At period 8 the `seq` arms carried ~75-95k
transmissions per 20 minutes against ~120-160k ungated: pairing is bought with
raw frame rate. Absolute paired transmissions per minute still rose (2629 vs
2524), because the discarded frames are disproportionately ones only one node
would have kept. Whether that trade is right depends on what consumes your
stream. On-device edge processing was **unaffected** here -- frames processed
stayed ~1900 per node per 4 minutes in both modes, because that pipeline is
separately rate-limited -- so the cost fell on the uplink, not on local sensing.
If your uplink is the scarce resource, that changes the answer.

**The two options differ in kind, not just in effect.** Mesh alignment is an
agreement between receivers: it needs a shared clock, so it inherits whatever
the sync topology does. Selecting by `rx_seq` needs no agreement at all. The
802.11 sequence control field is assigned by the TRANSMITTER and is identical
at every receiver, so it is a coordination-free name for one transmission --
two nodes that have never heard of each other, sit in different sync domains,
or run with no mesh whatsoever still keep the same subset of frames.

In the code the seq test runs before any sync check and reads only the frame;
only the mesh branch calls `c6_sync_espnow_is_valid()`. That asymmetry is the
reason `seq` had nothing to degrade in our runs, and it is worth weighing
directly: if your fleet's sync is fragile, `seq` is not merely the better
performer, it is the one with no dependency that can quietly fail.

**A limitation of our own measurement, which cuts against the mesh arm.** The
mesh gate uses mesh-time buckets only while `c6_sync_espnow_is_valid()`; a node
with no valid sync falls back to the elapsed gate rather than gating on a
meaningless epoch. That is the right behaviour, but it means our `mesh` arms
were partly `elapsed` arms: three of nine nodes never synchronised, so for those
three the setting under test was silently not in effect. Read the +0.5 pp as
"mesh where sync held, elapsed where it did not" rather than as a clean
fleet-wide measurement of alignment.

This is itself a house-dependent property. Time sync here is leader-elected and
hub-and-spoke -- one node broadcasts and the rest follow. Whether every node can
hear that leader is a question about your building, not about the firmware. In a
larger or more divided house the leader may not reach everyone, the fleet
fragments into sync domains, and any option that depends on shared time degrades
quietly for the nodes that fell out. Check `is_leader` and sync validity in
`/api/v1/mesh` before concluding that an alignment setting did or did not help.

**A prediction we got wrong, kept because it is the useful part.** Our mesh had
split into two leader domains with three nodes never synchronising at all. We
expected `seq` to win most on pairs spanning that split. Per-pair yield said
otherwise: same-domain 5.82x, cross-domain 6.24x, involving a never-synced node
**5.16x** -- the lowest, the opposite of the prediction -- and within-group
spread (3.6x-8.1x) dwarfed every between-group difference.

So `seq` is **not** a workaround for broken time sync; it wins roughly
uniformly, including where sync is healthy. The consequence is negative and
worth having: repairing the mesh split would probably not have closed the gap,
and that effort would have been spent for nothing.

### Measuring rather than guessing

A gate comparison needs a fixed environment, repeated blocks, and every arm run
on every node -- otherwise node placement is confounded with the setting. The
harness used for the published comparison is in the repository; use it rather
than eyeballing a frame rate, which moves for reasons unrelated to the setting.