Testing Strategy

Embedded testing budgets are finite; spending them where bugs actually live is the strategy. Safi's split: pure logic gets automated tests, hardware integration gets instrumented verification, and one technique (host harnesses) bridges the gap when a bug lives in logic but manifests on hardware.

On-target unit tests

test/ is a standalone Unity app running on the real chip:

idf.py -C test -B build-test build flash monitor

The suites, chosen because each guards logic with a history or a hazard:

Suite	Guards	Why it earns its slot
test_event_bus	subscribe/publish, unsubscribe, async payload copying, the 12-subscriber overflow returning ESP_ERR_NO_MEM	The bus is load-bearing for everything; its failure modes must be contractual, not emergent
test_mp3_prescan	ID3 skip, Xing TOC parse, seek interpolation, CBR fallback, oversized-tag rejection, garbage safety	Binary parsing of untrusted files is exactly where off-by-ones live; tests run against synthetic frames built byte-by-byte in the test itself
test_battery_monitor	the voltage curve (including monotonicity) and the shutdown latch (increment, reset on charge, reset on recovery, ignore invalid sense)	The latch decides when the device kills itself; every path is a one-liner to test and a field incident if wrong

Running on target rather than a host emulator is deliberate: the compiler, the esp_err_t semantics, and the FreeRTOS primitives under the event bus are the production ones. The cost (a board on the desk) is one this project pays anyway.

The test app carries its own partition table (test/partitions.csv) so a test run never touches production state: the factory image lands at the production ota_0 slot (0x20000) and the scratch nvs partition sits in the dead zone (0x1c000), clear of the production encrypted NVS region. Before this split, ESP-IDF's default single-app table put the test image inside the production NVS, so flashing the suite wiped live NVS pages and the next dev boot regenerated the API token, the prayer location and the saved WiFi credentials. The partition table plus the other test sdkconfig customizations are encoded in test/sdkconfig.defaults, because the HIL runner regenerates the sdkconfig from defaults and would otherwise discard them.

Note the pattern in what is tested: mp3_prescan and the battery latch were written as pure functions (bytes/state in, result out, no RTOS or filesystem calls inside) precisely so they could be tested this exhaustively. Testability was a design input, not an afterthought; where a module resisted testing, the fix was refactoring it toward purity, not mocking around it.

Host-side harnesses

When the MP3 tail-shear bug resisted on-device debugging (every experiment cost a flash cycle), the decisive move was compiling the unmodified codec_mp3.c on the desktop with three stub headers and a 30-line main() that replayed captured stream bytes in the exact chunk sizes the device uses. The bug reproduced byte-for-byte on the host; iteration dropped from minutes per attempt to seconds; the fix was verified against 600 real frames before ever touching the board.

The recipe generalizes to any pure-ish module:

1. Capture real input on the device (the stream bytes, in that case, with checksums proving fidelity).
2. Stub the esp_* includes (log macros to printf, esp_err_t to int).
3. Replay with production chunking; printf-debug like it is 1995, because now you can.

It is not committed CI (the stubs are throwaway by design); it is a documented technique, and the single highest-leverage one in this project's history.

Integration: instruments over assertions

Whole-system properties (does streaming hold realtime for an hour? does OTA round-trip?) are verified through the shipped observability instruments rather than a test framework: soak playback and require zero underruns and flat heap; run the OTA cycle and require boot-slot confirmation; screenshot-diff the UI via the debug endpoint. These checks are scripted against the REST API from a laptop, which is exactly what makes them possible against production builds on real WiFi, where the interesting failures live.

What is deliberately not tested

No mocked-peripheral unit tests for drivers (a mocked PN532 test verifies the mock), no UI snapshot suite beyond the screenshot checks (LVGL's rendering is upstream's contract), no coverage targets (coverage of glue code is a vanity metric; coverage of the prescan parser is table stakes and already 100 percent by construction). Every omission is a bet that the failure mode is either upstream's, or visible in integration instruments faster than a mock would catch it.

CI

Every push builds the firmware, the test app, and runs the static gates (CI chapter); on-target suites run at release points and after changes to their subjects, board-on-desk being the honest constraint of a hardware project.

Hardware-in-the-loop testing

The .github/workflows/hil.yml workflow automates the board-on-desk run: tools/hil/run_device_tests.sh builds and flashes the Unity test app, captures the serial run until the Unity summary, asserts 0 Failures, runs the stdio integrity gate (tools/stdio_stress_check.py) over the same capture, then restores the dev firmware and confirms it boots. The restore is an exit trap armed the moment the test app is flashed: any failure after that point (Unity failures, stdio gate, monitor timeout) still puts the dev firmware back, and if the restore itself fails the script exits non-zero with a "device left on test app" message. The PoP known-answer tests need no separate step: test_pop_derive_kat_* run inside the suite, and the end-to-end test compiles in when SAFI_POP_FACTORY_SECRET is set in the build environment (wired to a repository secret in the workflow).

One capture subtlety is handled explicitly: script(1) buffers its file output while the monitor is alive, so a Unity summary printed just before the test app exits can be invisible to the capture loop's grep. The runner re-checks the fully flushed capture after the monitor stops (with a sync) before declaring the summary pattern missing, which removes a source of false failures on green suites.

The workflow is dormant by design. It targets runs-on: [self-hosted, linux, safi-hil], a label set no GitHub-hosted runner carries, so on github.com the job never matches a runner and simply waits; it produces no failure and consumes no queue slot. There is one device in this project and it is a person's daily player, so nothing runs automatically until hardware is deliberately attached.

Two consequences of the dormancy:

• The workflow triggers on push and manual dispatch only. A pull-request trigger would leave a permanently pending check on every PR until a safi-hil runner exists. Do not make this workflow a required check before the runner is registered.
• The on-target suite runs against whatever branch was pushed; it is not a PR gate. Trigger it manually from the Actions tab to validate a branch before merging.

To activate:

1. Connect the device by USB-C so it enumerates as /dev/ttyACM0.
2. Give the runner account port access: an udev rule for the ESP32-S3 USB-Serial-JTAG vendor (SUBSYSTEM=="tty", ATTRS{idVendor}=="303a", MODE="0666") or dialout group membership.
3. Install EIM on the runner host: eim install -i v6.0.2 --idf-features=mcp --do-not-track true. The script sources the EIM activate file and falls back to eim run ... v6.0.2 when it is missing.
4. Register a self-hosted runner with the labels self-hosted, linux, safi-hil.
5. Optionally set the SAFI_POP_FACTORY_SECRET repository secret for the end-to-end PoP known-answer test.

From that point every push to main/develop touching firmware paths runs the real suite on the real chip, and the runner leaves the device on the dev firmware when it finishes, win or lose.