Observability and Debugging

A device in a bedroom has no attached debugger. This chapter is the instrumentation that ships in the firmware, and the house method for using it, demonstrated on the nastiest bug this project produced.

The instruments

The boot report

Peripheral init failures land in a small array shown on the splash screen and served by GET /api/system (implementation). Its job is cultural as much as technical: failures with a visible surface get fixed, warnings that scroll by do not. The otadata story is the case study.

The serial console and USB console commands

115200 baud on native USB. The firmware logs with intent, not spray: boot milestones with internal heap free/largest-block numbers after each init stage, every WiFi attempt with disconnect reason plus a post-mortem network scan, playback state changes, TLS session failures with mbedTLS error codes, and the API token on first boot.

The serial console also provides an interactive command prompt on stdin:

Command	Output / Action
token	Print the current Web API Bearer token
i2c / scan	Run live I2C bus diagnostics: tests GPIO electrical logic levels (SDA/SCL) and probes all addresses (PN532 at 0x24, DS3231 at 0x68, EEPROM at 0x57)
status	Display synchronized wall clock, free internal/PSRAM heap, SD mount state, RTC hardware status, and uptime
help	Display command summary

Core dumps

CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH writes a post-mortem (all task stacks, registers, per-task state) to the dedicated 64 KB partition on any panic. Retrieval needs nothing but USB:

idf.py coredump-info      # summary: crashed task, PC, per-task backtraces
idf.py coredump-debug     # full GDB session against the dump

The dump survives reboots until the next crash overwrites it, so "it crashed yesterday, here is the device" is a solvable ticket. Reading one effectively requires the matching ELF for the running build, which is why CI archives Safi.elf with every release binary.

Decoding backtraces

Panics print raw addresses; idf.py monitor decodes them live. For a pasted log:

xtensa-esp32s3-elf-addr2line -pfiaC -e build/Safi.elf 0x4201cb19 0x40385b2d ...

Memorize that line; it converts "gibberish from a user's screenshot" into file:line in seconds.

The screenshot endpoint

GET /api/debug/screenshot returns the live screen as BMP (design): UI verification without glass, and remote "what does it show right now" support.

Runtime telemetry

GET /api/system (heap, uptime, boot report, NVS stats), GET /api/battery, and the WebSocket's live underrun counter: the audio pipeline's single most predictive health number.

A worked example

The bug: audio streaming crashed the device seconds after decode started. Each crash's corpse differed: an interrupt watchdog spinning on a queue lock; a LoadProhibited in the system event loop; once, silent decode garbage. Classic symptoms of "the bug is not where it dies."

The method, step by step, exactly as it ran:

1. Read the corpse precisely. The watchdog dump showed a task spinning to acquire a queue spinlock whose owner field held neither "free" nor any valid core ID: not contention, corruption.
2. Get the full post-mortem. idf.py coredump-info listed every task; none held the lock. Corruption confirmed, source unknown.
3. Make the invisible visible. A linker trick, -Wl,--wrap=xPortEnterCriticalTimeout, wrapped the kernel's own lock-acquire in project code that, on a stall, printed the mux address and raw owner word before the watchdog could fire. No SDK sources were modified; the wrapper lives entirely in the app. This is the reusable trick of the story: when the kernel is the crime scene, wrap it, do not fork it.
4. Identify the victim. The printed address matched the audio command queue; a boot-time log of object addresses (temporarily added) proved it, and a hex dump of the corrupted words showed... the MP3 decoder's own saved registers. Stack frames inside a queue struct.
5. Name the mechanism. The queue was allocated directly below the audio task's stack; the stack, under minimp3's ~7 KB per-frame appetite plus an interrupt frame, punched through its canary into the neighbor. Different neighbor each build, hence the corpse-hopping. Fix and full story: tasks and memory.

Elapsed instrument time: minutes each. Elapsed staring-without-instruments time before them: hours. The moral the team wrote down: instruments first, hypotheses second, and every instrument added for a bug stays in the tree for the next one (the heap milestones and the boot report both started life this way).

Field diagnosis flow

A user reports a problem. In order of increasing effort:

1. Boot report and system info from the dashboard (no cable, user can screenshot).
2. Serial log of one boot plus the failing action (user pastes; you addr2line anything hex).
3. coredump-info if it crashed (the dump is still in flash days later).
4. Screenshot endpoint if the complaint is visual.

Four steps, no debugger, no "can you build with more logging", which on a shipped appliance is the difference between support and archaeology.