Factory Validation Runbook

This runbook turns the provisioning chapter into a repeatable manufacturing line: every step a production unit passes through between "firmware builds" and "a validated, labeled device leaves the bench". Each stage names the tool that enforces it, what to verify, and where a human must sign off.

eFuse burns are IRREVERSIBLE

A burned eFuse cannot be undone, rewritten, or worked around. A wrong key digest, a wrong key purpose, or a wrong block turns the unit into a brick that no amount of reflashing fixes. Every irreversible step in this runbook sits behind a sign-off gate, and the entire flow must be validated on a sacrificial board before it is applied to any unit that matters.

Current status

The full flow below is validated in writing, not yet on hardware. The device on hand is a person's daily player, not a candidate for irreversible burns, so executing stages 3 and 4 is deliberately deferred until a sacrificial device exists. Until then: stages 0 through 2 can be rehearsed end to end (they are reversible), and this document is the checklist the day the sacrificial board arrives.

Stage 0: environment and key material

Everything that can be checked before a device is plugged in lives in tools/production_preflight.sh, which refuses to continue unless all of the following hold:

Input	Requirement
SAFI_SECURE_BOOT_SIGNING_KEY	Absolute path to the RSA-3072 PEM private key
SAFI_FLASH_ENCRYPTION_KEY	Absolute path to the 32-byte flash encryption key file
SAFI_POP_FACTORY_SECRET	Even-length hex string, at least 32 chars, from which each device's provisioning PoP is derived
sdkconfig.production	Present, with every required gate on and every forbidden option off
Tooling	idf.py (exported IDF environment) and espsecure.py on PATH

Run it exactly as written:

tools/production_preflight.sh          # full check, expects the signing key
tools/production_preflight.sh --strict # add clean tree + release tag requirements

Any failure here stops the line. The key material itself lives in your vault, never in the repository or on the bench PC.

Stage 1: the production build

The production build is a CMake preset, not a menuconfig conversation:

idf.py --preset production build

The preset pins SDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.production and keeps its scratch sdkconfig at build-prod/sdkconfig, so a production build can never accidentally consume or overwrite a developer's build/ configuration. The preset also selects Secure Boot V2 (RSA-3072), flash encryption in release mode, NVS encryption, the certificate bundle, and anti-rollback with secure version 1.

Production uses partitions_production.csv with the partition table at offset 0x10000, not the development partitions.csv at 0x8000. The Secure Boot V2 bootloader grows to 0x9000 bytes, which does not fit under the 0x8000 development offset, so sdkconfig.production overrides the table offset and filename; the development layout is untouched so existing devices keep their NVS offsets.

The preset is not the last word: CONFIG_SAFI_ENFORCE_PRODUCTION_GATES makes main/app_config.h refuse to compile if any required gate is missing or any forbidden option is set (the #error list covers insecure OTA pull, development-mode flash encryption, and JTAG after secure boot). A production binary that exists is therefore, by construction, one that passed the security checklist. tools/production_preflight.sh re-checks the same surface at flash time for defense in depth.

Record for each release: the git commit, the v* tag if any, and the SHA-256 of the resulting build-prod/Safi.bin.

Stage 2: bench functional test, before any burn

This stage is fully reversible and must complete before stage 3 begins.

1. Flash the production image set without burning anything: tools/provision_device.sh without --burn-efuses (pass the build dir, certs dir, NVS image, serial, and port; run --dry-run first to review the plan).
2. Power the unit and watch the boot: the splash-screen boot report must be empty (no SD, NFC, or audio failures).
3. Run the full field validation checklist. Everything audio, network, NFC, and dashboard related is testable here, with one exception: the OTA rollback drill in the checklist requires secure boot, so it is exercised in stage 4 instead.
4. Confirm the label PoP: tools/provision_device.sh derives HMAC-SHA256(factory secret, base WiFi MAC), maps the first 16 digest bytes onto [A-Za-z0-9], and prints the 16-character PoP together with the provisioning username safi. Verify the same PoP comes out of tools/nvs_mass_provisioning.py (labels.csv carries serial,cn,fingerprint,pop), and that it matches what the firmware derives at boot (main/pop_derive.c). A PoP mismatch means the factory secret differs between build and label tooling, and the unit will refuse provisioning sessions.

Gate A: sign-off required before stage 3. The bench unit passed the full checklist, the PoP matches across tool and firmware, and this unit is understood to be sacrificial.

Stage 3: the eFuse burns

This is the irreversible stage. tools/provision_device.sh only burns when it is told twice: --burn-efuses plus the literal --i-understand-irreversible flag, followed by typing yes on the prompt. Read the plan it prints before confirming.

The burn sequence, in the order the tool runs it:

Order	Operation	eFuse	Effect
1	Secure Boot V2 digest key	BLOCK_KEY0	ROM verifies the bootloader against RSA-3072
2	Flash encryption key	BLOCK_KEY1, XTS_AES_128_KEY	External flash reads as noise without it
3	NVS HMAC key	BLOCK_KEY2, HMAC_UP	NVS encryption key; key id 2, never 1
4	Enable fuses	SECURE_BOOT_EN, SPI_BOOT_CRYPT_CNT=7, DIS_DOWNLOAD_*	Locks the boot and download paths

The key id matters: the NVS HMAC key uses CONFIG_NVS_SEC_HMAC_EFUSE_KEY_ID=2 because key id 1 addresses BLOCK_KEY1, which holds the XTS key. Burning the HMAC key into BLOCK_KEY1 instead would reuse a key purpose and break NVS at best.

The tool asserts the build directory carries the production partition table with nvs_key at 0xB50000 before it will burn anything, so a development layout cannot be frozen into the unit by mistake. The production table (partitions_production.csv) sits at offset 0x10000 for the Secure Boot V2 bootloader headroom, while the development partitions.csv keeps its 0x8000 layout.

The first boot after the encryption fuses are enabled re-encrypts flash in place. Power must not fail during that boot. This happens on the bench, never at a customer.

Gate B: sign-off required before the batch. On the sacrificial board only: the burn completed, the first encrypted boot finished cleanly, and stage 4 passes. Until Gate B is signed, no other unit gets burned.

Stage 4: post-burn verification

On the freshly burned sacrificial unit:

1. Boot normally and confirm the boot report is still empty and the dashboard comes up.
2. Negative test, secure boot: attempt to flash or OTA an unsigned image. The bootloader must refuse it, and the unit must come back on the signed firmware with no intervention.
3. Positive test, rollback: run the full OTA drill from the field validation checklist, including forcing a bad signed image and watching the bootloader roll back to the previous slot (anti-rollback secure version 1).
4. Negative test, provisioning: start a provisioning session and enter a wrong PoP; the session must be refused. Enter the label PoP; it must be accepted (security 2, SRP6a plus AES-GCM).
5. Confirm the device's identity items survive a factory reset: the reset regenerates the API token and wipes settings, while the burned key material is untouched by design.

Gate C: sign-off required before shipment. All stage 4 checks pass on the sacrificial unit, and the same signer records the batch roll-out decision.

Stage 5: batch roll-out

Only after Gate C: repeat stages 2 through 4 per unit, without re-validating the firmware image itself (the release artifact is frozen at stage 1). Every unit gets:

• Its production image set from the same frozen build.
• Its per-device NVS image and certificate (tools/nvs_mass_provisioning.py, tools/generate_production_certs.sh).
• Its printed label: serial, common name, cert fingerprint, PoP, and the provisioning username safi.
• Its entry in the sign-off log.

Sign-off log

Gate	When	Requires	Signer	Date
A	Before the first eFuse burn	Stage 2 complete, PoP verified, unit is sacrificial
B	Before any other unit is burned	Sacrificial burn clean, stage 4 passed
C	Before shipment	Full checklist passed on the unit

A gate without a signer does not open. That rule, not any single step above, is the runbook.

English-only page

The Arabic translation of this page is out of scope for now and tracked as todo in safi-docs/TRANSLATIONS.md.