ESP32-XX: fix hardware flash encryption issue when updating images

[almir-okato]

When hardware flash encryption is enabled, force expected erased value (0xFF) into flash when erasing a region, and also always do a real erase before writing data into flash.

This is handled on this implementation because MCUboot's state machine relies on erased valued data (0xFF) readed from a previously erased region that was not written yet, however when hardware flash encryption is enabled, the flash read always decrypts whats being read from flash, thus a region that was erased would not be read as what MCUboot expected (0xFF).

This PR also bumps used IDF code version to v5.1.6.

How to test it within Zephyr on ESP32-S3 using a "zeroed" key (DEVELOPMENT AND TEST ONLY, so the device will not be bricked) and updating through Wi-Fi:

Use Zephyr branch from this PR:
zephyrproject-rtos/zephyr#90442 (already merged into Zephyr's main)
Use hal_espressif branch from this PR:
zephyrproject-rtos/hal_espressif#478 (already merged into hal_espressif's main)

Prepare and enable Flash Encryption on ESP32-S3:

• Erase the flash:

esptool.py -p /dev/ttyUSB0 erase_flash --force

• Create a "zeroed" key (TEST ONLY, so the device will not be bricked):

dd if=/dev/zero of=zero_key.bin bs=1 count=32

• Burn the key into the eFuses:

espefuse.py -p /dev/ttyUSB0 burn_key BLOCK_KEY4 zero_key.bin XTS_AES_128_KEY

• Burn the eFuse that enables Flash Encryption:

espefuse.py burn_efuse SPI_BOOT_CRYPT_CNT

Building the SMPMGR sample application on Zephyr:

cd <ZEPHYRPROJECT_DIR>/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/

• Create an overlay file for ESP32-S3 with the content:
  <ZEPHYRPROJECT_DIR>/zephyr/samples/subsys/mgmt/mcumgr/smp_svr/boards/esp32s3_devkitm_procpu.overlay

&flash0 {
       write-block-size = <32>;
};

• Build the first image, replacing <IP_ADDRESS> with a valid IP on your network:

west build -b esp32s3_devkitm/esp32s3/procpu -p --build-dir build_1st -- -DEXTRA_CONF_FILE="overlay-udp.conf" -DCONFIG_MAIN_STACK_SIZE=8192 -DCONFIG_MCUMGR_TRANSPORT_UDP_STACK_SIZE=4096  -DCONFIG_NET_CONFIG_MY_IPV4_ADDR=\"<IP_ADDRESS>\" -DCONFIG_NET_CONFIG_MY_IPV4_NETMASK=\"255.255.255.0\" -DCONFIG_NET_CONFIG_MY_IPV4_GW=\"\" -DCONFIG_WIFI=y -DCONFIG_NET_SHELL=y -DCONFIG_NET_L2_WIFI_SHELL=y -DCONFIG_BOOTLOADER_MCUBOOT=y -DCONFIG_MCUBOOT_EXTRA_IMGTOOL_ARGS=\"--align\ 32\ --max-align\ 32\ --pad\" -DCONFIG_MCUBOOT_BOOTLOADER_MODE_SWAP_SCRATCH=y -DCONFIG_ESP_FLASH_ENCRYPTION=y

• Flash the image (will be force encrypted, since the encryption is already enabled on the device and we are using the DEVELOPMENT configs):

esptool.py -p /dev/ttyUSB0 -b 2000000 --no-stub --after no_reset write_flash  --flash_mode dio  --flash_freq 40m --encrypt 0x0020000 build_1st/zephyr/zephyr.signed.bin --force

• Build the update image, it can be the same sample on other build_dir:

west build -b esp32s3_devkitm/esp32s3/procpu -p --build-dir build_2nd -- -DEXTRA_CONF_FILE="overlay-udp.conf" -DCONFIG_MAIN_STACK_SIZE=8192 -DCONFIG_MCUMGR_TRANSPORT_UDP_STACK_SIZE=4096  -DCONFIG_NET_CONFIG_MY_IPV4_ADDR=\"<IP_ADDRESS>\" -DCONFIG_NET_CONFIG_MY_IPV4_NETMASK=\"255.255.255.0\" -DCONFIG_NET_CONFIG_MY_IPV4_GW=\"\" -DCONFIG_WIFI=y -DCONFIG_NET_SHELL=y -DCONFIG_NET_L2_WIFI_SHELL=y -DCONFIG_BOOTLOADER_MCUBOOT=y -DCONFIG_MCUBOOT_EXTRA_IMGTOOL_ARGS=\"--align\ 32\ --max-align\ 32\ --pad\" -DCONFIG_MCUBOOT_BOOTLOADER_MODE_SWAP_SCRATCH=y -DCONFIG_ESP_FLASH_ENCRYPTION=y

Building and flashing MCUboot:

cd <MCUBOOT_DIR>/boot/espressif

(MCUBOOT_DIR should be the directory from Zephyr environment: <ZEPHYRPROJECT_DIR>/bootloader/mcuboot)

• Modify <MCUBOOT_DIR>/boot/espressif/port/esp32s3/bootloader.conf with:

CONFIG_ESP_FLASH_SIZE=8MB
CONFIG_ESP_BOOTLOADER_SIZE=0xF000
CONFIG_ESP_BOOTLOADER_OFFSET=0x0000
CONFIG_ESP_IMAGE0_PRIMARY_START_ADDRESS=0x20000
CONFIG_ESP_APPLICATION_SIZE=0x150000
CONFIG_ESP_IMAGE0_SECONDARY_START_ADDRESS=0x170000
CONFIG_ESP_MCUBOOT_WDT_ENABLE=y
CONFIG_ESP_SCRATCH_OFFSET=0x3E0000
CONFIG_ESP_SCRATCH_SIZE=0x1F000

CONFIG_ESP_CONSOLE_UART=y
CONFIG_ESP_CONSOLE_UART_NUM=0

# Hardware Flash Encryption related options
CONFIG_SECURE_FLASH_ENC_ENABLED=1
CONFIG_SECURE_FLASH_UART_BOOTLOADER_ALLOW_ENC=1
CONFIG_SECURE_FLASH_UART_BOOTLOADER_ALLOW_DEC=1
CONFIG_SECURE_FLASH_UART_BOOTLOADER_ALLOW_CACHE=1
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT=1
CONFIG_SECURE_BOOT_ALLOW_JTAG=1
CONFIG_SECURE_BOOT_ALLOW_ROM_BASIC=1

• Build:

rm -rf build/ && cmake -DCMAKE_TOOLCHAIN_FILE=tools/toolchain-esp32s3.cmake -DMCUBOOT_TARGET=esp32s3  -DMCUBOOT_FLASH_PORT=/dev/ttyUSB0 -DESP_HAL_PATH=<ZEPHYRPROJECT_DIR>/modules/hal/espressif -B build -GNinja && ninja -C build 

• Flash:

esptool.py -p /dev/ttyUSB0 -b 2000000 --no-stub --after no_reset write_flash  --flash_mode dio  --flash_freq 40m --encrypt 0x0000 build/mcuboot_esp32s3.bin --force

Workaround for empty slot1 and scratch regions when testing:

Testing like described, as Flash Encryption would be already enabled, reading an empty region (physical 0xFF values) would result in garbage as the read decryption would decrypt the data unconditionally. So generate a 0xFFs binary file and flash it to the regions in order to avoid an unexpected behavior:

dd if=/dev/zero ibs=1 count=126976 | LC_ALL=C tr "\000" "\377" >ffff.bin
esptool.py -p /dev/ttyUSB0 -b 2000000 --no-stub --after no_reset write_flash  --flash_mode dio  --flash_freq 40m --encrypt 0x3e0000 ffff.bin --force
dd if=/dev/zero ibs=1 count=1376256 | LC_ALL=C tr "\000" "\377" >ffff_slot1.bin
esptool.py -p /dev/ttyUSB0 -b 2000000 --no-stub --after no_reset write_flash  --flash_mode dio  --flash_freq 40m --encrypt 0x170000 ffff_slot1.bin --force

• Now open the serial monitor and press the devkit's reset button.

Upload the update image through wifi using smpmgr:

• Using the serial monitor, after the Zephyr application boot and starts the shell, connect the device to the wifi:

wifi connect --key-mgmt 1 --ssid "<SSID>" --passphrase "<PASSWORD>"

• After the connection, on another terminal, upload the update image to the device:

smpmgr --timeout 10 --ip <IP_ADDRESS> upgrade --slot 1 build_2nd/zephyr/zephyr.signed.bin

• After upload completes, reset the device and wait the swap process finishes:

smpmgr --timeout 10 --ip <IP_ADDRESS> os reset

• The new image can be confirmed through a smpmgr command. Connect the device to wifi again, fetch the image information and use its hash to confirm the image.

smpmgr --timeout 10 --ip <IP_ADDRESS> image state-read

smpmgr --timeout 10 --ip <IP_ADDRESS> image state-write <SLOT0_IMAGE_HASH> true

• If the new image isn't confirmed as above, the next boot will revert it and swap to the first one.

[almir-okato]

@utzig @marekmatej could you please take a look?

[almir-okato]

@d3zd3z could you please take a look? It's missing a codeowner review.

[almir-okato]

@de-nordic or @nvlsianpu would you mind to take a look on this, please? It's pending a review of someone with write access on the repository.
The changes affects only the Espressif Port and its CI pipeline.

[gustavonihei]

PR looks good, just found some minor nitpicks.