Use merged_bin without overwriting NVS. (ESPTOOL-1032)

[MYCAMEL222]

Operating System

Windows 11

Esptool Version

esptool.py v4.6.2

Python Version

3.13

Chip Description

esp32wroom32

Device Description

I want to use the merged_bin function (or elf2image) to create a single file for the end user to upload. The problem is the partition.bin writes to 0x8000 and the factory.bin writes to 0x10000. NVS is at 0x9000. The users configuration settings are in NVS. When I write a merged bin file at 0x0, the NVS gets overwritten (I assume by padding). Is there a way to avoid that? I don't want the end user writing multiple .bin files if at all possible....Thanks for any help!

Hardware Configuration

No response

How is Esptool Run

terminal in VScode

Full Esptool Command Line that Was Run

No response

Esptool Output

More Information

No response

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

• I confirm I have read the troubleshooting guide.

[github-actions]

Hi @MYCAMEL222! Please be aware that (1) the following suggestions are generated by a bot and haven't been fact-checked by Espressif Systems, (2) burning eFuses and enabling security features are irreversible operations and can damage your ESP32 device. We hope that this message will help you until an Espressif Engineer looks at your issue.

---

Based on the GitHub issue, the user wants to create a merged binary file without overwriting the NVS (Non-Volatile Storage) partition that contains user configuration settings. This is a common concern when using the merge-bin command in esptool.

The issue occurs because when using merge-bin, the gaps between individual files are padded with 0xFF bytes in the default RAW format, which effectively erases any existing data in those regions when flashed.

According to the documentation, there are a couple of approaches to address this:

1. Use HEX output format instead of RAW format:
   The HEX format doesn't include padding between input files, which would avoid overwriting the NVS partition.

   esptool.py --chip ESP32 merge-bin --format hex -o merged-flash.hex --flash-mode dio --flash-size 4MB 0x1000 bootloader.bin 0x8000 partition-table.bin 0x10000 app.bin
   

   As mentioned in the ESP32 esptool documentation, the Intel HEX format offers advantages including:

   • Data is allocated to specific memory addresses without unnecessary padding
   • Each line has a checksum for error detection

2. Alternative approach - Flash files individually:
   The documentation explicitly notes: "Because gaps between the input files are padded with 0xFF bytes, when the merged binary is written then any flash sectors between the individual files will be erased. To avoid this, write the files individually." ESP32 RAW format documentation

3. Include the NVS partition in the merge:
   If you have a default NVS binary, you could include it in the merge at the correct offset (0x9000).

For the user's specific case, the HEX format approach is likely the best solution since they want to provide a single file to end users. The command would look something like:

esptool.py --chip ESP32 merge-bin --format hex -o merged-flash.hex --flash-mode dio --flash-size 4MB 0x1000 bootloader.bin 0x8000 partition-table.bin 0x10000 app.bin

Then users would flash this with:

esptool.py --chip ESP32 write-flash 0x0 merged-flash.hex

This should preserve the NVS data at 0x9000 since the HEX format doesn't include padding between the input files.

[igrr]

Linking the previous issue espressif/esptool-js#193

[peterdragun]

Hello, I am afraid that the response from the bot is not entirely correct. As mentioned, the HEX format does not include padding, but the esptool converts the HEX to a bin before flashing. During this conversion, padding is added, which will result in overriding the segments between merged binaries. I will try to take a look if this can be improved, and we can do some analysis when creating the bin (needed for flashing) from HEX, but at the moment, esptool is not trying to understand what is inside the merged HEX file.

[MYCAMEL222]

Thanks for the reply. I tried everything and could not get the merged .bin or .hex to work. FYI I am using esptool-js to do the actual flashing.

If possible, it would be great if when using merge-bin, you could specify the offset address and maximum size of the bin file for that address. ie: "esptool.py --chip ESP32 merge-bin -o merged-flash.bin --flash-freq keep --flash-mode keep --flash-size keep 0x8000 {sys.getsizeof(0x8000.bin)} 0x8000.bin 0x10000 {sys.getsizeof(0x10000.bin)} 0x10000.bin"

I know nothing really about python. I just know if I could tell the esptool to write 0x8000 only as large as I need it to or just add padding to a desired address, that would be best. If I could say 0x8000_0x9000 where it would write the 0x8000 and pad up to 0x9000 that would solve the NVS overwrite.

For now, I have modified my esptool-js to pre-populate the correct flash addresses and all the user has to do is pick the 2 .bin files and match them to the correct addresses. That works fine for now.

I'll keep an eye on this repository to see if a better solution gets figured out. I'd offer a PR, but this is way over my old head.

Thanks again.

[peterdragun]

I tried everything and could not get the merged .bin or .hex to work. FYI I am using esptool-js to do the actual flashing.

If you are using esptool-js for flashing, I don't think HEX will help you in any way; AFAIK, the Intel HEX format is not supported by esptool-js.

If possible, it would be great if when using merge-bin, you could specify the offset address and maximum size of the bin file for that address. ie: "esptool.py --chip ESP32 merge-bin -o merged-flash.bin --flash-freq keep --flash-mode keep --flash-size keep 0x8000 {sys.getsizeof(0x8000.bin)} 0x8000.bin 0x10000 {sys.getsizeof(0x10000.bin)} 0x10000.bin"

I know nothing really about python. I just know if I could tell the esptool to write 0x8000 only as large as I need it to or just add padding to a desired address, that would be best. If I could say 0x8000_0x9000 where it would write the 0x8000 and pad up to 0x9000 that would solve the NVS overwrite.

Regarding this proposal, I don't think that it will work. Unlike Intel Hex, bin files do not contain information about the address at which the bytes should be placed in the flash. There is no way to "skip" part of the addresses. So, without the padding, you will end up with binary files placed right after each other in the flash, which is not what you want. (And it would still override the NVS partition.)

[MYCAMEL222]

I'll probably keep playing with esptool-js and just make it more foolproof so even if the user picks the wrong address for the .bin file, it self corrects and puts everything in the correct order. I'm trying to make it so my end user can't mess up...;-)

[igrr]

If you are using esptool-js for flashing, I don't think HEX will help you in any way; AFAIK, the Intel HEX format is not supported by esptool-js.

Should we reopen espressif/esptool-js#193 to track this?

During this conversion, padding is added, which will result in overriding the segments between merged binaries. I will try to take a look if this can be improved

@peterdragun That would be very useful, the current behavior you described wasn't intuitive both for me and for the chatbot. Will you create an internal tracking issue for this or use this public one?

@MYCAMEL222 By the way, one (possibly simpler) solution from the other issue you opened: you can rearrange the binaries in the partition table to put NVS as the last partition. Then the merged binary won't include the NVS partition and it won't get overwritten. You shouldn't need any changes to tools this way.

[MYCAMEL222]

By the way, one (possibly simpler) solution from the other issue you opened: you can rearrange the binaries in the partition table to put NVS as the last partition. Then the merged binary won't include the NVS partition and it won't get overwritten. You shouldn't need any changes to tools this way

I wish i would have done that before I started shipping product out. The NVS is already populated with unit data at 0x9000. If I send out updates with a new NVS address, the end user has to re-enter some data. My code allows for that, but i was looking for a simple solution. I thought about somehow to read 0x9000 then write it to a new location automatically, but id still need to update the PT to point to the new location. Either way, I need to upload a PT at 0x8000 and factory at 0x10000 without messing with the NVS.

The esptool-js is a neat tool for the end user to "reprogram" their device. It took me a while to figure out how to build and publish a "custom" tool, but I got it done.

My latest firmware has a lightweight OTA capability using WiFimanager (hence the need to rewrite the PT). I'll keep messing with esptool & esptool-js. No need reopen it.

Thanks for all the quick replies!

[peterdragun]

@peterdragun That would be very useful, the current behavior you described wasn't intuitive both for me and for the chatbot. Will you create an internal tracking issue for this or use this public one?

@igrr yes, I agree that the current behavior is not expected. I will use this ticket to track the issue. We can keep this issue open until it is resolved.

[Jason2866]

@MYCAMEL222 Implementation of OTA does solve your problem. Using merged bin as initial factory image. After that all updates are done only via OTA. No setting will be lost.
Doing this since years in project Tasmota. Users are perfectly fine with this way.