readFlash() fails when trying to read a lot of data

[idolpx]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

I'm trying to backup the firmware from an ESP32 WROVER module.
Calling readFlash() with size set to the number of bytes for a full partition should return all of the data.

Actual behavior (suspected bug)

It fails when doing large reads. If I only do small ranges of bytes it seems to be fine but when I call readFlash() with size set to the size of a partition it fails after returning some of the data. If I try with a reduced baud rate it sometimes gets further. Sometimes not though. It seems something with the serial protocol is getting out of sync.

Error logs or terminal output

esptool.js
Serial port WebSerial VendorID 0x1a86 ProductID 0x7523
Connecting...
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision 3)
Features: Wi-Fi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 7c:9e:bd:28:7e:c4
Uploading stub...
Running stub...
Stub running...
Changing baudrate to 460800
Changed
If the chip does not respond to any further commands, consider using a lower baud rate.
Flash ID: 1840c8
Configuring flash size...
Auto-detected Flash size: 16MB
Detected flash size set to 16MB
Reading chip information...
Connected to ESP32
Chip ID: 0x0
Flash Size: 16MB (16MB)
Backing up firmware with 6 partitions...
Reading partition boot_loader at 0x1000 (28KB)...
Reading BOOT_LOADER (4KB) from offset 0x2000 - 14.29%...
Reading BOOT_LOADER (8KB) from offset 0x3000 - 28.57%...
Reading BOOT_LOADER (12KB) from offset 0x4000 - 42.86%...
Reading BOOT_LOADER (15.75KB) from offset 0x4efc - 56.24%...
Reading BOOT_LOADER (19.75KB) from offset 0x5efc - 70.52%...
Reading BOOT_LOADER (23.75KB) from offset 0x6efc - 84.81%...
Reading BOOT_LOADER (27.62KB) from offset 0x7e78 - 98.63%...

Steps to reproduce the behavior

This is the call I'm using.

            const data = await appState.espTool.readFlash(partition.offset, partition.size, (packet, progress, totalSize) => {
                const pct = (progress / totalSize) * 100;
                const offset = partition.offset + progress;

                console.log(
                `Reading ${partition.name.toUpperCase()} (${formatBytes(
                    progress
                )}) from offset 0x${offset.toString(16)} - ${pct.toFixed(2)}%...`
                );

                updateProgress("Backup", pct);
            });

Project release version

latest

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

MacOS

Operating system version

macOS Ventura v13.7.8

Shell

ZSH

Additional context

My app is available at:

https://flash.meatloaf.cc

Once connected click the following under "Hardware Info" to see the "Current Firmware Info" and the button to backup the firmware.

Click this to open the "Current Firmware Info" section

You can also hit "/" on the keyboard to open a terminal at the top of the screen to see the log while the backup is running.

[idolpx]

It is not reading the MD5 after the SLIP packets are received. I think the SLIP decoding gets out of sync and that may be what is causing all of the issues with this tool. I don't think it's that the baud rate is too high and needs to be lowered as it suggests.

I don't have these issues with "esptool.py".

[daveallie]

Looking at the differences in implementation, I don't think it's to do with the MD5 packets. Both implementations follow the same pseudo code for main read loop, which is where I'm seeing the same issues:

1. Send bootloader command to the device to start the flash read, supplying offset, length, packetSize, maxInflightPackets
2. Create an empty buffer
3. In a loop while there is still data to be read:
   1. Read a packet from the device
   2. Add the packet data to the buffer
   3. Write the total length of data received to the device
4. [ONLY esptool.py] Read the digest frame
5. Return the data

So I don't think it's from the digest frame as that should happen after the entire set of requested data is retrieved.

The one notable difference I can see at this point is that esptool.py uses maxInflightPackets of 64, whereas esptool-js uses maxInflightPackets of 1024. I'm going to try lowering this to 64 and testing.

[daveallie]

That change didn't seem to have any effect, however #221 seems to have resolved the instability for me. Looking at the memory profiling, reading 4MB of flash uses about an additional 200MB of JS VM memory (when compared to running the same command using the patch from #221). All of this comes from the tracingLog which is holding a persistent log of the bytes as hex, along with their ascii representation for both small segments of the read, and then each full packet adding a lot of overhead.

Can you give #221 a go a see if it also resolves it for you?