Skip reflashing similar blocks

[SergioGasquez]

If it is possible to read the hash of a particular flash region, it should also be possible I believe to implement selective reflashing: for each flash block, read the hash, compare, and only reflash if differs.

Also, cc #259 as this issue is basically the meat of flash content verification.

Originally posted by @bugadani in #479 (comment)

[bjoernQ]

Will be quite a bit of work so probably we should check how much we can benefit from it beforehand. We could create images via save-image and check the diff

[bjoernQ]

I did a quick check with an esp-hal example (embassy_i2s_sound.rs on ESP32-C3) and changed a few things between release builds:

• change a gpio number
• changed size of a heap allocated buffer
• add a println

Then I checked the output of espflash save-image if there are same flash sectors.

The only sectors which are same are the gaps between code and data (since data needs to be 64k aligned). The fill-byte is 0x00. When the gap is quite big compared to code and data, then this would give us a nice boost. Otherwise, it won't

I wonder if we could somehow optimize for this in another way? Could we just avoid flashing whatever is in that gap completely? (i.e. no need to even check the md5)

[bjoernQ]

I wonder if we could somehow optimize for this in another way? Could we just avoid flashing whatever is in that gap completely? (i.e. no need to even check the md5)

I tried changing the padding byte to 0xff and made the flasher-stub skip the writing in that case. No measurable difference for me. So probably not worth it

[AnthonyGrondin]

I did a quick check with an esp-hal example (embassy_i2s_sound.rs on ESP32-C3) and changed a few things between release builds:

* change a gpio number

* changed size of a heap allocated buffer

* add a println

Then I checked the output of espflash save-image if there are same flash sectors.

The only sectors which are same are the gaps between code and data (since data needs to be 64k aligned). The fill-byte is 0x00. When the gap is quite big compared to code and data, then this would give us a nice boost. Otherwise, it won't

I was afraid the entropy of the generated image file would be too high for this kind of improvement to be possible :(
I might try some experiments around. How do you diff the images for sectors?

EDIT: I tried compiling with different profiles. With --release, and changing only 1 character in a string, the most notable similarity is the block of 0x00 between 0x3EA0 and 0x1000 in my case. This might be the most optimization we can do for this architecture.

[bjoernQ]

For diffing in sector-sized chunks I just wrote a few lines in Rust since I wasn't able to find some utility (but didn't tried hard to find one)

fn main() {
    let args: Vec<String> = std::env::args().collect();

    let f1 = std::fs::read(&args[1]).unwrap();
    let f2 = std::fs::read(&args[2]).unwrap();


    let mut chunk = 0;
    let mut same = 0;
    let mut diff = 0;
    for page in f1.chunks(4096) {
        let dl = usize::min(4096, f2.len() - chunk*4096);
        if page == &f2[chunk*4096..][..dl] {
            same += 1;
        } else {
            diff += 1;
        }
        chunk += 1;
    }

    println!("Chunks={chunk}, Same={same}, Different={diff}");
}

Then I feed it two images generated with espflash's save-image command

[MabezDev]

Imo this is probably not worth pursuing for the v3 release, removing from the milestone.

[jessebraham]

I think probably we will not go down this path at this time. We can always revisit this in the future if there is a need/desire for it.