flash-size agnostic builds

[d-a-v]

(edit 2021/12/01)

This PR adds a new option in the flash size configuration options list called
Mapping defined by Hardware and Sketch.

This option is useful because:

• Users need not to care about flash configuration
  (If the option is selected. It is not currently the default)

• A binary can indifferently be flashed over 512k or 16MB flash chips, or esp8285 if dout is selected

  • FS and EEPROM location will automatically match legacy addresses as if flash size was correctly selected

  • Three configurations are available, they reflects standard / current menus flash mapping

    • FLASH_MAP_OTA_FS (default if FLASH_MAP_SETUP_CONFIG() is not used)
    • FLASH_MAP_MAX_FS
    • FLASH_MAP_NO_FS

    Sketch can use FLASH_MAP_SETUP_CONFIG(config) to select a configuration, for example: FLASH_MAP_SETUP_CONFIG(FLASH_MAP_MAX_FS)

  • Sketch can also define another configuration by providing a custom mapping structure.

• No more FS overwrite / reformatting when bad configuration is selected and flashed

• Binary releases for projects can see their number of builds reduced

When not enabled, this PR makes no change. and flash size and configuration needs to be correctly selected as before (with the same symbols).

Documentation update:

• filesystem
• faq

When the option is enabled, the magic word at physical address 0x0 in flash which contains "software" flash size is ignored; its reading is disabled and is replaced by reading the "chip vendor id" (ESP.getFlashChipRealSize()).

Fixes: #6679
Fixes: #2321
Closes: #6806 via #6813

[d-a-v]

@TD-er for a start I ran the same binary DOUT lib-eeprom read/write sketch on esp8285(1M) and esp8266(4M) with good results.

#include <EEPROM.h>

#if AUTOFLASHSIZE
void flashinit()
{
  FLASHMAPCONFIG(FLASH_MAP_MAX_FS); // maximum for FS
  // or FLASH_MAP_OTA_FS (maximum space for OTA, rest of free space for FS)
  // or FLASH_MAP_NO_FS (no FS needed)
}
#endif

void setup() {
  Serial.begin(115200);

  Serial.printf("\n\nsize=%d\nsize=%d\naddr=0x%08x\n\n", ESP.getFlashChipSize(), ESP.getFlashChipRealSize(), EEPROM_start);

  EEPROM.begin(512);
  Serial.printf("%02x %02x %02x %02x\nwait 10s\n",
                EEPROM.read(0),
                EEPROM.read(1),
                EEPROM.read(2),
                EEPROM.read(3));
  delay(10000);
  Serial.printf("write\n");
  EEPROM.write(0, 0xde);
  EEPROM.write(1, 0xad);
  EEPROM.write(2, 0xbe);
  EEPROM.write(3, 0xef);
  EEPROM.commit();
}

void loop() {
}

esp8285:

size=1048576
size=1048576
addr=0x402fb000

esp8266:

size=4194304
size=4194304
addr=0x405fb000

[TD-er]

So, you've already started with it :)
The earliest test I can do with it is tomorrow evening. (I also have 2M modules)
Have got to finish something else now and will be away most of tomorrow.

[TD-er]

I have not yet tested it, but looked through the code.
The flashInit function is called from the core main function and does look at the flash size to pick a layout.
Can I later set it again to another layout at runtime?
Or is it possible to set the magic byte to define some alternate setting and then reboot to load the alternate setting?

The reason I'm asking is that I cannot support a single binary for 4M flash with either 1 MB SPIFFS (default for all nodes out there running my software) or 2 MB SPIFFS if it is only looking at the flash size.
So either I have to have my own detection algorithm to see where the SPIFFS is located and change at runtime, or only try once to find any existing 1M SPIFFS and if it isn't there switch it to the 4M2M option, write that to the magic byte and reboot.
If it still cannot find a valid SPIFFS, then format it as a 2M SPIFFS.,

N.B. the reason I have not tested it yet, is mainly because of the amount of work needed to get a PR ready within PlatformIO. I also just made a feature request for it: platformio/platformio-core#3239
In the retrieved packages directory of PIO, the git config is not complete or correct.
I do see the pull requests on origin, but when I want to checkout based on a PR, I get an error like "missing delta bases"

[d-a-v]

The flashInit function is called from the core main function and does look at the flash size to pick a layout.

That is on purpose.
The parameter (what you call layout) is in fact an array of layouts indexed by the flash chip size.
There are three hardcoded layout sets, but you can use a self-defined one.

Can I later set it again to another layout at runtime?

FSes and EEPROM should be closed before doing that, but yes I don't see any reason against that.

[d-a-v]

As @earlephilhower said elsewhere, we cannot autodetect SPIFFS because there is no signature. I don't know about LittleFS.
In either case the information about location and size can anyway be written somewhere and allow to build the right descriptor.

[TD-er]

For my own purpose, I could also use the EPROM part to store something with a very specific signature to help detect the used offset.
It does probably have 2 possible locations at most for any given flash size.

[d-a-v]

this file contains all the addresses computed by boards.txt.py.
I think the easier path is to use the example provided in OP then change FS addresses:

... (later)
close_all_FSes(); // or not
FS_start = myFS_start;
FS_stop = myFS_end;
SPIFFS.begin()...

[mcspr]

As mentioned in the chats, still wip but it might be much nicer to edit it in the tree instead of amending the PR ad-infinitum

Only two sore points

• we still have flash layout as symbols & defines, I wonder if we should move closer to something 'iterable' so we see all of things at once as [begin, end] ranges. at least of the sake of the reader, not necessary to be closer to partition schemes of the updated sdks
• boards.txt.py python could do better :>

[mcspr]

Suggested change

	#include <flash_hal.h>
	#include "flash_hal.h"

[d-a-v]

This one has been reverted because

• mock version of flash_hal.h needs to be loaded first, to redefine FS_start and FS_end

This is only needed in sources files

• located in cores/esp8266/
• which make use of FS_start or FS_end