[Feature Req] Auto-detect SPIFFS/partition layout

[TD-er]

For ESPEasy I have a lot of builds defined.
The number of builds can be more than halved by not having separate builds for 1M, 2M, 4M etc.
Lately I also added test builds for 4M2M layout (4M flash, 2M SPIFFS) and when rolled out to other build definitions the number of build environments will increase even more.

Also the chance of loading the wrong image on a node does increase with all these options.

Would it be possible to have the sketch detect the flash size and detect the location of the SPIFFS, instead of hard-coded in the build?

Then I can add a number of best practice offsets for different setups and chose the best one based on the found hardware when a SPIFFS has to be created (or LittleFS) and when there is one already present just use that one.

This will remove all the flash size related builds and make it just about selecting the processor + set of plugins you want.

If this is already possible, where can I find information on how to set it?

I guess the core libraries also need some fixed offsets for things like WiFi settings and the EPROM area. So these should then also be auto-probed and offer some function call to query their locations.

[d-a-v]

Very interesting

Flash size: Esp.h: uint32_t EspClass::getFlashChipRealSize(void)
Locations: the logic in python boards.txt builder can be duplicated
SPIFFS/LittleFS: a quick scan for FS signatures can be done in the possible locations computed by the above logic.

WiFi settings addresses is independently computed by nonos-sdk-fw's by a similar way. There's nothing to do here. The only symbols that we use and defined by flash chip size / menu configuration are put to tools/sdk/ld/eagle.flash.*:

PROVIDE ( _FS_start = 0x402BB000 );
PROVIDE ( _FS_end = 0x402FB000 );
PROVIDE ( _FS_page = 0x100 );
PROVIDE ( _FS_block = 0x1000 );
PROVIDE ( _EEPROM_start = 0x402fb000 );

[TD-er]

Those WiFi settings addresses, are they computed at compile time, or run time?
If at run time, then we don't need to set the flash size at compile time for those.
If at compile time, then we still have to differentiate between flash sizes, but may be a bit more flexible in sketch/SPIFFS/LittleFS layout.

[d-a-v]

The logic in the python script is there to respect rf-cal | sdk-wifi-settings region relatively to the flash chip size to leave it untouched.
nonos-sdk computes the address and uses it at run time.

[earlephilhower]

SPIFFS does not have anything like a FSINFO block. It also happily mounts things which only nominally look like a SPIFFS FS. And there is no sane way of telling where a SPIFFS FS ends.

I would say the only sane and proper thing here is the partition layout stuff EPS32 and other chips already use.

[devyte]

@TD-er what you're asking doesn't make sense to me.

• The FS end address ia always at the end of the flash size (minus some space for wifi sdk params).
• The user has to select the FS size desired (can't be automatic)
• The FS start address is automatically calculated from flash size and FS size
• For specific boards, the flash size is known (e.g.: wemos d1 r2 is 4MB, done).
• For the generic board menu, the flash size has to be set by the user, because that is the whole point of it.

Given this, and previous comments, I'm closing this. If we're missing the point, please clarify.

[TD-er]

Given this, and previous comments, I'm closing this. If we're missing the point, please clarify.

The main problem is that the location of SPIFFS is defined at compile time, but it would be great if we can override this at run time.
As I understand from @earlephilhower we cannot currently probe for SPIFFS location, so another thing that is needed then is some indicator somewhere to get a hint on the SPIFFS location.
This could be a single byte, as it is only needed to know the number of blocks from the end, or some other enum-type.

My point is, that it would be a great feature to have the SPIFFS defined not only at compile time, but also at run time.
This way I can have a single build, which would support a setup with different flash sizes and different SPIFFS sizes.
For example the 4M/1M sketch/SPIFFS ratio for backwards compatibility and 4M/2M for newer setups.
But also the same binary for 2M/512k or any other weird combination.

[d-a-v]

Flash size and other flash-relative configuration in the menu is necessary for:

• where to store EEPROM (that can actually be calculated at run time)
• where FS ends (that can actually be calculated at run time)
• where FS starts (that can be stored in a specific location, read at runtime)

[JAndrassy]

how does SPIFFS know where FS starts?

[earlephilhower]

&__FS_start is taken at compile time.

[devyte]

@JAndrassy saif:

how does SPIFFS know where FS starts?

I said:

The FS start address is automatically calculated from flash size and FS size

There are pre-calculated ld scripts based on the menu options (board variant, FS size). The symbols are defined in there and get linked during build.
As @d-a-v said, the logic for the symbol calculations is in the boards.txt.py script.

[JAndrassy]

@JAndrassy saif:

how does SPIFFS know where FS starts?

I said:

The FS start address is automatically calculated from flash size and FS size

There are pre-calculated ld scripts based on the menu options (board variant, FS size). The symbols are defined in there and get linked during build.
As @d-a-v said, the logic for the symbol calculations is in the boards.txt.py script.

yes, yes, it clicked in my head a little later :-)

[d-a-v]

This symbol could be replaced by another one computed at run time from a fixed-address configuration structure.

[d-a-v]

Actually after internal dicussion, it is today possible for a user to instantiate manually the FS with different start/end addresses.

Arduino/cores/esp8266/spiffs_api.cpp

Lines 138 to 143 in 6e51ef0

	FS SPIFFS = FS(FSImplPtr(new spiffs_impl::SPIFFSImpl(
	FS_PHYS_ADDR,
	FS_PHYS_SIZE,
	FS_PHYS_PAGE,
	FS_PHYS_BLOCK,
	SPIFFS_MAX_OPEN_FILES)));

Arduino/libraries/LittleFS/src/LittleFS.cpp

Line 192 in 6e51ef0

FS LittleFS = FS(FSImplPtr(new littlefs_impl::LittleFSImpl(FS_PHYS_ADDR, FS_PHYS_SIZE, FS_PHYS_PAGE, FS_PHYS_BLOCK, FS_MAX_OPEN_FILES)));

So if we have an internal core API offering the same calculation as what we have in the python script, we could dynamically process EEPROM_START.

Then a pre-compiled sketch would not anymore depend on flash size (the FS size or start address and page size would be stored into a table indexed by flash size).

@TD-er would you be ready to translate the python logic to the core api ?

[TD-er]

@TD-er would you be ready to translate the python logic to the core api ?

What is the current Python code used for generating the Eagle files?
Is there any difference in numbers between LittleFS and SPIFFS?
Is there a byte where I can store information about the filesystem? Or should I use the EPROM for it?

[d-a-v]

python code: #6679 (comment)
No difference between LittleFS and SPIFFS

Is there a byte where I can store information about the filesystem?

no,

Or should I use the EPROM for it?

possible, or:

A default table can be implemented, giving FS size according to Flash size.
The default value (in arduino IDE, first value for each flash size configuration) is a good candidate: keep the first 2MB free to allow OTA, allocate all the rest for FS.
And for flash size <= 2MB we use default values. Current are:

Arduino/tools/boards.txt.py

Lines 1303 to 1333 in 6e51ef0

	f1m.update( flash_map( 1024, 64 ))
	f1m.update( flash_map( 1024, 128 ))
	f1m.update( flash_map( 1024, 144 ))
	f1m.update( flash_map( 1024, 160 ))
	f1m.update( flash_map( 1024, 192 ))
	f1m.update( flash_map( 1024, 256 ))
	f1m.update( flash_map( 1024, 512 ))
	f1m.update( flash_map( 1024))
	f2m.update( flash_map( 2*1024, 64 ))
	f2m.update( flash_map( 2*1024, 128 ))
	f2m.update( flash_map( 2*1024, 256 ))
	f2m.update( flash_map( 2*1024, 512 ))
	f2m.update( flash_map( 2*1024, 1024 ))
	f2m.update( flash_map( 2*1024))
	f4m.update( flash_map( 4*1024, 2*1024 ))
	f4m.update( flash_map( 4*1024, 3*1024 ))
	f4m.update( flash_map( 4*1024, 1024 ))
	f4m.update( flash_map( 4*1024))
	f8m.update( flash_map( 8*1024, 6*1024 ))
	f8m.update( flash_map( 8*1024, 7*1024 ))
	f16m.update(flash_map( 16*1024, 14*1024 ))
	f16m.update(flash_map( 16*1024, 15*1024 ))
	f512.update(flash_map( 512, 32 ))
	f512.update(flash_map( 512, 64 ))
	f512.update(flash_map( 512, 128 ))
	f512.update(flash_map( 512))

1MB: 64KB FS
2MB: 64KB FS
4MB: 2MB FS
8MB: 6MB FS
16MB: 14MB FS
512K: 32KB FS

edit: That could be a default table. Other table (like no-fs-table) could be used by user. One particular sketch (like ESP-Easy) would select one table among the many.

edit2: a new entry in "flash size" menu would be available: "sketch defined" where when selected, user would need to create missing symbols (possibly from the default tables described above). Symbols are: (_FS_SIZE, _FS_PAGE, _FS_BLOCK). eeprom size is constant, and location can be calculated from flash size.

[d-a-v]

@TD-er In fact there's no need to port python code.
We can modify it to generate a .h file containing all constants that sketch would choose to use.

[TD-er]

If we can generate it, why then use a table? Why not compute it?

[d-a-v]

I see two reasons:

• we wouldn't need to keep in sync two versions (C, python) inside the core
• python would generate some set of defines / an indication,
  but user can use whatever is preferred, even include this calculation in the sketch itself if needed

[TD-er]

The reason I asked is because of binary size.
If it can be computed giving one or two variables, then that should be preferred, since it does not add a lot to the binary size.

[d-a-v]

@TD-er Would these changes help with your release process ?

[TD-er]

@d-a-v I hope so, but I haven't yet made a build with ESPEasy using these changes.

[d-a-v]

@TD-er Are you still interested in this feature ?

[TD-er]

@TD-er Are you still interested in this feature ?

Yes!
For sure it will decimate the number of builds I will have to make.
Or do you plan on declaring SPIFFS deprecated in favor of LittleFS?
I have not yet made a test build for ESPEasy with this PR, mainly because building based on a separate commit/branch of the esp8266/Arduino core always was such a hassle in PlatformIO, but while I write this I do realize that should no longer be an issue.

I will try to make a test build for it ASAP. (hopefully on Monday...)