[LittleFS] SPIFFS support will be removed from ESP32

[TD-er]

We use the ESP32 platform package which is maintained by @Jason2866
He just told me that from now on in newer platform packages the support for SPIFFS has been removed.
As I have ignored the upcoming deprecation status of SPIFFS for a bit too long, this announcement came a bit as a surprise to me. (or wake-up call)

So this means we need to move along or else we cannot use the newer upcoming IDF 5.x needed for lots of bug fixes and support of newer platforms like the ESP32-C2/C6 etc.

For now we can still use the platform builds upto this one, which is based on Core 2.0.13 and is still newer than what we currently use in ESPEasy.
Also Jason offered to me to help out on a special platform build if there will be an important bug fix and we're still not done migrating away from SPIFFS.

Meaning this transition doesn't have to be immediate, but it must be done before support for other ESP32 nodes can be added.

There is a number of options to make such a transition easier for the user:

• Only use newer IDF on LittleFS builds (might be a lot of work for the developers)
• Create a special build for transitioning file systems (quite a lot of work for only 1-time transition)
• Build a separate tool to fetch and upload settings to ease transitioning to LittleFS

That last option is probably the best one as it can also be used for (mass)provisioning of nodes and as a backup tool.

If there are other ideas, please let me know.

I'm also thinking about simply removing SPIFFS support for the recently added platforms like ESP32-C3/S3 as it is probably not used a lot right now.
Or at least for the next build add LittleFS builds for those and add a huge warning.

[iz8mbw]

If I'm not wrong the support of SPIFFS will be removed and we will use LittleFS.
Some questions:
At the end, LittleFS will be used for all ESPEasy build (normal, climate, max, energy, etc.)?
At the end, LittleFS will be used for all kind of broads (ESP8266, ESP32, ESP32-S3, etc.)?
Can I update the Firmware using a build that uses LittleFS on a ESPEasy already installed firmware that uses SPIFFS?
If yes, I'll lost data/configs?

Thanks.

[TD-er]

There is LittleFS support for ESP8266, but right now I don't know if this will ever be a true replacement for SPIFFS on that platform.
SPIFFS is faster when updating something in the middle of a file (like we do when saving settings).
SPIFFS only becomes really slow when using larger file systems (> 2 MB) and noticable slower on file systems > 1 MB.
Almost all ESP8266 boards have 4MB flash of less, so I don't see any benefit for LittleFS on ESP8266.
However on ESP8266 with 4M flash, we do have some space to create a LittleFS partition, copy the files, delete SPIFFS file system and grow the LittleFS partition.
So that's probably easier to migrate compared to ESP32 if it would ever be needed to migrate ESP8266 to LittleFS.

For ESP32 it's a different story.
Currently we only have LittleFS builds for boards with larger flash sizes.
To be honest, I regret I did use SPIFFS on the recently added ESP32-S2/S3/C3 platforms.

Right now I don't really know how we can migrate from SPIFFS to LittleFS as ESP32 does use partitions.
Maybe I can make a conversion build which could use one of the OTA partitions to temporary copy the files, create a new LittleFS partition and copy the files back to the newly created file system.

Ton is also working on adding Tar support for ESPEasy.
This is also required to speed up saving changes to the settings file as we may need to split the settings file into separate chunks.
Maybe I can then create some migration tool to be executed from within the browser using JavaScript to cache the settings in the browser.
Or maybe we could use other ESPEasy nodes in the network to help keep a copy of the settings during the file system migration.

So lots of ideas.

I do think I will (at some point) no longer include the factory bin files for ESP32 SPIFFS builds on the web flash tool as those flashes are done via serial anyway.
But maybe people also upgrade their units via the web tool. So I have to think about this.

Very likely the next build will include a warning next to the update firmware button about the deprecation status of SPIFFS.
Maybe a tar export is good enough to help migrate to a different file system.

[iz8mbw]

Hi @TD-er.
Do you have a plan to fix "Access control based on client IP" for LittleFS (ESP-IDF5.1)?
Will it possible?
Thanks.

[TD-er]

Do you have a plan to fix "Access control based on client IP" for LittleFS (ESP-IDF5.1)?

Yep, only very recently I managed to get the remoteIP() again, but since it was only days ago I didn't dare to enable the access control again as it may lock out people if this getting the remote IP was just a fluke and not a proper fix.
So in the next builds it will be enabled again if it appears to be properly fixed.

[iz8mbw]

Great!

[fly74]

How can I make a test migration with an esp32 normal image?

[iz8mbw]

1. Enable all the check boxes in the page Tools -> Factory Reset (do NOT do a Factory Reset but make sure the check boxes are enabled)
2. Update the ESP32 board with latest "regular" (so SPIFFS) firmware (board will auto reboot after the update)
3. Wait 10 minutes
4. Download all the files you have on the File System from the page Tools -> File browser (this step is to make a backup of your configurations)
5. Update ESP32 board with latest "new" (so LittleFS) firmware (board will auto reboot after the update)
6. Upload the files downloaded at step 4 into the file system (from the page Tools -> File browser) and do NOT do any other things like change your settings
7. Reboot (immediately) you board

[tonhuisman]

4. Download the files c...

⚠️ Better mention explicitly to download all files from the file-system, as it won't ever be available or recoverable once the LittleFS build has formatted and used the storage! ⚠️

In PR #4777 I've added a Backup & Restore feature using a single TAR archive, but that hasn't been merged yet.

[iz8mbw]

Updated

[fly74]

It is not only download all files. Flash LittleFS and upload the files with reboot? Why a factory reset at first?

[iz8mbw]

NOT do a factory reset, as per my previous message verify that all the check boxes are enabled.

[iz8mbw]

It is not only download all files. Flash LittleFS and upload the files with reboot? Why a factory reset at first?

Yes, you must before update the LittleFS firmware (board auto reboot) and after you must upload all files you downloaded before in order to recover configurations.

[fly74]

This don't really work. I downloaded the files updated to LittleFS. WLAN NTP is OK. But the upload of the files don't restore the devices.

But an restore of an earlier config.dat does the job. Maybe my fault.

This is the target?

[iz8mbw]

I have done the procedure on 4 ESP32 boards with no issues.
Try to upload again the various files backupped before (made on step 4) and after all the uploads reboot the board.

[tonhuisman]

After uploading config.dat, DO NOT change any settings before rebooting the ESP, as that will overwrite the just restored config.dat file content. Just reboot immediately, as suggested (for any uploaded file, by the way, but that's not needed) on the upload page after the upload is completed.

[fly74]

After uploading config.dat, DO NOT change any settings before rebooting the ESP, as that will destroy the just restored config.dat file content. Just reboot immediately, as suggested (for any uploaded file, by the way, but that's not needed) on the upload page after the upload is completed.

Is there an upload order for the files ?

[tonhuisman]

I'd suggest to restore config.dat as the last file, but that's not explicitly specified. If you didn't do anything else between the uploads and the reboot, it should have worked just fine 🤔.

My TAR backup/restore PR restores the files in the order they are in the .tar file, but that's a nearly atomic operation, with the same explicit requirement to reboot afterward.

I'll update that PR, so we can do more tests with that. I expect some merge conflicts, but haven't looked at it yet.

[iz8mbw]

I restored files without any specific order and it worked.

[TD-er]

Just make sure that after uploading the config.dat file you do not change any settings but just reboot to make sure the new file is loaded.