Building an Espressif OTA (user{1,2}.bin) compatible Tasmota FW image

[mirko]

This is slightly off-topic, however -- in case my poor google skills didn't leave me in the lurch and indeed nobody managed to do so before -- might ease flashing the Tasmota supported itead/sonoff devices without HW/serial access.

I took a closer look at the original firmware, reversed engineered a few pieces and implemented a basic server which can communicate with the Sonoff devices (basically a subset of what the amazon services are doing for itead).
Goal is to use the original internal upgrade mechanism to flash custom firmware onto the devices without opening them, attaching a serial cable, potential soldering, pulling down GPIO0, you know the game...

By now I managed to trick a Sonoff device to download a custom FW from servers under my control, passing the verification, and eventually flashing the image. That's some fair progress, however the device doesn't boot.

My first naive try consisted of providing a sonoff.ino.bin instead of the original upgrade file the device would usually fetch.
Then I noticed, the original upgrade image doesn't contain the bootloader section, so I chopped off the first 0x1000 bytes.
Still, the device doesn't boot.

That could mean that
a) the original bootloader can't load the tasmota firmware (e.g. tasmota requires a modified bootloader to be loaded appropriately)?
b) the internal upgrade mechanism expects a different image format

After some research I figured the original FW uses the Espressif OTA functionality. This apparently makes use of some Ping-Pong, meaning, it splits the usable flash and always flashes the inactive part.
That also matches with the requests for files named "user1.bin" and/or "user2.bin".

Those -- the original upgrade files -- start with (HEX):
EA 04 00 01 04 00 10 40
The Tasmota built starts with (HEX at offset 0x1000):
E9 04 02 00 04 00 10 40

As you can see, the first 4 Bytes don't match, while the second 4 Bytes do.
The Tasmota image header matches the Espressif documentation[1] about how an image should like (always starting with E9).

I couldn't find any documentation on what the image header in the original upgrade images is supposed to represent.

Long story short: If case b) applies ("the internal upgrade mechanism expects a different image format"), what is the that image format / header and (how) could build I build a Tasmota FW matching that image format criteria? If a) applies ("the original bootloader can't load the tasmota firmware"), can we workaround that somehow, at least for an intermediate image providing the Tasmota OTA functionality?

[1]http://www.espressif.com/sites/default/files/esp8266-sdk_application_note_firmware_download_protocol_en.pdf

[khcnz]

The world is full of funny coincidences... after sometime thinking about this i had finally started looking into this last night as well.

I presume you are using a little DNS trickery and setting up a websocket which issues the update command (along with custom url). One thing to note is that i don't believe with standard tools you can update the bootloader OTA (but maybe you can by directly overwriting the flash... it wouldn't be 'safe' from possible corruption - but in this case the only thing that could happen is to have to flash it via serial which is where we have come from anyway).

I think the first question to determine is which bootloader they are using - using esptool i believe you can dump flash from an arbitrary memory location maybe we can get some clues on their bootloader.

PS: The latest firmware that i sniffed an upgrade to (2.0.2) seems to use SSL end to end for the websockets so this could close this avenue in the future if they start shipping with newer firmware

PPS: From sniffing it would also be possible to maintain compatibility with the ewelink app infrastructure so devices could be controlled by both mqtt and the ewelink - not sure if that would be usefull for anyone..?

[khcnz]

http://esp8266-re.foogod.com/wiki/SPI_Flash_Format#0xea_Header

Seems its a newer boot mode format (boot mode 1 vs boot mode 2)

[khcnz]

also of interest https://richard.burtons.org/2015/05/17/decompiling-the-esp8266-boot-loader-v1-3b3/

[mirko]

Glad to hear somebody else is also looking into this!
The hint towards the 2 image formats explains a lot - unfortunately it doesn't provide an easy solution to the issue.
From analyzing the original firmware there's no (proper) way of flashing the bootloader - only the "user1" and "user2" partitions - so flashing the bootloader from within the internal original upgrade procedure doesn't seem feasible.
So, if I'm understanding correctly, Tasmota images use the old image format (0xE9), including an bootloader which supports the old image type, while the original FW uses a bootloader which supports the new image format?
If correct, I guess the way to go would be to build (minimal) Tasmota images with the new 0xEA header(?)

Update: it seems esptool.py as well as the arduino esptool only create 0xE9 images. :/

[mirko]

Wohoo! I just managed to flash and boot a tasmota firmware image onto a sealed Sonoff Basic!
Tutorial and code will follow.

[khcnz]

(Assuming we can find a toolchain that support building the new format) I think you could maybe create a very minimal 0xEA firmware whose sole purpose is to flash the bootloader to the tasmota and enable the second stage firmware flash - however still dependent on being able to flash over the current bootloader while its running... my knowledge gets very slim here but from some reading I think the bootloader is loaded in ram... so it could be possible to flash while its running (maybe....).

[khcnz]

Edit - wow! nice! A wiki page would probably be the best place

Assuming you managed to compile tasmota with 0xEA bootloader I also guess this means we could retain backwards compatibility with the existing firmware for those who want/need to roll back?

[mirko]

Yes, I didn't touch the bootloader, just modified the image.

[davidelang]

please double-check that you can do an OTA upgrade from tasmota to a new tasmota image after this.

[mirko]

please double-check that you can do an OTA upgrade from tasmota to a new tasmota image after this.

Could you elaborate your concerns?

I'm still struggling with forcing the Sonoff to flash our custom image to the user1 partition, as -- depending what part is (in)active -- it might get flashed to user2 which is bad for several reasons.
My first thought was to figure out which image the device asks for and provide a faulty image to get the device booting from the other partition, however I noticed quite quickly that the only faulty thing in this logic is me:

When the currently running system is stored on user1 (user1 == active), then we'd flash user2. Providing a faulty image only result in the bootloader not being able to boot user2 and again boot into user1.

UPDATE:

The only way I come up with right now is, in case the device asks for the user2 image (meaning user1 == active), to override the internal data segment, which comes right after the user2 one, as the internal data section stores the information for the bootloader which partition is the currently active one.
But that sounds rather nasty and requires us to know the exact flash size to calculate the exact extra padding.

[davidelang]

you have been able to get tasmota running from the base build, but by not replacing the bootloader. my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota) If it's possible/required to do different update URLs for the two 'slots' in the router, that may not be a bad thing, we are getting to the point where size limits are pushing us to the need to do a 'two step' upgrade (first to a minimal build, then to a full build)

[mirko]

my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota)

I see, though I figured tasmota also flashes the bootloader, doesn't it?
If so, then -- once tasmota is running (even with the original bootloader) -- it would OTA-update the whole flash starting from 0x0 and therewith also replace the bootloader, wouldn't it?
Or does tasmota use the bootloader to perform the upgrade? That would indeed be an issue then.

If it's possible/required to do different update URLs for the two 'slots' in the router, that may not be a bad thing, we are getting to the point where size limits are pushing us to the need to do a 'two step' upgrade (first to a minimal build, then to a full build)

Good to know! That will definitely make certain things easier.
However the current problem is, that the tasmota build (with the 0xEA header) only boots when being flashed as user1.bin. When being flashed as user2.bin it doesn't.
Taking a closer look at the original upgrade images user1.bin and user2.bin also slightly differ.
I assume the place where they'll be flashed onto flash is somehow encoded and read/used by the bootloader.

[davidelang]

On Wed, 24 May 2017, Mirko Vogt wrote: > my concern is that since you didn't replace the bootloader, trying to d a tasmota -> tasmota OTA upgrade could possibly not work (or require changes to tasmota) I see, though I figured tasmota also flashes the bootloader, doesn't it? If so, then -- once tasmota is running (even with the original bootloader) -- it would OTA-updates the whole flash starting from 0x0 and therewith also replaces the bootloader, wouldn't it? Or does tasmota *use* the bootloader to perform the upgrade? That would indeed be an issue then.

I'm not sure. The max size you can OTA on tasmota is 1/2 (memory size - bootloader) so I think it doesn't replace the bootloader with an OTA upgrade. When arendst is back he can comment on this with a lot more knowledge. I'm just raising potential issues in the hope that you can check them and make this rock solid. David Lang k

[khcnz]

It's hard to say without knowing a little more about the process you are using to modify your tasmota image to boot?

I presume you have the user1.1024 and user2.1024 images from itead? When you diff them there are considerable differences (starting at the simple like byte 4 is 01 or 02 depending on which file).

I would guess that itead are using the espressif OTA mechanism - you can see more here http://www.espressif.com/sites/default/files/99c-esp8266_ota_upgrade_en_v1.6.pdf and it describes how to create the two versions aside from the bye mentioned above I believe memory addresses differ ti think..?