Long Running Futures Eventually Stall

[fko-kuptec]

I am currently developing a firmware for the ESP that needs to serve the camera feed as HTTP stream. For testing purposes, I reduced that further down to serving a simple JPEG animation. I initially used edge-executor::LocalExecutor to run my custom async HTTP server, handling every connection in a separate future spawned to the same thread-local executor. One stream to one client is therefore one (possibly infinitely) running async task.

Unfortunately, I ran into some weird issues: When running multiple streams in parallel, any stream might freeze randomly and never get back to live. Sometimes, the whole ESP crashes with some unhandled exception. It does not seem to be the same exception every time, but here is one:

Guru Meditation Error: Core  0 panic'ed (Unhandled debug exception). 
Debug exception reason: BREAK instr 
Core  0 register dump:
PC      : 0x420171d7  PS      : 0x00060b36  A0      : 0x82010e96  A1      : 0x3fccf060  
0x420171d7 - alloc::task::raw_waker::clone_waker
    at ??:??
A2      : 0x3fcd1154  A3      : 0x3fcd1154  A4      : 0x00000000  A5      : 0x3fccf3a0  
A6      : 0x3fc9ade8  A7      : 0x3fca12dc  A8      : 0xfefefefe  A9      : 0xffffffff  
0x3fc9ade8 - xKernelLock
    at ??:??
0x3fca12dc - pxReadyTasksLists
    at ??:??
A10     : 0xfefefefe  A11     : 0xfefefefe  A12     : 0xfefefefe  A13     : 0x00060b23  
A14     : 0xfffffffe  A15     : 0x0000cdcd  SAR     : 0x00000020  EXCCAUSE: 0x00000001  
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff  


Backtrace: 0x420171d4:0x3fccf060 0x42010e93:0x3fccf080 0x4201800a:0x3fccf9c0 0x4202bb7f:0x3fccf9f0 0x42030838:0x3fccfa10
0x420171d4 - alloc::task::raw_waker::clone_waker
    at ??:??
0x42010e93 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at ??:??
0x4201800a - core::ops::function::FnOnce::call_once{{vtable.shim}}
    at ??:??
0x4202bb7f - std::sys::pal::unix::thread::Thread::new::thread_start
    at ??:??
0x42030838 - pthread_task_func
    at ??:??

At some point I tried using a different executor. After replacing edge-executor by futures-executor::LocalPool, I could not reproduce these issues anymore. The streams seem to continue running without problems. Then, I've written my own version of LocalExecutor from scratch without using any third-party crates, and this also seems to work fine.

This makes me believe, that edge-executor has a bug... somewhere. I have not really a clue where it is. I just found this open issue in async-task talking about tasks not getting rescheduled randomly. That would fit my observations, at least.

Sorry for not providing sample code. I am developing the firmware as employee and therefore cannot just share our product's firmware. If you are interested, I can however try to build a minimal example, when I find the time.

[ivmarkov]

Aww, this sounds frightening :( . I do hope it is not really async-task as you suggest, or else the whole smol-rs ecosystem has a big trouble.

Perhaps it is related to our queues. Btw - unlike async-executor, edge-executor's queues are bound by default; what this means is that if you are spawning extra tasks during the life cycle of your app, and at some point in time the number of active tasks becomes bigger than the queue size, the app would crash in an unpredictable way, most likely precisely in the waker.

Anything you could share about that problem w.r.t. minimal reproducible case would be greatly appreciated.

If your employer is OK with that, sharing your own Local executor might also be interesting.

[ivmarkov]

Just for testing... what happens if you remove the heapless feature and add the unbounded feature?

[fko-kuptec]

I already tested it with the unbounded feature enabled, since I also assumed this to be the root cause. No luck, however. My last test today was with this configuration:

edge-executor = { version = "0.4", features = ["unbounded"] }

I talked to my supervisor and he is okay with providing a minial reproducable example and also with sharing my own executor implementation. The question would just be, how to handle the latter part. Maybe for starters, if I push it into a private Github repo and give you access? Then we could discuss if we want to merge it into edge-executor and how.

For now, I will try to build a minimal example for the bug.

[ivmarkov]

Yes - a private GH repo for the executor would be ideal, many thanks!

[ivmarkov]

Also, the following question: have you tried with async-executor? It is also based on async-task and - moreover - is API-compatible with edge-executor. If async-executor crashes on you as well, chances are there is a serious bug in async-task. Otherwise, there might just be something wrong in edge-executor itself, which should be easier to fix.

[fko-kuptec]

Interesting... I cannot reproduce the issue in my minimal example. My best guess would be, that this is a compiler bug again. I've seen them before in my firmware with stuff like:

• just including tokio as dependency without using it would make the streaming significantly more performant
• removing logging would make it slow again
• re-including the logging would make it fast again
• removing the logging again wouldn't impact performance... 0.o

I don't know whats happening. But maybe its not a bug with edge-executor or even async-task, but just some weird side-effects?

[ivmarkov]

I don't know whether this is even an option, but how about flashing (a subset of) your firmware on a RISCV MCU and trying that out instead? The compiler backend for RISCV is upstreamed in LLVM and I guess it is significantly more stable than xtensa.

We are currently having this miscompilation issue, which seems to be triggered by a very large alignment (padding) in ConcurrentQueue that LLVM xtensa does not like.

Coincidentally, edge-executor uses ArrayQueue by default, which has the very same large cache-padding.

Chances this to be the culprit are slim, but trying out with RISCV is one option to eliminate this hypothesis; another would be to enable heapless and try with the queue from heapless.

A third option is to try your full firmware with async-executor as I mentioned earlier.

[ivmarkov]

Also: are you using any async drivers from esp-idf-hal? Perhaps you can stop those during your test?

[ivmarkov]

Interesting... I cannot reproduce the issue in my minimal example. My best guess would be, that this is a compiler bug again. I've seen them before in my firmware with stuff like:

• just including tokio as dependency without using it would make the streaming significantly more performant

I think you might be getting paranoid :-) The linker would erase all traces of tokio if you don't call into it.

• removing logging would make it slow again
• re-including the logging would make it fast again
• removing the logging again wouldn't impact performance... 0.o

As I said - maybe trying it out with RISCV is an option. Definitely sounds weird though, but without at least seeing a portion of the code, I'm running out of hypotheses.

[fko-kuptec]

A short update for today: I finally can reproduce the issue with my minimal example. It happens with both edge-executor and async-executor, but only if the PSRAM is enabled, which is ... interesting. The exception messages make it look like heap corruptions, but enabling the integrity checks for the heap in the ESP IDF makes for less helpful panic messages.

Whatever, it will take me until tomorrow to give you a clean report and access to my private executor.

[ivmarkov]

Hmmmmm. This might be related. Are you confining the PSRAM to just one of the two CPUs, or are you happily using malloc with PSRAM enabled?

[ivmarkov]

@fko-kuptec Is this running on the stock ESP32, or on ESP32S3? Asking because - at least in theory - the ESP32S3 should not have the atomics-in-PSRAM issue I mentioned in my previous comment. (Also - and to my understanding - no RISCV chip should be affected by that.)

[fko-kuptec]

I am running the tests on ESP32-S3 :)

[ivmarkov]

Ok, so either the s3 is also affected, or there is something else going on with PSRAM. In any case, your reproducible example, as well as your infallible executor would help greatly in figuring out what is going in. Especially given that you have confirmed that async-executor is also affected.

[ivmarkov]

Also and if you have them handy - can you paste here other stack crashes you have recorded?