Watchdog unhappy with long requests

[erwan-privat]

Hi Mathieu,
I have developed a solar routeur for my father, I think the result is quite good. You can see my repo erwan-privat/rbr-magnac. I would like to know what you think of it if you have the time. I use your fork of the async web server because I had started to make corrections on the library code that you already did.
I also have a question: when dealing with long request, the watchdog will sometimes reboot the ESP32 if the computation/mostly delivery time of the content getting too long for the watchdog. I thought of setting the priority of the webserver as IDLE, but since I am not creating the tasks manually it is not obvious how to do so while trying not to modify the arduino libraries (or yours) so to have a packageable project. I have a html page with AJAX requests (I use direct JSON writing with no library, as I really think ArduinoJSON despite all the guarantees have memory leaks, I might misuse it but I doubt it), and if I launch my page in say five tabs the ESP32 reboots because of the watchdog complaining about a task taking too long. Since computing and generating the JSON cannot be deferred to an external call so no yield() is possible, I was wondering about what you think about that.
Cheers.

[mathieucarbou]

Hello,,

Fyi I am also developing solar routers:

• Shelly solar router: https://yasolr.carbou.me/blog/2024-07-01_shelly_solar_diverter
• Home Assistant solar router: https://yasolr.carbou.me/blog/2024-09-05_ha_diverter
• And YaSolR: https://yasolr.carbou.me/

YaSolR is the most optimized open-source solar router as you will be able to read. I've developed many libraries specialized around ZCD pulse analysys. (https://github.com/mathieucarbou/MycilaPulseAnalyzer), jsy, pzem, etc It uses a PID controller at a rate of more than 3 adjustments per second. You'll find them on my GitHub profile.

For you questions, just know that there is no issue in the ESPAsyncWebServer Library or Arduino Json, but only in user code.
The watchdog triggering is typically a smell for bae programming: it is a protection telling you that you have developed a piece of code which is blocking all the cpu for itself for too much time thus not allowing other important tasks to execute.

FreeRTOS is not a preemptive OS so your tasks, even if they have some priority, have to be co-operative.

Also, as a general rule, never do big operations in a handler or callback, whether it is a mqtt, async webserver or other one.
Callbacks/ handler are executing within the context of the caller (which is usually a more low-level API) and require to return as fast as possible.

You must find a way to redesign your code to adhere more to the Arduino spirit of doing things with the loop, and if necessary introducing tasks. MycilaTaskManager is a library I've made which can help you for that.

Good luck!

[mathieucarbou]

https://github.com/erwan-privat/rbr-magnac/blob/13cef33ea92807c50af5a0d225d38c5b9f0eba83/ServeurWeb.cpp#L63

Do not use stream responses to send back json data and do not yield.
Here, you are just accumulating data in memory which is not streamed but sent at once over the Network when the handler finishes.

Also, do not yield: building a Jason document is a fast operations so if you yield it potentially stop a handler which has to be executed fast to let another task with higher priority execute.

[mathieucarbou]

https://github.com/erwan-privat/rbr-magnac/blob/master/ServeurWeb.cpp#L97

Same problem: I think that what you wanted to do is improve memory usage
To do that, you can use chunk response with a callback filling the response buffer as the response is sent on the network

[mathieucarbou]

https://github.com/erwan-privat/rbr-magnac/blob/master/ServeurWeb.cpp#L168

What happens if your screen is updated on the background while you are reading the pointer ?

[mathieucarbou]

https://github.com/erwan-privat/rbr-magnac/blob/master/ServeurWeb.cpp#L317

With this Library there is a more optimal way to send big json documents

That's what we use in ESP-DASH which is able to send quite big websocket buffer (more than 6-8k) from a json documents.

[erwan-privat]

Thanks a lot for your review, that was very helpful.

[vortigont]

Here is an example on how to generate on-the-fly and send huge json response in chunks using only a small portion of WebServer's buffer without any intermediate date copies.

[mathieucarbou]

@vortigont : I am thinking that it would be nice to have an equivalent of the sendChunked / beginChunkedResponse, but with a callback taking as argument a Print object, where we can just call print and write and call serializeJson directly with it.

Problem is that I don't know how to control the fact that the client space of AsyncTCP can be lower than the json size.

[mathieucarbou]

@vortigont I bumped into your comment here:

https://github.com/vortigont/espem/blob/f26ef57fec4b87859a382d227de47cc70bc34e7d/espem/espem.cpp#L214

already seen that in equivalent code. The closure you're into is passed to the response object, so it is executed way after the handler finishes (when the whole middleware chain finishes).

by that time, the pointer captured on the iterator might point to a structure that has changed (or not there anymore):

  const auto ts = getTS(id);
  if (!ts)
    request->send(503, PGmimejson, "[]");

  auto iter = ts->cbegin();   // get const iterator

this code is only valid if 1) getTS returns a pointer or ref to a structure held on a wider scope (which I think is ok) and 2) the structure still does not change during the commit of the response on the network, causing the chunk callback to be called, and this happens way after the handle is completed. send() at the end of the handler does not commit on the wire but just "registers" the response on the request object, which will be made available for next middlewares acting on response code, headers , etc, and, when the chain is finished, asyncws will commit the response on the network and the chunk callback is called at that time.

so you might need mutex or something that is blocking the mutations of the list during its iteration.

[vortigont]

You are right, Matheiu, async nature of chunked response should be kept in mind at first place. However this comment of mine is of something different. I do not remember the details, it is related to dereferencing floats in sprintf somehow.
The iterator itself is quite solid, it points to an in-mem ring buffer object which allocation is immutable and iterator is never invalidates till the buffer itself exists. That was my implementation for generation really large jsons - 30 bytes of in-mem struct unfolds into 200 bytes json object multiplied by hundreds of array elements. This way I generate time-series data for WebUI charts.

[vortigont]

nice to have an equivalent of the sendChunked / beginChunkedResponse, but with a callback taking as argument a Print object

I was thinking about something similar that time ago but came with nothing solid. The problem is that Print blocks on send and chunked response needs a callback. The only solid idea that I found appropriate somewhere is to create an object that would do a very strange thing - it would take a JsonDocument than starts chunked responce and allocates a small buffer on it's own. Then it starts serializing object and copies the data to the chunk buffer unless it fills up. Then it continues to serialize data into internal buffer and simply DISCARDS unless the object ends. On a next callback it does the same from all over again, but that time discarding the beginning and the end of data. So piece by piece it sends chunked response but it has to do multiple full serializations each time and only sending a small piece of it. Quite efficient in terms of memory but very inefficient in terms of CPU time. ArduinoJson is very poor on partial serializations.

[mathieucarbou]

I was thinking about a sort of Pipe implementation, where we would have:

• beginResponse([](Print& output){ serializeJson(output) })

Or use a ArduinoJson CustomWriter (but at the end it is the same).

But for that to work, a task should be spawan to call callback(print) because this call will block until serializeJson() can finish writing. It should be fast though.

And when filling the buffer, we just have to return of the print implementation the size_t that was written in the client.

After that, this is matter of optimising : for example, could the async_task take care of calling this callback() if we send them to a sort of callbackQueue ? I don't know.

[erwan-privat]

I ended up doing that: https://github.com/erwan-privat/rbr-magnac/blob/6407b36dd2e272d73b8db5dd5e79874999defef5/ServeurWeb.cpp#L120

I do think there is a need for a simpler API, I am not satisfied with the process of coding with callbacks as it is needed now. I used some weird switches blocks before using default buffer size for chunks. I know I had to code a chunked delivery of html and script in this case for example, but a proper API relying on your response automagically would be nice. Especially one compatible with e.g. ArduinoJson static buffer, because I apparently don't understand it right now.

edit: messed up a copy-paste.