Launching the WebUI from crashes EMS-ESP on some environments

[proddy]

From the Discord chat both @glitter-ball and @JakeOfOz mentioned issues getting the WebUI working on Linux. I installed Firefox 84.02 on Ubuntu 21.10 (headless server) but couldn't reproduce. We're going to need to collect more data

[glitter-ball]

I wasn't using Linux! Firefox 93.0 on Windows 10 and Edge on Windows 10 are the worst-behaved. Chrome also does odd things. Best-behaved browser is Safari on iPhone.

Don't really know where to start with this! I've been having all sorts of weird behaviour for months...

• Hanging showing "Loading..." at the logon prompt. Hitting Refresh causes the login dialog to flash up and then disappear and be replaced by "Loading..." again. Only fix is to clear the browser cache and start again.
• Hanging during WiFi scan. Tried multiple times with Firefox. Tried again with iPhone and it worked straight off.
• Firmware upgrades never work properly. I drag and drop the .bin file on the firmware update page and the progress bar whizzes to 100% and nothing else happens. When I navigate away from the page I get "Update cancelled" or similar. Often a reboot will show that the update has actually happened, but I'm always nervous of it.
• Restart button on the System page doesn't work. Click the button on the Confirm Restart page and it goes from blue to grey. Nothing visible happens and the page just hangs. Usually the device will restart but it's just messy.

Don't want to sound like a whinger, but it's a grim user experience! Worst still, not really sure why it's happening. I was ready to blame my home network, but when it does it on a wiped device in access point mode, that rules that out. The fact that I can get working pages on iPhone via home network also suggests the network isn't the problem. Safari on iPhone does seem to work pretty well. Just tried the restart on Edge for iPhone and I did get the "System is restarting" popup that I just don't see on laptop. Scan WiFI Networks also works on Edge for iPhone. On Firefox on laptop, with a clear cache, it will work once. Thereafter, it will just hang at the "Scanning..." dialog with progress bar. If I clear the cache, it will work again - just once. Thereafter, if I leave the page or close the session and log in again, the Scan WiFi page just hangs at the progress bar again. Weird, weird, weird.

So, there's something odd going on around animations / popups and the browser cache but I can't pinpoint it. I could try and capture some video if GitHub will let me post. It's sad - we have a good-looking, slick UI but it's just really flaky when I use it. I also wonder why more people don't report issues if it's something big/widespread. I can replicate the issues on at least three different machines.

[proddy]

it is odd, but I'd like to know what is causing it. There are probably more than 1500 users with EMS-ESP out there, maybe more (@bbqkees knows the numbers) and I haven't heard of issues before.

Do you have any anti-virus software on your Windows 10 machine? If so try disabling them. EMS-ESP does support SSL, so only HTTP and maybe something doesn't like it. Is there another PC or laptop you can try out in your network (other than a mobile device) that will eliminate any network-related interference?

As for the Restart button, I'll see if there is something I can do with the UI like we had back in version 2 which shows a blank screen and tries to refresh in 10 seconds. The difficulty is that the web session is gone since the web server instance running on the little ESP32 is destroyed so there's no way for EMS-ESP to interact back with the browser page. I'll create a new GitHub issue for this.

[bbqkees]

@glitter-ball here it works fine. You are also the one with the most hardware issues it seems. Maybe that's related. Try another ESP32 board. A year or two back we had very strange connection issues that turned out to be a bad batch of ESP boards.
And perhaps an ad blockers on your system adds problems as well.

I did get a few emails about the web interface not loading over Ethernet while it worked fine over WiFi but that's it lately.

[glitter-ball]

Only anti-virus is Windows Defender. Lots of other people will be using that so I don't think that's the problem. I'm using some web filtering / parental controls stuff but I disabled that and it still didn't work. I have another client without this so I'll try that too.

Networking is tricky. I get the problems even with the EMS-ESP in access point mode, so there's nothing else in the way between the client and EMS-ESP. I also don't get issues when using the home network from an iphone. So, there's something about my PCs and/or their browsers which is breaking it.

I was having a lot of the UI issues with the ESP8266 too, which is a useful clue. I need to get another ESP32. The one which arrived recently is proving tricky to reflash so may be duff.

So, I think we probably have to blame my PC-based clients... Thanks for your patience. I'd love to crack the problem.

[proddy]

Pokito from Discord also is seeing issues. The EMS-ESP gets stuck and crashes with a async_tcp error when retrieving a large amount of data (like some of the JavsScript frontend). We're seeing if its only Client Wifi to Ethernet EMS-ESP or only Ethernet-to-Ethernet. Since I'm unable to reproduce I'll go ahead and make a few special branches with some tweaks to the TCP library to see what helps

[glitter-ball]

As I said in Discord, I could hook a stand-alone ESP32 to a PC and monitor the USB console when I try some stuff, in case it spits out any useful data.

[proddy]

One scenario we know that causes EMS-ESP to crash when loading the WebUI is on a (too) fast network. Pokito (from Discord) did some tests from an Ethernet-connected EMS-ESP to an Ethernet-connected laptop running Linux and was able to reproduce the crash with a wget command (no browser) when downloading a file > 150KB. By adding a 1ms latency (using sudo tc qdisc add dev $INTERFACE root netem delay 1ms) it solved the issue. And it looked like Ubuntu 18.04 is more stable than Ubuntu 20.04. So in this its seems the TCP async trips up. I'll make the special branch with some tweaks for experimentation.

@glitter-ball anything you could play with would help too, but I'm not sure right now where to begin

[cyberfox01]

I am Pokito from Discord.

Here is the script I use for testing. It is a little overengineered, but testing can be boring sometimes :-)
When started as root or with sudo, an artificial delay is added to the network interface (currently set to 1ms) and the whole communication is recorded with tcpdump.
Change to DELAY="" and/or RECORD_PCAP="" to disable that.

crash-test.txt

[VDRainer]

I'm also having this issue with EMS-ESP32 crashes, but only if i open the WebUI over a wired connection.

I'm using the Gateway E32 with Ethernet.

Pulling the cable on my Notebook and access the WebUI over WLan works without problems.
Latest Firefox and Chrome on Debian 11 (Ubuntu 18.04 until a month ago).

From a Windows 10 PC and a Virtualbox Windows 10 VM with bridged Network, there are no problems over wired connection.

Here's a debug log that i made a while ago, but it didn't show anything relevant to me and i was to lazy to report. 😄

Sep 13 15:22:05 ems-esp mqtt 000+00:42:20.298 D 215: Publishing topic ems-esp/thermostat_data (#1075, retain=1, retry=1, size=1023, pid=1)
Sep 13 15:22:05 ems-esp mqtt 000+00:42:20.399 D 216: Publishing topic ems-esp/mixer_data (#1076, retain=1, retry=1, size=89, pid=1)
Sep 13 15:22:09 ems-esp telegram 000+00:42:24.503 W 217: Incomplete Rx: 21 E8 00
Sep 13 15:22:14 ems-esp mqtt 000+00:42:30.062 D 218: Publishing topic ems-esp/boiler_data (#1077, retain=1, retry=1, size=669, pid=1)
Sep 13 15:22:14 ems-esp mqtt 000+00:42:30.163 D 219: Publishing topic ems-esp/boiler_data_ww (#1078, retain=1, retry=1, size=461, pid=1)
Sep 13 15:22:15 ems-esp mqtt 000+00:42:30.265 D 220: Publishing topic ems-esp/thermostat_data (#1079, retain=1, retry=1, size=1023, pid=1)
Sep 13 15:22:15 ems-esp mqtt 000+00:42:30.367 D 221: Publishing topic ems-esp/mixer_data (#1080, retain=1, retry=1, size=89, pid=1)
------------- CRASH ---------------------------------------------------------------
Sep 13 15:22:25 ems-esp emsesp 000+00:00:00.000 I 1: Ethernet initialized
Sep 13 15:22:25 ems-esp emsesp 000+00:00:00.000 I 0: Starting Syslog
Sep 13 15:22:25 ems-esp syslog 000+00:00:00.000 I 2: Log level set to DEBUG
Sep 13 15:22:26 ems-esp telegram 000+00:00:00.000 D 3: Tx read request to device 0x08 for type ID 0x07
Sep 13 15:22:26 ems-esp emsesp 000+00:00:00.000 I 4: EMS Device library loaded with 79 records
Sep 13 15:22:26 ems-esp emsesp 000+00:00:00.000 I 5: Starting Access Point with captive portal on 192.168.4.1
Sep 13 15:22:26 ems-esp telegram 000+00:00:01.411 W 6: Incomplete Rx: 21 08 00 07 00 03 01 00 02 00 00 00 00 00 00 00 00 00 00 00
Sep 13 15:22:26 ems-esp emsesp 000+00:00:01.720 D 7: No telegram type handler found for ID 0x19 (src 0x08)
Sep 13 15:22:26 ems-esp emsesp 000+00:00:02.402 D 8: No telegram type handler found for ID 0x18 (src 0x08)
Sep 13 15:22:26 ems-esp telegram 000+00:00:02.809 W 9: Incomplete Rx: 21 08 00 34 00 38 02 35 83 00 21 00 01 03 00 03 8C EA 00 6C D3
Sep 13 15:22:26 ems-esp emsesp 000+00:00:03.096 D 10: No telegram type handler found for ID 0x14 (src 0x08)
Sep 13 15:22:26 ems-esp emsesp 000+00:00:04.281 I 11: Starting NTP
Sep 13 15:22:26 ems-esp emsesp 000+00:00:04.305 I 12: Ethernet Connected with IP=192.168.77.14, speed 100 Mbps
Sep 13 15:22:30 ems-esp mqtt 000+00:00:04.305 I 13: MQTT connected
Sep 13 15:22:30 ems-esp mqtt 000+00:00:04.305 D 14: Subscribing to topic: ems-esp/system
Sep 13 15:22:30 ems-esp mqtt 000+00:00:04.409 D 15: Publishing topic ems-esp/info (#01, retain=1, retry=1, size=58, pid=1)
Sep 13 15:22:30 ems-esp mqtt 000+00:00:04.510 D 16: Publishing topic ems-esp/heartbeat (#02, retain=1, retry=1, size=171, pid=16408)
Sep 13 15:22:30 ems-esp mqtt 000+00:00:04.611 D 17: Subscribing to topic: ems-esp/system
Sep 13 15:22:31 ems-esp mqtt 000+00:00:04.712 D 18: Publishing topic ems-esp/status (#04, retain=1, retry=1, size=6, pid=1)
Sep 13 15:22:27 ems-esp mqtt 000+00:00:06.234 D 19: Publishing topic ems-esp/heartbeat (#05, retain=1, retry=1, size=171, pid=1)
Sep 13 15:22:28 ems-esp telegram 000+00:00:07.017 W 20: Incomplete Rx: 21 89 00
Sep 13 15:22:30 ems-esp telegram 000+00:00:08.740 W 21: Incomplete Rx: 10 21 00
Sep 13 15:22:30 ems-esp telegram 000+00:00:09.111 W 22: Incomplete Rx: 10 21 00
Sep 13 15:22:30 ems-esp emsesp 000+00:00:09.571 D 23: No telegram type handler found for ID 0x48 (src 0x10)
Sep 13 15:22:31 ems-esp emsesp 000+00:00:10.001 I 24: Stopping Access Point
Sep 13 15:22:32 ems-esp telegram 000+00:00:10.997 W 25: Incomplete Rx: 21 00 AB 00 05 01
Sep 13 15:22:33 ems-esp telegram 000+00:00:11.804 W 26: Incomplete Rx: 21 00 AB 00 05 01 26 00 88
Sep 13 15:22:33 ems-esp emsesp 000+00:00:11.914 D 27: No telegram type handler found for ID 0x18 (src 0x08)
Sep 13 15:22:33 ems-esp emsesp 000+00:00:12.195 D 28: No telegram type handler found for ID 0x18 (src 0x08)
Sep 13 15:22:34 ems-esp telegram 000+00:00:12.702 W 29: Incomplete Rx: 21 00 AB 00 05 01 26 00 9C 00
Sep 13 15:22:34 ems-esp emsesp 000+00:00:12.802 D 30: No telegram type handler found for ID 0x34 (src 0x08)

Please contact me if you need further informations.

[proddy]

@cyberfox01 could you try the build from the dev_webui_crash branch? I increased the queue size