[Bug]: WiFi / Network connectivity issues with 2.4.13+

[koliha]

Category

WiFi

Hardware

T-Beam, Heltec V3, Station G2

Firmware Version

2.5.13

Description

After a period of time (6-18hrs, no longer than 24hr), WiFi enabled ESP32 based devices on are losing network connectivity (not physical connectivity, just no longer passing traffic). I have tested this with 2.5.13 and 2.5.14, but not older 2.5.x builds. I'm going to setup a heltec v3 for testing so I can capture console traffic. I'll post that here when I'm able to do so.

When the issue occurs:

• You can de-auth the node or power off the access point that it's connected to and it reconnects to wifi (same or another AP)
• When it reconnects you can see the node request DHCP and get a response (dhcp server logs)
• The DHCP exchange is successfully sniffed by my Ubiquiti setup
• Traffic logs (ubiquiti) show WiFi rx but no tx for the node during this issue

I am seeing this on:

• Two different wireless networks using Ubiquiti hardware
• One wireless network running off a Netgear Nighthawk
• Station G2, Heltec V3, and a T-Beam

Relevant log output

No response

[koliha]

This is what I see from the WiFi side -- this pattern repeats.

It's not related to signal, here is a closer AP with the same pattern/behavior:

If I check my DHCP logs for today I see a DHCP lease renewals up until the wifi drop at 2:57pm. The 3:16pm wifi drop is seen as well, but nothing for the 3:24/26/27 reconnects.

Not sure if I can set a static IP, but that's my next step in attempting to troubleshoot.

[garthvh]

Are you connected to the public mqtt server? Do you have device logs?

[CTassisF]

Have you tried the latest "Bleeding" firmware? A bug affecting Wi-Fi connectivity (reported in #5387) was recently fixed in PR #5439.

[koliha]

Are you connected to the public mqtt server? Do you have device logs?

No device logs. Using a public MQTT server (used by tens of others), but not "the" public MQTT server. MQTT server resides on my local network (connecting via private address).

Have you tried the latest "Bleeding" firmware? A bug affecting Wi-Fi connectivity (reported in #5387) was recently fixed in PR #5439.

This seems extremely relevant. Only using pre-built/published builds, so I'll probably wait for a release to test.

I have a static IP set as of 8pm on the problematic node and will report back the results. Based on #5387 I would assume that it's still going to occur with a static. Rather than try to set something up to collect logs, I'll probably wait out the next alpha to see if it solves.

[CTassisF]

There were recent changes to MQTT servers in "private" (RFC1918) networks (discussed in #5203) that might be related to your issue. However, if I were to make an educated guess, most (if not all) of these Wi-Fi issues were likely caused by the memory leaks fixed in MQTT::onReceive.

[koliha]

There were recent changes to MQTT servers in "private" (RFC1918) networks (discussed in #5203) that might be related to your issue. However, if I were to make an educated guess, most (if not all) of these Wi-Fi issues were likely caused by the memory leaks fixed in MQTT::onReceive.

I am assuming the same re: MQTT::onReceive. I'm using public dns and IP and relying on a nat mirroring rule to route it back inside, so the RFC1918 issue shouldn't impact, but still great info to know.

[fifieldt]

Potentially also relevant: we were power saving even when wifi was connected. That's fixed now: #5443

[koliha]

Potentially also relevant: we were power saving even when wifi was connected. That's fixed now: #5443

Saw that but it's not related.

Static IP has kept it from falling off completely overnight, but it's disconnecting+reconnecting to wifi over and over and seems a little wonky overall (pretty sure it's not beaconing device metrics to MQTT consistently). 2 minutes of no ping responses, responds for 5-7 seconds, then back to no response.

I'll wait for a release that incorporates #5387 before I attempt to troubleshoot further.

[leshniak]

I'm also experiencing the issue on 2.5.14.f2ee0df and latest beta. Waiting for the mentioned fixes.

[LowVoltagePirate]

I've build the files from the master repo and can confirm that the issue seems fixed, my t-beam is now online >48 without any issues

[leshniak]

I’ve also decided to make my own build yesterday. So far so good, it maintains a stable WiFi connection and heap usage stabilized at 90% after 24h.

[leshniak]

Hmm, the same issue sometimes occurs while connecting to the node via web interface and TCP. The device rebooted itself after few minutes. No logs for now unfortunately, due to remote location.

I have ~70 nodes in my NodeDB, if that matters.

Edit: looks like all is fine if I connect to the node shortly after the reboot.

[CTassisF]

Meshtastic Firmware 2.5.15.79da236 Alpha was released with fixes for #5387.

[fifieldt]

Fixed by #5387