Panic on WPA2 Enterprise Connection

[Flole998]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-01
• Core Version: 3.0.0
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: Generic ESP8266 Module
• Flash Mode: dio
• Flash Size: 1M
• lwip Variant: v2 Lower Memory
• Reset Method: ck
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

Immediately after connecting to a WPA2 Enterprise encrypted network I receive an exception. This seems to be related to the way free is called from a function in eap.c (which seems to be part of the SDK). I am not sure if this is a bug here or in the SDK, but apparently an address that is unmapped was passed to free. I am not sure if maybe the stack got corrupted somewhere as that function hierarchy doesn't really make sense to me.....

I have everything working on the super old version 2.3.0 (which uses completely different functions for setting WPA Enterprise up) but I want to update.

I have reproduced this on all Espressif Firmware Versions that are available.

MCVE Sketch

void setup() {

	Serial.begin(115200);
	Serial.println("Startup!");

	Serial.print("Heap Free: ");
	Serial.println(system_get_free_heap_size());

	enableWiFiAtBootTime();

	wifi_set_opmode_current(STATION_MODE);

	struct station_config wifi_config;

	memset(&wifi_config, 0, sizeof(wifi_config));
	strcpy((char*)wifi_config.ssid, ssid);
	wifi_station_set_config_current(&wifi_config);

	wifi_station_set_wpa2_enterprise_auth(1);

	wifi_station_set_enterprise_identity((uint8_t*)identity, strlen(identity));
	wifi_station_set_enterprise_cert_key(esp_cert_pem, esp_cert_pem_len, esp_key_pem, esp_key_pem_len, NULL, 1);

	wifi_station_disconnect();
	wifi_station_connect();

	Serial.println(F("Waiting for connection..."));


	while (WiFi.status() != WL_CONNECTED) {
		if (millis() > 60000) {
			Serial.println(F("Took wayyy to long. Restarting..."));
			ESP.restart();
		}

		delay(1000);
	}
}

void loop() {
         Serial.println("Connected!");
}

Debug Messages

No poison after block at: 0x406e4a2f, actual data: 0x0 0x0 0x80 0x0

User exception (panic/abort/assert)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

 Error
   :?:::0x4024b080:etharp_output
   :?:::0x40201a52:raise_exception
   :?:::0x40201aaf:__panic_func
   0x40100e54 get_unpoisoned_check_neighbors
   0x401012f2 umm_free
   :\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266\umm_malloc/umm_malloc.cpp:574
   0x4010130d umm_poison_free_fl
   :?:::0x4024c4e0:etharp_output
   :?:::0x40229979:wpa_set_bss
   0x40100827 HeapSelectDram
   :\packages\esp8266\hardware\esp8266\3.0.0\cores\esp8266/heap.cpp:370
   :?:::0x4022aac7:wpabuf_free
   :?:::0x4022bb92:wpa2_sm_rx_eapol
   :?:::0x4022bba6:wpa2_sm_rx_eapol
   :?:::0x4022bbe1:wpa2_sm_rx_eapol
   :?:::0x4022b7ac:wpa2_sm_rx_eapol
   :?:::0x40223129:sta_input
   :?:::0x40240ccf:pp_tx_idle_timeout
   :?:::0x4024058f:ppPeocessRxPktHdr
   : ?? ??:0
   : ?? ??:0
   :?:::0x40105b88:call_user_start_local
   :?:::0x40105b8e:call_user_start_local
   :?:::0x4010000d:call_user_start
   0x40235458 cont_ret
   0x40235411 cont_continue
   

[redfast00]

Can confirm, I also have this issue

[Flole998]

@wujiangang Can you help with this issue? There is a bug introduced somewhere (seems like it's in wpabuf_free) in the nonos-sdk that frees memory that was never allocated (or to be more precise the pointer is pointing to a memory region that's invalid), would be great if that could get fixed.

[Flole998]

@d-a-v Unfortunately it looks like this SDK issue won't get fixed. What would you think about relaxing the free() check a little to work around this issue? Maybe just silently drop if free() is called on invalid memory (and optionally check if it's being called by wpabuf_free())? I could also patch out that call from the library, but I'm not sure if you want to have a modified binary in this repository?

Or maybe you know someone from espressif who you can contact to get this fixed in the SDK?

[d-a-v]

Or maybe you know someone from espressif who you can contact to get this fixed in the SDK?

I have no more power than you do :]

About relaxing the call to free() in this repo's binary, we should first get to know whether it is useful ?
Did you try it ?

[Flole998]

Yes, I have tried it (and others have aswell, see https://bbs.espressif.com/viewtopic.php?t=5962&start=20#p71905), it's really just this free()-call that's causing issues.

[d-a-v]

Thanks for pointing us !
It would be very interesting for some of us to have it working.
It wouldn't be the first firmware hack.

Unconditionally patching the check_poison() to return 1 will not be accepted.
But I think we can imagine a conditional build which would enable a new function which would temporarily force check_poison()'s return value.

I think something like this would be acceptable:
We'd have to use -DANTIDOTE=1 (thanks to incoming #8504), then:

umm_antidote(true); // disable umm_check_poison()
wifi_station_set_wpa2_enterprise_auth(1);
wifi_station_set_enterprise_identity((uint8*)username, strlen(username));
wifi_station_set_enterprise_username((uint8*)username, strlen(username));
wifi_station_set_enterprise_password((uint8*)password, strlen(password));
wifi_station_set_enterprise_ca_cert((byte*)ca_cert, strlen(ca_cert));
wifi_station_connect();
umm_antidote(false); // re/enable umm_check_poison()

(s/antidote/whatever/g)

[Flole998]

I thought about something like ignoring free() on unmapped memory (as this address is way to high) or ignoring based on the calling functions address (so that wpabuf_free() is basically whitelisted).

[d-a-v]

Making umm_free() a weak function calling __umm_free() (as it is currently done for yield()) might be another easier and less invasive solution as it allows users to redefine it to something else, to be able to conditionally call the real __umm_free().

[mhightower83]

No poison after block at: 0x406e4a2f, actual data: 0x0 0x0 0x80 0x0

0x406e4a2f is not a DRAM address. Earlier umm_malloc did not verify heap addresses on non-debug builds and I believe would have crashed with a corrupted heap. The current version I believe will abort or panic - I have lost track of where I am in pushing changes. The current version does range checking to determine which heap an allocation belongs on. A bad address should be caught earlier.

Since this is in the SDK any funny stuff could be moved to vPortFree 's thin wrapper.
https: //github.com/esp8266/Arduino/blob/732db594929965c7d298b6257fa74cfbd1be0398/cores/esp8266/heap.cpp#L372

[mhightower83]

Never mind it is not that simple. The poison check fails after the block. The block starts in DRAM and extends far beyond DRAM address space. Most likely umm_block data is invalid resulting in an oversized allocation calculation. umm_info() does not show the allocation being freed as allocated. So this could be a double free or a corrupt heap pointer.

[Flole998]

A double-free could probably easily be detected by adding a print() to free(), then we can see if it's indeed called twice (and also who called it the first time)? A corrupt heap pointer would be a lot harder to identify without having access to the source code...