HTTPClient leaking memory?

[Jeroen88]

Hardware:

Board: ESP32 NodeMCU
Core Installation version: ?1.0.0? ?1.0.1-rc4? ?1.0.1? ?1.0.1-git? ?1.0.2? ?1.0.3?
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 921600
Computer OS: Ubuntu

Description:

I believe HTTPClient is leaking memory. The sketch I provide here is the BasicHttpsClient example from the library. I set debug level to verbose and removed all of the debug info, except for the line:
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 262632
The Free internal heap before TLS drops at each iteration (see debug lines below):
-/- 376 bytes
-/- 104 bytes
-/- 100 bytes
-/- 96 bytes
-/- 96 bytes
-/- 96 bytes
-/- 96 bytes
-/- 96 bytes
-/- 96 bytes

Sketch: (leave the backquotes for code formatting)

//Change the code below by your sketch
/**
   BasicHTTPSClient.ino

    Created on: 14.10.2018

*/

#include <Arduino.h>

#include <WiFi.h>
#include <WiFiMulti.h>

#include <HTTPClient.h>

#include <WiFiClientSecure.h>

// This is GandiStandardSSLCA2.pem, the root Certificate Authority that signed 
// the server certifcate for the demo server https://jigsaw.w3.org in this
// example. This certificate is valid until Sep 11 23:59:59 2024 GMT
const char* rootCACertificate = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB\n" \
"iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
"cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n" \
"BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQw\n" \
"OTEyMDAwMDAwWhcNMjQwOTExMjM1OTU5WjBfMQswCQYDVQQGEwJGUjEOMAwGA1UE\n" \
"CBMFUGFyaXMxDjAMBgNVBAcTBVBhcmlzMQ4wDAYDVQQKEwVHYW5kaTEgMB4GA1UE\n" \
"AxMXR2FuZGkgU3RhbmRhcmQgU1NMIENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IB\n" \
"DwAwggEKAoIBAQCUBC2meZV0/9UAPPWu2JSxKXzAjwsLibmCg5duNyj1ohrP0pIL\n" \
"m6jTh5RzhBCf3DXLwi2SrCG5yzv8QMHBgyHwv/j2nPqcghDA0I5O5Q1MsJFckLSk\n" \
"QFEW2uSEEi0FXKEfFxkkUap66uEHG4aNAXLy59SDIzme4OFMH2sio7QQZrDtgpbX\n" \
"bmq08j+1QvzdirWrui0dOnWbMdw+naxb00ENbLAb9Tr1eeohovj0M1JLJC0epJmx\n" \
"bUi8uBL+cnB89/sCdfSN3tbawKAyGlLfOGsuRTg/PwSWAP2h9KK71RfWJ3wbWFmV\n" \
"XooS/ZyrgT5SKEhRhWvzkbKGPym1bgNi7tYFAgMBAAGjggF1MIIBcTAfBgNVHSME\n" \
"GDAWgBRTeb9aqitKz1SA4dibwJ3ysgNmyzAdBgNVHQ4EFgQUs5Cn2MmvTs1hPJ98\n" \
"rV1/Qf1pMOowDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYD\n" \
"VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCIGA1UdIAQbMBkwDQYLKwYBBAGy\n" \
"MQECAhowCAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNl\n" \
"cnRydXN0LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNy\n" \
"bDB2BggrBgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRy\n" \
"dXN0LmNvbS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZ\n" \
"aHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAWGf9\n" \
"crJq13xhlhl+2UNG0SZ9yFP6ZrBrLafTqlb3OojQO3LJUP33WbKqaPWMcwO7lWUX\n" \
"zi8c3ZgTopHJ7qFAbjyY1lzzsiI8Le4bpOHeICQW8owRc5E69vrOJAKHypPstLbI\n" \
"FhfFcvwnQPYT/pOmnVHvPCvYd1ebjGU6NSU2t7WKY28HJ5OxYI2A25bUeo8tqxyI\n" \
"yW5+1mUfr13KFj8oRtygNeX56eXVlogMT8a3d2dIhCe2H7Bo26y/d7CQuKLJHDJd\n" \
"ArolQ4FCR7vY4Y8MDEZf7kYzawMUgtN+zY+vkNaOJH1AQrRqahfGlZfh8jjNp+20\n" \
"J0CT33KpuMZmYzc4ZCIwojvxuch7yPspOqsactIGEk72gtQjbz7Dk+XYtsDe3CMW\n" \
"1hMwt6CaDixVBgBwAc/qOR2A24j3pSC4W/0xJmmPLQphgzpHphNULB7j7UTKvGof\n" \
"KA5R2d4On3XNDgOVyvnFqSot/kGkoUeuDcL5OWYzSlvhhChZbH2UF3bkRYKtcCD9\n" \
"0m9jqNf6oDP6N8v3smWe2lBvP+Sn845dWDKXcCMu5/3EFZucJ48y7RetWIExKREa\n" \
"m9T8bJUox04FB6b9HbwZ4ui3uRGKLXASUoWNjDNKD/yZkuBjcNqllEdjB+dYxzFf\n" \
"BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=\n" \
"-----END CERTIFICATE-----\n";

// Not sure if WiFiClientSecure checks the validity date of the certificate. 
// Setting clock just to be sure...
void setClock() {
  configTime(0, 0, "pool.ntp.org", "time.nist.gov");

  Serial.print(F("Waiting for NTP time sync: "));
  time_t nowSecs = time(nullptr);
  while (nowSecs < 8 * 3600 * 2) {
    delay(500);
    Serial.print(F("."));
    yield();
    nowSecs = time(nullptr);
  }

  Serial.println();
  struct tm timeinfo;
  gmtime_r(&nowSecs, &timeinfo);
  Serial.print(F("Current time: "));
  Serial.print(asctime(&timeinfo));
}


WiFiMulti WiFiMulti;

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("SSID", "PASSWORD");

  // wait for WiFi connection
  Serial.print("Waiting for WiFi to connect...");
  while ((WiFiMulti.run() != WL_CONNECTED)) {
    Serial.print(".");
  }
  Serial.println(" connected");

  setClock();  
}

void loop() {
  Serial.print("Free heap beginning loop:"); Serial.println(ESP.getFreeHeap());
  WiFiClientSecure *client = new WiFiClientSecure;
  if(client) {
    client -> setCACert(rootCACertificate);

    {
      // Add a scoping block for HTTPClient https to make sure it is destroyed before WiFiClientSecure *client is 
      HTTPClient https;
  
      Serial.print("[HTTPS] begin...\n");
      if (https.begin(*client, "https://jigsaw.w3.org/HTTP/connection.html")) {  // HTTPS
        Serial.print("[HTTPS] GET...\n");
        // start connection and send HTTP header
        int httpCode = https.GET();
  
        // httpCode will be negative on error
        if (httpCode > 0) {
          // HTTP header has been send and Server response header has been handled
          Serial.printf("[HTTPS] GET... code: %d\n", httpCode);
  
          // file found at server
          if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
            String payload = https.getString();
            Serial.println(payload);
          }
        } else {
          Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
        }
  
        https.end();
      } else {
        Serial.printf("[HTTPS] Unable to connect\n");
      }

      // End extra scoping block
    }
  
    delete client;
  } else {
    Serial.println("Unable to create client");
  }

  Serial.println();
  Serial.println("Waiting 10s before the next round...");
  Serial.print("Free heap end loop:"); Serial.println(ESP.getFreeHeap());
  delay(10000);
}

Debug Messages:

[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 262632
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 262256
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 262152
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 262052
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261956
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261860
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261764
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261668
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261572
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 261476 

[stickbreaker]

A possible locus: WifiSecureClient._streamLoad()

in your code you use setCACert() which just transfers a memory pointer, that points to your constant buffer(not a problem) But inside WiFiSecureClient() it has the potential to load from a Stream. I don't see anywhere in the code where it ever calls free() to match the malloc() in _streamLoad()

[Jeroen88]

@stickbreaker I agree that there is no free() for the malloc() in _streamLoad(). I am also surprised of the variable static char *dest = nullptr being local to the function _streamLoad(). Should be a private class variable and properly destructed.

However, I do not call loadCACert(), do this bug is not the cause of the leak.

Also, I have the impression that a lot of memory is leaked on a failed connection (~2k), causing the next call to GET to fail, but I am not sure yet. Maybe I can drop some debug info of another sketch (too big and to specialized to publish) that uses a PUT that fails first on the certificate that can not be verified and next on a lack of memory. No time tonight, so maybe tomorrow.

[stickbreaker]

My read through of WiFiSecureClient() showed me that issue, I haven't delved deeper into WiFiClient() yet, but the _streamLoad() issue could be a symptom of other questionable practices.

Chuck.

[stickbreaker]

@Jeroen88 can you try branch stickbreaker-patch-3? I think I found the Memory Leak.

Chuck.

[Jeroen88]

@stickbreaker thnx Chuck, I have a look at it later today!

[Jeroen88]

@stickbreaker your PR may be right, I did not check it, but it did not solve my problem. Your solution is in WiFiClient::stop() and I use a WiFiClientSecure. I double checked, although WiFiClientSecure is a public WiFiClient, the base WiFiClient::stop() is never called from WiFiClientSecure::stop(). It must be something else...

[Jeroen88]

Closing, because I found out that after 11 iterations the free heap stabilizes.