Improve available heap

[devyte]

Hardware

Hardware: all
Core Version: git / 2.4-rc2

Problem Description

Summary

This issue is meant to track a RAM optimization effort. The goal is to increase available RAM as much as possible.

Details

The following are initial ideas to investigate.

• Move all const char strings to flash, and use the the access macros to access them. Implement alternate methods/functions with the correct argument signatures where applicable. This covers both the core as well as the Arduino lib code.
• Move const declarations to flash. Example: CHARS_PER_LINE in libb64/cencode.c.
• Revise all singleton object declarations. Make sure that all are wrapped by #defines that allow compile-time configuration to not use the global singletons. Example: Use of the Serial, but not Serial1, frees up 256+ bytes of heap.
• Revise all uses of constant size char [] arrays. Arrays like that should not be declared on the stack, but rather created dynamically, and only on demand. Example: 2K buffer in ESP8266Webserver.
  -Revise all uses of String. When concatenating, the internal managed array will grow, sometimes far beyond what is needed. It is sometimes better to reserve, if there is an estimate of how much space will be needed.
• Revise object implementations. Make sure objects are passed by reference or pointer, and never by copy.

[Pablo2048]

Actually I suggest to modify debug messages also to store text into PROGMEM. I've write macro for this reason:
#ifdef DEBUG_ESP_PORT #define DEBUG_MSG(_1, ...) DEBUG_ESP_PORT.printf_P(PSTR(_1), ##__VA_ARGS__) #else #define DEBUG_MSG(...) #endif

[d-a-v]

There is this macro USE_OPTIMIZE_PRINTF in osapi.h which, if defined, makes all os_printf's fmt stored in flash.

[Pablo2048]

Yes, but IMHO this is not used in debug messages inside libraries - for example look into ESP8266WiFiGeneric.h macro DEBUG_WIFI_GENERIC. Every debug macro inside library uses DEBUG_ESP_PORT.printf() so it uses Serial, or Serial1 object, not os_printf...

[earlephilhower]

You probably already know this, but the following command lines are very useful for finding stuff in RODATA that's eating up RAM:

For finding constant strings: objdump  -s -j .rodata /tmp/arduino_build_<code>/*.elf

For finding symbols(aka variables): readelf -a /tmp/arduino_build_<code>/*.elf| grep 3f

Some wasteful things that stick out for me:

• ctype: 256 byte constant lookup table that appears in most apps using any string ops
• dns_table: 1.1KB to hold DNS entries for only 4 hostnames. This needs LWIP recompiled with a smaller #define'd hostname length.

[d-a-v]

@earlephilhower about dns_table, what would be reasonable? 4 hosts, 64 chars each ?
(check here and there, knowing that lwip/256chars is RFC-compliant)

[earlephilhower]

@d-a-v well, that's a good question. I do know the RFC says 256 octets, but practically on a tiny ESP8266 you're likely only initiating connections to 1 or 2 servers with 32-64 byte hostnames at any time.

With no changes to code, I like your 64-byte x 4 suggestion. If small LWIP patches can be entertained, then I'd either submit something that only stored hashes of the hostname (and thus only requiring 64-bytes for any length hostname) or did a dynamic allocation like @devyte is considering.

AFAIK there are already memory allocations in LWIP, so this second option may not break any contracts (but I've only spent a little bit going through the code so could be wrong here).

And if we're going heroic on it, valid DNS hostnames only allow something like 40 valid characters so you could even do a packed 6-bit encoding to save 25% of any space without any compromise on lengths.

[devyte]

@earlephilhower thank you for that readelf/objdump suggestions, I've actually done that, but only once before and a looong time ago. It didn't even cross my mind to apply it here.
Any further suggestions along those lines are welcome.