Sending GZIP HTML ~ 120kb+ (suggested enhancement)

[aaron-neal]

Before I start out, would just like to say thanks for this library it has helped me very quickly port a lot of work from the ESP to the STM32 platform where I can keep both systems fairly in sync!

I also do not know if this is a generic enhancment for both this repo and the non STM32 repo, that's up to you.

I have a HTML build process using GULP which Inlines, uglifies, minifies all of my HTML, CSS and JS . The output file is a single header file that puts the entire thing into PROGMEM.

#define index_html_gz_len 129855

const char PROGMEM index_html_gz[] = {
    0x1f,0x8b,0x08,0x00,0x00,0x00,0x00,0x00,....

In both the ESP32 / 8266 the following is used to send this as the index:

server.on(String(F("/index.html")).c_str(), HTTP_GET, [](AsyncWebServerRequest *request){
        if(authenticate(request)){
            return request->requestAuthentication();
        }
        
         //webPage setup
        char lastModified[50]; // Variable to hold the last modification datetime of website
        sprintf_P(lastModified, PSTR("%s %s GMT"), __DATE__, __TIME__); // Populate the last modification date based on build datetime
    
        if (request->header(F("If-Modified-Since")).equals(lastModified)) {
            request->send(304);
        } else {
            // Dump the byte array in PROGMEM with a 200 HTTP code (OK)
            AsyncWebServerResponse *response = request->beginResponse_P(200, F("text/html"), index_html_gz, index_html_gz_len);
            // Tell the browswer the contemnt is Gzipped
            response->addHeader(F("Content-Encoding"), F("gzip"));
            // And set the last-modified datetime so we can check if we need to send it again next time or not
            response->addHeader(F("Last-Modified"), lastModified);
            request->send(response);
        }
        WiFi.scanNetworks(true); //they might request wifi networks at some point
    });

Unforunately, this did not work with this library using the send_P command as I expected, the headers were send but no content. I have finally come up with a solution, I doubt that it is optimal, but for me it is working.

void EthernetWebServer::sendContent_P(PGM_P content, size_t size) 
{
  const char * footer = "\r\n";
  
  if (_chunked) 
  {
    char * chunkSize = (char *) malloc(11);
    
    if (chunkSize) 
    {
      sprintf(chunkSize, "%x%s", size, footer);
      _currentClient.write(chunkSize, strlen(chunkSize));
      free(chunkSize);
    }
  }

  uint16_t bufSize = 4096;
  uint8_t buffer[bufSize];
  uint16_t count = size / bufSize;
  uint16_t remainder = size % bufSize;
  uint16_t i = 0;

  for (i = 0; i < count; i++) {
    /* code */
    memcpy_P(buffer, &content[i*bufSize], bufSize);
    _currentClient.write(buffer, bufSize);
  }
  
  memcpy_P(buffer, &content[i*bufSize], remainder);
  _currentClient.write(buffer, remainder);
  
  if (_chunked) 
  {
    _currentClient.write(footer, 2);
  }
}

If there is a better way of doing this I would love to know, but for now I can carry on with my development, happy to offer a PR for something to work from if needed?

Now I can have my fully bootstrapped web interface that I use with my ESP devices. 👍

[aaron-neal]

see #4

[khoih-prog]

@porkyneal

Thanks so much for the good solution.
I'll test and merge soon into this EthernetWebServer_STM32 as well as EthernetWebServer, WiFiWebServer and WiFiNINA_Generic libraries.

Regards,

[khoih-prog]

@porkyneal

Could you please test to see this implementation in EthernetWebServer is OK in STM32.

void EthernetWebServer::sendContent_P(PGM_P content) 
{
  sendContent_P(content, strlen_P(content));
}

void EthernetWebServer::sendContent_P(PGM_P content, size_t size) 
{
  const char * footer = "\r\n";
  
  if (_chunked) 
  {
    char * chunkSize = (char *) malloc(11);
    
    if (chunkSize) 
    {
      sprintf(chunkSize, "%x%s", size, footer);
      _currentClient.write(chunkSize, strlen(chunkSize));
      free(chunkSize);
    }
  }
  _currentClient.write(content, size);
  
  if (_chunked) 
  {
    _currentClient.write(footer, 2);
  }
}

It's not currently implemented in STM32 (my fault). That's why you experienced the issue.

[khoih-prog]

@porkyneal

Which STM32 board are you using? Many STM32 (and other) boards don't support the memcpy_P command.
That might be the reason I dropped the PROGMEM related commands.

Could you also test the other version (I just draft, not compile / test) to use new, instead of local buffer, to allocate memory on heap.

void EthernetWebServer::sendContent_P(PGM_P content, size_t size) 
{
  const char * footer = "\r\n";
  
  if (_chunked) 
  {
    char * chunkSize = (char *) malloc(11);
    
    if (chunkSize) 
    {
      sprintf(chunkSize, "%x%s", size, footer);
      _currentClient.write(chunkSize, strlen(chunkSize));
      free(chunkSize);
    }
  }

#if 1
#define SENDCONTENT_P_BUFFER_SZ     4096
 
  uint8_t* buffer = new uint8_t[SENDCONTENT_P_BUFFER_SZ];
  
  if (buffer)
  {
    uint16_t count = size / SENDCONTENT_P_BUFFER_SZ;
    uint16_t remainder = size % SENDCONTENT_P_BUFFER_SZ;
    uint16_t i = 0;

    for (i = 0; i < count; i++) {
      /* code */
      memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], SENDCONTENT_P_BUFFER_SZ);
      _currentClient.write(buffer, SENDCONTENT_P_BUFFER_SZ);
    }
    
    memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], remainder);
    _currentClient.write(buffer, remainder);
    
    delete [] buffer;
    
  }
  else
  {
  }
#else

  uint16_t bufSize = 4096;
  uint8_t buffer[bufSize];
  uint16_t count = size / bufSize;
  uint16_t remainder = size % bufSize;
  uint16_t i = 0;

  for (i = 0; i < count; i++) {
    /* code */
    memcpy_P(buffer, &content[i*bufSize], bufSize);
    _currentClient.write(buffer, bufSize);
  }
  
  memcpy_P(buffer, &content[i*bufSize], remainder);
  _currentClient.write(buffer, remainder);
  
#endif
  
  if (_chunked) 
  {
    _currentClient.write(footer, 2);
  }
}

[khoih-prog]

@porkyneal

Could you verify these mods are OK in your case.

---

Compile OK, not yet tested:

1. EthernetWebServer_STM32.h

// For PROGMEM commands
#include <pgmspace.h>

#define memccpy_P(dest, src, c, n) memccpy((dest), (src), (c), (n))

#ifndef PGM_VOID_P
  #define PGM_VOID_P const void *
#endif

2. EthernetWebServer_STM32.h

void send_P(int code, PGM_P content_type, PGM_P content);
void send_P(int code, PGM_P content_type, PGM_P content, size_t contentLength);
    
void sendContent_P(PGM_P content);
void sendContent_P(PGM_P content, size_t size);

3. EthernetWebServer_STM32-impl.h

void EthernetWebServer::send_P(int code, PGM_P content_type, PGM_P content) 
{
  size_t contentLength = 0;

  if (content != NULL) 
  {
    contentLength = strlen_P(content);
  }

  String header;
  char type[64];
  
  memccpy_P((void*)type, (PGM_VOID_P)content_type, 0, sizeof(type));
  _prepareHeader(header, code, (const char* )type, contentLength);
  
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: len = "), contentLength);
  ET_LOGDEBUG1(F("content = "), content);
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: hdrlen = "), header.length());
  ET_LOGDEBUG1(F("header = "), header);
  
  _currentClient.write(header.c_str(), header.length());
  
  if (contentLength)
  {
    sendContent_P(content);
  }
}

void EthernetWebServer::send_P(int code, PGM_P content_type, PGM_P content, size_t contentLength) 
{
  String header;
  char type[64];
  
  memccpy_P((void*)type, (PGM_VOID_P)content_type, 0, sizeof(type));
  _prepareHeader(header, code, (const char* )type, contentLength);
  
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: len = "), contentLength);
  ET_LOGDEBUG1(F("content = "), content);
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: hdrlen = "), header.length());
  ET_LOGDEBUG1(F("header = "), header);

  _currentClient.write((const uint8_t *) header.c_str(), header.length());
  
  if (contentLength)
  {
    sendContent_P(content, contentLength);
  }
}


void EthernetWebServer::sendContent_P(PGM_P content) 
{
  sendContent_P(content, strlen_P(content));
}

void EthernetWebServer::sendContent_P(PGM_P content, size_t size) 
{
  const char * footer = "\r\n";
  
  if (_chunked) 
  {
    char * chunkSize = (char *) malloc(11);
    
    if (chunkSize) 
    {
      sprintf(chunkSize, "%x%s", size, footer);
      _currentClient.write(chunkSize, strlen(chunkSize));
      free(chunkSize);
    }
  }

#if 1
#define SENDCONTENT_P_BUFFER_SZ     4096
 
  uint8_t* buffer = new uint8_t[SENDCONTENT_P_BUFFER_SZ];
  
  if (buffer)
  {
    uint16_t count = size / SENDCONTENT_P_BUFFER_SZ;
    uint16_t remainder = size % SENDCONTENT_P_BUFFER_SZ;
    uint16_t i = 0;

    for (i = 0; i < count; i++) {
      /* code */
      memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], SENDCONTENT_P_BUFFER_SZ);
      _currentClient.write(buffer, SENDCONTENT_P_BUFFER_SZ);
    }
    
    memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], remainder);
    _currentClient.write(buffer, remainder);
    
    delete [] buffer;
    
  }
  else
  {
  }
#else

  uint16_t bufSize = 4096;
  uint8_t buffer[bufSize];
  uint16_t count = size / bufSize;
  uint16_t remainder = size % bufSize;
  uint16_t i = 0;

  for (i = 0; i < count; i++) {
    /* code */
    memcpy_P(buffer, &content[i*bufSize], bufSize);
    _currentClient.write(buffer, bufSize);
  }
  
  memcpy_P(buffer, &content[i*bufSize], remainder);
  _currentClient.write(buffer, remainder);
  
#endif
  
  if (_chunked) 
  {
    _currentClient.write(footer, 2);
  }
}

[aaron-neal]

Sorry, I have been away for a few days. I wont be able to test this for a day or so.

From memory the current Non STM32 specific EthernetWebServer implementation did not work, it compiles, didn't crash, but no data was transferred.

Again, will get on it when I back on that project.

[khoih-prog]

From memory the current Non STM32 specific EthernetWebServer implementation did not work, it compiles, didn't crash, but no data was transferred.

This happens because data is stored in PROGMEM and the original code doesn't support PROGMEM-related functions.

I already fixed in the previous post to add PROGMEM functions, waiting for you to test, before commit and release new version. Certainly with your contribution credit.

[aaron-neal]

@porkyneal

Could you verify these mods are OK in your case.

Compile OK, not yet tested:

1. EthernetWebServer_STM32.h

// For PROGMEM commands
#include <pgmspace.h>

#define memccpy_P(dest, src, c, n) memccpy((dest), (src), (c), (n))

#ifndef PGM_VOID_P
  #define PGM_VOID_P const void *
#endif

1. EthernetWebServer_STM32.h

void send_P(int code, PGM_P content_type, PGM_P content);
void send_P(int code, PGM_P content_type, PGM_P content, size_t contentLength);
    
void sendContent_P(PGM_P content);
void sendContent_P(PGM_P content, size_t size);

1. EthernetWebServer_STM32-impl.h

void EthernetWebServer::send_P(int code, PGM_P content_type, PGM_P content) 
{
  size_t contentLength = 0;

  if (content != NULL) 
  {
    contentLength = strlen_P(content);
  }

  String header;
  char type[64];
  
  memccpy_P((void*)type, (PGM_VOID_P)content_type, 0, sizeof(type));
  _prepareHeader(header, code, (const char* )type, contentLength);
  
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: len = "), contentLength);
  ET_LOGDEBUG1(F("content = "), content);
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: hdrlen = "), header.length());
  ET_LOGDEBUG1(F("header = "), header);
  
  _currentClient.write(header.c_str(), header.length());
  
  if (contentLength)
  {
    sendContent_P(content);
  }
}

void EthernetWebServer::send_P(int code, PGM_P content_type, PGM_P content, size_t contentLength) 
{
  String header;
  char type[64];
  
  memccpy_P((void*)type, (PGM_VOID_P)content_type, 0, sizeof(type));
  _prepareHeader(header, code, (const char* )type, contentLength);
  
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: len = "), contentLength);
  ET_LOGDEBUG1(F("content = "), content);
  ET_LOGDEBUG1(F("EthernetWebServer::send_P: hdrlen = "), header.length());
  ET_LOGDEBUG1(F("header = "), header);

  _currentClient.write((const uint8_t *) header.c_str(), header.length());
  
  if (contentLength)
  {
    sendContent_P(content, contentLength);
  }
}


void EthernetWebServer::sendContent_P(PGM_P content) 
{
  sendContent_P(content, strlen_P(content));
}

void EthernetWebServer::sendContent_P(PGM_P content, size_t size) 
{
  const char * footer = "\r\n";
  
  if (_chunked) 
  {
    char * chunkSize = (char *) malloc(11);
    
    if (chunkSize) 
    {
      sprintf(chunkSize, "%x%s", size, footer);
      _currentClient.write(chunkSize, strlen(chunkSize));
      free(chunkSize);
    }
  }

#if 1
#define SENDCONTENT_P_BUFFER_SZ     4096
 
  uint8_t* buffer = new uint8_t[SENDCONTENT_P_BUFFER_SZ];
  
  if (buffer)
  {
    uint16_t count = size / SENDCONTENT_P_BUFFER_SZ;
    uint16_t remainder = size % SENDCONTENT_P_BUFFER_SZ;
    uint16_t i = 0;

    for (i = 0; i < count; i++) {
      /* code */
      memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], SENDCONTENT_P_BUFFER_SZ);
      _currentClient.write(buffer, SENDCONTENT_P_BUFFER_SZ);
    }
    
    memcpy_P(buffer, &content[i*SENDCONTENT_P_BUFFER_SZ], remainder);
    _currentClient.write(buffer, remainder);
    
    delete [] buffer;
    
  }
  else
  {
  }
#else

  uint16_t bufSize = 4096;
  uint8_t buffer[bufSize];
  uint16_t count = size / bufSize;
  uint16_t remainder = size % bufSize;
  uint16_t i = 0;

  for (i = 0; i < count; i++) {
    /* code */
    memcpy_P(buffer, &content[i*bufSize], bufSize);
    _currentClient.write(buffer, bufSize);
  }
  
  memcpy_P(buffer, &content[i*bufSize], remainder);
  _currentClient.write(buffer, remainder);
  
#endif
  
  if (_chunked) 
  {
    _currentClient.write(footer, 2);
  }
}

This all seems to work OK. 👍

[khoih-prog]

Thanks. I'll merge into master then create a new release.

[aaron-neal]

Sweet, thanks.

[khoih-prog]

@porkyneal

Already merged then updated the code, then release v1.0.6

Release v1.0.6

1. Add support to PROGMEM-related commands, such as sendContent_P() and send_P()
2. Update Platform.ini to support PlatformIO 5.x owner-based dependency declaration.
3. Clean up code.

Your contribution is noted in Contributions-and-Thanks, but under the name of Aron N., not Nickname porkyneal ;-)

More contributions are awaited and very welcome.

Regards,