WebServer / StaticRequestHandler with mixed gz and non-gz contents

[tobozo]

Board

any

Device Description

any esp32 device

Hardware Configuration

any configuration using WebServer library

Version

3.1.0 (also on master)

IDE Name

Arduino IDE (any)

Operating System

Linux

Flash frequency

n/a

PSRAM enabled

yes

Upload speed

n/a

Description

Currentl StaticRequestHandler doesn't serve index.htm.gz file when index.htm exists.

&& !_fs.exists(path) prevents gz content to be served: when both compressed and uncompressed versions of same file are present, the uncompressed version is served.

arduino-esp32/libraries/WebServer/src/detail/RequestHandlersImpl.h

Line 207 in 51ef2a1

if (!path.endsWith(FPSTR(mimeTable[gz].endsWith)) && !_fs.exists(path)) {

Moreover, StaticRequestHandler::handle() completely ignores the 'Accept-Encoding' header sent by the client, which should be checked for gzip and/or deflate before deciding if it is worth looking for a compressed version of the file.

Sketch

server.serveStatic("/", LittleFS, "/");

Debug Message

filesystem contents:

File: /index.htm     29878 bytes
File: /index.htm.gz  13698 bytes

current behaviour:

$ curl -I -H 'Accept-Encoding: gzip,deflate' http://192.168.1.13
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 29878
Connection: close

expected behaviour:

$ curl -I -H 'Accept-Encoding: ' http://192.168.1.13
HTTP/1.1 200 OK
Content-Type: text/html
Content-Encoding: gzip
Content-Length: 13698
Connection: close

other expected behaviour (no accept-encoding headers):

$ curl -I http://192.168.1.13
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 29878
Connection: close

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

This is on purpose. The point is to be able to edit the HTML and when you are happy to turn it into gzip. Legacy from the FSBrowser. Firmware comes with some standard gzipped index and you can add your own uncompressed one through the editor/upload

[tobozo]

This logic only makes sense with FSBrowser. and it breaks the principles of content negociation (should use the "best-suited" method in every situation).

[me-no-dev]

I guess you are proposing to parse the Accept-Encoding header, as well as returning the GZIP by default (if present)?

[tobozo]

yup, I'm not sure how to do that without affecting the FSBrowser editing workflow though

that's what I'm using in a custom handler, I also have to add Accept-Encoding via server.collectHeaders():

    bool hasGz = false;

    if( server.hasHeader("Accept-Encoding") ) {
      String acceptEncoding = server.header("Accept-Encoding");
      if( acceptEncoding.indexOf("gzip") > 0 || acceptEncoding.indexOf("deflate") > 0 ) {
        hasGz = _fs.exists(pathWithGz);
      }
    }

    if( hasGz ) {
      path = pathWithGz;
    }

[tobozo]

Adding more feedback here as I found another anomaly in WebServer class:

WebServer::_prepareHeader() selects Chunked transfer encoding when the content length is unknown.

arduino-esp32/libraries/WebServer/src/WebServer.cpp

Lines 564 to 569 in 2f423af

	} else if (_contentLength == CONTENT_LENGTH_UNKNOWN && _currentVersion) { //HTTP/1.1 or above client
	//let's do chunked
	_chunked = true;
	sendHeader(String(F("Accept-Ranges")), String(F("none")));
	sendHeader(String(F("Transfer-Encoding")), String(F("chunked")));
	}

While this is correct with HTTP 1.1, it isn't valid with HTTP 1.0.

With HTTP 1.0 the Content-Length header is optional: chunked transfer encoding is only available since HTTP 1.1.

[me-no-dev]

we can not support requests with unknown lengths in HTTP1.0. Either send the length with 1.0 or use chunked with 1.1

[tobozo]

Cloudflare does it.

This is implicit in both HTTP/1.0 and HTTP/1.1: a server response without content-length ends when the stream closes.

https://datatracker.ietf.org/doc/html/rfc1945#section-7.2.2

If a Content-Length header field is present, its value in bytes represents the length of the Entity-Body. Otherwise, the body length is determined by the closing the connection by the server.

Moreover, a HTTP/1.1 response without content-length is not necessarily chunked, it can be multipart/byteranges or support other (!=identity) transfer encodings.

https://datatracker.ietf.org/doc/html/rfc2616#section-4.4

2.If a Transfer-Encoding header field (section 14.41) is present and has any value other than "identity", then the transfer-length is defined by use of the "chunked" transfer-coding (section 3.6), unless the message is terminated by closing the connection.

[me-no-dev]

Seems you have done quite a bit of research. Why not make a PR with the changes?

[tobozo]

Why not make a PR with the changes?

Because I'm biased by my use case (to mimic apache's mod_gzip+mod_cache) and therefore at risk of bloating the library with maybe-unnecessary features.

In my use case, skipping the content-length header only makes sense when streaming content of unknown size generated on the fly (such as gzip/deflate), so I overloaded/inherited instead of modifying the WebServer library:

• created a custom StaticRequestHandler with mod_gzip + mod_cache logic
• added missing support for HTTP_HEAD to that custom handler
• created a derivated class GzWebServer from WebServer
• implemented GzWebserver::gzStream() to build+send its own HTTP response without content-length
• added stream encoder callback to GzWebServer size_t(Stream *input, size_t len, Stream* output)
• added filesystem encoder callback to GzWebServer size_t(fs::FS&fs, const String &inputFilename, bool use_cache)
• upgraded to 3.1.1 and figured out a new MiddleWare class was introduced 🤦

I need study that new middleware class anyway, I haven't figured out yet if it should be involded.

[tobozo]

if an image is worth a thousand words, a WebServer_mod_gzip example is worth a thousand lines 🤭

https://github.com/tobozo/ESP32-targz/tree/LZ77/examples/ESP32/WebServer_mod_gzip