correctly update ContentLength for uncompressed response body

[ruitianzhong]

fix the bug that is found when reproducing in issue #491 .

When Content-Encoding is gzip, the uncompressed response body length is different from the compressed one. So the Content-Length need to be updated to stay consistent with the uncompressed body length.

More detail in #491 (comment)

[cfc4n]

Can you add a unit test?

[ruitianzhong]

• first I add three *.bin files. In particular, 952253698002000.bin file is response body which is compressed with gzip to detect the bug we fix in this PR.
  To see its content, run the following content

cat 952253698002000.bin | gunzip

• Also error detection is added in TestEventProcessor_Serve() function to explicitly detect error through log message.

With all steps above, it can detect all related bugs on version that is only added unit test related code with the bug not fixed.

=== RUN   TestEventProcessor_Serve
    processor_test.go:81: 2024/03/01 20:00:21 [http response] DumpResponse error: unexpected EOF
    processor_test.go:81: 2024/03/01 20:00:21 [http response] DumpResponse error: http: ContentLength=1145 with Body length 2443
    processor_test.go:89: some errors occurred
--- FAIL: TestEventProcessor_Serve (3.00s)

It can be verified through my demo branch https://github.com/ruitianzhong/ecapture/tree/demo

• Furthermore, with the added unit test, one more error is found during testing (i.e., DumpResponse error: unexpected EOF). This is because of the wrong value for Content-Length (74332, actually too large) in 952282712204824.bin.

• Finally, with all the bug fixes applied, all the unit tests are passed and it can be verified by checking out https://github.com/gojue/ecapture/actions/runs/8110905628/job/22169121779?pr=498#step:8:237 and https://github.com/gojue/ecapture/actions/runs/8110905628/job/22169121779?pr=498#step:8:247

[cfc4n]

PTAL

[ruitianzhong]

Further Explanation

On previous commit(before this PR):

When running:

wget -d --header 'Accept-Encoding: gzip' https://1.1.1.1

It works well and no DumpError occurred.

However, when running:

wget -d --header 'Accept-Encoding: gzip' https://www.baidu.com

DumpError occurred(mentioned in #491).

After further investigation, I found that it's because of the response structure.

for baidu, its response header looks like

HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 1145
Server: bfe
Date: Fri, 01 Mar 2024 08:24:27 GMT
Content-Type: text/html;charset=UTF-8

for 1.1.1.1, its response header looks like:

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked``

Obviously, Content-Length is different. According to the HTTP/1.1 standard, with Transfer-Encoding set, Content-Length is not set and in go it is zero, according to comment in http package comment:

// ContentLength records the length of the associated content. The
	// value -1 indicates that the length is unknown. Unless Request.Method
	// is "HEAD", values >= 0 indicate that the given number of bytes may
	// be read from Body.
	ContentLength [int64](https://pkg.go.dev/builtin#int64)

So DumpResponse() work well under this condition after response body is uncompressed.

However, when the Content-Length is set like first example, if we just uncompressed the response body without adjusting the Content-Length, error would be returned from DumpResponse.

Just to give more context info here.

Thanks!

[cfc4n]

LGTM, thanks.

In fact, when a larger string is called with SSL_Write, it will be split into multiple fragments and SSL_Write will be called multiple times. Moreover, there are concurrent scenarios in the upper-level business logic.
Therefore, there is an issue of disorder in the order of HOOK results for SSL_Write. The current solution cannot guarantee a perfect resolution.

Therefore, a better solution is to use the -m pcap mode.

[cfc4n]

merged

[ruitianzhong]

I agree with you. There exists a semantic gap between eCapture and the application which means that eCapture might not fully understand the application-specific behavior (like parsing different interleaving protocol or concurrent scenario). When that really happen, some errors occur which is expected though.