looks like there is another issue related to max response read size

[dogancanbakir]

          looks like there is another issue related to max response read size 

$  proxify -v

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

		projectdiscovery.io

[INF] Current proxify version v0.0.15 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Saving proxify logs to proxify_logs.jsonl
[WRN] responseChain: Error while dumping response: error reading response body: could not read response body: http: request body too large

$ curl -v -x http://127.0.0.1:8888 https://github.githubassets.com/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js

• we can use io.MultiReader along with ResponseChain, so that processing limit of proxify is 4 MB but whether to read all bytes or not depends on client and not proxify

cc: @dogancanbakir

Originally posted by @tarunKoyalwar in #519 (comment)

[dogancanbakir]

I'm unable to reproduce the issue, steps I followed:

$ curl -v -k -x http://127.0.0.1:8888 https://github.githubassets.com/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js
*   Trying 127.0.0.1:8888...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to github.githubassets.com:443
> CONNECT github.githubassets.com:443 HTTP/1.1
> Host: github.githubassets.com:443
> User-Agent: curl/8.1.2
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Content-Length: 0
* Ignoring Content-Length in CONNECT 200 response
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: O=Martian Proxy; CN=github.githubassets.com
*  start date: Mar 14 15:52:59 2024 GMT
*  expire date: Mar 14 17:52:59 2024 GMT
*  issuer: O=Proxify CA; CN=Proxify CA
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* using HTTP/1.1
> GET /asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js HTTP/1.1
> Host: github.githubassets.com
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Connection: close
< Content-Length: 18
< Accept-Ranges: bytes
< Access-Control-Allow-Origin: *
< Content-Type: text/html
< Cross-Origin-Resource-Policy: cross-origin
< Date: Thu, 14 Mar 2024 16:52:59 GMT
< Retry-After: 0
< Server: Varnish
< Strict-Transport-Security: max-age=31536000
< Via: 1.1 varnish
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Fastly-Request-Id: 76fad7d21f4911632224d77cb4f770f2d259e4b3
< X-Served-By: cache-vie6356-VIE
<
* Closing connection 0
<h1>Not Found</h1>%                                                                                                                                                                                                        

go run . -sr

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

                projectdiscovery.io

[INF] Current proxify version v0.0.15 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Saving proxify logs to proxify_logs.jsonl
[INF] Saving proxify logs (raw) to proxify_logs

$ cat proxify_logs.jsonl | jq
{
  "timestamp": "2024-03-14T19:52:59+03:00",
  "url": "https://github.githubassets.com:443",
  "request": {
    "header": {
      "Connection": "close",
      "User-Agent": "curl/8.1.2",
      "host": "github.githubassets.com:443",
      "method": "CONNECT",
      "path": "",
      "scheme": "https"
    },
    "raw": "CONNECT github.githubassets.com:443 HTTP/1.1\r\nHost: github.githubassets.com:443\r\nConnection: close\r\nUser-Agent: curl/8.1.2\r\n\r\n"
  },
  "response": {
    "raw": "HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"
  }
}
{
  "timestamp": "2024-03-14T19:52:59+03:00",
  "url": "https://github.githubassets.com/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js",
  "request": {
    "header": {
      "Accept": "*/*",
      "Connection": "close",
      "User-Agent": "curl/8.1.2",
      "host": "github.githubassets.com",
      "method": "GET",
      "path": "/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js",
      "scheme": "https"
    },
    "raw": "GET /asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js HTTP/1.1\r\nHost: github.githubassets.com\r\nAccept: */*\r\nConnection: close\r\nUser-Agent: curl/8.1.2\r\n\r\n"
  },
  "response": {
    "header": {
      "Accept-Ranges": "bytes",
      "Access-Control-Allow-Origin": "*",
      "Content-Length": "18",
      "Content-Type": "text/html",
      "Cross-Origin-Resource-Policy": "cross-origin",
      "Date": "Thu, 14 Mar 2024 16:52:59 GMT",
      "Retry-After": "0",
      "Server": "Varnish",
      "Strict-Transport-Security": "max-age=31536000",
      "Via": "1.1 varnish",
      "X-Cache": "MISS",
      "X-Cache-Hits": "0",
      "X-Fastly-Request-Id": "76fad7d21f4911632224d77cb4f770f2d259e4b3",
      "X-Served-By": "cache-vie6356-VIE"
    },
    "raw": "HTTP/1.1 404 Not Found\r\nConnection: close\r\nContent-Length: 18\r\nAccept-Ranges: bytes\r\nAccess-Control-Allow-Origin: *\r\nContent-Type: text/html\r\nCross-Origin-Resource-Policy: cross-origin\r\nDate: Thu, 14 Mar 2024 16:52:59 GMT\r\nRetry-After: 0\r\nServer: Varnish\r\nStrict-Transport-Security: max-age=31536000\r\nVia: 1.1 varnish\r\nX-Cache: MISS\r\nX-Cache-Hits: 0\r\nX-Fastly-Request-Id: 76fad7d21f4911632224d77cb4f770f2d259e4b3\r\nX-Served-By: cache-vie6356-VIE\r\n\r\n"
  }
}

@tarunKoyalwar

[dogancanbakir]

same as projectdiscovery/nuclei#4942