[DO NOT MERGE - wait 8.16 release][8.16](backport #5802) proxytest: proxy HTTPS request using MITM

[mergify]

What does this PR do?

It extends proxytest to support proxying HTTPS requests.

• The proxy can now act as a MITM (man-in-the-middle) proxy for HTTPS, generating certificates on the fly for the target host.
• The proxy assumes it should proxy an HTTPS request if it receives a CONNECT requests.

Why is it important?

It'll allow us to create integration tests for mTLS scenarios

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

• None

How to test this PR locally

Run TestRunHTTPSProxy passing the example build tag and follow the instructions to perform a cRUL request using the proxytest:

• run the test: go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
• run the provided curl command

Example:

$ go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
=== RUN   TestRunHTTPSProxy
    proxytest_test.go:451: target server running on https://127.0.0.1:34355
    proxytest.go:100: [proxy-https] time=2024-10-16T11:14:33.982+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:236 msg="running on https://127.0.0.1:40241 -> https://localhost:40241"
    proxytest_test.go:473: proxy running on https://localhost:40241
    proxytest_test.go:479: make request to "https://not-a-server.co" using proxy "https://localhost:40241"
    proxytest_test.go:481: curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
    proxytest_test.go:496: CTRL+C to stop
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:183 msg="STARTING - CONNECT '//not-a-server.co:443' HTTP/1.1 127.0.0.1:40496"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:22 msg="handling CONNECT" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:34 msg="hijacked request" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.092+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:282 msg="original URL: https://not-a-server.co:443/, new URL: https://127.0.0.1:34355/"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.097+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:129 msg="EOF reached, finishing HTTPS handler" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] [8a3b5c0f-50c0-40d1-84af-59412ed8bdc3] DONE 0 - CONNECT //not-a-server.co:443 HTTP/1.1 127.0.0.1:40496
^Csignal: interrupt

$ curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
It works!

Related issues

• Closes Improve test proxy/mock fleet server to proxy HTTPS connections #4903

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #5802 done by [Mergify](https://mergify.com).

[mergify]

This pull request has not been merged yet. Could you please review and merge it @AndersonQ? 🙏

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube