Skip to content

Conversation

@abhinavsingh
Copy link
Owner

@abhinavsingh abhinavsingh commented Jan 13, 2022

@abhinavsingh abhinavsingh added the bot:chronographer:skip PR using this label is exempted from CHANGELOG management label Jan 13, 2022
@abhinavsingh
Copy link
Owner Author

Following works

Gen cert

$ make ca-certificates

Run proxy

$ python -m proxy --port 9000 --plugins proxy.plugin.CacheResponsesPlugin --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key ca-signing-key.pem --enable-web-server --threadless
2022-01-13 16:34:21,954 - pid:23607 [I] plugins.load:85 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2022-01-13 16:34:21,955 - pid:23607 [I] plugins.load:85 - Loaded plugin proxy.http.server.HttpWebServerPlugin
2022-01-13 16:34:21,955 - pid:23607 [I] plugins.load:85 - Loaded plugin proxy.plugin.CacheResponsesPlugin
2022-01-13 16:34:21,955 - pid:23607 [I] listener.setup:97 - Listening on 127.0.0.1:9000
2022-01-13 16:34:22,011 - pid:23607 [I] pool.setup:106 - Started 16 acceptors in threadless (local) mode

2022-01-13 16:34:26,906 - pid:23614 [I] server.access_log:400 - 127.0.0.1:60346 - CONNECT httpbin.org:443 - 486 bytes - 1216.18ms
2022-01-13 16:34:26,906 - pid:23614 [I] disk.close:60 - Cached response at /var/folders/ry/tkb2qyj13qv1mdwf6z7r13l00000gn/T/httpbin.org-5-0-29.txt

Curl

$ curl -v -x localhost:9000 --cacert ca-cert.pem https://httpbin.org/get             ─╯
*   Trying ::1:9000...
* connect to ::1 port 9000 failed: Connection refused
*   Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to httpbin.org:443
> CONNECT httpbin.org:443 HTTP/1.1
> Host: httpbin.org:443
> User-Agent: curl/7.77.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: ca-cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=httpbin.org
*  start date: Jan 13 11:03:03 2022 GMT
*  expire date: Jan 13 11:03:03 2024 GMT
*  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
*  issuer: CN=example.com
*  SSL certificate verify ok.
> GET /get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 13 Jan 2022 11:04:26 GMT
< Content-Type: application/json
< Content-Length: 256
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.77.0", 
    "X-Amzn-Trace-Id": "Root=1-61e0073a-6fdb4457650af7fd2ac13448"
  }, 
  "origin": "106.202.255.39", 
  "url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact

@codecov
Copy link

codecov bot commented Jan 13, 2022

Codecov Report

Merging #979 (1989085) into develop (dd2476f) will decrease coverage by 0.18%.
The diff coverage is 38.46%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #979      +/-   ##
===========================================
- Coverage    85.90%   85.72%   -0.19%     
===========================================
  Files          145      145              
  Lines         6252     6257       +5     
  Branches       636      639       +3     
===========================================
- Hits          5371     5364       -7     
- Misses         758      769      +11     
- Partials       123      124       +1     
Flag Coverage Δ
pytest 85.58% <38.46%> (-0.19%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
proxy/core/work/remote.py 96.42% <ø> (ø)
proxy/core/work/pool.py 88.52% <33.33%> (-3.15%) ⬇️
proxy/core/work/threaded.py 75.00% <33.33%> (-11.67%) ⬇️
proxy/core/work/threadless.py 79.16% <42.85%> (-1.59%) ⬇️
proxy/core/base/tcp_server.py 84.48% <0.00%> (-1.73%) ⬇️
proxy/http/handler.py 72.84% <0.00%> (-1.30%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dd2476f...1989085. Read the comment docs.

@abhinavsingh abhinavsingh changed the title [TlsInterception] v2.4.x vs v2.3.1 differences fixes [TlsInterception] v2.4.x vs v2.3.1 differences Jan 13, 2022
@abhinavsingh abhinavsingh merged commit 41ba504 into develop Jan 13, 2022
@abhinavsingh abhinavsingh deleted the tls-ca-cert branch January 13, 2022 13:25
abhinavsingh added a commit that referenced this pull request Jan 14, 2022
* Remove menubar (#930)

Remove `menubar`

* Use `128 KB` as default value for `DEFAULT_BUFFER_SIZE` (#926)

* Add `TlsInterception` acceptance test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Generate CA cert if not available

* Fix lint and tests

* Fix args

* Remove acceptance tests for now

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* pip prod(deps): bump wheel from 0.37.0 to 0.37.1 (#934)

Bumps [wheel](https://github.com/pypa/wheel) from 0.37.0 to 0.37.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](pypa/wheel@0.37.0...0.37.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* npm: bump jasmine from 3.10.0 to 4.0.0 in /dashboard (#933)

Bumps [jasmine](https://github.com/jasmine/jasmine-npm) from 3.10.0 to 4.0.0.
- [Release notes](https://github.com/jasmine/jasmine-npm/releases)
- [Commits](jasmine/jasmine-npm@v3.10.0...v4.0.0)

---
updated-dependencies:
- dependency-name: jasmine
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* npm: bump ws from 8.3.0 to 8.4.0 in /dashboard (#936)

Bumps [ws](https://github.com/websockets/ws) from 8.3.0 to 8.4.0.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](websockets/ws@8.3.0...8.4.0)

---
updated-dependencies:
- dependency-name: ws
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* pip prod(deps): bump types-paramiko from 2.8.4 to 2.8.6 (#937)

Bumps [types-paramiko](https://github.com/python/typeshed) from 2.8.4 to 2.8.6.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-paramiko
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* `DescriptorsHandlerMixin` and `Descriptors`, `SelectableEvents` types (#938)

* Add `Descriptors` type

* Add a `DescriptorsHandlerMixin` class used throughout the http framework

* Remove dependency upon `HasFileno` ie `typing_extension` too

* Define `SelectableEvents` type

* Fix doc

* Blank line

* Remove dep on `typing-extensions`

* Discover base plugin class

* await on now async handlers

* Fix doc spell

* Add `--port-file` flag (#942)

* Add `--port-file` flag

* Use `--port-file` flag for integration tests using `get_available_port`

* Use temp dir

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix `base_klass` variable related lint issues

* Fix main tests

* Fix integration

* Use timeout when terminating proc

* Skip integration on win instead of xmark

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Expose `UpstreamConnectionPool` to web & proxy plugins (#946)

* Expose conn pool to plugins

* Fix reusable state handling

* Separate `release` and `retain` methods

* Fix conn pool tests

* Fix tests

* npm: bump chrome-devtools-frontend in /dashboard (#949)

Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.952865 to 1.0.956881.
- [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases)
- [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md)
- [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits)

---
updated-dependencies:
- dependency-name: chrome-devtools-frontend
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* pip prod(deps): bump sphinxcontrib-towncrier from 0.2.0a0 to 0.2.1a0 (#941)

Bumps [sphinxcontrib-towncrier](https://github.com/sphinx-contrib/sphinxcontrib-towncrier) from 0.2.0a0 to 0.2.1a0.
- [Release notes](https://github.com/sphinx-contrib/sphinxcontrib-towncrier/releases)
- [Commits](sphinx-contrib/sphinxcontrib-towncrier@v0.2.0a0...v0.2.1a0)

---
updated-dependencies:
- dependency-name: sphinxcontrib-towncrier
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* `isort` everything except lib (for now) (#952)

* isort the tests folder

* Carry over changes from #672

* Disable pre-commit

* Revert flake8 config change

* isort examples too

* pip prod(deps): bump httpx from 0.20.0 to 0.21.3 (#951)

Bumps [httpx](https://github.com/encode/httpx) from 0.20.0 to 0.21.3.
- [Release notes](https://github.com/encode/httpx/releases)
- [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md)
- [Commits](encode/httpx@0.20.0...0.21.3)

---
updated-dependencies:
- dependency-name: httpx
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* Decouple transport framework from dashboard plugin (#953)

* Decouple transport framework from dashboard plugin

* Move `InspectTrafficPlugin` within `http.inspector` module

* Avoid exporting plugins within `__init__.py` files

* Use `/transport/` prefix to avoid #945 conflict issue

* Add todo

* Build containers in parallel and pre-check (#954)

* Build containers in parallel and pre-check

* Fix loop in workflow

* Test container needs build

* Invoke `WebSocketTransportBasePlugin` connected and disconnected callbacks (#956)

* [HttpProtocolHandler] Handle invalid request parsing exceptions (#957)

* Handle invalid request parsing exception when raised, log the bytes for later inspection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* npm: bump jasmine-ts from 0.3.0 to 0.4.0 in /dashboard (#958)

Bumps [jasmine-ts](https://github.com/svi3c/jasmine-ts) from 0.3.0 to 0.4.0.
- [Release notes](https://github.com/svi3c/jasmine-ts/releases)
- [Changelog](https://github.com/svi3c/jasmine-ts/blob/master/CHANGELOG.md)
- [Commits](https://github.com/svi3c/jasmine-ts/commits/v0.4.0)

---
updated-dependencies:
- dependency-name: jasmine-ts
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Ignore `utf-8` decoding errors within event core (#961)

* [WebServer] Refactor routing to allow same path for websocket and web requests (#962)

* Switch to WS

* Refactor

* Ignore utf-8 decode error during logging (#966)

* On-demand `TlsInterception` capability, driven by plugins (#965)

* Add `TlsInterceptionPropertyMixin`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add `do_intercept` hook

* call super init

* No super from mixin as it is followed by abc?

* type ignore

* spell

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [WebServer] Fix routing (#968)

* Fix web server routing

* Fix mypy

* pip prod(deps): bump furo from 2021.11.23 to 2022.1.2 (#959)

Bumps [furo](https://github.com/pradyunsg/furo) from 2021.11.23 to 2022.1.2.
- [Release notes](https://github.com/pradyunsg/furo/releases)
- [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md)
- [Commits](pradyunsg/furo@2021.11.23...2022.01.02)

---
updated-dependencies:
- dependency-name: furo
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* pip prod(deps): bump paramiko from 2.9.1 to 2.9.2 (#970)

Bumps [paramiko](https://github.com/paramiko/paramiko) from 2.9.1 to 2.9.2.
- [Release notes](https://github.com/paramiko/paramiko/releases)
- [Changelog](https://github.com/paramiko/paramiko/blob/main/NEWS)
- [Commits](paramiko/paramiko@2.9.1...2.9.2)

---
updated-dependencies:
- dependency-name: paramiko
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* Test submodule and refactor (#971)

* Refactor tests into submodules

* isort tests

* Add malicious request headers test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Handle `OSError` on shutdown & `TimeoutError` on recv (#974)

* Expect `TimeoutError` during `recv`

* Expect `OSError` during socket shutdown, can happen if other end has already closed the socket

* pip prod(deps): bump mypy from 0.920 to 0.931 (#976)

Bumps [mypy](https://github.com/python/mypy) from 0.920 to 0.931.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](python/mypy@v0.920...v0.931)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* npm: bump @types/jquery from 3.5.4 to 3.5.13 in /dashboard (#975)

Bumps [@types/jquery](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jquery) from 3.5.4 to 3.5.13.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jquery)

---
updated-dependencies:
- dependency-name: "@types/jquery"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>

* Refactor into separate `Work` module (#977)

* work module

* Fix imports

* String based typing for multiprocessing.synchronize

* Fix `test_accepts_client_from_server_socket`

* Move staticmethod outside of threadless pool class

* Fix doc build

* Fix test mock

* mp grouped together

* pylint happy

* import only for type checking

* doc build

* wrong import order

* Type fixes (#979)

* Type fixes

* Type fix

* Py class role

* unused any import

* [TlsInterception] GHA integration tests (#981)

* Add TLS interception integration tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixture to gen certificate once for the `test_integration` module

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use https endpoints in tls interception tests

* Fix modify post data integration test

* Only start 3 acceptor & 3 workers during integration run

* disable chunk response

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* npm: bump typescript from 3.9.7 to 4.5.4 in /dashboard (#982)

Bumps [typescript](https://github.com/Microsoft/TypeScript) from 3.9.7 to 4.5.4.
- [Release notes](https://github.com/Microsoft/TypeScript/releases)
- [Commits](microsoft/TypeScript@v3.9.7...v4.5.4)

---
updated-dependencies:
- dependency-name: typescript
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* pip prod(deps): bump types-paramiko from 2.8.6 to 2.8.9 (#983)

Bumps [types-paramiko](https://github.com/python/typeshed) from 2.8.6 to 2.8.9.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-paramiko
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Pass separate `--ca-cert-dir` flag for parallel TLS interception tests (#984)

* Pass separate `--ca-cert-dir` flag for parallel TLS interception tests

* Temp disable `test_modify_post_response_integration`

* mock ca cert dir

* Is threaded an issue with TLS interception?

* Disable modify chunk response for python < 3.10

* Disable modify chunk response for python < 3.10

* [TlsInterception] Fix broken `ChunkedResponsePlugin` for `Python < 3.10` (#986)

* Add TLS interception integration tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI

* unused

* Well 3.9 just worked locally

* Dispatching empty byte to client results in OSError for Python < 3.10

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Uncomment old integration tests

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add `# pragma: no cover` for unnecessary code (#987)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:skip PR using this label is exempted from CHANGELOG management

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TlsInterception] OSError when sending empty chunk to clients under Python < 3.10

2 participants