Skip to content

Conversation

gareth-ellis
Copy link
Member

Initial support for proxies.

The aim is to rely on existing support from requests or aiohttp.

Next steps:

  • more thorough testing
  • documentation

@gareth-ellis gareth-ellis requested review from pquentin and a team March 31, 2025 16:26
Comment on lines +133 to +134
# If we have proxy env vars, we need to use the requests transport
# to ensure that we use the correct proxy settings.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered using requests inconditionally here? Without any proxy tests, requests support could be subtly broken.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered it - my aim here though was to try and retain as much current behaviour when not using proxies as possible. I was aiming to get in a set of proxy tests to cover the proxy use case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen with any of these variables set to empty value? Example:

http_proxy=

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requests would be used, but requests likely would not use the proxy. Requests actually looks at any env var ending with _proxy - we can check if there is a value if you feel that is necessary, though i don't want to start doing any other validation on the input, since that will automatically be up to the client that we are using (requests for synchronus, aiohttp for async).

Copy link
Contributor

@fressi-elastic fressi-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit test(s) for this fix so we can test it without having to deal with actually making requests to a server. Also please validate what would happen with any of these variables set as an empty value.

Comment on lines +133 to +134
# If we have proxy env vars, we need to use the requests transport
# to ensure that we use the correct proxy settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen with any of these variables set to empty value? Example:

http_proxy=

# If we have proxy env vars, we need to use the requests transport
# to ensure that we use the correct proxy settings.
self.logging.warning("Proxy settings detected. Using requests transport.")
super().__init__(*args, node_class=RequestsHttpNode, **kwargs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If kwargs already contains "node_class" key, then this would raise an exception. I think you should add node_class as an expecific parameter at the beginning of this method so that in case is None then we can replace its value. Example:

def __init__(self, node_class = None, **kwargs):
   
   # ...
  if node_class is None and any(x in os.environ for x in proxy_env_vars):
      node_class = RequestsHttpNode
  super().__init__(*args, node_class=node_class, **kwargs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants