Add trace option for better API call debugging

[justinpolygon]

This commit introduces the --trace-api-calls flag, which enables users to see the full URL, query parameters, and headers for every API request made by the library. This functionality is particularly useful when working with additional parameters like .lt, .lte, .gt, and .gte.

With this flag, users can easily debug the API calls, compare them to the web UI or make manual calls to the API, and uncover any potential mistakes in the usage of the library.

Example script:

from polygon import RESTClient

client = RESTClient()

options_chain = []
for o in client.list_snapshot_options_chain(
    "HCP",
    params={
        "expiration_date.gte": "2024-03-16",
        "strike_price.gte": 20,
    }):
    options_chain.append(o)

print(options_chain)

Example useage:

python3 example.py --trace-api-calls

Output with the flag:

Full URL: https://api.polygon.io/v3/snapshot/options/HCP?expiration_date.gte=2024-03-16&strike_price.gte=20
Headers: {'Authorization': 'Bearer XXXXXX', 'Accept-Encoding': 'gzip', 'User-Agent': 'Polygon.io PythonClient/1.8.5'}
Full URL: https://api.polygon.io/v3/snapshot/options/HCP?cursor=YXA9TyUzQUhDUDI1MDExN1AwMDAyMjUwMCUzQTIyLjUwJmFzPSZleHBpcmF0aW9uX2RhdGUuZ3RlPTIwMjUtMDEtMTcmbGltaXQ9MTAmc29ydD10aWNrZXImc3RyaWtlX3ByaWNlLmd0ZT0yMC4wMDAwMDA
Headers: {'Authorization': 'Bearer XXXXXX', 'Accept-Encoding': 'gzip', 'User-Agent': 'Polygon.io PythonClient/1.8.5'}
<normal response here>

This feature empowers both client library developers and end users to better understand what's happening under the hood and solve problems more efficiently.

[justinpolygon]

Was thinking about this a bit over the weekend and we could probably do the same thing by adding something to client = RESTClient(). Maybe client = RESTClient(trace=true) or something like that. I thought it was nice to just turn this on and off via a command flag without a code change but I'm open to whatever you folks suggest.

[mmoghaddam385]

Was thinking about this a bit over the weekend and we could probably do the same thing by adding something to client = RESTClient(). Maybe client = RESTClient(trace=true) or something like that. I thought it was nice to just turn this on and off via a command flag without a code change but I'm open to whatever you folks suggest.

A client library exposing a command line flag feels a bit odd, is this something you've seen in other Python libraries before? I would feel better with something like you're suggesting here where it's a property you pass when creating a RESTClient. A property in the client is more flexible and doesn't make any assumptions about how users are using this library. The question then becomes should we have a more generalized log_level attribute for RESTClient? If log_level=trace then the client would log API calls like this. I can't think off the top of my head what other logging would be useful in the library though so maybe a more pointed trace=true attribute will do just as well.

Either way, I don't think we should be logging API keys, even if it's opt-in. We should redact the Authorization header in the logs like you did in your example in the PR description (I assume that's something you did by hand before posting the PR?).

[justinpolygon]

Hey, your feedback here makes sense. I really like the idea of using trace=true and sanitizing the API key. Yeah, I just did it by hand before but I'll add some logic there and move the cmd flag to use the trace=true idea. Changes incoming.

[justinpolygon]

Hey @mmoghaddam385, updated this PR to remove the --trace-api-calls feature flag and introduces a new trace option in the RESTClient constructor. Users can now enable tracing by passing trace=True when initializing the RESTClient instance.

Furthermore, this updated PR adds functionality to redact the API key in the Authorization header when printing request and response headers to the console for debugging purposes. Also, the response headers are now included in the trace output, providing additional insight into the API communication, along with the X-Request-Id which is sometimes needed on our end for debugging. Thanks for the suggestions, as these updates improve the overall debugging experience for users.

Example script:

from polygon import RESTClient

client = RESTClient(trace=True)  # POLYGON_API_KEY environment variable is used

options_chain = []
for o in client.list_snapshot_options_chain(
    "HCP",
    params={
        "expiration_date.gte": "2024-03-16",
        "strike_price.gte": 20,
    },
):
    options_chain.append(o)
print(options_chain)

Output (not modified):

Request URL: https://api.polygon.io/v3/snapshot/options/HCP?expiration_date.gte=2024-03-16&strike_price.gte=20
Request Headers: {'Authorization': 'Bearer REDACTED', 'Accept-Encoding': 'gzip', 'User-Agent': 'Polygon.io PythonClient/1.8.5'}
Response Headers: {'Server': 'nginx/1.19.2', 'Date': 'Fri, 05 May 2023 17:00:32 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Link': '<https://api.polygon.io/v3/snapshot/options/HCP?cursor=YXA9TyUzQUhDUDI1MDExN1AwMDAyMjUwMCUzQTIyLjUwJmFzPSZleHBpcmF0aW9uX2RhdGUuZ3RlPTIwMjUtMDEtMTcmbGltaXQ9MTAmc29ydD10aWNrZXImc3RyaWtlX3ByaWNlLmd0ZT0yMC4wMDAwMDA>; rel="next"', 'Vary': 'Accept-Encoding', 'X-Item-Count': '10', 'X-Request-Id': '09a949b3933ca9b2db2e6dd423e72bb8', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains'}
Request URL: https://api.polygon.io/v3/snapshot/options/HCP?cursor=YXA9TyUzQUhDUDI1MDExN1AwMDAyMjUwMCUzQTIyLjUwJmFzPSZleHBpcmF0aW9uX2RhdGUuZ3RlPTIwMjUtMDEtMTcmbGltaXQ9MTAmc29ydD10aWNrZXImc3RyaWtlX3ByaWNlLmd0ZT0yMC4wMDAwMDA
Request Headers: {'Authorization': 'Bearer REDACTED', 'Accept-Encoding': 'gzip', 'User-Agent': 'Polygon.io PythonClient/1.8.5'}
Response Headers: {'Server': 'nginx/1.19.2', 'Date': 'Fri, 05 May 2023 17:00:32 GMT', 'Content-Type': 'application/json', 'Content-Length': '1343', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Item-Count': '6', 'X-Request-Id': '1eaac7e01e1fdd8d89f8bdacbc075349', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains'}
<normal response down here>