Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry #25

Merged
merged 13 commits into from
Jan 29, 2024
Merged

Retry #25

merged 13 commits into from
Jan 29, 2024

Conversation

zmoon
Copy link
Owner

@zmoon zmoon commented Jan 27, 2024

  • retry decorator with backoff until max time (60 s) reached, a simplified (and currently inflexible) version of what backoff does, applied to the readers and cases of explicit requests usage
  • add 10-s timeouts for explicit requests usage
  • add timeout to the pytest step in CI

Resolves #21

applied to the readers

catching connection exceptions that I saw in the GH logs
(don't seem to happen locally in my testing on Win...)
Copy link

codecov bot commented Jan 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (edd5466) 97.44% compared to head (2158e60) 98.69%.

❗ Current head 2158e60 differs from pull request most recent head c8ced8e. Consider uploading reports for the commit c8ced8e to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
+ Coverage   97.44%   98.69%   +1.25%     
==========================================
  Files           3        4       +1     
  Lines         430      461      +31     
==========================================
+ Hits          419      455      +36     
+ Misses         11        6       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@zmoon
Copy link
Owner Author

zmoon commented Jan 28, 2024

Maybe just the requests.get usages that were sometimes failing, will have to check the logs more closely. I did see one case of urllib error though, will have to check if that came from read_csv or not.

Update: yes there was one case from get-data daily where read_csv had connection error (don't see the warning message for retry though, which is kind of weird). But the others were from requests.get.

would need some more complex testing with mocking
Trying CI multiple times, got `requests.exceptions.ReadTimeout`
for URL /pub/data/uscrn/products/daily01/2001/
(in `get_year_urls` with `get_data`)
if many URLs are retried, could take long

for now, for some runs this is < 1 min,
for some, as long as 3 min
internal utilities
uscrn/data.py Outdated Show resolved Hide resolved
the above one applies to this too
@zmoon zmoon marked this pull request as ready for review January 29, 2024 17:20
@zmoon zmoon merged commit db37121 into main Jan 29, 2024
5 checks passed
@zmoon zmoon deleted the retry branch January 29, 2024 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Robustness
1 participant