Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust curl retry and connection timeout handling #1017

Merged
merged 1 commit into from
Jun 13, 2022

Conversation

edmorley
Copy link
Member

In the shimmed CNBs used in heroku/builder we have been seeing quite a few transient errors related to buildpacks downloading from S3.

Adding appropriate retries and connection timeouts to all of our buildpack's curl usages should help with these, as well as make builds more reliable in general for users on Heroku, plus also anyone using a shimmed CNB locally with Pack CLI (where the network connection may be even less reliable).

The --retry-connrefused option has been used since otherwise curl doesn't retry cases where the connection was refused. Ideally we would use --retry-all-errors which takes that one step further, however that option was only added in curl 7.71, so is only supported by Heroku-22+.

For more on curl options, see:
https://curl.se/docs/manpage.html

GUS-W-11283397.

@edmorley edmorley self-assigned this Jun 13, 2022
In the shimmed CNBs used in `heroku/builder` we have been seeing
quite a few transient errors related to buildpacks downloading from S3.

Adding appropriate retries and connection timeouts to all of our
buildpack's curl usages should help with these, as well as make builds
more reliable in general for users on Heroku, plus also anyone using a
shimmed CNB locally with Pack CLI (where the network connection may
be even less reliable).

The `--retry-connrefused` option has been used since otherwise curl
doesn't retry cases where the connection was refused. Ideally we would
use `--retry-all-errors` which takes that one step further, however that
option was only added in curl 7.71, so is only supported by Heroku-22+.

For more on curl options, see:
https://curl.se/docs/manpage.html

GUS-W-11283397.
@edmorley edmorley force-pushed the edmorley/reliable-curl branch from 147cc51 to 401965b Compare June 13, 2022 19:39
@edmorley edmorley marked this pull request as ready for review June 13, 2022 19:47
@edmorley edmorley requested a review from a team as a code owner June 13, 2022 19:47
@joshwlewis joshwlewis merged commit 7881511 into main Jun 13, 2022
@joshwlewis joshwlewis deleted the edmorley/reliable-curl branch June 13, 2022 20:02
edmorley added a commit that referenced this pull request Jun 15, 2022
@edmorley edmorley mentioned this pull request Jun 15, 2022
edmorley added a commit that referenced this pull request Jun 15, 2022
@kqiang
Copy link

kqiang commented Jun 17, 2022

@edmorley
--retry-connrefused is only available on the newer curl version. My code is impacted by this change because my curl version is 7.35.0. Could you do a curl version check before applying --retry-connrefused. ie.
if curl_version < XXX
do the old way
else
do the new way

@edmorley
Copy link
Member Author

edmorley commented Jun 17, 2022

@kqiang Hi! --retry-connrefused is supported in curl 7.52+, which is therefore supported by Ubuntu 18.04+ and so all current Heroku stack versions. If you are using this buildpack with an older (and thus unsupported) operating system, you will need to pin to an older buildpack version. You are also likely at risk of security issues, if you are using old/unsupported Linux versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants