Repository navigation
fix(downloader): bound git network hangs and retry transient 504s - #2463
Erik Osterman (Cloud Posse) (osterman) wants to merge 6 commits into
Conversation
…hook log
- Prepend `-c http.lowSpeedLimit=1000 -c http.lowSpeedTime=30` to every
network-touching git invocation (clone/fetch/pull/ls-remote/submodule
update) so stalled HTTP(S) transfers surface within ~30s instead of
consuming the full 5-minute context budget x 3 commands = ~15 min hang.
- Always set GIT_SSH_COMMAND with ConnectTimeout=30, ServerAliveInterval=15,
ServerAliveCountMax=4 -- even without an sshKeyFile -- to bound SSH
transports the same way.
- Add "returned error: 5", "could not resolve host", and "early eof" to
isRetryableGitError. The previous "gateway timeout" pattern did not
match git's real wire format (fatal: ... The requested URL returned
error: 504), so retries never fired on the reported incident.
- Apply a conservative default RetryConfig (3 attempts, exponential
2s->30s with jitter) in pkg/provisioner/source/vendor.go when
sourceSpec.Retry is nil, so auto-provisioned sources retry on
transient 5xx/DNS failures instead of failing the whole apply.
- Demote log.Info("Running hooks", "event", event) to log.Debug in
cmd/terraform/utils.go -- matches the sibling debug log in
pkg/hooks/hooks.go and removes per-invocation noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Tip Atmos Pro
No affected stacks workflow was detected for this pull request. |
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
📝 WalkthroughWalkthroughHardens Git transport and retries: adds HTTP low-speed git settings and shared command construction, enforces SSH timeouts for all git commands, broadens retryable-git error detection, provides a default exponential-backoff retry for vendor/source provisioning, and lowers a hook log to debug. ChangesGit network resilience and SSH timeout enhancements
Default source provisioning retry configuration
Logging adjustment
Sequence Diagram(s)sequenceDiagram
participant Caller
participant gitCommandContext
participant GitServer
Caller->>gitCommandContext: build git cmd with -c http.lowSpeedLimit/time and setupGitEnv (GIT_SSH_COMMAND)
gitCommandContext->>GitServer: execute network git subcommand (ls-remote/clone/fetch/pull/submodule update)
GitServer-->>gitCommandContext: output or transient error
gitCommandContext-->>Caller: return output/exit status (retry decision uses isRetryableGitError)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsStopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
pkg/provisioner/source/vendor.go (1)
34-52: 💤 Low valueTiny nit: locals are only needed for addressability.
The intermediate locals exist solely so you can take addresses of literal values. That's fine and idiomatic, but if you want to trim a few lines you could inline via a small helper like
ptr[T any](v T) *T { return &v }. Totally optional — current form is perfectly readable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/provisioner/source/vendor.go` around lines 34 - 52, The local variables in defaultProvisionRetryConfig (maxAttempts, initialDelay, maxDelay, multiplier, jitter) are only created to take addresses of literals; replace them by using a small generic helper ptr[T any](v T) *T and call ptr(defaultProvisionRetryMaxAttempts), ptr(defaultProvisionRetryInitialDelay), etc., when constructing the schema.RetryConfig in defaultProvisionRetryConfig so you can remove the intermediate locals and keep the same addressable values.pkg/provisioner/source/vendor_test.go (1)
705-719: ⚡ Quick winConsider also asserting the defaults flow through
VendorSource.The test pins the field values nicely, but there's no coverage confirming that
VendorSourceactually falls back todefaultProvisionRetryConfig()whensourceSpec.Retryis nil (and uses the user-provided one when set). A small test with an injectable/fake downloader, or asserting the option list passed toNewGoGetterDownloader, would lock down the wiring you just added in lines 117-121 ofvendor.go. Happy to sketch one if useful.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/provisioner/source/vendor_test.go` around lines 705 - 719, Add a unit test that verifies VendorSource uses defaultProvisionRetryConfig() when sourceSpec.Retry is nil and uses the provided Retry when set by creating a VendorSource (or calling the factory function that constructs it) with an injectable/fake downloader or by spying the options passed to NewGoGetterDownloader; assert that the retry-related option values passed into NewGoGetterDownloader (or observed on the fake downloader) equal the defaults from defaultProvisionRetryConfig() in the nil case and equal the user-provided values in the non-nil case, referencing VendorSource, defaultProvisionRetryConfig, NewGoGetterDownloader and sourceSpec.Retry to locate the wiring to test.pkg/downloader/get_git_network_test.go (1)
24-33: ⚡ Quick winAssert arg order directly for the prepend contract.
Current contains-checks won’t fail if
-cflags move after the subcommand. Since this test’s contract is ordering, assert positional args.Suggested patch.
- joined := strings.Join(cmd.Args, " ") - assert.Contains(t, joined, "-c http.lowSpeedLimit="+gitHTTPLowSpeedLimit) - assert.Contains(t, joined, "-c http.lowSpeedTime="+gitHTTPLowSpeedTime) - - // The actual subcommand must still be present after the -c flags. - assert.Contains(t, joined, "clone") - assert.Contains(t, joined, "https://example.com/repo.git") + assert.GreaterOrEqual(t, len(cmd.Args), 8) + assert.Equal(t, "-c", cmd.Args[1]) + assert.Equal(t, "http.lowSpeedLimit="+gitHTTPLowSpeedLimit, cmd.Args[2]) + assert.Equal(t, "-c", cmd.Args[3]) + assert.Equal(t, "http.lowSpeedTime="+gitHTTPLowSpeedTime, cmd.Args[4]) + assert.Equal(t, "clone", cmd.Args[5]) + assert.Equal(t, "https://example.com/repo.git", cmd.Args[6])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/downloader/get_git_network_test.go` around lines 24 - 33, The test currently only checks substrings in joined (strings.Join(cmd.Args, " ")) which won't guarantee ordering; update the assertions to check positional ordering on cmd.Args directly: verify that the two -c flags ("-c http.lowSpeedLimit="+gitHTTPLowSpeedLimit and "-c http.lowSpeedTime="+gitHTTPLowSpeedTime) appear in cmd.Args before the subcommand "clone" and the repo URL "https://example.com/repo.git". Use index/position checks on cmd.Args (not joined) so the prepend contract for the network-tuning flags is enforced.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/downloader/get_git_network_test.go`:
- Around line 41-47: The test currently captures the first GIT_SSH_COMMAND it
sees by breaking out of the loop, which can return a stale value when multiple
entries exist; update the lookup logic for cmd.Env so it iterates the whole
slice and assigns sshEnv to the last matching "GIT_SSH_COMMAND=" entry (i.e.,
remove the break and keep overwriting sshEnv), and/or make the test
deterministic by sanitizing cmd.Env before running (remove duplicate
GIT_SSH_COMMAND entries or explicitly set the expected single value); apply the
same fix to the other identical loop that inspects cmd.Env around the other
assertion.
---
Nitpick comments:
In `@pkg/downloader/get_git_network_test.go`:
- Around line 24-33: The test currently only checks substrings in joined
(strings.Join(cmd.Args, " ")) which won't guarantee ordering; update the
assertions to check positional ordering on cmd.Args directly: verify that the
two -c flags ("-c http.lowSpeedLimit="+gitHTTPLowSpeedLimit and "-c
http.lowSpeedTime="+gitHTTPLowSpeedTime) appear in cmd.Args before the
subcommand "clone" and the repo URL "https://example.com/repo.git". Use
index/position checks on cmd.Args (not joined) so the prepend contract for the
network-tuning flags is enforced.
In `@pkg/provisioner/source/vendor_test.go`:
- Around line 705-719: Add a unit test that verifies VendorSource uses
defaultProvisionRetryConfig() when sourceSpec.Retry is nil and uses the provided
Retry when set by creating a VendorSource (or calling the factory function that
constructs it) with an injectable/fake downloader or by spying the options
passed to NewGoGetterDownloader; assert that the retry-related option values
passed into NewGoGetterDownloader (or observed on the fake downloader) equal the
defaults from defaultProvisionRetryConfig() in the nil case and equal the
user-provided values in the non-nil case, referencing VendorSource,
defaultProvisionRetryConfig, NewGoGetterDownloader and sourceSpec.Retry to
locate the wiring to test.
In `@pkg/provisioner/source/vendor.go`:
- Around line 34-52: The local variables in defaultProvisionRetryConfig
(maxAttempts, initialDelay, maxDelay, multiplier, jitter) are only created to
take addresses of literals; replace them by using a small generic helper ptr[T
any](v T) *T and call ptr(defaultProvisionRetryMaxAttempts),
ptr(defaultProvisionRetryInitialDelay), etc., when constructing the
schema.RetryConfig in defaultProvisionRetryConfig so you can remove the
intermediate locals and keep the same addressable values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 72b7837e-e5c6-4771-9481-fa7444a78228
📒 Files selected for processing (7)
cmd/terraform/utils.gopkg/downloader/get_git.gopkg/downloader/get_git_network_test.gopkg/downloader/get_git_retry_test.gopkg/downloader/get_git_test.gopkg/provisioner/source/vendor.gopkg/provisioner/source/vendor_test.go
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2463 +/- ##
==========================================
+ Coverage 78.54% 78.59% +0.05%
==========================================
Files 1144 1144
Lines 109984 110037 +53
==========================================
+ Hits 86387 86485 +98
+ Misses 18807 18760 -47
- Partials 4790 4792 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
setupGitEnv only filters non-empty inherited GIT_SSH_COMMAND values, so when the parent env has GIT_SSH_COMMAND="" the resulting cmd.Env carries both the empty entry and the new augmented one. A first-match loop reads the wrong value. - Pull the lookup into a tiny helper that walks the whole slice and returns the last GIT_SSH_COMMAND= entry (last-wins is how effective env actually evaluates in practice). - Scrub GIT_SSH_COMMAND via t.Setenv at the top of both network tests so the baseline is deterministic regardless of CI shell config. Addresses CodeRabbit review feedback on PR #2463. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI's gofumpt is stricter than the locally-installed version and required: - log.Debug call's first argument moved to its own line in cmd/terraform/utils.go. - append() and fmt.Errorf() multi-line arg lists with a leading break and a trailing comma + closing paren on their own lines in pkg/downloader/get_git.go. No behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
| gitHTTPLowSpeedLimit = "1000" | ||
| gitHTTPLowSpeedTime = "30" |
There was a problem hiding this comment.
Should we maybe support envars to control overrides for this?
| sshConnectTimeoutSeconds = "30" | ||
| sshServerAliveInterval = "15" | ||
| sshServerAliveCountMax = "4" |
There was a problem hiding this comment.
Same here, env var overrides
Defaults stay the same (1 KB/s for 30 s on HTTP, 30 s connect / 15 s keepalive on SSH) but each knob is now overridable per-environment: - ATMOS_GIT_HTTP_LOW_SPEED_LIMIT (bytes/sec) - ATMOS_GIT_HTTP_LOW_SPEED_TIME (seconds) - ATMOS_GIT_SSH_CONNECT_TIMEOUT (seconds) - ATMOS_GIT_SSH_SERVER_ALIVE_INTERVAL (seconds) - ATMOS_GIT_SSH_SERVER_ALIVE_COUNT_MAX (count) A shared numericEnvOr() helper validates the override is a non-negative integer; anything else (typos like "60s", negatives) logs a warning and falls back to the default so a misconfigured env var can never silently disable the timeout. Unit tests cover the override path, the validator's rejection of bad input, and the empty-string fallthrough. These are runtime-tuning env vars, not Atmos config — read via os.Getenv with //nolint:forbidigo matching the precedent in pkg/config/git_root.go and pkg/config/utils.go (ATMOS_VERSION_ENFORCEMENT). Addresses review feedback on PR #2463. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/downloader/get_git_network_test.go (1)
21-33:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAssert argument order directly in the prepend test.
This currently proves presence, not ordering. The test can pass even if
-cflags drift afterclone, which weakens the prepend contract.Suggested patch.
func TestGitCommandContext_PrependsNetworkArgs(t *testing.T) { cmd := gitCommandContext(t.Context(), "clone", "https://example.com/repo.git", "/tmp/dst") - // cmd.Args[0] is the resolved git binary path; subsequent args must lead - // with the network-tuning -c flags before the subcommand. - joined := strings.Join(cmd.Args, " ") - assert.Contains(t, joined, "-c http.lowSpeedLimit="+defaultGitHTTPLowSpeedLimit) - assert.Contains(t, joined, "-c http.lowSpeedTime="+defaultGitHTTPLowSpeedTime) - - // The actual subcommand must still be present after the -c flags. - assert.Contains(t, joined, "clone") - assert.Contains(t, joined, "https://example.com/repo.git") + // cmd.Args[0] is the resolved git binary path; subsequent args must lead + // with the network-tuning -c flags before the subcommand. + assert.Equal(t, "-c", cmd.Args[1]) + assert.Equal(t, "http.lowSpeedLimit="+defaultGitHTTPLowSpeedLimit, cmd.Args[2]) + assert.Equal(t, "-c", cmd.Args[3]) + assert.Equal(t, "http.lowSpeedTime="+defaultGitHTTPLowSpeedTime, cmd.Args[4]) + assert.Equal(t, "clone", cmd.Args[5]) + assert.Equal(t, "https://example.com/repo.git", cmd.Args[6]) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/downloader/get_git_network_test.go` around lines 21 - 33, The test TestGitCommandContext_PrependsNetworkArgs currently only checks presence of the -c flags; change it to assert ordering by inspecting cmd.Args (from gitCommandContext) and ensuring the indices of "-c http.lowSpeedLimit="+defaultGitHTTPLowSpeedLimit and "-c http.lowSpeedTime="+defaultGitHTTPLowSpeedTime are both less than the index of the subcommand "clone"; use index lookup on cmd.Args (or iterate to find positions) and assert pos(flag) < pos("clone") for each flag so the test verifies the flags are prepended, not just present.
🧹 Nitpick comments (1)
pkg/downloader/get_git_network_test.go (1)
77-97: ⚡ Quick winUse a table-driven test for numeric override scenarios.
These four tests cover one function with scenario variations; folding into one table-driven test reduces repetition and makes future cases easier to add.
As per coding guidelines: "Use table-driven tests for testing multiple scenarios in Go."Suggested refactor.
-func TestNumericEnvOr_AcceptsValidOverride(t *testing.T) { - t.Setenv(envGitHTTPLowSpeedTime, "60") - assert.Equal(t, "60", gitHTTPLowSpeedTime(), "valid override should win over default") -} - -func TestNumericEnvOr_RejectsNonNumericOverride(t *testing.T) { - // A typo like "60s" must not silently disable the timeout (git would error - // or interpret it as 0). Fall back to the safe default. - t.Setenv(envGitHTTPLowSpeedTime, "60s") - assert.Equal(t, defaultGitHTTPLowSpeedTime, gitHTTPLowSpeedTime()) -} - -func TestNumericEnvOr_RejectsNegativeOverride(t *testing.T) { - t.Setenv(envGitHTTPLowSpeedTime, "-1") - assert.Equal(t, defaultGitHTTPLowSpeedTime, gitHTTPLowSpeedTime()) -} - -func TestNumericEnvOr_EmptyValueFallsBackToDefault(t *testing.T) { - t.Setenv(envGitHTTPLowSpeedTime, "") - assert.Equal(t, defaultGitHTTPLowSpeedTime, gitHTTPLowSpeedTime()) -} +func TestNumericEnvOr_GitHTTPLowSpeedTime(t *testing.T) { + tests := []struct { + name string + value string + want string + }{ + {name: "accepts valid override", value: "60", want: "60"}, + {name: "rejects non-numeric override", value: "60s", want: defaultGitHTTPLowSpeedTime}, + {name: "rejects negative override", value: "-1", want: defaultGitHTTPLowSpeedTime}, + {name: "empty value falls back to default", value: "", want: defaultGitHTTPLowSpeedTime}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Setenv(envGitHTTPLowSpeedTime, tc.value) + assert.Equal(t, tc.want, gitHTTPLowSpeedTime()) + }) + } +}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/downloader/get_git_network_test.go` around lines 77 - 97, Combine the four similar tests into a single table-driven test (e.g., TestNumericEnvOr) that iterates over cases with fields like name, envValue, and expected; for each case call t.Run(case.name, func(t *testing.T) { t.Setenv(envGitHTTPLowSpeedTime, case.envValue); assert.Equal(t, case.expected, gitHTTPLowSpeedTime()) }). Use the existing symbols envGitHTTPLowSpeedTime, defaultGitHTTPLowSpeedTime and the gitHTTPLowSpeedTime() helper to build the cases for "60" -> "60", "60s" -> defaultGitHTTPLowSpeedTime, "-1" -> defaultGitHTTPLowSpeedTime, and "" -> defaultGitHTTPLowSpeedTime. Ensure each case uses t.Run so failures are reported per scenario.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@pkg/downloader/get_git_network_test.go`:
- Around line 21-33: The test TestGitCommandContext_PrependsNetworkArgs
currently only checks presence of the -c flags; change it to assert ordering by
inspecting cmd.Args (from gitCommandContext) and ensuring the indices of "-c
http.lowSpeedLimit="+defaultGitHTTPLowSpeedLimit and "-c
http.lowSpeedTime="+defaultGitHTTPLowSpeedTime are both less than the index of
the subcommand "clone"; use index lookup on cmd.Args (or iterate to find
positions) and assert pos(flag) < pos("clone") for each flag so the test
verifies the flags are prepended, not just present.
---
Nitpick comments:
In `@pkg/downloader/get_git_network_test.go`:
- Around line 77-97: Combine the four similar tests into a single table-driven
test (e.g., TestNumericEnvOr) that iterates over cases with fields like name,
envValue, and expected; for each case call t.Run(case.name, func(t *testing.T) {
t.Setenv(envGitHTTPLowSpeedTime, case.envValue); assert.Equal(t, case.expected,
gitHTTPLowSpeedTime()) }). Use the existing symbols envGitHTTPLowSpeedTime,
defaultGitHTTPLowSpeedTime and the gitHTTPLowSpeedTime() helper to build the
cases for "60" -> "60", "60s" -> defaultGitHTTPLowSpeedTime, "-1" ->
defaultGitHTTPLowSpeedTime, and "" -> defaultGitHTTPLowSpeedTime. Ensure each
case uses t.Run so failures are reported per scenario.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 93b017c6-9c8c-46b1-af2f-af99b3f10e74
📒 Files selected for processing (3)
pkg/downloader/get_git.gopkg/downloader/get_git_network_test.gopkg/downloader/get_git_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- pkg/downloader/get_git.go
- pkg/downloader/get_git_test.go
|
💥 This pull request now has conflicts. Could you fix it Erik Osterman (Cloud Posse) (@osterman)? 🙏 |
what
-c http.lowSpeedLimit=1000 -c http.lowSpeedTime=30to every network-touching git invocation inpkg/downloader/get_git.go(clone/fetch/pull/ls-remote/submodule update) so stalled HTTP(S) transfers surface within ~30 s instead of consuming the full context budget.GIT_SSH_COMMANDwithConnectTimeout=30,ServerAliveInterval=15,ServerAliveCountMax=4— even without ansshKeyFile— to bound SSH transports the same way."returned error: 5","could not resolve host", and"early eof"toisRetryableGitError. The previous"gateway timeout"pattern did not match git's real wire format (fatal: ... The requested URL returned error: 504), so retries never fired.RetryConfig(3 attempts, exponential 2 s → 30 s with jitter) inpkg/provisioner/source/vendor.gowhensourceSpec.Retryisnil, so auto-provisioned sources retry on transient 5xx/DNS failures instead of failing the whole apply.log.Info("Running hooks", "event", event)tolog.Debugincmd/terraform/utils.goto match the sibling debug log inpkg/hooks/hooks.goand remove per-invocation noise.why
before.terraform.applyand then failed with a GitLab504from go-getter. The same atmos invocation succeeded a second later, confirming the failure was transient.references
Summary by CodeRabbit
Bug Fixes
Tests