Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
1599832	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6041/merge to s	Log	dotnet/diagnostics#6041
1599997	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6042/merge to s	Log	dotnet/diagnostics#6042
1599665	dotnet/sdk	Checkout dotnet/sdk@refs/pull/56050/merge to s	Log	dotnet/sdk#56050
1599117	dotnet/maui	Checkout dotnet/maui@refs/pull/37319/merge to s	Log	dotnet/maui#37319
1599112	dotnet/maui	Checkout dotnet/maui@refs/pull/38550/merge to s	Log	dotnet/maui#38550
1598962	dotnet/runtime	Checkout dotnet/runtime@refs/pull/134037/merge to s	Log	dotnet/runtime#134037
1583451	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68787/merge to s	Log	dotnet/aspnetcore#68787
1597264	dotnet/sdk	Checkout dotnet/sdk@refs/pull/56296/merge to s	Log	dotnet/sdk#56296
1597222	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1729/merge to s	Log	dotnet/xharness#1729
1597223	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133447/merge to s	Log	dotnet/runtime#133447
1596699	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6706/merge to s	Log	dotnet/arcade-services#6706
1596631	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133918/merge to s	Log	dotnet/runtime#133918
1594555	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68714/merge to s	Log	dotnet/aspnetcore#68714
1594556	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68714/merge to s	Log	dotnet/aspnetcore#68714
1594558	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68714/merge to s	Log	dotnet/aspnetcore#68714
1593999	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1593551	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133741/merge to s	Log	dotnet/runtime#133741
1593401	dotnet/performance	Checkout dotnet/performance@refs/pull/5308/merge to s	Log	dotnet/performance#5308
1592721	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133606/merge to s	Log	dotnet/runtime#133606
1592315	dotnet/sdk	Checkout dotnet/sdk@refs/pull/56241/merge to s	Log	dotnet/sdk#56241
1592143	dotnet/runtime	Checkout dotnet/runtime@release/9.0 to s	Log
1591878	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133535/merge to s	Log	dotnet/runtime#133535
1591529	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/69205/merge to s	Log	dotnet/aspnetcore#69205
1591382	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6022/merge to s	Log	dotnet/diagnostics#6022
1591021	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2865/merge to s	Log	maestro-auth-test/build-result-analysis-test#2865
1591022	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2865/merge to s	Log	maestro-auth-test/build-result-analysis-test#2865
1591008	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2864/merge to s	Log	maestro-auth-test/build-result-analysis-test#2864
1591009	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2864/merge to s	Log	maestro-auth-test/build-result-analysis-test#2864
1590235	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133471/merge to s	Log	dotnet/runtime#133471
1589991	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17413/merge to s	Log	#17413
1589898	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133458/merge to s	Log	dotnet/runtime#133458
1589425	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6015/merge to s	Log	dotnet/diagnostics#6015
1589440	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6017/merge to s	Log	dotnet/diagnostics#6017
1589524	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133514/merge to s	Log	dotnet/runtime#133514
1587312	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/69118/merge to s	Log	dotnet/aspnetcore#69118
1586433	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133378/merge to s	Log	dotnet/runtime#133378
1585959	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/6000/merge to s	Log	dotnet/diagnostics#6000
1585890	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55717/merge to s	Log	dotnet/sdk#55717
1584323	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133300/merge to s	Log	dotnet/runtime#133300
1584228	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133291/merge to s	Log	dotnet/runtime#133291
1583937	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5999/merge to s	Log	dotnet/diagnostics#5999
1583291	dotnet/sdk	Checkout dotnet/sdk@release/8.0.1xx to s	Log
1581968	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5980/merge to s	Log	dotnet/diagnostics#5980
1581806	dotnet/sdk	Checkout dotnet/sdk@refs/pull/56124/merge to s	Log	dotnet/sdk#56124
1581935	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/69038/merge to s	Log	dotnet/aspnetcore#69038
1581475	dotnet/runtime	Checkout dotnet/runtime@release/9.0 to s	Log
1580914	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133134/merge to s	Log	dotnet/runtime#133134
1581066	dotnet/sdk	Checkout dotnet/sdk@refs/pull/56122/merge to s	Log	dotnet/sdk#56122
1579779	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5998/merge to s	Log	dotnet/diagnostics#5998
1577852	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17472/merge to s	Log	#17472
1577738	dotnet/maui	Checkout dotnet/maui@refs/pull/38082/merge to s	Log
1577389	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1577359	dotnet/runtime	Checkout dotnet/runtime@refs/pull/133036/merge to s	Log	dotnet/runtime#133036
1576703	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131789/merge to s	Log	dotnet/runtime#131789
1575733	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6662/merge to s	Log	dotnet/arcade-services#6662
1575682	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132958/merge to s	Log	dotnet/runtime#132958
1575231	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132965/merge to s	Log	dotnet/runtime#132965
1572494	dotnet/maui	Checkout dotnet/maui@main to s	Log
1572383	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132885/merge to s	Log	dotnet/runtime#132885
1572321	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132884/merge to s	Log	dotnet/runtime#132884
1571918	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68086/merge to s	Log	dotnet/aspnetcore#68086
1571631	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55977/merge to s	Log	dotnet/sdk#55977
1571630	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55976/merge to s	Log	dotnet/sdk#55976
1571629	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55975/merge to s	Log	dotnet/sdk#55975
1571140	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132847/merge to s	Log	dotnet/runtime#132847
1570971	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132665/merge to s	Log	dotnet/runtime#132665
1570690	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132739/merge to s	Log	dotnet/runtime#132739
1570570	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1569528	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132632/merge to s	Log	dotnet/runtime#132632
1569068	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68829/merge to s	Log	dotnet/aspnetcore#68829
1569103	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132795/merge to s	Log	dotnet/runtime#132795
1568612	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55940/merge to s	Log	dotnet/sdk#55940
1568485	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132633/merge to s	Log	dotnet/runtime#132633
1568526	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55520/merge to s	Log	dotnet/sdk#55520
1568432	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132769/merge to s	Log	dotnet/runtime#132769
1568519	dotnet/maui	Checkout dotnet/maui@refs/pull/37802/merge to s	Log	dotnet/maui#37802
1567862	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132737/merge to s	Log	dotnet/runtime#132737
1567394	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68784/merge to s	Log	dotnet/aspnetcore#68784
1567253	dotnet/maui	Checkout dotnet/maui@release/11.0.1xx-rc1 to s	Log
1566335	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1565575	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68754/merge to s	Log	dotnet/aspnetcore#68754
1565264	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1562783	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132633/merge to s	Log	dotnet/runtime#132633
1562675	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1562372	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132438/merge to s	Log	dotnet/runtime#132438
1561613	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1561525	dotnet/performance	Checkout dotnet/performance@refs/pull/5287/merge to s	Log	dotnet/performance#5287
1561578	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132593/merge to s	Log	dotnet/runtime#132593
1561116	dotnet/maui	Checkout dotnet/maui@refs/pull/37688/merge to s	Log	dotnet/maui#37688
1560580	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132556/merge to s		dotnet/runtime#132556
1560388	dotnet/maui	Checkout dotnet/maui@refs/pull/37642/merge to s	Log	dotnet/maui#37642
1559803	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132529/merge to s	Log	dotnet/runtime#132529
1558545	dotnet/runtime	Checkout dotnet/runtime@release/11.0-rc1 to s	Log
1557990	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132432/merge to s	Log	dotnet/runtime#132432
1557800	dotnet/runtime	Checkout dotnet/runtime@refs/pull/132424/merge to s	Log	dotnet/runtime#132424
1557752	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55841/merge to s	Log	dotnet/sdk#55841
1557745	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55837/merge to s	Log	dotnet/sdk#55837
1557283	dotnet/maui	Checkout dotnet/maui@main to s	Log
1551774	dotnet/maui	Checkout dotnet/maui@refs/pull/37111/merge to s	Log	dotnet/maui#37111

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	29	99

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?

[premun]

GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
We saw problems cloning all over. Maybe it's related to that?

[dougbu]

This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔