Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
1538085	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131768/merge to s	Log	dotnet/runtime#131768
1537538	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131750/merge to s	Log	dotnet/runtime#131750
1536977	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55563/merge to s	Log	dotnet/sdk#55563
1536577	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5937/merge to s	Log	dotnet/diagnostics#5937
1534890	dotnet/maui	Checkout dotnet/maui@refs/pull/36130/merge to s	Log	dotnet/maui#36130
1533768	dotnet/maui	Checkout dotnet/maui@darc-release/11.0.1xx-preview7-89a633a6-cbda-4d07-9007-2984c7617fd1 to s	Log
1533646	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1533645	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1533573	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
3035203	dotnet-maui	Checkout dotnet-maui@main to s	Log
1533054	dotnet/maui	Checkout dotnet/maui@refs/pull/35846/merge to s	Log	dotnet/maui#35846
1532867	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131577/merge to s	Log	dotnet/runtime#131577
1532809	dotnet/maui	Checkout dotnet/maui@refs/pull/36657/merge to s	Log	dotnet/maui#36657
1532371	dotnet/maui	Checkout dotnet/maui@inflight/candidate to s	Log
1532391	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131027/merge to s	Log	dotnet/runtime#131027
1532341	dotnet/maui	Checkout dotnet/maui@refs/pull/36876/merge to s	Log	dotnet/maui#36876
1532009	dotnet/maui	Checkout dotnet/maui@refs/pull/36671/merge to s	Log	dotnet/maui#36671
1531897	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131546/merge to s	Log	dotnet/runtime#131546
1531991	dotnet/maui	Checkout dotnet/maui@refs/pull/35491/merge to s	Log	dotnet/maui#35491
1531789	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68093/merge to s	Log	dotnet/aspnetcore#68093
3034461	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/63356/merge to s	Log	#63356
3034449	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/63356/merge to s	Log	#63356
3034447	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/63356/merge to s	Log	#63356
1531515	dotnet/maui	Checkout dotnet/maui@refs/pull/36633/merge to s	Log	dotnet/maui#36633
1531491	dotnet/maui	Checkout dotnet/maui@refs/pull/34519/merge to s	Log	dotnet/maui#34519
1531271	dotnet/maui	Checkout dotnet/maui@refs/pull/36876/merge to s	Log	dotnet/maui#36876
1531076	dotnet/maui	Checkout dotnet/maui@refs/pull/36654/merge to s	Log	dotnet/maui#36654
1530987	dotnet/maui	Checkout dotnet/maui@refs/pull/36654/merge to s	Log	dotnet/maui#36654
1530988	dotnet/maui	Checkout dotnet/maui@refs/pull/36657/merge to s	Log	dotnet/maui#36657
1530985	dotnet/maui	Checkout dotnet/maui@refs/pull/36653/merge to s	Log	dotnet/maui#36653
1530938	dotnet/maui	Checkout dotnet/maui@refs/pull/36657/merge to s	Log	dotnet/maui#36657
1530762	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131294/merge to s	Log	dotnet/runtime#131294
1530536	dotnet/runtime	Checkout dotnet/runtime@refs/pull/130373/merge to s	Log	dotnet/runtime#130373
1530552	dotnet/maui	Checkout dotnet/maui@refs/pull/36653/merge to s	Log	dotnet/maui#36653
1530245	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131489/merge to s	Log	dotnet/runtime#131489
1530464	dotnet/maui	Checkout dotnet/maui@refs/pull/36885/merge to s	Log	dotnet/maui#36885
1530333	dotnet/maui	Checkout dotnet/maui@refs/pull/36892/merge to s	Log	dotnet/maui#36892
3033587	dotnet-maui	Checkout dotnet-maui@inflight/candidate to s	Log
1530019	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131477/merge to s	Log	dotnet/runtime#131477
1529872	dotnet/maui	Checkout dotnet/maui@refs/pull/36882/merge to s	Log	dotnet/maui#36882
1529460	dotnet/maui	Checkout dotnet/maui@refs/pull/36676/merge to s	Log	dotnet/maui#36676
1529462	dotnet/maui	Checkout dotnet/maui@refs/pull/36750/merge to s	Log	dotnet/maui#36750
1529358	dotnet/maui	Checkout dotnet/maui@refs/pull/36869/merge to s	Log	dotnet/maui#36869
1529175	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1529145	dotnet/maui	Checkout dotnet/maui@refs/pull/36854/merge to s	Log	dotnet/maui#36854
1529047	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6524/merge to s	Log	dotnet/arcade-services#6524
1528976	dotnet/maui	Checkout dotnet/maui@darc-release/11.0.1xx-preview7-861953bf-0702-4736-ac3d-bff1ef704d24 to s	Log
3032704	dotnet-maui	Checkout dotnet-maui@net11.0 to s	Log
1528714	dotnet/maui	Checkout dotnet/maui@darc-release/11.0.1xx-preview7-861953bf-0702-4736-ac3d-bff1ef704d24 to s	Log
1528141	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131265/merge to s	Log	dotnet/runtime#131265
1528078	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17197/merge to s	Log	#17197
1527989	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1527967	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68039/merge to s	Log	dotnet/aspnetcore#68039
1527968	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/68039/merge to s	Log	dotnet/aspnetcore#68039
1527839	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131306/merge to s	Log	dotnet/runtime#131306
1527727	dotnet/maui	Checkout dotnet/maui@refs/pull/36603/merge to s	Log	dotnet/maui#36603
3032226	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/63242/merge to s	Log	#63242
1527694	dotnet/maui	Checkout dotnet/maui@inflight/current to s	Log
1527363	dotnet/maui	Checkout dotnet/maui@refs/pull/36658/merge to s	Log	dotnet/maui#36658
1527189	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131392/merge to s	Log	dotnet/runtime#131392
1527055	dotnet/maui	Checkout dotnet/maui@refs/pull/36577/merge to s	Log	dotnet/maui#36577
1526898	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1526757	dotnet/maui	Checkout dotnet/maui@refs/pull/36771/merge to s	Log	dotnet/maui#36771
1526573	dotnet/maui	Checkout dotnet/maui@refs/pull/36657/merge to s	Log	dotnet/maui#36657
1526489	dotnet/maui	Checkout dotnet/maui@refs/pull/36733/merge to s	Log	dotnet/maui#36733
1526009	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17187/merge to s	Log	#17187
3030888	dotnet-sdk	Checkout dotnet-sdk@dev/marcpop/nuget-proxy-network-isolation to s
1525789	dotnet/maui	Checkout dotnet/maui@refs/pull/36774/merge to s	Log	dotnet/maui#36774
1525647	dotnet/maui	Checkout dotnet/maui@refs/pull/36773/merge to s	Log	dotnet/maui#36773
1525401	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131326/merge to s	Log	dotnet/runtime#131326
1525353	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128148/merge to s	Log	dotnet/runtime#128148
1525229	dotnet/maui	Checkout dotnet/maui@refs/pull/36733/merge to s	Log	dotnet/maui#36733
1524845	dotnet/maui	Checkout dotnet/maui@refs/pull/36700/merge to s	Log	dotnet/maui#36700
3030409	dotnet-maui	Checkout dotnet-maui@inflight/current to s	Log
3030259	dotnet-installer	Clone dotnet/installer	Log	#63194
1524446	dotnet/maui	Checkout dotnet/maui@darc-net11.0-ff6e3b4a-44ee-4a3b-94d3-908c4a8e5d91 to s	Log
1524314	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55438/merge to s	Log	dotnet/sdk#55438
1524209	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17182/merge to s	Log	#17182
3029959	dotnet-installer	Clone dotnet/installer	Log	#63181
1523628	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
3029830	dotnet-sdk	Checkout dotnet-sdk@feature/dotnetup-crossbuild-runtime-arch to s	Log
1523488	dotnet/maui	Checkout dotnet/maui@refs/pull/36755/merge to s	Log	dotnet/maui#36755
1523420	dotnet/sdk	Checkout dotnet/sdk@refs/pull/55429/merge to s	Log	dotnet/sdk#55429
1523408	dotnet/maui	Checkout dotnet/maui@refs/pull/36753/merge to s	Log	dotnet/maui#36753
1523184	dotnet/maui	Checkout dotnet/maui@refs/pull/36109/merge to s	Log	dotnet/maui#36109
1522917	dotnet/maui	Checkout dotnet/maui@refs/pull/36682/merge to s	Log
1522788	dotnet/maui	Checkout dotnet/maui@darc-net11.0-ff6e3b4a-44ee-4a3b-94d3-908c4a8e5d91 to s	Log
1522681	dotnet/runtime	Checkout dotnet/runtime@refs/pull/130998/merge to s	Log	dotnet/runtime#130998
1522279	dotnet/diagnostics	Checkout dotnet/diagnostics@refs/pull/5938/merge to s	Log	dotnet/diagnostics#5938
3029142	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@dev/loganbussell/netiso-disable to s	Log
3029134	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@dev/loganbussell/netiso-disable to s	Log
1521814	dotnet/maui	Checkout dotnet/maui@refs/tags/10.0.90 to s	Log
1521891	dotnet/maui	Checkout dotnet/maui@refs/pull/36727/merge to s	Log	dotnet/maui#36727
1521693	dotnet/maui	Checkout dotnet/maui@refs/pull/36698/merge to s	Log	dotnet/maui#36698
1521662	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1521091	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67954/merge to s	Log	dotnet/aspnetcore#67954
1520957	dotnet/maui	Checkout dotnet/maui@darc-net11.0-cdff04bc-18d9-4d19-87ce-a704293a36c4 to s	Log
1520756	dotnet/runtime	Checkout dotnet/runtime@refs/pull/131150/merge to s	Log	dotnet/runtime#131150
1520464	dotnet/maui	Checkout dotnet/maui@refs/pull/36633/merge to s	Log	dotnet/maui#36633
1520236	dotnet/maui	Checkout dotnet/maui@darc-net11.0-cdff04bc-18d9-4d19-87ce-a704293a36c4 to s	Log
Displaying 100 of 208 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	44	208

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?