Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1207513	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121743/merge to s		dotnet/runtime#121743
1206903	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206915	dotnet/sdk	Checkout dotnet/sdk@release/8.0.4xx to s	Log
1206822	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120589/merge to s	Log	dotnet/runtime#120589
1206922	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16307/merge to s	Log	#16307
1206918	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1206917	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1206919	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119432/merge to s	Log	dotnet/runtime#119432
1203938	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51659/merge to s	Log	dotnet/sdk#51659
1206885	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206908	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206906	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1206901	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206902	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121646/merge to s	Log	dotnet/runtime#121646
1206859	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64426/merge to s	Log	dotnet/aspnetcore#64426
1206892	dotnet/performance	Checkout dotnet/performance@refs/pull/5035/merge to s/performance	Log	dotnet/performance#5035
1206889	dotnet/aspire	Checkout dotnet/aspire@refs/pull/13036/merge to s	Log	dotnet/aspire#13036
1206801	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51419/merge to s	Log	dotnet/sdk#51419
1206601	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5518/merge to s	Log	dotnet/arcade-services#5518
1206531	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5517/merge to s	Log	dotnet/arcade-services#5517
1206513	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5513/merge to s	Log	dotnet/arcade-services#5513
1205901	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64271/merge to s	Log	dotnet/aspnetcore#64271
1203492	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64369/merge to s	Log	dotnet/aspnetcore#64369
1203494	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64369/merge to s	Log	dotnet/aspnetcore#64369
1203422	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16310/merge to s	Log	#16310
1202792	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121603/merge to s	Log	dotnet/runtime#121603
1202736	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121334/merge to s	Log	dotnet/runtime#121334
1202427	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64350/merge to s	Log	dotnet/aspnetcore#64350
1200180	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64313/merge to s	Log	dotnet/aspnetcore#64313
1200181	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64313/merge to s	Log	dotnet/aspnetcore#64313
1197636	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121396/merge to s	Log	dotnet/runtime#121396
1197105	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121446/merge to s		dotnet/runtime#121446
1196011	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121278/merge to s	Log	dotnet/runtime#121278
1196008	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120598/merge to s	Log	dotnet/runtime#120598
2831733	dotnet-dotnet-monitor	Checkout dotnet-dotnet-monitor@refs/pull/54935/merge to s	Log	#54935
2831406	dotnet-dotnet-monitor	Checkout dotnet-dotnet-monitor@refs/pull/54883/merge to s	Log	#54883
2831393	dotnet-dotnet-monitor	Checkout dotnet-dotnet-monitor@refs/pull/54882/merge to s	Log	#54882
1193364	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51549/merge to s	Log	dotnet/sdk#51549
1193359	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51548/merge to s	Log	dotnet/sdk#51548
1193365	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51550/merge to s	Log	dotnet/sdk#51550
1193321	dotnet/sdk	Clone dotnet/sdk	Log	dotnet/sdk#51547
1193343	dotnet/installer	Clone dotnet/installer	Log	dotnet/installer#20690
1193344	dotnet/installer	Checkout dotnet/installer@refs/pull/20690/merge to s	Log	dotnet/installer#20690
1193345	dotnet/installer	Checkout dotnet/installer@refs/pull/20691/merge to s	Log	dotnet/installer#20691
1193346	dotnet/installer	Checkout dotnet/installer@refs/pull/20692/merge to s	Log	dotnet/installer#20692
1193071	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5427/merge to s	Log	dotnet/arcade-services#5427
1192205	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51533/merge to s	Log	dotnet/sdk#51533
1191654	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1191698	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121263/merge to s	Log	dotnet/runtime#121263
1190586	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120895/merge to s	Log	dotnet/runtime#120895
1190220	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120976/merge to s	Log	dotnet/runtime#120976
1190096	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120887/merge to s	Log	dotnet/runtime#120887
1189797	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121094/merge to s	Log	dotnet/runtime#121094
1187846	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51475/merge to s	Log	dotnet/sdk#51475
1187864	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51447/merge to s	Log	dotnet/sdk#51447
1187640	dotnet/installer	Clone dotnet/installer	Log	dotnet/installer#20688
2826348	dotnet-installer	Clone dotnet/installer	Log	#54674
1186932	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121126/merge to s	Log	dotnet/runtime#121126
1186960	dotnet/machinelearning	Checkout dotnet/machinelearning@refs/pull/7534/merge to s	Log	dotnet/machinelearning#7534
1186961	dotnet/machinelearning	Checkout dotnet/machinelearning@refs/pull/7534/merge to s	Log	dotnet/machinelearning#7534
1182065	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120972/merge to s	Log	dotnet/runtime#120972
1182016	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51180/merge to s	Log	dotnet/sdk#51180
1181499	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1086/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1086
2821505	dotnet-installer	Clone dotnet/installer	Log	#54554
1181135	dotnet/runtime	Checkout dotnet/runtime@refs/pull/120330/merge to s	Log	dotnet/runtime#120330

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	28	65

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?