Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1064398	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1023/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1023
1064038	dotnet/sdk	Checkout dotnet/sdk@refs/pull/49316/merge to s	Log	dotnet/sdk#49316
1064083	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1022/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1022
1063966	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48964/merge to s	Log	dotnet/sdk#48964
1063967	dotnet/sdk	Clone dotnet/sdk	Log	dotnet/sdk#48964
1063496	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/4942/merge to s	Log	dotnet/arcade-services#4942
1063216	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115577/merge to s	Log	dotnet/runtime#115577
1062997	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116450/merge to s	Log	dotnet/runtime#116450
1063085	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1021/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1021
2726631	dotnet-sdk	Checkout dotnet-sdk@refs/pull/50639/merge to s	Log	#50639
1061433	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1019/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1019
1061253	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1017/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1017
1061009	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116377/merge to s	Log	dotnet/runtime#116377
1060716	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1015/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1015
1059230	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1012/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1012
1058330	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1011/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1011
1057532	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116242/merge to s	Log	dotnet/runtime#116242
1057018	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1009/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1009
1054257	dotnet/runtime	Checkout dotnet/runtime@refs/pull/113790/merge to s	Log	dotnet/runtime#113790
2719241	dotnet-performance	Checkout dotnet-performance@main to s/performance	Log
1053490	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/4894/merge to s	Log	dotnet/arcade-services#4894
1053276	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1007/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1007
1052602	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1006/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1006
1051118	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
2716122	dotnet-dotnet-monitor	Checkout dotnet-dotnet-monitor@main to s	Log
1050259	dotnet/runtime	Checkout dotnet/runtime@refs/pull/113576/merge to s	Log	dotnet/runtime#113576
1049515	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115935/merge to s	Log	dotnet/runtime#115935
1047975	dotnet/machinelearning	Checkout dotnet/machinelearning@refs/pull/7463/merge to s	Log	dotnet/machinelearning#7463
1047974	dotnet/machinelearning	Checkout dotnet/machinelearning@refs/pull/7463/merge to s	Log	dotnet/machinelearning#7463
1047579	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115545/merge to s	Log	dotnet/runtime#115545
1047055	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62040/merge to s	Log	dotnet/aspnetcore#62040
1047056	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62040/merge to s	Log	dotnet/aspnetcore#62040
1046940	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62032/merge to s	Log	dotnet/aspnetcore#62032
1046941	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62032/merge to s	Log	dotnet/aspnetcore#62032
1046627	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115791/merge to s	Log	dotnet/runtime#115791
1045738	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62000/merge to s	Log	dotnet/aspnetcore#62000
1045739	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/62000/merge to s	Log	dotnet/aspnetcore#62000
1045352	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115730/merge to s	Log	dotnet/runtime#115730
1045351	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115730/merge to s	Log	dotnet/runtime#115730
1044174	dotnet/runtime	Checkout dotnet/runtime@refs/pull/115545/merge to s	Log	dotnet/runtime#115545
1044014	dotnet/sdk	Checkout dotnet/sdk@refs/pull/49024/merge to s	Log	dotnet/sdk#49024
1044015	dotnet/sdk	Clone dotnet/sdk	Log	dotnet/sdk#49024
1043401	dotnet/sdk	Checkout dotnet/sdk@refs/pull/49006/merge to s	Log	dotnet/sdk#49006
1042896	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1042015	dotnet/runtime	Checkout dotnet/runtime@refs/pull/114867/merge to s	Log	dotnet/runtime#114867
1041617	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48959/merge to s	Log	dotnet/sdk#48959
1041565	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48954/merge to s	Log	dotnet/sdk#48954
1041566	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48955/merge to s	Log	dotnet/sdk#48955
1040784	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/61922/merge to s	Log	dotnet/aspnetcore#61922
1040024	dotnet/runtime	Checkout dotnet/runtime@refs/pull/114881/merge to s	Log	dotnet/runtime#114881
1039879	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/61889/merge to s	Log	dotnet/aspnetcore#61889
1039880	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/61889/merge to s	Log	dotnet/aspnetcore#61889
1039878	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/61889/merge to s	Log	dotnet/aspnetcore#61889
1039353	dotnet/sdk	Checkout dotnet/sdk@refs/pull/48887/merge to s	Log	dotnet/sdk#48887

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	15	54

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?