Description
What happened?
Error occurs in Initialize Job when attempting to delete a file in the s directory. We did not have any issues with v3.251.0.
After running into the issue, we reverted back to v3.251.0 and the error went away. This occurred on Linux 22-04 x86/64 and ARM64 systems.
Logs attached (some names were changed for the sake of public sharing, but all of the information is as it occurred in the logs).
Bullseye is referenced as it includes both the ARM and AMD CPU types. It failed on all distributions we compiled for (Ubuntu and Debian)
Setup:
Base Linux 22.04 Build agent
Bullseye container for both x86/64 and ARM64
Build process Debian make as the build step to compile C/++ code and bundle it into a .deb
Steps:
- Run a build consisting of Stage > Job > (Container) > Steps
- Rerun the same build on the same agent
Result:
Build failed during initialize Job step
Expected result:
Successfully start build
See the following for more information:
build_agent_success_log.txt
Direct_system_log.txt
build_agent_error_log.txt
Versions
Azure Devops Version 4.248.0 Ubuntu 22.04
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
No response
Version controll system
No response
Relevant log output
Starting: Initialize job
Agent name: 'agent-a64'
Agent machine name: 'agent-a64'
Current agent version: '4.248.0'
Agent running as: 'root'
##[debug]Triggering repository: Internal_Library. repository type: Git
Prepare build directory.
##[debug]Delete existing build directory: '/home/BuildAgent/_work/2'
##[debug]Deleting build directory: '/home/BuildAgent/_work/2'
##[error]One or more errors occurred. (Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.)
##[debug]System.AggregateException: One or more errors occurred. (Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.)
---> System.IO.FileNotFoundException: Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.
File name: '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
at Interop.CheckIo(Int64 result, String path, Boolean isDirError)
at System.IO.FileStatus.SetAttributes(SafeFileHandle handle, String path, FileAttributes attributes, Boolean asDirectory)
at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.SetAttributesWithDiagnostics(FileSystemInfo item, FileAttributes newAttributes) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 538
at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.RemoveReadOnly(FileSystemInfo item) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 565
at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.<>c__DisplayClass12_0.<DeleteDirectory>b__0(FileSystemInfo item) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 150
at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
at System.Linq.Parallel.SpoolingTaskBase.Work()
at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of inner exception stack trace ---
at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
at System.Linq.Parallel.SpoolingTask.SpoolForAll[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, TaskScheduler taskScheduler)
at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
at System.Linq.Parallel.MergeExecutor`1.Execute()
at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
at System.Linq.Parallel.ForAllOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings)
at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream)
at System.Linq.Parallel.ScanQueryOperator`1.ScanEnumerableQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
at System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings)
at System.Linq.Parallel.ForAllOperator`1.RunSynchronously()
at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.DeleteDirectory(String path, Boolean contentsOnly, Boolean continueOnContentDeleteError, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 141
at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.DeleteDirectory(String path, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 98
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.DeleteDirectory(IExecutionContext executionContext, String description, String path) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 268
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.CreateDirectory(IExecutionContext executionContext, String description, String path, Boolean deleteExisting) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 250
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.PrepareDirectory(IExecutionContext executionContext, IList`1 repositories, WorkspaceOptions workspace) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 100
at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.InitializeJobExtension(IExecutionContext executionContext, IList`1 steps, WorkspaceOptions workspace) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildJobExtension.cs:line 163
at Microsoft.VisualStudio.Services.Agent.Worker.JobExtension.InitializeJob(IExecutionContext jobContext, AgentJobRequestMessage message) in /mnt/vss/_work/1/s/src/Agent.Worker/JobExtension.cs:line 123
Finishing: Initialize job
Activity