Skip to content

[BUG]: v4.248.0 Unable to perform source delete of build directory #5104

Open
@rfolkker

Description

What happened?

Error occurs in Initialize Job when attempting to delete a file in the s directory. We did not have any issues with v3.251.0.
After running into the issue, we reverted back to v3.251.0 and the error went away. This occurred on Linux 22-04 x86/64 and ARM64 systems.
Logs attached (some names were changed for the sake of public sharing, but all of the information is as it occurred in the logs).
Bullseye is referenced as it includes both the ARM and AMD CPU types. It failed on all distributions we compiled for (Ubuntu and Debian)

Setup:
Base Linux 22.04 Build agent
Bullseye container for both x86/64 and ARM64

Build process Debian make as the build step to compile C/++ code and bundle it into a .deb

Steps:

  1. Run a build consisting of Stage > Job > (Container) > Steps
  2. Rerun the same build on the same agent

Result:
Build failed during initialize Job step
Expected result:
Successfully start build

See the following for more information:
build_agent_success_log.txt
Direct_system_log.txt
build_agent_error_log.txt

Versions

Azure Devops Version 4.248.0 Ubuntu 22.04

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

Starting: Initialize job
Agent name: 'agent-a64'
Agent machine name: 'agent-a64'
Current agent version: '4.248.0'
Agent running as: 'root'
##[debug]Triggering repository: Internal_Library. repository type: Git
Prepare build directory.
##[debug]Delete existing build directory: '/home/BuildAgent/_work/2'
##[debug]Deleting build directory: '/home/BuildAgent/_work/2'
##[error]One or more errors occurred. (Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.)
##[debug]System.AggregateException: One or more errors occurred. (Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.)
 ---> System.IO.FileNotFoundException: Could not find file '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'.
File name: '/home/BuildAgent/_work/2/s/debian/tmp/usr/lib/aarch64-linux-gnu/customlibrary.so.0'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Interop.CheckIo(Int64 result, String path, Boolean isDirError)
   at System.IO.FileStatus.SetAttributes(SafeFileHandle handle, String path, FileAttributes attributes, Boolean asDirectory)
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.SetAttributesWithDiagnostics(FileSystemInfo item, FileAttributes newAttributes) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 538
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.RemoveReadOnly(FileSystemInfo item) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 565
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.<>c__DisplayClass12_0.<DeleteDirectory>b__0(FileSystemInfo item) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 150
   at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
   at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   --- End of inner exception stack trace ---
   at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
   at System.Linq.Parallel.SpoolingTask.SpoolForAll[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, TaskScheduler taskScheduler)
   at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
   at System.Linq.Parallel.MergeExecutor`1.Execute()
   at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
   at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
   at System.Linq.Parallel.ForAllOperator`1.WrapPartitionedStream[TKey](PartitionedStream`2 inputStream, IPartitionedStreamRecipient`1 recipient, Boolean preferStriping, QuerySettings settings)
   at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.ChildResultsRecipient.Receive[TKey](PartitionedStream`2 inputStream)
   at System.Linq.Parallel.ScanQueryOperator`1.ScanEnumerableQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
   at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
   at System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings)
   at System.Linq.Parallel.ForAllOperator`1.RunSynchronously()
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.DeleteDirectory(String path, Boolean contentsOnly, Boolean continueOnContentDeleteError, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 141
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.DeleteDirectory(String path, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 98
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.DeleteDirectory(IExecutionContext executionContext, String description, String path) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 268
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.CreateDirectory(IExecutionContext executionContext, String description, String path, Boolean deleteExisting) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 250
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildDirectoryManager.PrepareDirectory(IExecutionContext executionContext, IList`1 repositories, WorkspaceOptions workspace) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildDirectoryManager.cs:line 100
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.BuildJobExtension.InitializeJobExtension(IExecutionContext executionContext, IList`1 steps, WorkspaceOptions workspace) in /mnt/vss/_work/1/s/src/Agent.Worker/Build/BuildJobExtension.cs:line 163
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtension.InitializeJob(IExecutionContext jobContext, AgentJobRequestMessage message) in /mnt/vss/_work/1/s/src/Agent.Worker/JobExtension.cs:line 123
Finishing: Initialize job

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions