Test failure System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

[VincentBu]

Run: runtime-libraries-coreclr outerloop 20211124.6

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

Error message:

Assert.Throws() Failure
Expected: typeof(System.IO.DirectoryNotFoundException)
Actual:   typeof(System.IO.IOException): Access to the path '/tmp/Directory_Delete_str_bool_e2szzobu.04s/ReadOnly_FileSystemHelper_104_cc389631/DoesNotExist' is denied.
---- System.IO.IOException : Access to the path '/tmp/Directory_Delete_str_bool_e2szzobu.04s/ReadOnly_FileSystemHelper_104_cc389631/DoesNotExist' is denied.


Stack trace
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.Directory.Delete(String path, Boolean recursive) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 304
   at System.IO.Tests.Directory_Delete_str_bool.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 226
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214
----- Inner Stack Trace -----
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.Directory.Delete(String path, Boolean recursive) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 304
   at System.IO.Tests.Directory_Delete_str_bool.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 226
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime-libraries-coreclr outerloop 20211124.6

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume

Error message:

Assert.Throws() Failure
Expected: typeof(System.IO.DirectoryNotFoundException)
Actual:   typeof(System.IO.IOException): Access to the path '/tmp/Directory_Delete_str_bool_e2szzobu.04s/ReadOnly_FileSystemHelper_104_cc389631/DoesNotExist' is denied.
---- System.IO.IOException : Access to the path '/tmp/Directory_Delete_str_bool_e2szzobu.04s/ReadOnly_FileSystemHelper_104_cc389631/DoesNotExist' is denied.


Stack trace
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.Directory.Delete(String path, Boolean recursive) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 304
   at System.IO.Tests.Directory_Delete_str_bool.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 226
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214
----- Inner Stack Trace -----
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.Directory.Delete(String path, Boolean recursive) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 304
   at System.IO.Tests.Directory_Delete_str_bool.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 226
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214

Author:	VincentBu
Assignees:	-
Labels:	area-System.IO, os-linux, arch-x64
Milestone:	-

[carlossanlop]

Hi @VincentBu , do you know when it started happening? Is it intermittent or is it happening consistently?

[carlossanlop]

We haven't introduced any recent changes in code related to directory deletion: https://github.com/dotnet/runtime/commits/09c1a1f7b0c477890b04912d8dd4f742f80faffc/src/libraries/System.Private.CoreLib/src/System/IO

[VincentBu]

Hi @VincentBu , do you know when it started happening? Is it intermittent or is it happening consistently?

This is the first time. We have checked the previous work. Since this problem was raised, it will appear every day, which may occur continuously

[VincentBu]

Failed again in：runtime-libraries-coreclr outerloop 20211202.6

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume

Error message：

Assert.Throws() Failure
Expected: typeof(System.IO.DirectoryNotFoundException)
Actual:   typeof(System.IO.IOException): Access to the path '/tmp/Directory_Delete_str_siexdwdb.a1t/ReadOnly_FileSystemHelper_104_969b38bf/DoesNotExist' is denied.
---- System.IO.IOException : Access to the path '/tmp/Directory_Delete_str_siexdwdb.a1t/ReadOnly_FileSystemHelper_104_969b38bf/DoesNotExist' is denied.


Stack trace
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 492
   at System.IO.Directory.Delete(String path) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 298
   at System.IO.Tests.Directory_Delete_str.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 18
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214
----- Inner Stack Trace -----
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 492
   at System.IO.Directory.Delete(String path) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Directory.cs:line 298
   at System.IO.Tests.Directory_Delete_str.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 18
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214

[danmoseley]

TestResults
| join kind=inner WorkItems on WorkItemId
| join kind=inner Jobs on JobId
| where Finished >= now(-120d)
| where Result == "Fail"
| where  Method == "Unix_NotFoundDirectory_ReadOnlyVolume"
| project 
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  Pipeline_Configuration = tostring(parse_json(Properties).configuration),
  OS = QueueName,
  Arch = tostring(parse_json(Properties).architecture),
  Test = Type1,
  Result,Finished,Started,
  Duration,
  Method,
  Build = tostring(parse_json(Properties).BuildNumber),
  Message
| sort by Started asc
| take 1

Pipeline	Pipeline_Configuration	OS	Arch	Test	Result	Finished	Started	Duration	Method	Build	Message
runtime-libraries-coreclr outerloop-linux	Release	redhat.7.amd64.open.rt	x64	test/functional/cli/outerloop/	Fail	2021-11-24 12:20:59.6950000	2021-11-24 12:20:28.5180000	0.183	Unix_NotFoundDirectory_ReadOnlyVolume	20211124.2	Assert.Throws() Failure Expected: typeof(System.IO.DirectoryNotFoundException) Actual: typeof(System.IO.IOException): Access to the path '/tmp/Directory_Delete_str_bzvbnoah.pap/ReadOnly_FileSystemHelper_104_850771f9/DoesNotExist' is denied. ---- System.IO.IOException : Access to the path '/tmp/Directory_Delete_str_bzvbnoah.pap/ReadOnly_FileSystemHelper_104_850771f9/DoesNotExist' is denied.

It started no later than 11/24.

C:\git\runtime\src\libraries>git log --since="Nov 22 2021" --until="Nov 25 2021" --abbrev-commit -- *IO**cs
commit a2af8ae7575
Author: Carlos Sanchez <1175054+carlossanlop@users.noreply.github.com>
Date:   Thu Nov 25 16:38:38 2021 -0600

    Re-enable Windows test that verifies DriveInfo.VolumeLabel setter fails on SUBST'd drive (#59850)

    * Re-enable test that verifies DriveInfo.VolumeLabel setter fails on SUBST'd drive

commit a0fdc256404
Author: Tom Deseyn <tom.deseyn@gmail.com>
Date:   Tue Nov 23 15:51:37 2021 +0100

    FileSystem.Unix: Directory.Delete: remove per item syscall. (#59520)

    * FileSystem.Unix: Directory.Delete: remove per item syscall.

    By recursing using FileSystemEnumerable we know the file type and
    can omit the stat calls made by DirectoryInfo.Exists.

    For the top level path, we can omit the call also and handle
    non-directories when rmdir errno is ENOTDIR.

    For the recursive case we can avoid recursion when the top level path rmdir
    succeeds immediately.

    FileSystemEntry is updated so IsSymbolicLink remembers the file is symbolic link
    and does not make a syscall for it.

I suspect #59520 cc @tmds

[danmoseley]

This test has [Trait(XunitConstants.Category, XunitConstants.RequiresElevation)] so it might be that it didn't run during PR validation (?)

[danmoseley]

@carlossanlop example of how to do such a query
https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAI1RTW%2fCMAy9I%2fEfLE5UYtPYvZO2ISQmDRiw7ViF1lBDG1dOIlZpP35p2g0OO3BK7Pdh52WDxq7QuMKafu8bDkwajqSzmLRGgU%2bW48xiaYD1XzHL%2fqO%2b8Daw%2fNkSTjkKwpQ0mRwzeIhB82l4M76%2fy6Iz3g6HOIbBVFExOCPwijbnLEDvmr6SOdspO51NSDC1LHWyQpUtdFF%2fcOFKDNpK%2bOBR6PcAllRhQRohBsvGCun9sFJiMDkY1sOlcIViCU10O8GdX9QS67kqMRpdypNn1jvaO1ENfoVZeslvvRZrr3tz6LDxD61HSfMrzJSnkfVPctLttfF%2f5oWbusJxaLQZjn6jHq2tEuvPBpt0a4SiTTRcnxwV2RXjA2%2fuyi1K1HkYo%2fbYZG1YLGxr6OaBMmnTtuqIMP4B9qaBqFoCAAA%3d&web=0

[VincentBu]

Failed again in: runtime-libraries-coreclr outerloop 20211212.2

Failed test:

net7.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open

- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

net7.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open

- System.IO.Tests.DirectoryInfo_Delete.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.DirectoryInfo_Delete_bool.Unix_NotFoundDirectory_ReadOnlyVolume
- System.IO.Tests.Directory_Delete_str_bool.Unix_NotFoundDirectory_ReadOnlyVolume

Error message:

Assert.Throws() Failure
Expected: typeof(System.IO.DirectoryNotFoundException)
Actual:   typeof(System.IO.IOException): Access to the path '/tmp/DirectoryInfo_Delete_ugygquy5.zi0/ReadOnly_FileSystemHelper_104_ee861342/DoesNotExist' is denied.
---- System.IO.IOException : Access to the path '/tmp/DirectoryInfo_Delete_ugygquy5.zi0/ReadOnly_FileSystemHelper_104_ee861342/DoesNotExist' is denied.


Stack trace
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.DirectoryInfo.Delete() in /_/src/libraries/System.Private.CoreLib/src/System/IO/DirectoryInfo.cs:line 229
   at System.IO.Tests.DirectoryInfo_Delete.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/Delete.cs:line 12
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214
----- Inner Stack Trace -----
   at System.IO.FileSystem.RemoveEmptyDirectory(String fullPath, Boolean topLevel, Boolean throwWhenNotEmpty) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Unix.cs:line 501
   at System.IO.DirectoryInfo.Delete() in /_/src/libraries/System.Private.CoreLib/src/System/IO/DirectoryInfo.cs:line 229
   at System.IO.Tests.DirectoryInfo_Delete.Delete(String path) in /_/src/libraries/System.IO.FileSystem/tests/DirectoryInfo/Delete.cs:line 12
   at System.IO.Tests.Directory_Delete_str.<>c__DisplayClass22_0.<Unix_NotFoundDirectory_ReadOnlyVolume>b__1() in /_/src/libraries/System.IO.FileSystem/tests/Directory/Delete.cs:line 214