[CI]: Windows failure on pull: `UtilityVM: Incorrect function.: unknown`

[apostasie]

Description

On occasions, windows nerdctl pull will fail with UtilityVM: Incorrect function.: unknown

Windows people, do you have a clue what that means / why it happens?

@sondavidb @jsturtevant @TinaMor

Steps to reproduce the issue

Seen on CI.

Describe the results you received and expected

=== NAME  TestImageInspectSimpleCases
    command.go:112: assertion failed: expect.ExitCode is not result.ExitCode: Expected exit code: 0
        
        Command:  C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=nerdctl-test pull --quiet gcr.io/k8s-staging-e2e-test-images/busybox:1.29-2
        ExitCode: 1
        Error:    exit status 1
        Stdout:   
        Stderr:   time="2024-10-08T23:21:46Z" level=fatal msg="failed to extract layer sha256:f2032cdb21dfb9f3d6fdd3797ea992ec05415e5a6a57f9eb36b6646496db8d09: hcsshim::ProcessUtilityVMImage \\\\?\\C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\2\\UtilityVM: Incorrect function.: unknown"
        
        Env:
        ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE=C:\actionarchivecache\
        ALLUSERSPROFILE=C:\ProgramData
        ANDROID_HOME=C:\Android\android-sdk
        ANDROID_NDK=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_HOME=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_LATEST_HOME=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_ROOT=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_SDK_ROOT=C:\Android\android-sdk
        ANT_HOME=C:\ProgramData\chocolatey\lib\ant\tools\apache-ant-1.10.14
        APPDATA=C:\Users\runneradmin\AppData\Roaming
        AZURE_CONFIG_DIR=C:\azureCli
        AZURE_DEVOPS_CACHE_DIR=C:\azureDevOpsCli\cache
        AZURE_EXTENSION_DIR=C:\Program Files\Common Files\AzureCliExtensionDirectory
        AZ_DEVOPS_GLOBAL_CONFIG_DIR=C:\azureDevOpsCli
        CABAL_DIR=C:\cabal
        CI=true
        COBERTURA_HOME=C:\cobertura-2.1.1
        COMMONPROGRAMFILES=C:\Program Files\Common Files
        COMPUTERNAME=fv-az1435-31
        COMSPEC=C:\Windows\system32\cmd.exe
        CONDA=C:\Miniconda
        CONTAINERD_VERSION=2.0.0-rc.5
        ChocolateyInstall=C:\ProgramData\chocolatey
        ChromeWebDriver=C:\SeleniumWebDrivers\ChromeDriver
        CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
        CommonProgramW6432=C:\Program Files\Common Files
        DEPLOYMENT_BASEPATH=C:\actions
        DOTNET_MULTILEVEL_LOOKUP=0
        DOTNET_NOLOGO=1
        DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
        DriverData=C:\Windows\System32\Drivers\DriverData
        EXEPATH=C:\Program Files\Git\bin
        EdgeWebDriver=C:\SeleniumWebDrivers\EdgeDriver
        GCM_INTERACTIVE=Never
        GHCUP_INSTALL_BASE_PREFIX=C:\
        GHCUP_MSYS2=C:\msys64
        GITHUB_ACTION=__run_5
        GITHUB_ACTIONS=true
        GITHUB_ACTION_REF=
        GITHUB_ACTION_REPOSITORY=
        GITHUB_ACTOR=apostasie
        GITHUB_ACTOR_ID=142371135
        GITHUB_API_URL=https://api.github.com
        GITHUB_BASE_REF=main
        GITHUB_ENV=D:\a\_temp\_runner_file_commands\set_env_8d607278-332f-49c5-8e49-2f94bd071026
        GITHUB_EVENT_NAME=pull_request
        STATS_V3PS=true
        STATS_VMD=true
        STATS_VMFE=true
        SYSTEMDRIVE=C:
        SYSTEMROOT=C:\Windows
        TEMP=C:\Users\RUNNER~1\AppData\Local\Temp
        TERM=xterm-256color
        TMP=C:\Users\RUNNER~1\AppData\Local\Temp
        UBUNTU_VERSION=24.04
        USERDOMAIN=fv-az1435-31
        USERDOMAIN_ROAMINGPROFILE=fv-az1435-31
        USERNAME=runneradmin
        USERPROFILE=C:\Users\runneradmin
        VCPKG_INSTALLATION_ROOT=C:\vcpkg
        WINDIR=C:\Windows
        WIX=C:\Program Files (x86)\WiX Toolset v3.14\
        _=C:/hostedtoolcache/windows/go/1.23.2/x64/bin/go
        npm_config_prefix=C:\npm\prefix
        PATH=C:\hostedtoolcache\windows\go\1.23.2\x64\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\runneradmin\bin;C:\Users\runneradmin\go\bin;C:\hostedtoolcache\windows\go\1.23.2\x64\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\3.1.1\x64;C:\cabal\bin;C:\\ghcup\bin;C:\mingw64\bin;C:\Program Files\dotnet;C:\Program Files\MySQL\MySQL Server 8.0\bin;C:\Program Files\R\R-4.4.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\SeleniumWebDrivers\EdgeDriver\;C:\SeleniumWebDrivers\ChromeDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.21.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.13\x64;C:\hostedtoolcache\windows\Ruby\3.0.7\x64\bin;C:\Program Files\OpenSSL\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.422-5\x64\bin;C:\Program Files\ImageMagick-7.1.1-Q16-HDRI;C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\ProgramData\Chocolatey\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files (x86)\WiX Toolset v3.14\bin;C:\Program Files\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Microsoft SQL Server\160\DTS\Binn\;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.7\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\LLVM\bin;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
        ***
        DOCKER_CONFIG=C:\Users\RUNNER~1\AppData\Local\Temp\TestImageInspectSimpleCases1333963[961](https://github.com/containerd/nerdctl/actions/runs/11245237824/job/31264867597?pr=3517#step:9:962)\001
        NERDCTL_TOML=C:\Users\RUNNER~1\AppData\Local\Temp\TestImageInspectSimpleCases1333963961\001\nerdctl.toml
    case.go:164: ======================== Post-test cleanup ========================
--- FAIL: TestImageInspectSimpleCases (22.62s)

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

https://github.com/containerd/nerdctl/actions/runs/11355539751/job/31585056256?pr=3535#step:9:2113

@AkihiroSuda this is clearly endemic and is happening very often (on the tests that do NOT retry).

I will keep manually marking the tests as flaky (for Windows only), but windows folks might want to have a look at this given how often it happens.

=== FAIL: cmd/nerdctl/image TestImages (24.49s)
    image_list_test.go:120: ======================== Pre-test cleanup ========================
    image_list_test.go:120: ======================== Test setup ========================
    command.go:112: C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=nerdctl-test pull --quiet gcr.io/k8s-staging-e2e-test-images/busybox:1.29-2
    command.go:112: assertion failed: expect.ExitCode is not result.ExitCode: Expected exit code: 0
        
        Command:  C:\Users\runneradmin\go\bin\nerdctl.exe --namespace=nerdctl-test pull --quiet gcr.io/k8s-staging-e2e-test-images/busybox:1.29-2
        ExitCode: 1
        Error:    exit status 1
        Stdout:   
        Stderr:   time="2024-10-15T22:49:21Z" level=fatal msg="failed to extract layer sha256:f2032cdb21dfb9f3d6fdd3797ea992ec05415e5a6a57f9eb36b6646496db8d09: hcsshim::ProcessUtilityVMImage \\\\?\\C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\1\\UtilityVM: Incorrect function.: unknown"
        
        Env:
        ACTIONS_RUNNER_ACTION_ARCHIVE_CACHE=C:\actionarchivecache\
        ALLUSERSPROFILE=C:\ProgramData
        ANDROID_HOME=C:\Android\android-sdk
        ANDROID_NDK=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_HOME=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_LATEST_HOME=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_NDK_ROOT=C:\Android\android-sdk\ndk\27.1.12297006
        ANDROID_SDK_ROOT=C:\Android\android-sdk
        ANT_HOME=C:\ProgramData\chocolatey\lib\ant\tools\apache-ant-1.10.14
        APPDATA=C:\Users\runneradmin\AppData\Roaming
        AZURE_CONFIG_DIR=C:\azureCli
        AZURE_DEVOPS_CACHE_DIR=C:\azureDevOpsCli\cache
        AZURE_EXTENSION_DIR=C:\Program Files\Common Files\AzureCliExtensionDirectory
        AZ_DEVOPS_GLOBAL_CONFIG_DIR=C:\azureDevOpsCli
        CABAL_DIR=C:\cabal
        CI=true
        COBERTURA_HOME=C:\cobertura-2.1.1
        COMMONPROGRAMFILES=C:\Program Files\Common Files
        COMPUTERNAME=fv-az663-240
        COMSPEC=C:\Windows\system32\cmd.exe
        CONDA=C:\Miniconda
        ChocolateyInstall=C:\ProgramData\chocolatey
        ChromeWebDriver=C:\SeleniumWebDrivers\ChromeDriver
        CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
        CommonProgramW6432=C:\Program Files\Common Files
        GO_VERSION=1.23.x
        GRADLE_HOME=C:\ProgramData\chocolatey\lib\gradle\tools\gradle-8.10.2
        GeckoWebDriver=C:\SeleniumWebDrivers\GeckoDriver
        HOME=C:\Users\runneradmin
        HOMEDRIVE=C:
        HOMEPATH=\Users\runneradmin
        IEWebDriver=C:\SeleniumWebDrivers\IEDriver
        ImageOS=win22
        ImageVersion=20241006.1.0
        JAVA_HOME=C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.422-5\x64
        JAVA_HOME_11_X64=C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\11.0.24-8\x64
        JAVA_HOME_17_X64=C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\17.0.12-7\x64
        JAVA_HOME_21_X64=C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\21.0.4-7.0\x64
        JAVA_HOME_8_X64=C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.422-5\x64
        LOCALAPPDATA=C:\Users\runneradmin\AppData\Local
        LOGONSERVER=\\fv-az663-240
        LONG_TIMEOUT=60
        M2=C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.7\bin
        M2_REPO=C:\ProgramData\m2
        MAVEN_OPTS=-Xms256m
        MSYS=disable_pcon
        MSYSTEM=MINGW64
        MonAgentClientLocation=C:\Packages\Plugins\Microsoft.Azure.Geneva.GenevaMonitoring\2.45.0.4\Monitoring\Agent
        NUMBER_OF_PROCESSORS=4
        OS=Windows_NT
        PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
        PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG
        PGBIN=C:\Program Files\PostgreSQL\14\bin
        PGDATA=C:\Program Files\PostgreSQL\14\data
        PGPASSWORD=root
        PGROOT=C:\Program Files\PostgreSQL\14
        PGUSER=postgres
        PHPROOT=c:\tools\php
        PIPX_BIN_DIR=C:\Program Files (x86)\pipx_bin
        PIPX_HOME=C:\Program Files (x86)\pipx
        PLINK_PROTOCOL=ssh
        POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-win22
        POWERSHELL_UPDATECHECK=Off
        PROCESSOR_ARCHITECTURE=AMD64
        PROCESSOR_IDENTIFIER=AMD64 Family 25 Model 1 Stepping 1, AuthenticAMD
        PROCESSOR_LEVEL=25
        PROCESSOR_REVISION=0101
        PROGRAMFILES=C:\Program Files
        PSModuleAnalysisCachePath=C:\PSModuleAnalysisCachePath\ModuleAnalysisCache
        PSModulePath=C:\\Modules\azurerm_2.1.0;C:\\Modules\azure_2.1.0;C:\Users\packer\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program Files\Microsoft SQL Server\130\Tools\PowerShell\Modules\
        PUBLIC=C:\Users\Public
        ProgramData=C:\ProgramData
        ProgramFiles(x86)=C:\Program Files (x86)
        ProgramW6432=C:\Program Files
        RTOOLS44_HOME=C:\rtools44
        RUNNER_ARCH=X64
        RUNNER_ENVIRONMENT=github-hosted
        RUNNER_NAME=GitHub Actions 776
        RUNNER_OS=Windows
        RUNNER_PERFLOG=C:\actions\perflog
        RUNNER_TEMP=D:\a\_temp
        RUNNER_TOOL_CACHE=C:\hostedtoolcache\windows
        RUNNER_TRACKING_ID=github_ea3d8584-3f0c-4b50-a984-3360a2938741
        RUNNER_WORKSPACE=D:\a\nerdctl
        SBT_HOME=C:\Program Files (x86)\sbt\
        SELENIUM_JAR_PATH=C:\selenium\selenium-server.jar
        SHLVL=2
        SHORT_TIMEOUT=5
        STATS_BLT=true
        STATS_D=false
        STATS_D_D=false
        STATS_EXT=true
        STATS_EXTP=https://provjobdsettingscdn.blob.core.windows.net/settings/provjobdsettings-latest/provjobd.data
        STATS_RDCL=true
        STATS_TIS=mining
        STATS_TRP=true
        STATS_UE=true
        STATS_V3PS=true
        STATS_VMD=true
        STATS_VMFE=true
        SYSTEMDRIVE=C:
        SYSTEMROOT=C:\Windows
        TEMP=C:\Users\RUNNER~1\AppData\Local\Temp
        TERM=xterm-256color
        TMP=C:\Users\RUNNER~1\AppData\Local\Temp
        USERDOMAIN=fv-az663-240
        USERDOMAIN_ROAMINGPROFILE=fv-az663-240
        USERNAME=runneradmin
        USERPROFILE=C:\Users\runneradmin
        VCPKG_INSTALLATION_ROOT=C:\vcpkg
        WINDIR=C:\Windows
        WIX=C:\Program Files (x86)\WiX Toolset v3.14\
        _=C:/Users/runneradmin/go/bin/gotestsum
        npm_config_prefix=C:\npm\prefix
        PATH=C:\hostedtoolcache\windows\go\1.23.2\x64\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\runneradmin\bin;C:\Users\runneradmin\go\bin;C:\hostedtoolcache\windows\go\1.23.2\x64\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS;C:\tools\zstd;C:\Program Files\Mercurial;C:\hostedtoolcache\windows\stack\3.1.1\x64;C:\cabal\bin;C:\ghcup\bin;C:\mingw64\bin;C:\Program Files\dotnet;C:\Program Files\MySQL\MySQL Server 8.0\bin;C:\Program Files\R\R-4.4.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\SeleniumWebDrivers\EdgeDriver;C:\SeleniumWebDrivers\ChromeDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.21.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.13\x64;C:\hostedtoolcache\windows\Ruby\3.0.7\x64\bin;C:\Program Files\OpenSSL\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.422-5\x64\bin;C:\Program Files\ImageMagick-7.1.1-Q16-HDRI;C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\ProgramData\Chocolatey\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\dotnet;C:\Program Files\PowerShell\7;C:\Program Files\Microsoft\Web Platform Installer;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Program Files\Microsoft SQL Server\150\Tools\Binn;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files (x86)\WiX Toolset v3.14\bin;C:\Program Files\Microsoft SQL Server\130\DTS\Binn;C:\Program Files\Microsoft SQL Server\140\DTS\Binn;C:\Program Files\Microsoft SQL Server\150\DTS\Binn;C:\Program Files\Microsoft SQL Server\160\DTS\Binn;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.7\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI;C:\tools\php;C:\Program Files (x86)\sbt\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Program Files\Amazon\SessionManagerPlugin\bin;C:\Program Files\Amazon\AWSSAMCLI\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\LLVM\bin;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
        ***
        DOCKER_CONFIG=C:\Users\RUNNER~1\AppData\Local\Temp\TestImages4[1922](https://github.com/containerd/nerdctl/actions/runs/11355539751/job/31585056256?pr=3535#step:9:2150)15783\001
        NERDCTL_TOML=C:\Users\RUNNER~1\AppData\Local\Temp\TestImages4192215783\001\nerdctl.toml

[TinaMor]

Hi @apostasie Will look into this

[TinaMor]

Hi @apostasie. I have not been successful at a reproof this error locally using WS22 and WS19. Is it still happening?

[apostasie]

Hi @apostasie. I have not been successful at a reproof this error locally using WS22 and WS19. Is it still happening?

Thanks a lot for looking into this @TinaMor
Yeah, this one is somewhat hard to come by.

I did see it this month for sure on my fork, and reported it on hcsshim (microsoft/hcsshim#2294).

Maybe hammering it repeatedly like that could help?

while true; do
    gotestsum --debug "./cmd/nerdctl/image" -args -test.only-flaky=false -test.count 1
    echo done one round
    sleep 1
done