When using a PSRepository backed by Azure Devops Artifact feed that upstreams PSGallery, Install-Module fails on specific module versions

[tristanbarcelon]

Steps to reproduce

1. Create an Azure Devops Artifact feed (org or team project scoped) called privatepsgallery and upstream powershellgallery.
2. Generate a personal access token with Package read/write permission within the organization containing the Artifact feed
3. Execute the following in PSCore, remembering to replace yourazdopat and yourazdoorganization placeholders with relevant values.

$VSCred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList @('youremail@yourdomain.com', (ConvertTo-SecureString -String 'yourazdopat' -AsPlainText -Force))
Register-PSRepository -Name 'PrivatePSGallery' -SourceLocation 'https://pkgs.dev.azure.com/yourazdoorganization/_packaging/privatepsgallery/nuget/v2' -Credential $VSCred -InstallationPolicy Trusted
Find-Module -Name 'AzurePipelinesPS' -AllVersions -Repository psgallery -ErrorAction SilentlyContinue | Where-Object { $_.Version -inotlike '*-*' -and ([System.Version] $_.Version -ge [System.Version] '1.0.0' ) } | Select-Object -Property Name, Version, @{ label = 'SortableVersion'; expression = { [System.Version] $_.Version }} | Sort-Object -Property SortableVersion | Foreach-object {
         Write-Host "Installing module $($_.Name) and version $($_.Version)"
         Install-Module -Name $_.Name -RequiredVersion $_.Version -Repository privatepsgallery -Credential $vscred -Scope allusers -AllowClobber -SkipPublisherCheck
}

To reproduce the issue encountered with AWS.Tools.* modules, use this script below.

 [string[]] $Modules = @('AWS.Tools.ElasticLoadBalancingV2', 'AWS.Tools.Pricing', 'AWS.Tools.RDS', 'AWS.Tools.Route53', 'AWS.Tools.S3', 'AWS.Tools.SecurityToken', 'AWS.Tools.SimpleSystemsManagement')
 foreach ($Module in $Modules)
 {
    Find-Module -Name $Module -AllVersions -Repository psgallery -ErrorAction SilentlyContinue | Where-Object { $_.Version -inotlike '*-*' -and ( [System.Version] $_.Version -ge [System.Version] '4.0.0')} | Select-Object -Property Name, Version, @{ label = 'SortableVersion'; expression = { [System.Version] $_.Version }} | Sort-Object -Property SortableVersion | Foreach-object {
        Install-Module -Name $_.Name -RequiredVersion $_.Version -Repository privatepsgallery -Credential $vscred -Scope allusers -AllowClobber -SkipPublisherCheck
    }
 }

Expected behavior

When the module or requiredversion is not found in azure devops feed, it should fetch the module from powershellgallery and make it visible in Azure Devops.

Actual behavior

I get an error like this:

Install-Package: No match was found for the specified search criteria and module name 'AzurePipelinesPS'. Try Get-PSRepository to see all available registered module repositories.

Looking at Azure Devops, I can tell where it's failed, starting with version 3.1.42 of AzurePipelinesPS. I experience the same issues with SqlServerDSC on any version newer than 15.0.1 (e.g., 15.1.0, 15.1.1, 15.2.0). AWS.Tools.Common, AWS.Tools.EKS, AWS.Tools.EC2, etc all fail as well between versions 4.1.99 and 4.1.86, inclusive and 4.1.9.0. Due to the consistency of versions affected in AWS.Tools.* modules, I'm leaning towards corruption of version metadata stored in PSGallery's index/database when AWS pushed these versions to PSGallery.

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      7.2.4
PSEdition                      Core
GitCommitId                    7.2.4
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

> Get-Module -ListAvailable PowerShellGet,PackageManagement
ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-M…

    Directory: C:\program files\powershell\7\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     1.4.7                 PackageManagement                   Desk      {Find-Package, Get-Package, Get-Package…
Script     2.2.5                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-M…

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Binary     1.0.0.1               PackageManagement                   Desk      {Find-Package, Get-Package, Get-Package…
Script     1.0.0.1               PowerShellGet                       Desk      {Install-Module, Find-Module, Save-Modu…

> Get-PackageProvider -ListAvailable
Name                     Version          DynamicOptions
----                     -------          --------------
NuGet                    3.0.0.1          Destination, ExcludeVersion, Scope, SkipDependencies, Headers, FilterOnTag, …
PowerShellGet            2.2.5.0          PackageManagementProvider, Type, Scope, AllowClobber, SkipPublisherCheck, In…
PowerShellGet            1.0.0.1

[SteveL-MSFT]

cc @SydneyhSmith @alerickson

[tristanbarcelon]

I tried installing versions of dbatools >= 1.0 module in verbose mode

Find-Module -Name dbatools -AllVersions -Repository PSGallery | Where-Object { [System.Version] $_.Version -ge [System.Version] '1.0' } | Foreach-Object {
     $_.Version
     Install-Module -Name $_.Name -RequiredVersion $_.Version -Scope AllUsers -SkipPublisherCheck -Repository PrivatePSGallery -Credential $VSCred -Verbose
}

log entry for nuget package v2 protocol uses FindPackagesById() seems correct. At the end of this logging statement, it is searching for something and it is empty. Is that the version and why is the total package yield 0? When I copy and paste https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/FindPackagesById()?id='dbatools' into my browser and authenticate to Azure Devops, this returns a proper result.

VERBOSE: Repository details, Name = 'PrivatePSGallery', Location = 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2'; IsTrusted = 'True'; IsRegistered = 'True'.
VERBOSE: Using the provider 'PowerShellGet' for searching packages.
VERBOSE: Using the specified source names : 'PrivatePSGallery'.
VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
VERBOSE: The specified Location is 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2' and PackageManagementProvider is 'NuGet'.
VERBOSE: Searching repository 'https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/FindPackagesById()?id='dbatools'' for ''.
VERBOSE: Total package yield:'0' for the specified package 'dbatools'

When getting a specific version of dbatools using Nuget V2 protocol the rest query should look like this: https://pkgs.dev.azure.com/[redactedorg]/_packaging/privatepsgallery/nuget/v2/Packages(Id='dbatools',Version='0.8.952').

[anamnavi]

@tristanbarcelon thanks for reaching out with this! We're looking into this scenario and seeing what's happening and will get back soon.

[tristanbarcelon]

Hi @anamnavi, I'm curious whether you've been able to reproduce this issue I'm observing. It started affecting newer versions of AWS.Tools.* modules starting with 4.1.116.

[anamnavi]

@tristanbarcelon I'll test and get back on this early next week, sorry for the delay, just coming back from vacation.

[tristanbarcelon]

@anamnavi , does your team collaborate with Azure Devops artifacts team? I noticed another pattern whenever these errors happen. Specifically, the errors happen within a few hours after a new version of a module is published in PSGallery. I don't know whether Azure Devops artifacts caches metadata from PSGallery rather than querying it directly for the presence of a module and version. Re-attempting the same script later works, though for some modules, the upstreaming behavior still remains broken (dbatools, sqlserverdsc, azurepipelinesps) and now it affects ImportExcel version 7.8.0.