[New Feature]: Warn before removing an entire package from repo

[mdanish-kh]

Description of the new feature/enhancement

Inspired by #120261 (comment)

If a package only has a single version in the repo, and that is getting removed in a PR, the pipelines should warn the author / moderators in the PR. Most packages that have a single version in the repo are ones that use a "vanity" URL. Because of hash mismatches / 404 URLs, those tend to get auto-removed by wingetbot or other automations set up by community contributors.

This offers a bad UX to end-users that get confused as to why the package was entirely removed from the repo. In reality, the package may simply require an update to address the hash mismatch or URL issue. The validation pipelines should have ways to protect against this scenario. Retaining the package in the repository, albeit with an outdated hash / URL means that the contributors can report against the correct problem and request a package update.

Proposed technical implementation details (optional)

No response

[mdanish-kh]

[Policy] Area-Validation-Pipelines
[Policy] Area-Bots

@stephengillie - Could you please remove the Package-Update and Error-Hash-Mismatch labels?

[mdanish-kh]

@stephengillie -

Also, out of curiosity, is this something that can be achieved through your automation? Example of it being #136974 (comment)

[stephengillie]

@stephengillie -

Also, out of curiosity, is this something that can be achieved through your automation? Example of it being #136974 (comment)

Yes, I can build a check that will verify if previous manifest versions exist before removing. Not all approvals go through my system, but most do. We have a ceiling in the form of API calls, but I'm working to mitigate that as well.

[Trenly]

@stephengillie -
Also, out of curiosity, is this something that can be achieved through your automation? Example of it being #136974 (comment)

Yes, I can build a check that will verify if previous manifest versions exist before removing. Not all approvals go through my system, but most do. We have a ceiling in the form of API calls, but I'm working to mitigate that as well.

@stephengillie - When querying which versions exist, you could directly interact with the Source.msix package to extract the database file and run queries against that directly. It won’t contain any of the manifest content, but will give you a list of the PackageIdentifiers, versions, publishers, commands, and a few other pieces of data. You could use that to check for the number of parts within a version string, how many versions of a package exist, and it can give you the PathParts required to directly fetch the latest manifest to check for AppsAndFeaturesEntries, all without hitting the GitHub API

[stephengillie]

I added a Versions switch to my Get-ManifestListing function. This function generally gets the contents of a GitHub directory. The switch strips the version number from the path, so it returns a list of versions instead of a list of files in that version.

Function Get-ManifestListing {
	param(
		$PackageIdentifier,
		$Version = (Find-WinGetPackage $PackageIdentifier | Where-Object {$_.id -eq $PackageIdentifier}).version,
		$Path = ($PackageIdentifier -replace "[.]","/"),
		$FirstLetter = ($PackageIdentifier[0].tostring().tolower()),
		$Uri = "https://api.github.com/repos/microsoft/winGet-pkgs/contents/manifests/$FirstLetter/$Path/$Version/",
		[Switch]$Versions
	)
	If ($Versions) {
		$Uri = "https://api.github.com/repos/microsoft/winGet-pkgs/contents/manifests/$FirstLetter/$Path/"
	}
	try{
		$out = (Invoke-GitHubRequest -Uri $Uri -JSON).name
	}catch{
		$out = "Error"
	}
	return $out -replace "$($PackageIdentifier)[.]",""
}

I know that I have both Version and Versions, and this is probably not a best practice. I'm somewhat running out of names, in an odd way.

[stephengillie]

Script in action: #136730 (comment)

• Faster than checking the client or extracting from source.msix as it doesn't have to wait for the Publish pipeline to process.
• Only checks when "Remove" or "Automatic Deletion" is in the title - otherwise will need to be manually checked.

[Trenly]

Script in action: #136730 (comment)

• Faster than checking the client or extracting from source.msix as it doesn't have to wait for the Publish pipeline to process.
• Only checks when "Remove" or "Automatic Deletion" is in the title - otherwise will need to be manually checked.

It may be faster, but if its pushing you up against your rate limits then it may not be worth it. I think it would be rare that a true race condition where changes which have been merged but not published would impact the results of an index check. In fact, only one comes to mind - When there are two or more versions of a package and they are all being removed, using source.msix wouldn’t update between the removals. But, this race condition still exists querying GitHub directly if the manual checks for both PRs are run close enough together that neither PR is merged before the checks of the other is completed

[vedantmgoyal9]

@stephengillie suppose I'm removing TeamViewer.TeamViewer, the script will still pass because there's TeamViewer TeamViewer.Host still present and your script will detect "Host" as a version, although it is another (sub-)package in itself.

[stephengillie]

@stephengillie suppose I'm removing TeamViewer.TeamViewer, the script will still pass because there's TeamViewer TeamViewer.Host still present and your script will detect "Host" as a version, although it is another (sub-)package in itself.

It's true that Find-WinGetPackage will return this kind of wildcard response:

PS C:\ManVal> Find-WinGetPackage TeamViewer.TeamViewer

Name            Id                         Version Source
----            --                         ------- ------
TeamViewer      TeamViewer.TeamViewer      15.50.5 winget
TeamViewer Host TeamViewer.TeamViewer.Host 15.50.5 winget

Which is why I filter for the exact PackageIdentifier when finding the version, to build that part of the path:

$Version = (Find-WinGetPackage $PackageIdentifier | Where-Object {$_.id -eq $PackageIdentifier}).version,

But the PackageIdentifier itself is parsed directly from user input:

$Path = ($PackageIdentifier -replace "[.]","/"),