New Manifest Schema and Structure

[Demitrius Nelon (denelon)]

All,

We have been working through several of the features on the backlog, and looking towards the future. Several new and some existing features would be greatly improved by updating our manifest schema. We’ve looked at some inconsistencies in terminology as well as some issues with Pascal casing identifiers. The list below should help to identify the key names, the field descriptions, some data type information and constraints/hints to the constraints. Several of the new fields are required for new features or they help with pulling a bit more of the meta-data from packages/Apps in the Microsoft Store so we can get closer to parity for displaying meta-data and supporting install for all packages in the store. In addition we’re looking to begin partitioning the directories under “manifests” by the (lower case) first letter of the publisher. GitHub.com has some limitations on the number of entries rendered on the website. The table below shows our current thinking. We're also looking into the possibility of splitting a manifest into multiple files to separate installers from localized meta-data.

Feedback is welcome.

Edited
Initial table deleted in favor of JSON schemas:
https://github.com/microsoft/winget-cli/tree/master/schemas/JSON/manifests/v1.0.0

[Heath Stewart (heaths)]

1. Please not YAML. Everyone hates YAML. Tooling is poor (no schema validation) and it's harder to approach (e.g. when to include a dash or not).
2. In addition to installer success exit codes, you need to have a list of exit codes (e.g. 3010) that signal a required reboot. It will happen. You should consider refusing to install more packages (with overrides) until that happens or you risk torn state (loading a mix of newer and older dependencies).
3. Nothing for uninstall? Are you not going to support true package management (this was in the roadmap at one point)? It's important to note that some products require uninstalling old before installing new, so no true "upgrade".

I'd be happy to walk you through some examples offline (we can codify them here). I was one of the original devs on WiX's burn installer and the architect on the Visual Studio setup engine for Dev15+ and we handled very diverse first- and third-party installers by taking all this into account.

[Demitrius Nelon (denelon)]

YAML was originally chosen to be "easier" for humans to write. We're aware of the tooling challenges with it. We're building support for both YAML and JSON in the REST API that's currently in development with the "same" YAML keys.

[Heath Stewart (heaths)]

Understandable, but history has shown it's not. ;) It's mainly the lack of scheme support in editors that makes it harder to consume. You have to keep going back to documentation, getting whitespace right, etc. I mean, it's a nit, but I imagine most devs would agree.

[Mahdi Hosseini (ghost1372)]

Hi Demitrius Nelon (@denelon) Is API available now? I have created a GUI for Winget I get the manifests by cloning manifest repository, so Can I get a list of all the manifests by API?

[Demitrius Nelon (denelon)]

It's not quite ready. We're building a reference implementation and testing with the client. Once we believe we have parity with PreIndexed sources, we will share the API and the reference will also be open-source. I hope to get it out next month.

[Mahdi Hosseini (ghost1372)]

thank you

[Demitrius Nelon (denelon)]

There has been some discussion about "reboot required". What is the scenario related to overrides you have in mind?

[Heath Stewart (heaths)]

When an installer returns an error code like 3010 (or custom ones meaning the same thing) indicating a reboot is required, it's best not to ignore it in many cases. For example, with the "new" VSSetup engine, we have vital packages (must be installed successfully; if any return a reboot required, we prompt before continuing) and non-vital packages (optional features; reboots can be done at the end). The main source of reboots (initially or updates) are .NET Framework (though, that's mainly a non-issue now) and VCRedist. Because of the amount of managed and native code that uses the VC runtime, if we know we need to load them and continue installing, we risk loading some newer files that were replaced with older files that weren't replaced. This was the number 1 cause of torn state issues investigated by VS teams and why we stopped allowing for it. People hate reboots, sure, but it's even more troubling if your application doesn't work right later because of torn state issues that are incredibly hard (and expensive) to diagnose.

So by allowing for an array of exit codes that translate to "reboot required", you can prompt for a reboot at the end and, for example, refuse to install more applications until they do unless they force it, i.e. customer takes responsibility for saying, "no, I want to install more software anyway". It's a gamble. 20+ years experience has shown most of the time the gamble pays off, but when it doesn't - it can be a latent problem that, by the time you discover it, could have corrupted data and the install from weeks or even months ago is likely not going to be on your radar of causes.

[Demitrius Nelon (denelon)]

Upgrade includes a pair of behaviors. One of them is "Install" which essentially just installs the new version, the other is "UninstallPrevious" which would uninstall the previous version before installing the new version.

[Heath Stewart (heaths)]

Great! But I see no uninstall command listed here, so how do you know to properly uninstall the application?

[Heath Stewart (heaths)]

For applications like VS that have optional features (since I imagine someone will create unofficial installers for it), will there be support in winget for it, or at least a way to pass additional command line parameters? In winget v0.2.2521 Preview I see an install --override command but 1) no information on how to use it (does everything after it just replace the install command options? how much of tit?), and 2) isn't necessarily what people would want to pass additional options. A common approach (git, dotnet, et. al.) is to have users specify -- and everything after that is sent directly to the installer as additional command line options. E.g.:

winget install vs_buildtools --silent -- --add Microsoft.Component.MSBuild

[Demitrius Nelon (denelon)]

https://github.com/microsoft/winget-cli/tree/master/schemas/JSON/manifests/v1.0.0

We've got an initial version of the new schemas checked in. They may still change, but this should give an overview of what we're looking at for the next version of the manifest schemas.

[Demitrius Nelon (denelon)]

Easton Pillay (@jedieaston) I know you're working on some PowerShell helpers. I just wanted to make sure you're aware of some big changes coming. We're also working on a CLI that will help generate manifests by downloading installers and trying to auto-populate fields (we will be making it open-source).

[Easton Pillay (jedieaston)]

This looks great so far, I'll keep an eye on it!

[Yigit Emre Sahinoglu (yigitemres)]

Can we also get custom messages (after successful installs)? Sometimes users asks particular questions about particular programs (like this: microsoft/winget-pkgs/issues/656) and telling users that "this problem still exist and in order to fix this issue, you can do these..." is a good way to reduce issues number on the winget-pkg repo.

Also, can we get after install requirements? Some installers calls driver installers (microsoft/winget-pkgs/pull/8521) after they finish initial install process and this is cause stalls/timed outs.

[Heath Stewart (heaths)]

Demitrius Nelon (@denelon), that really depends on what you do with them. As inferred in my last reply (which I wrote up before I saw your reply), there are cases where subsequent calls to winget will fail. Should winget care? Probably not. It's not technically a chainer - it's a bootstrapper. But it can be used to chain installs, which I believe is a goal. So if an install returns an error, what do you do? If it returned that a reboot is required, will you allow other installs to proceed? If winget itself doesn't really react to previous return codes, I don't think there's much point in even classifying the return codes themselves. I think this is a more interesting discussion about whether or not winget should care.

For example, in the new setup engine and all previous chainers, we often classified certain packages as "vital": VS can't work (even start) without them. Others (like tools, project systems, etc.) are non-vital. But that's across a set of packages we curate, effectively. I don't know how you'd do that here. In WiX burn we developed a system of related packages, which approximate dependencies or extensions. Should there maybe exist something like that for winget manifests? An app could declare it's an extension for another app, and if that other app requires a reboot (which you'd have to keep track of ephemerally; we use PendingFileRenames to delete such state), refuse to install the "extension" app. Of course, that might be up to the extension or the original app.

Personally, given the goal of getting things on a box quickly and efficiently (e.g. through scripting), this is probably more than what winget needs to do. Delegating mitigations for torn state, missing entry points, or whatever, might best be left to the installers themselves. Burn handles this well (we have related bundle types that work like hooks) and other installers can too if it really matters.

[Heath Stewart (heaths)]

We seem to be answering each others questions / addressing concerns. :)

I worked a lot on Burn's related dependency system and tied in the package ref-counting feature and would be happy to discuss further - even pointing out code where needed in Burn - if that's a route worth pursuing. I still wonder if that's within the goals of winget in either the short or long terms.

Conceptually, they work great in a more closed ecosystem. With VS, we control the manifest for the core product that the new engine installs. With Burn, it's typically used (and was used for VS for a couple versions) for an extensible but rather small ecosystem. For something as open as winget, it could easily devolve into a cyclic graph or outright abused to effective DOS a system with tons of dependencies.

[tristanbarcelon]

Heath Stewart (@heaths) and Demitrius Nelon (@denelon), when installing Microsoft.VisualStudio.2019.Enterprise via winget, how should I be passing additional parameters to vs_setup? I see there's an optional --override parameter to winget install command. Is this documented somewhere internally or public?

[Demitrius Nelon (denelon)]

Heath Stewart (@heaths) the "--override" parameter does pass the string specified to the installer. This is an area that does need attention in documentation https://docs.microsoft.com/en-us/windows/package-manager/winget/install.

[tristanbarcelon]

Thank you Demitrius Nelon (@denelon) and Heath Stewart (@heaths). Currently, we use powershell to create an offline snapshot of a specific VS2019 version with workload ids to include and componentids to exclude on a folder using the --layout parameter, --include, --add, --lang, --keepLayoutVersion parameters. At a later point in time, we also use powershell to pass following parameters to vs_setup.exe: --productKey, --nocache, --norestart, --noWeb, --wait, --installPath, --in. It would be good if we can retire maintenance of these scripts and utilize winget to create a an offline layout or install specific versions of Microsoft.VisualStudio.2019.Enterprise by way of the --override parameter.

Demitrius Nelon (@denelon), is there a winget command for displaying available versions of a specific package id? Currently, winget only shows latest. I suppose it's possible to download the winget source index from https://winget.azureedge.net/cache/source.msix, extract the index.db sqldatabase and query from tables directly but that would be going too far.

[Demitrius Nelon (denelon)]

tristanbarcelon, yes. winget show <package> --versions will list all of the versions we have in the repository.

[tristanbarcelon]

Demitrius Nelon (@denelon) Agree with you on the open/closed principle. For OS build numbers that support winget but do not have built-in store support, how do we install winget properly? For example, installing on Windows Server 2022 proved to be more challenging and required installing its dependencies first. Windows Server 2019 appears to be the same way based on this issue 144 comment. Are there plans to create an installer that will package the vclibs dependencies and publishing them in releases instead of just the msixbundle?

[Demitrius Nelon (denelon)]

It's nice to meet another fan of SOLID principles. We're sticking with MSIX. Installing the Windows Package Manager on Windows Server is challenging. We've had several internal discussions regarding the plan for Windows Server. Long term support is one of the most complex business drivers on those SKUs. We added the license file to releases for offline installation, but the other dependencies and understanding the implications for servicing the cartesian has made us pause. The future work with Evolving the Microsoft Store for Business should lead us on the right path.

[tristanbarcelon]

Demitrius Nelon (@denelon), Is there any guidance or idiom/convention for package authors to follow regarding version numbers when publishing manifest? For example, nuget version numbers with alpha values and/or dashes are typically considered unstable releases. Same thing for apps that follow semver 2 semantics. However, when I look at package id Mirantis.Lens, they've been publishing version numbers that would have been considered unstable using nuget or semver semantics.

Mirantis just made it really difficult for their consumers to pin their installations to specific versions with values that are too long.

[Easton Pillay (jedieaston)]

We have to use whatever they put in the Add and Remove Programs table (until schema 1.1 when we'll have other ways to put version numbers). Unfortunately, Mirantis does that.

[Demitrius Nelon (denelon)]

Easton Pillay (@jedieaston) is correct. Not all software uses semantic versioning.