Packaging and distributing apps

[killercup]

(moderated summary by @epage)

Context:

• "Rust makes writing crossplatform, tested, modern command line applications frictionless while incorporating industry best practices and providing great documentation." (goal)
• Focus is on CLI apps (see README for definition)

Assumptions and Implications

• Majority of CLIs will at more than just Rust developers (for targeting rust-developers, see Improving binary distribution through cargo. #20)
• Users of CLIs will be somewhat technical considering they are using a terminal
• Use case priorities
  1. Using the app on the command line via the path
  2. Glued together with other apps in a script
  3. Drag and drop files onto an icon for the CLI
• Examples of content to distribute
  • one or more rust binaries
  • completion files
  • man pages (See Documenting CLIs #23)

Other solutions

• Go has equinox

Plan of attack

1. "How to package" documentation

• Spec out and document how to package

2. Automation

• Implement stager
• Reach out to binary packaging tools to consolidate effort with stager

In-ecosystem resources

• trust: example project that tarballs pre-built binaries releases via github releases
• crate-ci docs: A home for process automation documentation
• crate-ci meta: A home for discussing tooling and documentation, including discussing stager
• stager
• self-update
• cargo-deb
• cargo-wix
• cargo-arch
• cargo-rpm
• cargo-ebuild
• cargo-bundle

Open Issues

• (Windows) Adding manifests, see Support application manifests on Windows rust-lang/rfcs#721

Open Questions

• Prioritized list of package formats per platform?
  • Linux: deb, rpm, flatpak, snap
    • Are flatpak / snap's apps added to PATH?
  • Windows: MSI via wix, chocolatey
  • Mac: homebrew
    • How easily are pkg's available from command line?
• PPAs?
• Use case, challenges, and priority of source packages?
  • See Build src deb mmstick/cargo-deb#40
• Importance of remote administration and provisioning?

@killercup's original post

In the first meeting, we identified packaging and distributing CLI apps a something we should improve.

We need to have a best-practice solution for getting from "I have a crate with a binary" to "I can ship this app and don't have to worry about cross-platform installation stuff".

Edit 2018-03-19: This issue is about distribution of rust binaries without using cargo-install.

[killercup]

• cf. https://github.com/japaric/trust/
• https://github.com/crate-ci/crate-ci.github.io by @epage
• cargo-install doesn't respect Cargo.lock
  • help fix Packages for binary crates should include Cargo.lock rust-lang/cargo#2263

[killercup]

Distribute documentation:

• documentation of env vars a CLI app repsects/uses
  • auto-generate from something like https://docs.rs/configure?
  • how to incorporate external crates, like env_logger?
• there seems to be no cohesive answer for man pages yet
  • ripgrep is using aciidoc: auto generate man page, update clap, CI cleanup BurntSushi/ripgrep#776
  • issue in clap Auto-generate manpage, help docs, etc. clap-rs/clap#552 (@kbknapp, awaiting clap v3-alpha)

[kbknapp]

• ref: man pages in clap v3-alpha the overall tracking issue (Tracking Issue for 3.x clap-rs/clap#1037)

[killercup]

From the etherpad:

• Guides/How-Tos
  • Deb, RPM, AUR, Homebrew, Snap, Flatpak, AppImage, nixpkgs
• Automation
  • cargo subcommands
  • Using CI

If cargo-install is an important use case for dev tools, how much should we look at improving it?

[XAMPPRocky]

If cargo-install is an important use case for dev tools, how much should we look at improving it?

I think this is part of a lot broader area of "crates.io's treatment of executables.". Which is something that previously has been very vague. Part of distribution should be codifying how crates.io handles and promotes executables or lack thereof.

I'm unsure how we should go about this as there isn't really a process for requesting crates.io features.

[jaemk]

For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

[spacekookie]

@killercup When it comes to automatically generating debian packages (and there is already cargo-deb for that) I think it's even more important to generate source packages for distributions to include (or to build a PPA with). And that is something that cargo-deb can't yet to.

See cargo-deb#40

[matklad]

For windows specifically, it would be nice to be able to add manifest to the binary: rust-lang/rfcs#721.

[sophiajt]

A couple more thoughts after brainstorming with @killercup:

With CLI apps that are built by doing "cargo install", we have a pretty straightforward path we could go down with regards to building installers for people.

For example, what if we had a new cargo plugin, or separate tool, that could take a "cargo install"-able CLI app and then, using a bit of convention over configuration, build you an installable package for a distro or use a simple Windows installer .exe?

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem. @killercup mentioned that we might even want to go a step further and offer some project templates that drop in whatever metadata is required for building .deb, homebrew, etc. From there maybe the cargo plugin can either be configured and/or searches for packaging tools it knows about and builds using them.

These combined with something like https://github.com/japaric/trust/ means folks could take an easy extra step to see what tweaks they'd need to make to get it to work across more platforms.

[Screwtapello]

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem.

Eh, that kind of depends on the tool, and how polished an experience you want it to be. Even static binaries might want to include manpages, generic documentation, default config files, command-line completion integration, data files...

I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms. However, it would be great to have a set of base conventions that platform-specific tools can extend in different ways. For example, there could be a base convention that documentation lives in a docs/ subdirectory in Markdown format, and the build-posix-package tool could have an extra convention that a file named docs/manpage.md will be installed as the manpage for the crate's main executable.

[russel]

I got pointed at this issue by @killercup on https://users.rust-lang.org/t/cargo-and-installation-of-ancilliary-files/15974

My specific problem is installing a GNOME application. So not a CLI application per se, but very similar. Whilst many of the resources needed by the application are embedded in the executable, there are some that need installing. Manual pages and user documentation certainly, but also some image files and a GNOMEShell launcher.

Whilst my need is very specific, I am sure it acts as a microcosm of the more general situation.

Meson makes it very easy to install application – my experience is with D and C++, I have not tried using Meson with Rust. It is also fairly straightforward, but not always easy, to set up installation using SCons and CMake – but again my experience is with D and C++, to date I have only used Cargo with Rust.

[kbknapp]

@Screwtapello: I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms.

This is my exact concern. I agree very much with the rest of your comment that having separate tools or conventions that people can pick/choose/extend would go a lot further than an "all in one" tool. This is with the caveat that those "separate tools" are well known, widely supported, well documented, etc. Not just one offs.

@jaemk: For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

I love this! It may not work for everyone, or for complex use cases, but making it easy to have a "self update" feature of Rust CLI applications for "small utilties and binaries" would probably go a long way. Of course this doesn't solve the initial installation piece, but I still think it'd be great to include in a full blown solution. This particular piece could just be a standard guide on how to do this, such as a cookbook like recipe.

[about cargo-install in general as an install tool]

I'm on record saying I don't really like leaning on cargo-install for the general public, or general purposes. I think it's great for dev tools, or things that make sense to use during Rust development, but I personally don't want to expand it's use case to a general purpose installer.

I think there's too many nuances, and too much complexity to do everything correctly for it to all be contained in cargo. I'd like to keep cargo a Rust build tool, not a general purpose package manager. This isn't to say we shouldn't add certain features to cargo-install such as post build scripts...I'd just like to have proper guides/tools doing things the correct way first so that people don't just hit the easy button of cargo install which is what people are starting to do already.

Why?

Here's a few of the things I think would need to be sorted out (conversely why I think they shouldn't sorted out). This isn't meant to be an all encompassing list, just a quick search of some relevant issues. There may be more I can't think of off the top of my head, but here's a few:

Advanced or Post Build Steps

My primary concern with using cargo install too generally is that because of all the added complexity of correctly installing software, cargo-install takes the easy route and doesn't support complex/advanced use cases. Which is perfectly fine to me because it's not a general purpose installer. Until recently cargo-install didn't even support use of Cargo.lock files. As far as I know it still doesn't (easily) support more advanced build steps (like setting RUSTFLAGS, or like many have mentioned post build steps).

• Pass -Ctarget-cpu=native during cargo install rust-lang/cargo#4150 (RUSTFLAGS="-C target-cpu=native" by default on cargo-install)
• cargo install does not support --target CLI param, unlike other build params rust-lang/cargo#5068 (cargo install doesn't support --target options)
• Add an install.rs that lets a file perform installation setup, as well as a permanent installation outdir rust-lang/cargo#2386 (Request install.rs similar to build.rs)

Crate Size

Including things like docs, manpages, completion scripts, licenses, axillary files, etc. which will be required for installation of end user products will bloat the crate packages. I'd actually like to do the opposite of de-bloating crates

• Exclude more files in Cargo.toml clap-rs/clap#1186
• cargo shouldn't check for dev-dependencies when running cargo build or cargo install rust-lang/cargo#4988 (cargo install requires downloading and building dev-dependencies)

Binary Only Features/Deps

Until cargo supports binary only features and deps, crates that include both a lib and bin will suffer from the above points when used as a lib (compile times, crate size, etc.)

Also, being able to specify binary dependencies is huge for a general purpose package manager.

• Allow dependencies that only apply to specific cargo targets (bin, example, etc.) rust-lang/cargo#1982 (bin only features)
• Enable 'cargo install' to install binary-crates according to a spec of dependencies rust-lang/cargo#5120 (cargo-install binary deps, which is part of the equation, but still leaves out general binary deps)
• Need a way of specifying features for dependencies with cargo install rust-lang/cargo#4555 (Specify deps for cargo install)
• Support a first-class format for declaring external dependencies rust-lang/cargo#3816 (General sense of allowing "external deps" in cargo)
• Proposal: allow crates to depend on a binary crate rust-lang/cargo#4316 (crates depending on binary crates)

Code Signing

This is another pretty big hole if we'd want to properly use cargo install as a general purpose installer.

• End-to-end integrity of crates in registries rust-lang/cargo#4768

[vitiral]

Just to include packagers, nixpkgs has pretty good support for rust ATM. I managed to get the novault nixpkg created, which required custom external dependencies.

The advantage of nixpkgs is that they can run alongside other package managers on any Unix distribution (including Mac) and I believe they are trying to support windows as well. This allows for at least some installation path for almost any platform/distribution, which is very valuable.

The process is pretty straightforward, with carnix creating the package from the Cargo.lock file. I think it's a workflow worth emulating for other packaging scripts as well.