Is `--out-dir` justified or can we find another way of accomplishing it?

[withoutboats]

I noticed that cargo build has two subcommands, both added this year. I believe this accurately describes their behavior:

• --target-dir: Replaces $PWD/target with this directory (e.g. a debug build will be built into target-dir/debug). Stable flag, added in April 2018. Add target directory parameter --target-dir #5393 In addition to CARGO_TARGET_DIR env var, support --target-dir command line option #5308. Mimicks the behavior of CARGO_TARGET_DIR
• --out-dir: Things will be built into $PWD/target as normal, but copies some of the artifacts into the directory specified by out-dir (not a profile specific subdirectory). Unstable flag, added in March 2018. cargo build --out-dir #5203 Ability to specify output artifact name #4875. Mimicks the behavior of OUT_DIR.

(I'm not 100% certain I've accurately described their behavior in relationship to the profile specific subdirectories.)

--out-dir is still unstable, but --target-dir was instantly stabilized (by mistake, I believe).

I can't help but notice that these two flags do very similar things! They also seem to have been designed for very similar purposes (build integration, the meson project is mentioned specifically in both of them).

I'm not certain I agree that either of them should have been added as flags (an argument is made that flags are better than env vars, and while I agree in principle for user oriented APIs, I don't think that build system integration is a user oriented API in the relevant sense), but I'm definitely suspicious of having both of these very similar flags provided. Since target-dir is stable, I'm not sure if there's an available avenue to unify these flags other than just only providing target-dir.

cc @rust-lang/cargo

[joshtriplett]

I do think it makes sense to keep these separate. I like the idea of having --out-dir to put the final build artifacts and not all the intermediate bits.

[matklad]

I think it is important that we have both of these features:

• ability to override "scratch space" dir for intermediate build artifacts
• ability to override the place where the final build results go

UI-wise, using flags for these seems better: the "build system" code, like ci-scripts, tends to be poked by many people, so it's important to keep it easy to understand without prior knowledge, and flags are better than env vars in this respect.

I do agree that for most of the users, interacting with Cargo's CLI, these flags are irrelevant. However, this could be said about some other flags as well, like --manifest-path, --message-format, --build-plan, --frozen, --locked. Perhaps we should hide such "useful in very specific circumstances" flags from cargo cmd --help by default?

[withoutboats]

I do agree that for most of the users, interacting with Cargo's CLI, these flags are irrelevant. However, this could be said about some other flags as well, like --manifest-path, --message-format, --build-plan, --frozen, --locked. Perhaps we should hide such "useful in very specific circumstances" flags from cargo cmd --help by default?

I am unhappy with the UX of having all of these flags also. For a long time we've been creeping in the direction of having flags to customize every user visible aspect of the behavior, because this has been how CLI tools traditionally handle users wanting to customize their behavior. However, I think this has a bad UX overall, because it makes the tool overwhelming and ultimately makes all of the flags undiscoverable. I think hiding these tools from --help unless you pass another flag is a half measure, which improves the situation but is probably not the best solution.

I think a preferable approach to what we've done would be to provide some lower level command which takes as many flags as you want and can be used to build up the user-oriented commands like cargo build, cargo run, and cargo test, but with all of the flags to customize the precise nature of its operations. The higher level commands could be defined in terms of that lower level command, and people writing very bespoke integrations can also build their own build command out of it.

So I noticed a few days ago that when I ran cargo build --help it went beyond the length of my terminal. This surprised and concerned me, so I took statistics of how cargo build --help's output has grown over time:

It was in the process of investigating how the flags to cargo build had proliferated that I noticed these two flags. I think there's a larger problem here, but these two flags in particular have such similar behavior that keeping them both has unique negative UX consequences aside from the general concern. From the --help output I could not understand how they were different, and I don't think their names indicate that the most material difference between them is that one excludes the intermediate artifacts. Its the kind of customization creep that makes me groan and turn to stack overflow instead whenever I try to read the man pages of any of the standard UNIX utilities: their man pages are an information overload that make it too difficult to learn things.

An obvious way to help resolve their close similarity is to clarify the differences in the --help output. But as we've already touched on, these are not useful for most users, and so in the common case these two commands are both cruft I am trying to sort through to figure out the flag I actually want - increasing the amount of time we dwell on them makes the situation worse. So I don't think that would be a solution either.

[alexcrichton]

I'm personally fine adding these flags basically wherever makes sense, but to echo @matklad I don't think we'll want to remove the functionality. In that sense I would want to make sure that these sorts of configurations (which you're 100% right @withoutboats, will continue to get added to seemingly no end) have some home somewhere. They're definitely not important to surface in high-profile locations, just need to be located somewhere!

[withoutboats]

We discussed this & also the overarching issue a bit in the cargo meeting yesterday. The sort of overarching viewpoint was that it would be ideal to expose these kinds of knobs through a set of lower level primitive commands on top of which sit the more user oriented commands (git has this distinction for example, calling them "porcelain" and "plumbing").

However, the problem with doing that is that it requires major refactors which seem difficult or infeasible. On the other hand, by adding all of these knobs incrementally we end up with the same functionality, but with worse UX (from having the functionality exposed through a bunch of flags on the main commands), and worse maintainability (because those flags are being threaded through the entire codebase). But back to the problem of pulling off a refactor, and so we ended at a standstill.

[alexcrichton]

Was there a good idea of what the porcelain/plumbing mode would actually look like in Cargo? In abstract it makes sense but if there's a concrete idea where to go it may not be so bad to draw up an issue and start soliciting help for making progress

[mark-i-m]

Is there any chance of --out-dir being stabilized any time soon? It is really useful when integrating cargo with another build system, e.g. Makefile.

[kornelski]

It seems to be related, but a bit backwards, to temp vs non-temp separation of directories proposed in #6227

[elichai]

I would also like to see --target-dir for cargo install

[igrep]

I found the --out-dir option is very useful for me to copy .so files built for Android to app/src/main/jniLibs/<arch_dir>.
--target-dir emits too many unnecessary things, and I have to resolve the dylib's name (in debug? in release? etc.) by myself in addition...

[yaahc]

I recently had a couple of people express an interest in --out-dir being stabilized and from my initial digging it seems like what they may actually want is to switch to OUT_DIR, which is already stable. If we don't intend to stabilize --out-dir since flags are the "user oriented API" then I think we might want to more clearly document how to get the functionality provided by --out-dir on stable to make the environment variable more discoverable.

[sophie-h]

@yaahc

from my initial digging it seems like what they may actually want is to switch to OUT_DIR, which is already stable

OUT_DIR is read only if I understand the docs correctly. (I also tried to set it and it didn't worked.)