during incr. comp. place `.o` files in incr. comp. directory and give predictable name

[nikomatsakis]

As part of the incr. comp. work, we need to place the intermediate .o files for each partition into the incr. comp. directory (and not erase them after we're done). They should be given names that reflect the name of the partition they represent. This will let us reuse them in later compilations when possible.

cc @michaelwoerister

[nagisa]

Seems to me like using the hash generated as part of #34037 should suffice? Not only, then, it would be trivial to check for need to regenerate only the necessary files (doesn’t exist? rebuild), but also easy to keep clean (didn’t use that hash? remove.)

[nikomatsakis]

@nagisa yeah, we could use the hashes, though the partitions already have stable names at present, so that works too.

[michaelwoerister]

I'd opt for including a human-readable name into the obj file name for debugging reasons.

[nikomatsakis]

OK, so, I made a branch that places .o files (and other temporary artifacts) into the incremental compilation directory. It also opts not to delete .o files when the work is done (at least those corresponding to modules). I was debating though whether it'd be better to instead copy the files into the incremental directory when the work is done (and then copy them back out again if we want to re-use them). The idea would be that the "external interface" of the compiler remains the same -- if you run in incremental mode, and you pass --save-temps (or whatever we call it now), you still wind up with your temporary files in the output directory (whereas in my branch they will be in the incremental directory instead). The cost is that we have to copy some files.

[nikomatsakis]

I will open the PR once #33890 lands.

[nikomatsakis]

cc @rust-lang/tools curious to get your take on this comment

[eddyb]

I believe that in the long term, in a situation without dynamic libraries, we would want to avoid generating intermediary rlibs (at least Cargo doesn't need them) and always link together .o files directly into the final build product, and this is a good first step towards that.

[michaelwoerister]

I made a branch that places .o files (and other temporary artifacts) into the incremental compilation directory.

What does it do when no incremental compilation directory is given? Just use the output directory?

[nikomatsakis]

@michaelwoerister yes, as before.

[alexcrichton]

It also opts not to delete .o files when the work is done (at least those corresponding to modules).

If you run in incremental mode, how would this work if we deleted the object files? Don't we have to leave them around to reuse them?

I do think though that rustc foo.rs should still not litter the current directory, but rustc foo.rs --incremental bar should be able to do whatever it wants to the directory bar.

I was debating though whether it'd be better to instead copy the files into the incremental directory when the work is done (and then copy them back out again if we want to re-use them).

Oh so we always leave it in the incremental directory, and only maybe move it out? -C save-temps isn't really specified so I'd be ok if we didn't actively move it out, but if we normally leave it in the incremental dir this seems fine.

The cost is that we have to copy some files.

I've found a great way to handle this is:

fs::hard_link(src, dst).or_else(|_| fs::copy(src, dst))

That way it attempts to do the super fast thing first which normally succeeds, but in weird cases it'll copy some files.

[nikomatsakis]

@alexcrichton I can't really tell which design you are advocating for :) but yes that hard-link thing seems elegant. Maybe it's better then to keep everything in the output directory (as today) but also copy/link things into or out of the incr. dir in incremental mode. Obviously not a huge deal either way.

[alexcrichton]

Oh hm I may just actually be confused about what the possible designs here are. To me the title of this issue is an clear "we can literally never be incremental without doing that". The next piece:

It also opts not to delete .o files when the work is done (at least those corresponding to modules).

So right now we spray a bunch of files in the output directory, and do you mean that we never delete these files if incremental is turned on? If that's the case then I'd instead recommend what I think your next strategy was with always emit object files into the incremental directory directly, and then only if -C save-temps is passed are they copied out (although that's somewhat optional in my opinion).

Does that make sense?

[michaelwoerister]

If that's the case then I'd instead recommend what I think your next strategy was with always emit object files into the incremental directory directly, and then only if -C save-temps is passed are they copied out (although that's somewhat optional in my opinion).

This is the option that makes the most sense to me.