Excessive inputs: why libs are inputs for another libs?

[yanok]

Look at this line:

rules_d/d/private/rules/common.bzl

Line 165 in 73f2cd8

[d.libraries for d in d_deps] +

(also the next one with c_libs), why do we need compiled libraries to be inputs for compiling libraries?

This is not a big deal actually, since all the transitive dependency sources are already in the inputs, so it is even impossible to produce an example that works "wrong".

But in our fork we support headers and implementation_deps, so interface_srcs are not always all the transitive dependency sources. (BTW, is there some interest to get these changes merged back into this repo?)

And with that, having libs as inputs starts to hurt: if we have app <- libA <- libB dependency, and I change a source in libB which is not exported to libA, I expect no recompilation for libA. And with the compiled libB being an input, libA will still be recompiled.

[dcarp]

Thanks, @yanok, for reporting this. Please take a look at the PR that addresses this issue.

To be honest, I don’t think this change will have much impact, since it doesn’t affect the compiler invocation itself, but only the size of the compile action sandbox.

If you’re looking to optimize further, one possible approach could be to generate the .di sources and use those in dependent targets. This should automatically strip out the implementation details. I’m not entirely sure how reliable .di generation is in practice, though, since it doesn’t seem to be widely used.

And with that, having libs as inputs starts to hurt: if we have app <- libA <- libB dependency, and I change a source in libB which is not exported to libA, I expect no recompilation for libA. And with the compiled libB being an input, libA will still be recompiled.

I’m not sure I fully follow this example. If the libB target changes, that implies one of its sources has changed. Since libA imports one of libB’s sources, it will be recompiled as well, even if the resulting libA binary ends up being identical.

[yanok]

To be honest, I don’t think this change will have much impact

yes, right now it won't: as a said, because all transitive imports sources are inputs to the compilation, there will be no difference.

But it doesn't have to be the case: I'm doing a couple of things in https://github.com/weka/rules_d to make things slightly better:

1. Explicit exports and hdrs attributes in d_library. Now if I have libA that depends on libB, only libB's exports and hdrs are inputs to libA, not the whole srcs. exports are for public API and hdrs are pre-generated headers. There is a catch: exports must be closed under imports. So one could think: what's the point when? It will end up being the whole library anyway. That's not true for two reasons: a. headers help; b. even without headers, we can have api.d module that doesn't import impl.d, instead impl.d imports api.d and uses static constructors to register the implementation.
2. implementation_deps: if you import libC from libB, but not from the exported part, you should put libC into implementation_deps instead of deps: this way no part of libC goes into inputs of libA, not even the exported bits.
3. As you mentioned, automatic header generation. We have a macro that runs header generation on the exported modules, so only the distilled headers are exported, not the whole module. This helps with shrinking the exported part, since imports from the function bodies are now removed. That works, but have some quirks:
   • requires a patched compiler, since in the upstream compiler there is no way to run just header generation. The patch is trivial, but I never had time to open a discussion to include it upstream.
   • as you correctly guessed, header generation is full of bugs :) They are mostly trivial, but I didn't have time recently to sit and fix them
   • it doesn't help with templates :) but that's by design.

The code in the repo is somewhat messy (I forked before the recent update/refactor). It also contains one a bit more controversial feature. But if there is interest, I can try rebasing feature by feature and send PRs.

I’m not sure I fully follow this example. If the libB target changes, that implies one of its sources has changed. Since libA imports one of libB’s sources, it will be recompiled as well, even if the resulting libA binary ends up being identical.

Exactly! As I said, it's impossible to produce a "bad" example with the current dependency tracking setup. You need to imagine you have the features described above (at least the first one). So libB would have exports = ["api.d"] and we only change impl.d. This way we need to recompile libB, but not libA.

[dcarp]

D doesn’t have a header (API declaration) / source (API implementation) separation like C or C++.
Because of that, its build model often has to resemble the rules_cc approach.

D aims for simplicity, and rules_d follows the same principle.
From what I see, your use case can be handled cleanly by splitting libB into two libraries :

• libB containing the public API, and
• libBImpl containing the implementation.

The resulting dependency graph would look like this:

graph TD
    app --> libA
    libA --> libB
    libBImpl --> libB
    libBImpl --> libC
    app --> libBImpl

Loading

In this way libA will not be recompiled when libBImpl changes.

I think that for a power user who cares about build speed, it’s easier to separate the implementation and the interface into distinct libraries, rather than making every user decide whether their sources and dependencies belong in srcs, exports, hdrs, deps, implementation_deps, and so on.

[yanok]

Ok, I take that answer as we have to keep our own fork :) That's totally fine.

D doesn’t have a header (API declaration) / source (API implementation) separation like C or C++.

Oh, I'm very much aware of that. That's why we can't do some build optimizations that are possible with C/C++...

I don't like your solution, since even in your simple example app depends on libBImpl for no obvious reason. Imagine what happens if we have many layers with a lot of dependencies on each layer.

BTW, in our fork hdrs/exports and implementation_deps are tools for power users: one can use only srcs/deps, getting back the same behavior as before.

Also, rules_cc also supports implementation_deps, so it's not like I've invented it :)

[dcarp]

Thank you again for your constructive feedback!
I’m currently working on adding DUB support, and if that ends up requiring any of the attributes you proposed, I’ll take inspiration from your approach and aim to implement it in a way that stays compatible with your implementation.