-
Notifications
You must be signed in to change notification settings - Fork 427
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't use include_str!()
macro with Bazel-generated data files
#459
Comments
Looks like the same issue as #79 and #222.
https://docs.bazel.build/versions/master/build-ref.html#data suggests that
data is runfiles only, and probably explains why this behaves
inconsistently.
Does this work if you add the generated data file to sources? Presumably
this requires some more hacking to rules_rust, or giving the file a .rs
suffix.. If so, that might inform the implementation of #222
…On Wed, Oct 21, 2020 at 12:06 PM David Johnston ***@***.***> wrote:
The Problem
The data attribute appears on each of the core Rust rules (e.g.
rust_binary). The documentation for data says
This attribute can be used to specify any data files that are embedded
into the library, such as via the include_str! macro.
I have found that include_str()! works only for "source" files (i.e.
files included in the Bazel source tree). Unfortunately, the
include_str!() macro does *not* work if the data files are generated by
some other Bazel target.
Example
To demonstrate this, I've added a small test to my rules_rust fork
<dwtj@b67ce49>
.
In this test, one data file is a source file and one data file is
generated by a genrule. The contents of both files are embedded in the
rust_test via the include_str!() macro.
On my linux system, this rust_test fails to compile with the following
rustc error:
error: couldn't read test/generated_data/include_str/generated_data.txt: No such file or directory (os error 2)
--> test/generated_data/include_str/include_str.rs:9:13
|
9 | include_str!("generated_data.txt").trim()
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
error: aborting due to previous error
Notice that rustc has no problem finding source_data.txt.
I believe that the problem can be seen by inspecting this rustc action's
sandbox. On my Linux system, if I run the test with bazel test
--sandbox_debug, and then change directory to
<bazel_workspace_output_base>/sandbox/linux-sandbox/<action_id>/execroot/io_bazel_rules_rust,
the root of the problem becomes clear: the library and the source data are
sibling files
./test/generated_data/include_str/include_str.rs
./test/generated_data/include_str/source_data.txt
but the generated file is not
./bazel-out/k8-fastbuild/bin/test/generated_data/include_str/generated_data.txt
So, the generated data file is an input to the rustc Bazel action, but it
isn't anywhere that the rustc executable can find it. Thus, a compiler
error occurs.
Temporary Workaround
In my project where I actually want this feature I have an
awful-dreadful-ugly hack to overcome this problem: I add a symlink from my
source tree pointing to where I expect the generated file should be under
the Bazel workspace's output base; then I add both the generated file and
its symlink to the data attribute.
A Very Rough Idea for a Solution
I have a little bit of experience writing Bazel rules that also needed to
deal with this problem. My solution there
<https://github.com/dwtj/dwtj_rules_hugo/blob/8617d3455cf8cc9787aced9ea2a86045efefeb7d/hugo/private/hugo_website/defs.bzl#L57>
was for each target to add various ctx.actions.symlink() actions to place
both generated files and source files into a new directory under their
logical paths before calling the tool.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#459>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAROG6EC5QG5S33V3PZHAUDSL4BGXANCNFSM4SZ7S65A>
.
|
Oh shucks. I think you're right. I completely missed these. Sorry for the possible duplicate. I'll read #79 in a bit.
Nope. I just tried using a
and then including
Again, looking at the sandbox contents, this generated file is over in
To quote the Rust documentation for
|
I'm thinking that make variable expansion in rustc_env might be a cleaner way to solve this, and will try to get a PoC PR up soon. In the mean time, you can use make variable expansion to get one level up from a build script's OUT_DIR, which your script can write files to, eg: genrule(
name = "protobuf_gen",
srcs = [
"//proto:backend.proto",
],
outs = [
"backend_proto.rs",
],
cmd = """\
PROTOC=$(location @com_google_protobuf//:protoc) \
OUT_DIR_PARENT=$(RULEDIR) \
SRCFILE=$(location //proto:backend.proto) \
$(location :generator_bin)""",
tools = [
":generator_bin",
"@com_google_protobuf//:protoc",
],
) Then in your code: include!(concat!(env!("OUT_DIR"), "/../backend_proto.rs")); This requires the crate to have a build script in order for OUT_DIR to be set. |
This makes it possible to pass in the path to generated files and external tools. This potentially closes bazelbuild#459, closes bazelbuild#454, and closes bazelbuild#79. The docs seem to indicate there's precedent for this in rules_cc: https://docs.bazel.build/versions/master/be/make-variables.html#predefined_label_variables
@dae Thanks for your workaround approach. This is sufficient for my needs at the moment. I've added a full example of this workaround approach to a branch of my fork and reproduced it here:
#[cfg(test)]
mod test {
#[test]
pub fn include_str_test() {
let hello_world = format!(
"{}, {}!",
include_str!("source_data.txt").trim(),
include_str!(concat!(env!("OUT_DIR"), "/../generated_data.txt")).trim()
);
println!("{}", hello_world);
assert_eq!("Hello, world!", hello_world);
}
} The key to this hack is that this crate's @dae One nitpick about your suggestion. You said
However, I don't really see how either make variable substitution or using the cargo build script to write the files is really necessary here. My example, for instance, doesn't use either. |
The Problem
The
data
attribute appears on each of the core Rust rules (e.g.rust_binary
). The documentation fordata
saysI have found that
include_str()!
works only for "source" files (i.e. files included in the Bazel source tree). Unfortunately, theinclude_str!()
macro does not work if the data files are generated by some other Bazel target.Example
To demonstrate this, I've added a small test to my
rules_rust
fork.In this test, one data file is a source file and one data file is generated by a
genrule
. The contents of both files are embedded in therust_test
via theinclude_str!()
macro.On my linux system, this
rust_test
fails to compile with the followingrustc
error:Notice that
rustc
has no problem findingsource_data.txt
.I believe that the problem can be seen by inspecting this
rustc
action's sandbox. On my Linux system, if I run the test withbazel test --sandbox_debug
, and then change directory to<bazel_workspace_output_base>/sandbox/linux-sandbox/<action_id>/execroot/io_bazel_rules_rust
, the root of the problem becomes clear: the library and the source data are sibling filesbut the generated file is not
So, the generated data file is an input to the
rustc
Bazel action, but it isn't anywhere that therustc
executable can find it. Thus, a compiler error occurs.Temporary Workaround
In my project where I actually want this feature I have an awful-dreadful-ugly hack to overcome this problem: I add a symlink from my source tree pointing to where I expect the generated file should be under the Bazel workspace's output base; then I add both the generated file and its symlink to the
data
attribute.A Very Rough Idea for a Solution
I have a little bit of experience writing Bazel rules that also needed to deal with this problem. My solution there was for each target to add various
ctx.actions.symlink()
actions to place both generated files and source files into a new directory under their logical paths before calling the tool.The text was updated successfully, but these errors were encountered: