PLEASE_BENCH is broken

[nagisa]

~> set -x PLEASE_BENCH 1
~> make check-stage1
cfg: version 1.0.0-dev (d2516fc09 2015-02-13 20:50:31 +0200)
cfg: build triple x86_64-unknown-linux-gnu
…
rustc: x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustdoc
run rpass-full [x86_64-unknown-linux-gnu]: x86_64-unknown-linux-gnu/stage1/bin/compiletest
thread '<main>' panicked at 'UnrecognizedOption("bench")', /home/nagisa/files/rust/rust/src/compiletest/compiletest.rs:107
/home/nagisa/files/rust/rust/mk/tests.mk:702: recipe for target 'tmp/check-stage1-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rpass-full.ok' failed
make: *** [tmp/check-stage1-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rpass-full.ok] Error 101

[Gankra]

Can confirm. Saw this a week or so ago but assumed my local install was busted.

[bluss]

I've used it successfully with make check-stage1 PLEASE_BENCH=1. Is there a difference?

[nagisa]

Yes, make check-stage1 PLEASE_BENCH=1 is the same thing as make check-stage1, because for this particular variable our scripts react to environment variable, not the make argument.

EDIT: I was wrong, make check-stage1 PLEASE_BENCH=1 fails exactly the same way as it does with an environment variable.

[Gankra]

Oh wait, I was getting this failure while using make check PLEASE_BENCH=1

[fhahn]

I had a closer look at this issue. PLEASE_BENCH=1 adds --bench to the arguments passed to the executable to be tested. At the moment, compiletest does not accept a --bench argument, because it is not needed for any test in src/test/, even the tests in src/test/bench do not use #[bench].

I have updated the makefile to only add the --bench flag for crate tests. I do not think that changing compiletest add --bench to the run arguments of all compile tests makes sense, because it would mess up tests which check command line arguments. Also the bench option can be added as comment in a compile test as well.