[Test] Test runner doesn't seem to honor weak/strong flag

[leafpetersen]

See detailed transcript below, but in summary:

Running test.py on a test with --nnbd=weak passed as an argument results in this command line:

DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --enable-experiment=non-nullable --ignore-unrecognized-flags --packages=/Users/leafp/src/dart-repo/sdk/.packages /Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart

Note that this command line does not set the mode to weak mode.

Running with --nnbd=strong results in the same command line.

cc @munificent @sortie

leafp-macbookpro:sdk leafp$ python tools/test.py -c dartk -m release --enable-experiment=non-nullable --nnbd=weak tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart 
Test configuration:
    custom configuration(architecture: x64, compiler: dartk, mode: release, runtime: vm, system: mac, nnbd: weak, enable-experiment: [non-nullable])
Suites tested: language

FAILED: dartk-vm release_x64 language/nnbd/normalization/generic_function_type_object_normalization_test
Expected: Pass
Actual: RuntimeError

--- Command "vm" (took 602ms):
DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --enable-experiment=non-nullable --ignore-unrecognized-flags --packages=/Users/leafp/src/dart-repo/sdk/.packages /Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart

exit code:
255

stderr:
Unhandled exception:
Expect.notEquals(unexpected: <<R0 extends Future<Never>, R1 extends Never, R2 extends Null>() => Null>, actual:<<R0 extends Future<Never>, R1 extends Never, R2 extends Null>() => Null>) fails.
#0      Expect._fail (package:expect/expect.dart:685:5)
#1      Expect.notEquals (package:expect/expect.dart:306:5)
#2      checkNotEquals2 (file:///Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/type_builder.dart:136:10)
#3      checkTypeNotEquals2 (file:///Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart:22:3)
#4      neverBoundTests (file:///Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart:175:5)
#5      main (file:///Users/leafp/src/dart-repo/sdk/tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart:199:3)
#6      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:301:19)
#7      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

--- Re-run this test:
python tools/test.py -m release -c dartk --nnbd weak --enable-experiment non-nullable language/nnbd/normalization/generic_function_type_object_normalization_test
[00:00 | 100% | +    0 | -    1]

=== 0 tests passed, 1 failed ===

[athomas]

@alexmarkov What would be the correct command line arguments for weak and strong for the VM here?

[athomas]

@leafpetersen I looked at what we have in the test matrix, and there for the weak/strong we also add the vm-options --no-null-safety (weak) and --null-safety (strong) respectively.

The recommended way to run what you tried to run above would be:

$ tools/test.py -ndartk-weak-asserts-mac-release-x64 tests/language/nnbd/normalization/generic_function_type_object_normalization_test.dart

@munificent It seems that using vm-options/dart2js-options in the test matrix is how people set this up. Should this be moved from test_matrix configuration to test runner logic?

[leafpetersen]

I haven't found a way to make use of the -n option. In my scenario, I usually have a test (often one that I just wrote), and I want to run it on a particular platform (say the VM), in a particular mode (say release, with nnbd strong). I don't know how to map from that to something to pass to -n. So I always end up using the -c options.

If the nnbd=??? are no-ops, should we get rid of them, replace them?

[athomas]

The -n options are defined in the test matrix (https://github.com/dart-lang/sdk/blob/master/tools/bots/test_matrix.json#L751).

I think it would be more helpful to make the nnbd options do what you expected them to do (they are also not no-ops, they affect how tests that require a particular mode are run).

@whesse FYI, this is also a problem related to "how to test my code".

[munificent]

Yeah. :-/ This is a consequence of how the VM team added weak/strong mode to the test runner. Instead of basing it on command line arguments passed to the test runner (and there were already existing arguments in place for this), special arguments were added directly to certain named configurations in the text matrix file.

This means there is no way at all to correctly run weak/strong mode VM tests without going through a named configuration. But figuring out what named configuration to use is really painfully difficult. The test runner is in a really messy state right now and could use some love.

I think it would be more helpful to make the nnbd options do what you expected them to do

+1.

I also have some ideas to improve the usability of running named configurations. Once I'm done with the test migration, I'll see if I can write something down.