Inconsistent commands in the cli

[adbridge]

We now have for non tests

mbed compile *options*
mbed export *options*

whereas for tests we have

mbed test --compile *options*
mbed test --export *options*      (proposed but not yet implemented)

This is very inconsistent and is likely to be a poor user experiece for a customer/partner. Professional tools are consistent and make logical sense. The above does not.
I also believe we are going to get

mbed test

Which will compile and run all the tests. So how are we going to do the same for applications? This hasn't been thought about.

It is essential we provide a consistent and logic interface if this is going to be the one stop tool.

Aside from this if there are multiple ways of compiling / running tests then this will make life very dificult for the CI tools. This is not a position we want to see.

[PrzemekWirkus]

We also have `mbed compile --tests' etc.

We used to have in yotta

$ yotta target XXX
$ yotta build
$ mbedgt -VS 

Setup, build + test.

We had before Partner Workshop

$ mbed-cli target K64F
$ mbed-cli toolchain GCC_ARM
$ mebd-cli compile
$ mbedgt -VS 

Now we have

$ mbed-cli target K64F
$ mbed-cli toolchain GCC_ARM
$ mebd-cli compile --tests (?)
$ mbed-cli test --run -VS  (?)

I probably missed stuff.

[PrzemekWirkus]

• We've also specified test_spec.json file and its format, where it belongs and this contract was broken. I caused Greentea test tool to stop working. We had a mbed test replacement which added complexity.
• We do not inform each other about mbed-cli interface changes. Yes, command line INTERFACE is also an interface :)
• We do not tag mbed-os so we do not know which of two versions of mbed-cli works with which version of mbed-os.
  • Yes, mbed-cli bleeding edge works with mbed-os bleeding edge (when you are not on "push to master spree".
• Now if I put a job in CI how I correlate between mbed-cli version I should use and mbed-os version ?
  • How about in near future I have a snapshot from 2 months ago, I want to rebuild for partner and reproduce and error? I for sure will not know which version of mbed-cli to use with that mbed-os.
  • Yes, there are commits in Git, they are not labels, they are not tag or releases, or milestones.
• If mbed-cli interface changes so rapidly how do I make a stable job which is calling build system ?

[PrzemekWirkus]

What could happen is that somehow there will be a mutual agreement on how we organize ourselves between TWO repositories: monolith and build system. This is the same agreement you will have with partners so you can hit two birds with one stone.

[screamerbg]

@bridadan @PrzemekWirkus @adbridge @sg-

mbed CLI currently offers a full test workflow.

In cased you've missed the documentation (that was already available during the SiP Partner Workshop)

https://github.com/ARMmbed/mbed-cli#testing

You used to have in yotta:

$ yotta target XXX
$ yotta build
$ mbedgt -VS 

Now you have:

$ mbed target K64F
$ mbed toolchain GCC_ARM
$ mbed test --compile
$ mbed test --run

It's in the documentation already. How is this inconsistent?

For me it seems reasonable to have mbed test --export as well.

[screamerbg]

And by the way mbed test -m K64F -t GCC_ARM will execute everything listed above - 4 commands instead of 1! :)

[adbridge]

@screamerbg If you fully read my initial comment, the inconsistency is between commands -

mbed compile *options*
mbed test --compile *options*

and

mbed export *options*
mbed test --export *options*

As you say you are now bringing in

mbed test *run options*

To compile and run tests. But how will this be duplicated for applications? My suspicion is that it will be again an inconsistent command.

[screamerbg]

Agreed that mbed compile --tests when there is mbed test that handles the whole testing workflow.
This was left there not to break the CI and agreed with @sg- and @bridadan during the integration workshop. Totally agree that this should be removed to avoid confusion.

I'll send a PR for this.

[adbridge]

@screamerbg You're still missing the point . Just removing mbed compile --tests is not the answer! There should not be one way of compiling for tests and a different way for the application. The same argument applies to subsequent commands (such as export) !! If you are going to go down the
mbed test --compile route then you need to deprecate mbed compile and replace it with something like mbed app --compile etc!

[screamerbg]

The application lives in the root of the project, tests are not. Tests are multiple applications with specific requirements toward testing framework etc. They are not the same thing.

[bridadan]

I might be able to shed some more light on the current debate.

Echoing @adbridge, there are inconsistencies between commands, the following being an example:

mbed compile *options*
mbed test --compile *options*

Having a "compile" option in two different places does feel a bit strange. However, at the moment compiling tests and compiling an app ARE different technical processes.

1. Compiling an app is a relatively "simple" process compared to compiling tests.
   1. To compile an app, all the source is lumped together, compiled, and linked. Done! You get a binary and off you go. Really simple and really easy
2. Compiling tests is quite a bit more complicated and there were a few design choices taken here.
   1. First, all the code except for the code contained under TESTS directories is compiled but not linked. This is done to reduce the compile time for each test.
   2. Next, each test folder is built and linked against the previously compiled code
   3. Finally, a test specification (test_spec.json) is created and placed in the test build directory. This lets the test tools know what tests are available for running.
      1. Location: .build/tests/<TARGET>/<TOOLCHAIN>/test_spec.json

As you can see, compiling tests and apps are currently two separate beasts. This all boils down to the issue that the tools don't know where the application lives since it can be (technically) anywhere in the tree besides an ignored directory (TESTS, .mbedignored directory, etc). So in a sense, compiling the two things ARE different processes.

So, all this led to the decision to separating the testing functionality into a different CLI command, since they have different implications. Doing this led to interface changes. As @PrzemekWirkus mentioned, the test_spec.json location was moved which broke existing CI setups (this was on me). It helps underline @PrzemekWirkus's statement that the mbed Command Line Interface is an Interface, and as such it will need to go under proper versioning since so many things will depend on it.

HOWEVER, this project is still in the early stages and small tweaks may be necessary (0.x.x). And this is probably necessary for the next few weeks. But because mbed-cli is so integral to all workflows (contributing, development, CI, etc), consistency and stability are the most important things. This was the message I believe @adbridge (please correct me if I've taken too many liberties here!) was trying to make.

Finally, making large changes to the CLI now might be a bit too last minute. If this were done, we would need a clear outline on all necessary changes, covering all workflows, so we can minimize the surprises to interested parties. Otherwise we'll probably cause more problems again.