Adding new framework (plugin?)

[jackorp]

I'd like to work on an RSpec plugin someday but I don't know how to go about it and there seems to be no documentation present on the topic of writing a plugin. Do I need to open a PR to this repo or can it be a pip library?

Regarding documentation I am missing answers mainly to these questions:
How to implement a plugin of this sort anyway?
What needs to be done?
How to register it with tmt?
Can I define my own fmf fields for configuration?

[psss]

Thanks for the issue, @jackorp. Yeah, the documentation has still a few white spaces... Let's use this issue to track creating a new section for writing plugins.

But to quickly answer your questions: If your new plugin would be useful to more users it would make sense to contribute a new pull request here. The rspec plugin could then be distributed as a new subpackage if necessary.

How to implement a plugin of this sort anyway?

Is that supposed to be a plugin for the discover step?

What needs to be done?

Plugins are detected from the step folders or from the TMT_PLUGINS variable (hm, which needs to be documented as well).

How to register it with tmt?

Your plugin just needs to inherit from one of the plugin classes e.g. here and define the supported methods.

Can I define my own fmf fields for configuration?

Your plugin will implement the method as registered above and each method defines its own set of keys. For example provision --how container defines the --pull option.

Let us know if anything is not clear.

[jackorp]

Thanks for the issue, @jackorp. Yeah, the documentation has still a few white spaces... Let's use this issue to track creating a new section for writing plugins.

But to quickly answer your questions: If your new plugin would be useful to more users it would make sense to contribute a new pull request here. The rspec plugin could then be distributed as a new subpackage if necessary.

How to implement a plugin of this sort anyway?

Is that supposed to be a plugin for the discover step?

I am not sure. discover seems correct, however, @pvalena has told me there exists framework. Maybe both should be implemented? IIRC shell is an option for both the framework and the discover tags.

What needs to be done?

Plugins are detected from the step folders or from the TMT_PLUGINS variable (hm, which needs to be documented as well).

Hmm, yes that would be nice if documented.

What step folder do you mean? Do you mean tmt's project folder or the project's fmf definitions? I have limited knowledge on tmt so I am not sure.

How to register it with tmt?

Your plugin just needs to inherit from one of the plugin classes e.g. here and define the supported methods.

So all I need to do is inherit the class in my plugin, have it somewhere where tmt looks for plugins and then it gets loaded and is ready to be used?

Can I define my own fmf fields for configuration?

Your plugin will implement the method as registered above and each method defines its own set of keys. For example provision --how container defines the --pull option.

I am not sure what do you mean by this. IOW the example you provided also implies --pull which does not need to be specified by the user because the plugin implements it?

[FrNecas]

I am not sure. discover seems correct, however, @pvalena has told me there exists framework. Maybe both should be implemented? IIRC shell is an option for both the framework and the discover tags.

Depends on how the plugin is intended to work. shell inside framework and shell inside discover are not very related (other than the fact that they both use shell ;) ). The discover steps finds tests to be executed, in this case using a shell script. The framework attribute is used during the execute step to select the "method" of execution (it basically defines how the test is intended to execute - try running tmt test create -t beakerlib and tmt test create -t shell and see the difference).

I am not very familiar with RSpec so I can't really help more. What is the intended way of integration into tmt? Am I right to assume that you would like to write a test using RSpec and then run it using tmt? In that case, the framework attribute sounds like what you are looking for - you will have to add support for execution using RSpec into the execute step, see for example this: https://github.com/psss/tmt/blob/master/tmt/steps/execute/internal.py#L148

What step folder do you mean? Do you mean tmt's project folder or the project's fmf definitions? I have limited knowledge on tmt so I am not sure.

Inside tmt repo, there is tmt/steps folder which contains the varios steps. If you wanted to add a new discover method, you would simply add a module to tmt/steps/discover, create a class inheriting from DiscoverPlugin and implement the required methods (seems like options and tests in the case of discover).

I am not sure what do you mean by this. IOW the example you provided also implies --pull which does not need to be specified by the user because the plugin implements it?

I think what Petr meant here is that you can create fmf fields for a plugin using options, See this https://github.com/psss/tmt/blob/master/tmt/steps/provision/podman.py#L31, lets take a look at --image. The user can provide the options through fmf, e.g.:

provision:
    how: container
    image: fedora

Or through the command-line (provision -h container --image fedora) or a default is used.

[thrix]

This seems like a nice topic to discuss on tmrw's hacking meeting. I believe we should discuss the general guidance here, how much integration with tmt makes sense for different test frameworks. Use case or rspec in this case can be similar to pytest, and other test frameworks for other languages ....

[jackorp]

I am not very familiar with RSpec so I can't really help more. What is the intended way of integration into tmt? Am I right to assume that you would like to write a test using RSpec and then run it using tmt? In that case, the framework attribute sounds like what you are looking for - you will have to add support for execution using RSpec into the execute step, see for example this: https://github.com/psss/tmt/blob/master/tmt/steps/execute/internal.py#L148

Yes that sounds like I want to make use of that. Basically, what you said, I want to run RSpec inside tmt and have the results reported using tmt's methods of reporting.

I also think that it should support choosing tests to be run. With RSpec, you can define the test file's path and the lines of tests to be run, e.g. rspec path/to/test_spec.rb:44 -- that would run test that is on line 44 from file test_spec. Other test suites also have other methods of filtering, e.g. cucumber uses "tags". You can then run exclusively tests that have that tag or exclude those tests from running.

However, all that is just a theoretical discussion of what it could / should do at this point.

What step folder do you mean? Do you mean tmt's project folder or the project's fmf definitions? I have limited knowledge on tmt so I am not sure.

Inside tmt repo, there is tmt/steps folder which contains the varios steps. If you wanted to add a new discover method, you would simply add a module to tmt/steps/discover, create a class inheriting from DiscoverPlugin and implement the required methods (seems like options and tests in the case of discover).

I see, thanks.

I think what Petr meant here is that you can create fmf fields for a plugin using options, See this https://github.com/psss/tmt/blob/master/tmt/steps/provision/podman.py#L31, lets take a look at --image. The user can provide the options through fmf, e.g.:

provision:
    how: container
    image: fedora

Or through the command-line (provision -h container --image fedora) or a default is used.

Right, thanks. Now I understand.

[jscotka]

In teemtee/fmf#136

I've tried to do some another way how to solve similar issue for pytest. it is based on changes of FMF what allows custom plugins for discovering data in the files. (then it could be supported inside TMT) so theoretically you can use some already existing plugins inside FMF or create custom one directly in your project.

I can imagine something like:

/discover:
   how: fmf
   extension: ./path/to/your/plugin (or some_plugin inside FMF without path)

then this discovery step will use FMF custom plugin and allows you e.g. inspect python/ruby test files.
And test itself run in shell e.g. like: test: pytest -v python_file.py::cls::test_name (see linkek PR, to see how data looks like)