tests: add basic pytest for plugins repo

[daywalker90]

As you know, we will require all plugins in the plugins repo to have atleast some basic test to show the users that this plugin is working. Unfortunately i don't quite understand what this plugin does and how to test it. All rpc methods i tried failed for different reasons. The one from the README, metrics_one, seems to be no longer in the code?! And the ones i found in the code did not work either. The setup.sh will be used by the plugins repo to get a release binary so we don't have to compile 100 plugins every CI run. If you could get the test working somehow that would be great! Otherwise @chrisguida will move this plugin into the archived section soon.

[vincenzopalazzo]

I guess this does not work in my ci? because it is missing the requirements.txt

[vincenzopalazzo]

Otherwise @chrisguida will move this plugin into the archived section soon.

Mh! I still do not like this rule, because lnmetrics is used in production with v24.05 already, maybe it is one of the few plugin that works in the list 😏

But yeah I will try to make a monkey test just to make @chrisguida happy

[chrisguida]

@vincenzopalazzo if you're actively maintaining it and it works in production with the latest CLN, then I should think it would be pretty straightforward to write a basic test that shows it working.

We just want to make sure we can automatically detect when plugins break. When plugins don't have tests, plugins can stay broken for a long time, and the longer they stay broken, the harder they are to fix. We have seen this on several plugins so far and this ultimately harms the confidence users have in the plugins repo.

I don't doubt that this is one of the best plugins on the list 😁

Thanks for helping us maintain a high standard for the plugins repo, where users can confidently install high-quality, actively maintained plugins🙏

[vincenzopalazzo]

@vincenzopalazzo if you're actively maintaining it and it works in production with the latest CLN, then I should think it would be pretty straightforward to write a basic test that shows it working.

You can just do it by running make check, but I will add the pytest architecture to work also on my CI, plus the plugins CI should build the plugin from source, and not fetch the binary

[chrisguida]

@daywalker90 I'm down to add make check as an alternative to pytest?

[vincenzopalazzo]

Ah you are also assuming that I plugin will always use pytest? mh this can be problematic because I was going to write the integration test in rust

[chrisguida]

@vincenzopalazzo happy to use whatever framework you like, as long as you incorporate it into the plugins repo CI ;)

[vincenzopalazzo]

happy to use whatever framework you like, as long as you incorporate it into the plugins repo CI ;)

Why you do not run a make check instead inside the repo CI? in this way you make happy everyone

[chrisguida]

Just need to think of a good way to incorporate that... maybe if the plugin uses coffee then we can pull the test command out of the coffee.yml?

[vincenzopalazzo]

Just need to think of a good way to incorporate that... maybe if the plugin uses coffee then we can pull the test command out of the coffee.yml?

This can be a really good option IMHO, also because coffee will implement the code to help plugin developers too

[chrisguida]

So I've rethought this and talked it over with @vincenzopalazzo and I think it's best if we require integration tests only, as this is the only type of test that tells us whether CLN upstream has broken the plugin.

Since for now, @vincenzopalazzo does not want to use pytest, the only way to fulfill this requirement is to use some other testing framework, or to remove the plugin from the list.

So @daywalker90 unless you can think of a way to use something other than Python to run integration tests in the plugins repo CI, I think we should archive this plugin for now until a better solution can be found.

[vincenzopalazzo]

if you give me time till june I will implement it in Rust, and for now you can remove it from the list

[chrisguida]

Sounds good, thanks @vincenzopalazzo! I guess this PR can be closed then..