Convert from Litmus to Cucumber #89

ekohl · 2023-02-23T16:36:51Z

This switches away from Litmus because it's the wrong tool. Cucumber can easily run locally without having to spin up a completely separate machine. It also doesn't create a circular dependency (because puppetlabs_spec_helper depends on puppet-lint).

This works locally, but I don't know what CI runs. It also has one TODO, with the version expectation.

ekohl · 2023-02-23T16:41:00Z

Rakefile

 end
+
+task default: [:spec, :features]


Note there is no rubocop because it's not defined, despite including rubocop/rake_task on line 2. Ideally that should actually be there.

ekohl · 2023-02-23T16:41:38Z

Rakefile

-desc 'Run acceptance tests'
-task :acceptance do
-  Rake::Task['litmus:acceptance:localhost'].invoke
+Cucumber::Rake::Task.new(:features) do |t|


Note that right now this isn't run in CI because that explicitly calls spec.

smortex · 2023-02-23T18:53:00Z

.github/workflows/ci.yml

@@ -25,17 +25,3 @@ jobs:
    secrets: "inherit"


Not sure about your comment:

This works locally, but I don't know what CI runs.

The run commands are from the line just above this comment (github does not allow me to comment on the right line). This "expand" to:

https://github.com/puppetlabs/cat-github-actions/blob/main/.github/workflows/gem_ci.yml

Overall 💯 🤩 for cucumber AFAIAC!

ekohl · 2023-02-23T18:59:09Z

features/manifest.feature

+    Then it has 1 error and 0 warnings
+
+    Scenario: containing a control statement
+      Given a file named "ignore.pp" with:


I first did have a version which loaded fixtures, but decided to inline things here for now.

I think it is more readable to have the content here 💯 .

smortex · 2023-02-23T22:43:42Z

features/step_definitions/stepdefs.rb

@@ -0,0 +1,7 @@
+Then('it has {int} error/errors and {int} warning/warnings') do |error, warning|


to follow cucumber style convention, we should use conditional here and use the point of view of the user (I run, ...). I also think the foo/bar syntax is less suited than foo(s) here:

Suggested change

Then('it has {int} error/errors and {int} warning/warnings') do |error, warning|

Then('I should see {int} error(s) and {int} warning(s)') do |error, warning|

I looked at the style, but first tried regexes. That didn't work, but this looks cleaner.

Hehe, I should probably have quoted "style" because there is no real reference / ultimate source of truth that I am aware of: much more experience that piles up with the years 🤷 (even if I only do some cucumber 3 of 4 times a year).

I don't remember which version of gherkin changed the game and allowed this flexible syntax instead of bare regular expressions.

Anyway, the doc points to this repository that is handy:
https://github.com/cucumber/cucumber-expressions

smortex · 2023-02-23T22:44:49Z

features/manifest.feature

+      "test" # lint:ignore:double_quoted_strings
+      """
+      When I run `puppet-lint ignore.pp`
+      Then it has 0 errors and 0 warnings


then we can update the manifest:

Suggested change

Then it has 0 errors and 0 warnings

Then I should see 0 errors and 0 warnings

ekohl · 2023-10-03T10:17:35Z

RuboCop failures should be fixed by #157.

david22swan · 2024-02-16T11:43:25Z

Not sure that I see the issue behind this in regard to the localhost, Litmus is running the test on the local machine as shown here:

puppet-lint/Rakefile

Line 44 in d0e5560

Rake::Task['litmus:acceptance:localhost'].invoke

And as to the circular dependency, while I can somewhat see this, adding an additional testing framework for this one instance seems to us to be to be to much of an increased overhead in order to fix something that does not seem to be impacting anything.

If we are missing anything in regards to this situation please point it out and we can take another look, but at this moment it does not seem to be of enough benefit to match the costs.

ekohl · 2024-02-16T12:46:13Z

I don't want Litmus running on my machine because it uses Bolt, which has opt-out analytics (https://www.puppet.com/docs/bolt/latest/analytics.html). So out of principle I will never use Litmus. It's developer hostile to me. That is a big motivation for me to move to Cucumber.

bastelfreak · 2024-02-16T12:55:37Z

I totally agree with @ekohl here.

davidsandilands · 2024-02-16T15:58:25Z

I guess this is difficult for us, we ultimately need to make informed decisions on how to develop on our products and can't do that without analytics.

It doesn't sound like detailing the data and purpose would help here but it may be worth while us discussing with bolt team because the PDK had been on google analytics before the version upgrade meant this wasn't useable for applications as it was

davidsandilands · 2024-02-16T17:02:19Z

So just to confirm Bolt is now in the same place as PDK and google analytics isn't available

ekohl · 2024-02-16T18:19:36Z

I guess this is difficult for us, we ultimately need to make informed decisions on how to develop on our products and can't do that without analytics.

I think this is a strong statement that needs proper argumentation.

I'm in the position that I rather maintain an alternative development stack in Vox Pupuli than use the Puppet tooling because of analytics. So I'm not touching PDK, Bolt and Litmus but rather maintain modulesync and Beaker. It's true I largely make sure my own workflows work well, which is different from running a product.

One of the results of Puppet's choice to mandate analytics is that the user base is split into two camps. It's one way to develop a product, but is it the best?

bastelfreak · 2024-02-17T12:10:14Z

I guess this is difficult for us, we ultimately need to make informed decisions on how to develop on our products and can't do that without analytics.

As a maintainer I understand this, but I still think it should be opt-in, not opt-out.

davidsandilands · 2024-07-05T10:33:47Z

Hello things have moved on in this debate @ekohl @bastelfreak @smortex and we have removed google analytics from PDK and other products since it was no longer valid and was a community issue. As I go into building out the full q3 plan for Bolt, is it a safe assumption that if bolt similarly removes analytics this is no longer an issue?

We will review new analytics in the future but from my discussions with legal an opt in model would be our approach in future implementations.

bastelfreak · 2024-07-05T11:11:41Z

Hi @davidsandilands , thanks for the update!

discussions with legal an opt in model would be our approach

Thanks, I appreciate that.

if bolt similarly removes analytics this is no longer an issue

As @ekohl mentioned in the initial PR description: I think litmus is the wrong tool. It just makes the test setup more complex and this complexity isn't required. Also what's the future of litmus? I don't think it makes sense to maintain litmus and beaker and the last statement from Josh was that all the puppet agent/PE related pipelines won't move away from beaker.

ekohl · 2024-07-05T11:16:54Z

As @bastelfreak said, I appreciate removing the telemetry but Litmus is a very heavy weight solution that just isn't needed here. You don't need a full machine just to run Ruby application code.

This switches away from Litmus because it's the wrong tool. Cucumber can easily run locally without having to spin up a completely separate machine. It also doesn't create a circular dependency (because puppetlabs_spec_helper depends on puppet-lint).

danadoherty639 · 2024-10-11T10:53:46Z

@ekohl following on from our conversation at community triage we don't have the resources to implement this right now given the impact this would have across our estate. Thanks

bastelfreak · 2024-10-11T12:13:58Z

@danadoherty639 can you please elaborate on the 'impact across your estate'?

ia-content added the community label Feb 23, 2023

ekohl mentioned this pull request Feb 23, 2023

(CONT-339) Update CI for ruby 2.5 #88

Merged

ekohl force-pushed the move-to-cucumber branch from 997725b to 9a64917 Compare February 23, 2023 16:40

ekohl commented Feb 23, 2023

View reviewed changes

smortex reviewed Feb 23, 2023

View reviewed changes

ekohl commented Feb 23, 2023

View reviewed changes

smortex reviewed Feb 23, 2023

View reviewed changes

ekohl mentioned this pull request Oct 2, 2023

(CAT-1434) Update README #156

Merged

5 tasks

ekohl force-pushed the move-to-cucumber branch from 9a64917 to b645225 Compare October 2, 2023 14:48

ekohl marked this pull request as ready for review October 3, 2023 10:17

ekohl requested review from bastelfreak and a team as code owners October 3, 2023 10:17

ekohl force-pushed the move-to-cucumber branch from b645225 to fd8ec30 Compare October 5, 2023 09:21

Convert from Litmus to Cucumber

046c9a0

This switches away from Litmus because it's the wrong tool. Cucumber can easily run locally without having to spin up a completely separate machine. It also doesn't create a circular dependency (because puppetlabs_spec_helper depends on puppet-lint).

ekohl force-pushed the move-to-cucumber branch from fd8ec30 to 046c9a0 Compare July 5, 2024 11:17

ekohl requested a review from a team as a code owner July 5, 2024 11:17

danadoherty639 closed this Oct 11, 2024

		@@ -0,0 +1,7 @@
		Then('it has {int} error/errors and {int} warning/warnings') do \|error, warning\|

	Then('it has {int} error/errors and {int} warning/warnings') do \|error, warning\|
	Then('I should see {int} error(s) and {int} warning(s)') do \|error, warning\|

	Then it has 0 errors and 0 warnings
	Then I should see 0 errors and 0 warnings

Convert from Litmus to Cucumber #89

Convert from Litmus to Cucumber #89

Uh oh!

Conversation

ekohl commented Feb 23, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekohl commented Oct 3, 2023

Uh oh!

david22swan commented Feb 16, 2024

Uh oh!

ekohl commented Feb 16, 2024

Uh oh!

bastelfreak commented Feb 16, 2024

Uh oh!

davidsandilands commented Feb 16, 2024

Uh oh!

davidsandilands commented Feb 16, 2024

Uh oh!

ekohl commented Feb 16, 2024

Uh oh!

bastelfreak commented Feb 17, 2024

Uh oh!

davidsandilands commented Jul 5, 2024

Uh oh!

bastelfreak commented Jul 5, 2024

Uh oh!

ekohl commented Jul 5, 2024

Uh oh!

danadoherty639 commented Oct 11, 2024

Uh oh!

bastelfreak commented Oct 11, 2024

Uh oh!

Uh oh!