From 3a9766a3f051cac0931e97a8c70e99c18c0ac170 Mon Sep 17 00:00:00 2001 From: Matijs van Zuijlen Date: Sun, 20 Jun 2021 12:19:31 +0200 Subject: [PATCH] Re-organize and improve output check scenarios - Move scenarios for checking success + output to a separate feature - Unify order and implementation of scenarios for checking output, stderr and stdout --- .../check_for_success_and_output.feature | 96 ++++++++ .../command/check_output_of_command.feature | 208 +++--------------- .../command/check_stderr_of_command.feature | 82 ++++--- .../command/check_stdout_of_command.feature | 91 ++++---- 4 files changed, 231 insertions(+), 246 deletions(-) create mode 100644 features/03_testing_frameworks/cucumber/steps/command/check_for_success_and_output.feature diff --git a/features/03_testing_frameworks/cucumber/steps/command/check_for_success_and_output.feature b/features/03_testing_frameworks/cucumber/steps/command/check_for_success_and_output.feature new file mode 100644 index 000000000..531154698 --- /dev/null +++ b/features/03_testing_frameworks/cucumber/steps/command/check_for_success_and_output.feature @@ -0,0 +1,96 @@ +Feature: Checking success and output together + + In order to specify expected output and success + As a developer using Cucumber + I want to use the "it should pass with" step family + + Background: + Given I use a fixture named "cli-app" + + Scenario: Match passing exit status and partial output + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `echo hello world` + Then it should pass with: + \"\"\" + hello + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Match passing exit status and exact output + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `ruby -e 'puts "hello\nworld"; exit 0'` + Then it should pass with exactly: + \"\"\" + hello + world + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Match failing exit status and partial output + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `ruby -e 'puts "hello\nworld"; exit 1'` + Then it should fail with: + \"\"\" + hello + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Match failing exit status and exact output + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `ruby -e 'puts "hello\nworld"; exit 1'` + Then it should fail with exactly: + \"\"\" + hello + world + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Match failing exit status and output with regex + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `ruby -e 'puts "hello\nworld"; exit 1'` + Then it should fail with regex: + \"\"\" + hello\s*world + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Detect output from all processes + Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `echo hello` + And I run `echo good-bye` + Then the stdout should contain exactly: + \"\"\" + hello + good-bye + \"\"\" + """ + When I run `cucumber` + Then the features should all pass diff --git a/features/03_testing_frameworks/cucumber/steps/command/check_output_of_command.feature b/features/03_testing_frameworks/cucumber/steps/command/check_output_of_command.feature index 69bc6920c..e8ae8a79f 100644 --- a/features/03_testing_frameworks/cucumber/steps/command/check_output_of_command.feature +++ b/features/03_testing_frameworks/cucumber/steps/command/check_output_of_command.feature @@ -14,35 +14,11 @@ Feature: All output of commands which were executed Scenario: Run command When I run `echo hello world` Then the output should contain "hello world" + And the output should not contain "good-bye" """ When I run `cucumber` Then the features should all pass - Scenario: Detect absence of inline subset of output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `echo hello world` - Then the output should not contain "good-bye" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Failed detection of inline subset of output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `echo hello world` - Then the output should contain "goodbye world" - """ - When I run `cucumber` - Then the features should not all pass with: - """ - expected "hello world" to string includes: "goodbye world" - """ - Scenario: Detect subset of output with multiline string Given a file named "features/output.feature" with: """cucumber @@ -53,57 +29,73 @@ Feature: All output of commands which were executed \"\"\" hello \"\"\" + And the output should not contain: + \"\"\" + good-bye + \"\"\" """ When I run `cucumber` Then the features should all pass - Scenario: Detect absence subset of output with multiline string + Scenario: Detect exact multiline output Given a file named "features/output.feature" with: """cucumber Feature: Run command Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"'` - Then the output should not contain: + When I run `ruby -e 'puts "hello\nworld"; warn "good-bye"'` + Then the output should contain exactly: \"\"\" + hello + world good-bye \"\"\" """ When I run `cucumber` Then the features should all pass - Scenario: Detect exact multiline output - Given a file named "features/output.feature" with: - """cucumber + Scenario: Detect combined output from normal and interactive processes + Given an executable named "bin/aruba-test-cli" with: + """ + #!/usr/bin/env ruby + + while input = gets do + break if "" == input + puts input + end + """ + And a file named "features/output.feature" with: + """ Feature: Run command Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"'` - Then the output should contain exactly: + When I run `echo hello` + When I run `aruba-test-cli` interactively + And I type "good-bye" + And I type "" + Then the stdout should contain exactly: \"\"\" hello - world + good-bye \"\"\" """ When I run `cucumber` Then the features should all pass - Scenario: Failed detection of exact multi-line output + Scenario: Detect output from named source Given a file named "features/output.feature" with: - """cucumber + """ Feature: Run command Scenario: Run command - When I run `ruby -e 'puts "goodbye\nworld"'` - Then the output should contain exactly: + When I run `echo hello` + And I run `echo good-bye` + Then the output from "echo hello" should contain "hello" + And the output from "echo good-bye" should contain exactly "good-bye" + And the output from "echo hello" should contain exactly: \"\"\" hello - world \"\"\" """ When I run `cucumber` - Then the features should not all pass with: - """ - expected "goodbye\nworld" to output string is eq: "hello\nworld" - Diff: - """ + Then the features should all pass Scenario: Detect exact output with ANSI output Given a file named "features/output.feature" with: @@ -191,121 +183,6 @@ Feature: All output of commands which were executed When I run `cucumber` Then the features should all pass - Scenario: Match passing exit status and partial output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `echo hello world` - Then it should pass with: - \"\"\" - hello - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Match passing exit status and exact output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"; exit 0'` - Then it should pass with exactly: - \"\"\" - hello - world - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Match failing exit status and partial output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"; exit 1'` - Then it should fail with: - \"\"\" - hello - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Match failing exit status and exact output - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"; exit 1'` - Then it should fail with exactly: - \"\"\" - hello - world - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Match failing exit status and output with regex - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `ruby -e 'puts "hello\nworld"; exit 1'` - Then it should fail with regex: - \"\"\" - hello\s*world - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Detect output from all processes - Given a file named "features/output.feature" with: - """cucumber - Feature: Run command - Scenario: Run command - When I run `echo hello` - And I run `echo good-bye` - Then the stdout should contain exactly: - \"\"\" - hello - good-bye - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - - Scenario: Detect combined output from normal and interactive processes - Given an executable named "bin/aruba-test-cli" with: - """ - #!/usr/bin/env ruby - - while input = gets do - break if "" == input - puts input - end - """ - And a file named "features/output.feature" with: - """ - Feature: Run command - Scenario: Run command - When I run `echo hello` - When I run `aruba-test-cli` interactively - And I type "good-bye" - And I type "" - Then the stdout should contain exactly: - \"\"\" - hello - good-bye - \"\"\" - """ - When I run `cucumber` - Then the features should all pass - Scenario: Check size of output Given a file named "features/flushing.feature" with: """cucumber @@ -316,16 +193,3 @@ Feature: All output of commands which were executed """ When I run `cucumber` Then the features should all pass - - Scenario: Detect output from named source - Given a file named "features/output.feature" with: - """ - Feature: Run command - Scenario: Run command - When I run `echo hello` - And I run `echo good-bye` - Then the output from "echo hello" should contain exactly "hello" - And the output from "echo good-bye" should contain exactly "good-bye" - """ - When I run `cucumber` - Then the features should all pass diff --git a/features/03_testing_frameworks/cucumber/steps/command/check_stderr_of_command.feature b/features/03_testing_frameworks/cucumber/steps/command/check_stderr_of_command.feature index 4eebfd64e..e85a9bb45 100644 --- a/features/03_testing_frameworks/cucumber/steps/command/check_stderr_of_command.feature +++ b/features/03_testing_frameworks/cucumber/steps/command/check_stderr_of_command.feature @@ -7,43 +7,75 @@ Feature: STDERR of commands which were executed Background: Given I use a fixture named "cli-app" - Scenario: Detect stderr from each process + Scenario: Detect inline subset of stdout Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `echo hello world` + Then the stdout should contain "hello" + And the stderr should not contain "hello" """ + When I run `cucumber` + Then the features should all pass + + Scenario: Detect subset of stdout with multiline string + Given a file named "features/output.feature" with: + """cucumber Feature: Run command Scenario: Run command - When I run `bash -c 'printf "hello world!\n" >&2'` - And I run `bash -c 'cat >&2 '` interactively - And I type "hola" - And I type "" - Then the stderr should contain: + When I run `ruby -e 'puts "hello\nworld"; warn "good-bye"'` + Then the stdout should contain: \"\"\" - hello world! + hello \"\"\" - And the stderr should contain: + And the stdout should not contain: \"\"\" - hola + good-bye \"\"\" - And the stdout should not contain anything """ When I run `cucumber` Then the features should all pass - Scenario: Detect stderr from all processes + Scenario: Detect exact multiline stdout Given a file named "features/output.feature" with: + """cucumber + Feature: Run command + Scenario: Run command + When I run `ruby -e 'puts "hello\nworld"; warn "good-bye"'` + Then the stdout should contain exactly: + \"\"\" + hello + world + \"\"\" + """ + When I run `cucumber` + Then the features should all pass + + Scenario: Detect combined stderr from normal and interactive processes + Given an executable named "bin/aruba-test-cli" with: + """ + #!/usr/bin/env ruby + + while input = gets do + break if "" == input + puts input + warn input + end + """ + And a file named "features/output.feature" with: """ Feature: Run command Scenario: Run command - When I run `bash -c 'printf "hello world!\n" >&2'` - And I run `bash -c 'cat >&2 '` interactively - And I type "hola" + When I run `ruby -e 'warn "hello"` + When I run `aruba-test-cli` interactively + And I type "good-bye" And I type "" - Then the stderr should contain: + Then the stderr should contain exactly: \"\"\" - hello world! - hola + hello + good-bye \"\"\" - And the stdout should not contain anything """ When I run `cucumber` Then the features should all pass @@ -53,16 +85,16 @@ Feature: STDERR of commands which were executed """ Feature: Run command Scenario: Run command - When I run `bash -c 'printf hello >&2'` - And I run `printf goodbye` - Then the stderr from "bash -c 'printf hello >&2'" should contain "hello" - And the stderr from "bash -c 'printf hello >&2'" should contain exactly "hello" - And the stderr from "bash -c 'printf hello >&2'" should contain exactly: + When I run `ruby -e 'warn "hello"'` + And I run `ruby -e 'warn "goodbye"'` + Then the stderr from "ruby -e 'warn "hello"'" should contain "hello" + And the stderr from "ruby -e 'warn "hello"'" should contain exactly "hello" + And the stderr from "ruby -e 'warn "hello"'" should contain exactly: \"\"\" hello \"\"\" - And the stdout from "bash -c 'printf hello >&2'" should not contain "hello" - And the stderr from "printf goodbye" should not contain "hello" + And the stdout from "ruby -e 'warn "hello"" should not contain "hello" + And the stderr from "ruby -e 'warn "goodbye"" should not contain "hello" """ When I run `cucumber` Then the features should all pass diff --git a/features/03_testing_frameworks/cucumber/steps/command/check_stdout_of_command.feature b/features/03_testing_frameworks/cucumber/steps/command/check_stdout_of_command.feature index 98712ffa7..879097a02 100644 --- a/features/03_testing_frameworks/cucumber/steps/command/check_stdout_of_command.feature +++ b/features/03_testing_frameworks/cucumber/steps/command/check_stdout_of_command.feature @@ -7,81 +7,75 @@ Feature: STDOUT of commands which were executed Background: Given I use a fixture named "cli-app" - Scenario: Match output in stdout - Given an executable named "bin/aruba-test-cli" with: - """bash - #!/usr/bin/env bash - - echo -e "hello\nworld" - """ - And a file named "features/output.feature" with: + Scenario: Detect inline subset of stdout + Given a file named "features/output.feature" with: """cucumber Feature: Run command Scenario: Run command - When I run `aruba-test-cli` + When I run `echo hello world` Then the stdout should contain "hello" - Then the stderr should not contain "hello" + And the stderr should not contain "hello" """ When I run `cucumber` Then the features should all pass - Scenario: Match stdout on several lines - Given an executable named "bin/aruba-test-cli" with: - """bash - #!/usr/bin/env bash - - echo 'GET /' - """ - And a file named "features/output.feature" with: + Scenario: Detect subset of stdout with multiline string + Given a file named "features/output.feature" with: """cucumber Feature: Run command Scenario: Run command - When I run `aruba-test-cli` + When I run `ruby -e 'puts "hello\nworld"; warn "good-bye"'` Then the stdout should contain: \"\"\" - GET / + hello + \"\"\" + And the stdout should not contain: + \"\"\" + good-bye \"\"\" """ When I run `cucumber` Then the features should all pass - Scenario: Detect stdout from each process including interactive ones + Scenario: Detect exact multiline stdout Given a file named "features/output.feature" with: - """ + """cucumber Feature: Run command Scenario: Run command - When I run `printf "hello world!\n"` - And I run `cat` interactively - And I type "hola" - And I type "" - Then the stdout should contain: - \"\"\" - hello world! + When I run `ruby -e 'puts "hello\nworld"; warn "good-bye"'` + Then the stdout should contain exactly: \"\"\" - And the stdout should contain: - \"\"\" - hola + hello + world \"\"\" - And the stderr should not contain anything """ When I run `cucumber` Then the features should all pass - Scenario: Detect stdout from all processes - Given a file named "features/output.feature" with: + Scenario: Detect combined stdout from normal and interactive processes + Given an executable named "bin/aruba-test-cli" with: + """ + #!/usr/bin/env ruby + + while input = gets do + break if "" == input + puts input + warn input + end + """ + And a file named "features/output.feature" with: """ Feature: Run command Scenario: Run command - When I run `printf "hello world!\n"` - And I run `cat` interactively - And I type "hola" + When I run `echo hello` + When I run `aruba-test-cli` interactively + And I type "good-bye" And I type "" - Then the stdout should contain: + Then the stdout should contain exactly: \"\"\" - hello world! - hola + hello + good-bye \"\"\" - And the stderr should not contain anything """ When I run `cucumber` Then the features should all pass @@ -91,16 +85,15 @@ Feature: STDOUT of commands which were executed """ Feature: Run command Scenario: Run command - When I run `printf 'hello'` - And I run `printf 'goodbye'` - Then the stdout from "printf 'hello'" should contain "hello" - And the stdout from "printf 'hello'" should contain exactly "hello" - And the stdout from "printf 'hello'" should contain exactly: + When I run `echo hello` + And I run `echo good-bye` + Then the stdout from "echo hello" should contain "hello" + And the stdout from "echo good-bye" should contain exactly "good-bye" + And the stdout from "echo hello" should contain exactly: \"\"\" hello \"\"\" - And the stderr from "printf 'hello'" should not contain "hello" - And the stdout from "printf 'goodbye'" should not contain "hello" + And the stderr should not contain anything """ When I run `cucumber` Then the features should all pass