Dynamic Test Case Generation

[CozThatsWhy]

We are working on transitioning from our current Mocha implementation of tests to venom, and one of the things we perform is a for each loop with our endpoints that then performs a test case. This way when we add more endpoints, we can add to the array of endpoints to test.

We tried some functionality with Venom, but saw that if we are using a ranged value in a single step will only print the name of the test case and it's status (with a ranged=X) if the verbosity is set to > 1. Setting it to 0 will not print the step at all, and only print the testcase status.

Is there a way that we can generate test cases off a range so we can write one test case but it is created via a range? Or maybe be able to print out the steps of a test case in 0 Verbosity if it is a range (and without the range=X).

Thank you!

[fsamin]

Hello,
I don't understand what you mean by "generate test cases". The "range" feature is available in a testcase to perform teststep over an array, map... https://github.com/ovh/venom#iterating-over-data

I think it can match your use case "This way when we add more endpoints, we can add to the array of endpoints to test." without having to generate testcases.

Also you are talking about the verbosity, can you explain what you would like to see ?

[CozThatsWhy]

Hello,

I believe you are right, it will match our use case to generate multiple tests over the range. So that part is good.

The one thing we would like to see or if there is a way to do it, is to print out the name of the Ranged test (without the range=X) in the text (with the 0 verbose case). So for example:

name: Ranged Test Generation
version: 2
vars:
  data:
    services:
    - service1
    - service2
    - service3	
testcases:
- name: 'Testing Array of Services' 
  steps:
  - name: '{{.value}} Exists'
    range: '{{.data.services}}'
    type: http
    method: GET
    url: 'http://localhost'
    path: '/api/services/{{.value}}'
    parameters:
    headers:
      Accept: 'application/json'
    assertions:
    - result.statuscode ShouldEqual 200
	
Verbose = 0: (current)
Prints:
 Testing Array of Services   PASS
 
Verbose = 0: (ideal or configurable)
Prints:
 Testing Array of Services
   service1 Exists   PASS
   service2 Exists   PASS
   service3 Exists   PASS

Verbose = 1: (current)
Prints:
 Testing Array of Services
   service1 Exists (range=0)  PASS
   service2 Exists (range=1)  PASS
   service3 Exists (range=2)  PASS

Maybe a configuration value or if name is detected print it out?

[yesnault]

with the fix, it's now:

without range and all success

❯ venom run exec.yml
          [trac] writing venom.83.log
 • Exec testsuite (exec.yml)
        • testA PASS
          [info] this a first info (exec.yml:7)
          [info] and a second... (exec.yml:9)
        • testB PASS
        • sleep-1 PASS
        • cat-json PASS
          [info] the value of result.systemoutjson is {"foo":"bar"} (exec.yml:34)
        • verify-default PASS
final status: PASS

without range and some errors:

❯ venom run exec.yml
          [trac] writing venom.84.log
 • Exec testsuite (exec.yml)
        • testA PASS
          [info] this a first info (exec.yml:7)
          [info] and a second... (exec.yml:9)
        • testB FAIL
Testcase "testB", step #0-0: Assertion "result.code ShouldEqual 0" failed. expected: 0  got: 1 (exec.yml:23)
        • sleep-1 PASS
        • cat-json PASS
          [info] the value of result.systemoutjson is {"foo":"bar"} (exec.yml:34)
        • verify-default PASS
final status: FAIL

with range and all ok

❯ venom run verbose-range.yml
          [trac] writing venom.86.log
 • Ranged Test Generation (verbose-range.yml)
        • Testing-Array-of-Services
                • service1 Exists (range=0) PASS
                • service2 Exists (range=1) PASS
                • service3 Exists (range=2) PASS

final status: PASS

with range and some errors:

❯ venom run verbose-range.yml
          [trac] writing venom.82.log
 • Ranged Test Generation (verbose-range.yml)
        • Testing-Array-of-Services
                • service1 Exists (range=0) FAIL
                  Testcase "Testing Array of Services", step #0-0: Assertion "result.code ShouldEqual 0" failed. expected: 0  got: 1 (verbose-range.yml:0)
                • service2 Exists (range=1) FAIL
                  Testcase "Testing Array of Services", step #0-1: Assertion "result.code ShouldEqual 0" failed. expected: 0  got: 1 (verbose-range.yml:0)
                • service3 Exists (range=2) FAIL
                  Testcase "Testing Array of Services", step #0-2: Assertion "result.code ShouldEqual 0" failed. expected: 0  got: 1 (verbose-range.yml:0)

final status: FAIL

[CozThatsWhy]

Looks really good! Can't wait for it to be in!