Skip to content

Missing formats for --list_content #397

Open

Description

@fsimonis

There are currently 2 formats for --list_content:

  1. The human-readable format, great visually finding a test
  2. The dot format, great for visualizing the test-suite

Over the years, I found 2 formats to be missing:

List of test-cases

For simple tools or integration with external test runners like CTest, it would be super useful to have an output listing all testcases in an easy to consume way:

testsuite1/testcase1
testsuite1/testcase2
testsuite2/testcase3

Generating this information from the existing formats is rather tedious.

Use cases:

  • Register testcases as separate tests external test runners, such as CTest
  • Writing custom tools that work only on test cases, such as bisection of test cases, running them with GNU parallel, or running them individually in various docker containers.
  • Use fuzzy search to quickly find a testcase and run it ./test --list_content=LIST | fzf | xargs ./test -t

Detailed machine-readable output

For advanced tools, a detailed dump of the test tree would be useful.
JSON would be suitable as it can easily be generated without special libraries.

{
  "type": "suite",
  "name": "testsuite1",
  "fileName": "...",
  "fileLine": 21,
  "labels": [],
  "children": [
    {
      "type": "case",
      "name": "testcase1",
      "fileName": "...",
      "fileLine": 23,
      "labels": []
    },
    {
      "type": "case",
      "name": "testcase2",
      "fileName": "...",
      "fileLine": 33,
      "labels": []
    }
  ]
}

The DOT format includes some of this information in a less structured way as most of it is encoded in the label.

Use cases:

  • Custom linters
  • Merging the detailed test setup with external data
  • Intermediate format to generate other custom formats

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions