Skip to content

Make parsers self-documenting #2346

Open
@karkhaz

Description

@karkhaz

All modules under parsers should have a public get_metrics method that returns a dict documenting what metrics that parser supports. That method should also be used when constructing that parser's actual output. The parser should also have another method that returns some help text.
This should all be rendered into a summary in the help text of benchcomp run, like this:

$ benchcomp run -h
# ...
You can use the following parsers for the `parser` key of each suite:

kani_perf
=========
Parses the output files after running the kani performance regression suite.
== Available metrics ==
* runtime
* symex_runtime
* solver_runtime
* passed

litani_cbmc
======
Parses the output of a Litani run of CBMC proofs using the AWS CBMC Starter kit.
== Available metrics ==
* runtime
* symex_runtime
* solver_runtime
* memory_kiB
* passed```

Metadata

Metadata

Assignees

Labels

Z-BenchcompIssues related to the benchcomp performance comparison script

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions