Change benchmark output to CSV

[real-or-random]

Not crucial but nice to have. It's as readable as the current form for engineers but much easier to use in a spreadsheet if you want to look at the changes, which we do occasionally (that's what benchmarks are for, hm?).

Originally posted by @elichai in #678 (comment)

[gmaxwell]

I really dislike the bitcoin output, I think it's very hard to read and make sense of... if someone does this at least put the most important measurements first. I'm dubious about the value of just bulk running these as most of their performance is a consequence of random code/data alignment and such... they jump around at random unless effort is made to deal with that.

[siv2r]

I see running the bench_internal, bench_sign binaries produce the benchmark output to the terminal. I have the following two approaches in mind for CSV output.

Approach 1: Modify the run_benchmark function to write to a CSV file. So, we will have a file like bench_internal.csv instead of bench_internal binary.

Approach 2: Set the benchmark output format with a ./configure option (default will be terminal output) and create a binary or a CSV file accordingly. The google benchmark tool follows this pattern.

Thoughts?

[real-or-random]

I see running the bench_internal, bench_sign binaries produce the benchmark output to the terminal. I have the following two approaches in mind for CSV output.

There are more binaries, just look bench_*.c files in the src dir.

Approach 1: Modify the run_benchmark function to write to a CSV file. So, we will have a file like bench_internal.csv instead of bench_internal binary.

Approach 2: Set the benchmark output format with a ./configure option (default will be terminal output) and create a binary or a CSV file accordingly. The google benchmark tool follows this pattern.

Thoughts?

I think there's a fundamental misunderstanding here. You'll still need the binaries to produce the output. The question is just if the binaries write a table to the terminal or to a CSV file.

So I think the right approach is to add a boolean parameter/flag to the run_benchmark function such that the caller can choose CSV output. And then the user could specify --output=foo.csv on the command-line when running the binary.

[siv2r]

I think there's a fundamental misunderstanding here. You'll still need the binaries to produce the output. The question is just if the binaries write a table to the terminal or to a CSV file.

Makes sense. I don't know why I thought creating a CSV output was possible without a binary.

So I think the right approach is to add a boolean parameter/flag to the run_benchmark function such that the caller can choose CSV output. And then the user could specify --output=foo.csv on the command-line when running the binary.

Sounds good!

In addition to a command line and CSV output, Will a JSON output format be helpful? (like this)

[real-or-random]

In addition to a command line and CSV output, Will a JSON output format be helpful? (like this)

It'd nice to have if it's not much more work but I think it's a stretch goal.