New version error: Failed to convert results with adapter rust/rust_bench

[shenxiangzhuang]

The issue

Without change anything in a github action, the bencher ci raise an 400 error: "Message: Failed to convert results with adapter (rust_bench | JsonReportSettings { adapter: Some(RustBench), average: None, fold: None }): Failed to convert results".

I try to fix it by change with a specific rust adapter, but it fails too. Any suggestion to fix this? The CI log is as fellows:

Bencher New Report:
{
  "branch": "fix/ci",
  "end_time": "2024-05-14T10:25:07.505431153Z",
  "hash": "e310db8a939f6a09618d5fbbf5cca1ad2b1ae7b2",
  "results": [
    "running 13 tests\ntest bleu::test::test_add_vectors ... ignored\ntest bleu::test::test_bleu ... ignored\ntest bleu::test::test_stat_agg ... ignored\ntest ngram::test::test_get_ngram_long ... ignored\ntest ngram::test::test_get_ngram_short ... ignored\ntest ngram::test::test_get_token_ngram_long ... ignored\ntest ngram::test::test_get_token_ngram_short ... ignored\ntest tokenizer::test::test_tokenize_13a_regex ... ignored\ntest tokenizer::test::test_tokenize_regex ... ignored\ntest bleu::benchmark::bench_batch_bleu     ... bench:  13,967,756.30 ns/iter (+/- 266,769.64)\ntest bleu::benchmark::bench_bleu           ... bench:     298,794.10 ns/iter (+/- 2,639.98)\ntest ngram::benchmark::bench_ngram         ... bench:      49,480.28 ns/iter (+/- 501.33)\ntest tokenizer::benchmark::bench_tokenizer ... bench:      15,690.73 ns/iter (+/- 6,749.97)\n\ntest result: ok. 0 passed; 0 failed; 9 ignored; 4 measured; 0 filtered out; finished in 4.88s\n"
  ],
  "settings": {
    "adapter": "rust_bench"
  },
  "start_time": "2024-05-14T10:24:30.639097209Z",
  "testbed": "ubuntu-latest"
}

Bencher Report:

Failed to create new report: Error processing request:
Status: 400 Bad Request
Headers: {"content-type": "application/json", "x-request-id": "d7595436-f378-4286-88cd-7f461d0c782d", "content-length": "1182", "date": "Tue, 14 May [202](https://github.com/shenxiangzhuang/bleuscore/actions/runs/8957244547/job/24600098330#step:4:203)4 10:25:07 GMT", "server": "Fly/00e000b6 (2024-05-10)", "via": "2 fly.io", "fly-request-id": "01HXVA4VJHPEFSEN6H36VK61A7-sjc"}
Request ID: d7595436-f378-4286-88cd-7f461d0c782d
Message: Failed to convert results with adapter (rust_bench | JsonReportSettings { adapter: Some(RustBench), average: None, fold: None }): Failed to convert results: running 13 tests

[epompeii]

Hey @shenxiangzhuang sorry you're hitting an error. I'll take a look and let you know what I find!

[epompeii]

@shenxiangzhuang good news!
I figured out what the issue is. It seems like rust version 1.80.0-nightly introduces a change to the way libtest bench formats its results. The metrics are now floats instead of integers: rust-lang/rust#124774

Current format:

test tests::benchmark ... bench:             3,161 ns/iter (+/- 975)

New 1.80.0 format:

test bleu::benchmark::bench_bleu           ... bench:     298,794.10 ns/iter (+/- 2,639.98)

I will have a new release containing the fix out in the next day.

[shenxiangzhuang]

@epompeii Thanks a lot! I'm looking forward to use the new version.

[epompeii]

Fixed in v0.4.9: https://github.com/bencherdev/bencher/releases/tag/v0.4.9