Skip to content

Conversation

yifanmai
Copy link
Collaborator

@yifanmai yifanmai commented Nov 1, 2024

Fixes an error when trying to use aggregation_strategies: ["mean"] in schema.yaml with helm-summarize:

Traceback (most recent call last):
  File "/path/to/crfm-helm/bin/helm-summarize", line 8, in <module>
    sys.exit(main())
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 1315, in main
    summarizer.run_pipeline(skip_completed=args.skip_completed_run_display_json)
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 1223, in run_pipeline
    self.write_groups()
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 1167, in write_groups
    tables: List[Table] = self.create_group_tables_by_metric_group(group)
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 1060, in create_group_tables_by_metric_group
    table = self.create_group_table(
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 983, in create_group_table
    means = compute_aggregate_row_means(table)
  File "/path/to/crfm-helm/lib/python3.10/site-packages/helm/benchmark/presentation/summarize.py", line 279, in compute_aggregate_row_means
    total += float(cell.value)
ValueError: could not convert string to float: 'DBRX Instruct'

If the first column contains the names of models, helm-summarize should skip it instead of treating it like a value column when computing the mean.

@yifanmai yifanmai requested a review from farzaank November 2, 2024 01:09
@yifanmai yifanmai merged commit 119c555 into main Nov 4, 2024
8 checks passed
@yifanmai yifanmai deleted the yifanmai/fix-summarize-mean branch November 4, 2024 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants