Skip to content

Conversation

yifanmai
Copy link
Collaborator

This is a mathematical calculation benchmark based on ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (Chen ey al., 2022) from the IBM Enterprise benchmark.

@yifanmai yifanmai force-pushed the yifanmai/conv-fin-qa-calc branch from 62f3d69 to d3641dc Compare March 19, 2025 22:10
@yifanmai
Copy link
Collaborator Author

Hi @ryokawajp and @mtake, please review this pull request.

By approving, you acknowledge that:

  • You will be listed as a co-author of the merge commit
  • You agree to license your work under Apache 2.0

Main changes:

  • The scenario has been renamed from ConvFinQA to ConvFinQACalc to clarify that the task is different from the original paper's task. The new task is to output a number, rather than to write code for a calculation.
  • Scenario implementation has been simplified.
  • Scenario description and metadata have been added to the schema.
  • The metric has been added to the schema.
  • The Githash in the source file URL has been pinned.
  • Metrics code has been moved to its own file conv_fin_qa_calc_metrics.py.
  • Additional instructions have been added to help instruction following models.
  • Prompt formatting has been changed to fix inconsistent whitespace formatting.

@yifanmai yifanmai requested review from mtake and ryokawajp March 19, 2025 22:16
Copy link
Collaborator

@ryokawajp ryokawajp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! @yifanmai .
LGTM.

@yifanmai yifanmai merged commit 7f44dd3 into main Mar 25, 2025
8 checks passed
@yifanmai yifanmai deleted the yifanmai/conv-fin-qa-calc branch March 25, 2025 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants