Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models

This is the github repo for our ACL 2024 paper: Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models, (Link).

The included Python file replicates the results in Section 3.4 of the paper: Generalizing to other language models, which shows how FaR shows good calibration over various open-source language models, with StrategyQA as the data and Token. Prob is the confidence extraction method.

Running the code mainly requires installing vLLM. Hugginface_hub is required for models with restricted usage. Do not forget to add the desired model to your token in the Hugging Face settings.

The authors are working on preparing an easy-to-use github repository. Do not hesitate to send an email if you need the raw code.

Citation

  @inproceedings{zhao-etal-2024-fact,
      title = "Fact-and-Reflection ({F}a{R}) Improves Confidence Calibration of Large Language Models",
      author = "Zhao, Xinran  and
        Zhang, Hongming  and
        Pan, Xiaoman  and
        Yao, Wenlin  and
        Yu, Dong  and
        Wu, Tongshuang  and
        Chen, Jianshu",
      editor = "Ku, Lun-Wei  and
        Martins, Andre  and
        Srikumar, Vivek",
      booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
      month = aug,
      year = "2024",
      address = "Bangkok, Thailand and virtual meeting",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/2024.findings-acl.515",
      pages = "8702--8718",
  }

Others

If you have any other questions about this repo, you are welcome to open an issue or send me an email, I will respond to that as soon as possible.