This repository contains the code for the paper "Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation". This paper is the extention work from the poster From Symbolic Logic Reasoning to Soft Reasoning: A Neural-Symbolic Paradigm. We add details for building the large-scale deeper multi-step reasoning dataset PARARULE-Plus and more experiments by adding the PARARULE-Plus in training. The goal is to build up an end-to-end neural based reasoning engine. The existing neural based models lack a reasoning engine, and they are not end-to-end training process. The repository also incorporates extra code for research as part of future work.
We use PARARULE from Allen AI institute. The dataset has been published on Transformers as Soft Reasoners over Language, CONCEPTRULES V1 and CONCEPTRULES V2 from Tim Hartill and we published PARARULE-Plus in this paper. You can download the data from the dataset folder. PARARULE-Plus has also been collected by LogiTorch which is a PyTorch-based library for logical reasoning on natural language. Furthermore, PARARULE-Plus has been merged by OpenAI/Evals and collected by ReasoningNLP and Prompt4ReasoningPapers.
The main body of the code is following with my previous project From Symbolic Logic Reasoning to Soft Reasoning: A Neural-Symbolic Paradigm. We add more experiment analysis on CONCEPTRULES V1, CONCEPTRULE V2, and PARARULE-Plus. You can look and use the code in the source code folder.
If you want more details about this project, watch our presentation recording.
@inproceedings{bao2022multi,
title={Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation},
author={Qiming Bao and Alex Yuxuan Peng and Tim Hartill and Neset Tan and Zhenyun Deng and Michael Witbrock and Jiamou Liu},
booktitle={Proceedings of the 16th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 2nd International Joint Conference on Learning \& Reasoning (IJCLR 2022)},
pages={202-217},
month=sep,
year={2022},
address={Cumberland Lodge, Windsor Great Park, United Kingdom}
}
Thanks to the author of the DeepLogic: Towards End-to-End Differentiable Logical Reasoning, for his advice and help in understanding and reproducing his work. This is of great help to me in completing this research and future research. Also Thanks Tim Hartill who developed CONCEPTRULES V1 and CONCEPTRULES V2 datasets.
Thanks to the help from FacebookAI: fairseq for my replication to the experiment result from Transformers as Soft Reasoners over Language. Here is my replication notes Finetuning RoBERTa on RACE tasks to the Transformers as Soft Reasoners over Language.
[DMN+] Dynamic Memory Networks for Visual and Textual Question Answering https://arxiv.org/abs/1603.01417
[DMN] Ask Me Anything: Dynamic Memory Networks for Natural Language Processing https://arxiv.org/abs/1506.07285
[Hyperas] Bayesian optimization of automated hyperparameter tuning https://github.com/maxpumperla/hyperas
[MemN2N] End-To-End Memory Networks https://arxiv.org/abs/1503.08895
[MemN2N implementation on bAbI tasks with very nice interactive demo] End-To-End Memory Networks for Question Answering https://github.com/14H034160212/MemN2N-babi-python
Memory Networks https://arxiv.org/pdf/1410.3916.pdf