A Group of more Challenge Logical Reasoning Machine Reading Comprehension Datasets with Logical Reasoning Instruction For OpenAI EVAL

[14H034160212]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

Eval details 📑

Eval name

[reclor-logical-reasoning-plus]
[logiqa-logical-reasoning-plus]
[logiqav2-logical-reasoning-plus]

Eval description

[We reorganize the three most representative and hardest logical reasoning machine reading comprehension datasets (ReClor, LogiQA, LogiQA V2). The number of the dataset for reclor-logical-reasoning-plus, logiqa-logical-reasoning-plus and logiqav2-logical-reasoning-plus is 500, 651, and 1569, respectively. We made two changes for the three validation sets, and we committed the changed validation set in this pull request. One is to shuffle the order of options. We find ChatGPT can answer questions if those questions are public on the internet, but it fails when we shuffle the order of options. The other is we replace the correct option using None of the other options are correct. GPT-4 and ChatGPT both fail in this case, especially ChatGPT will find an option that is most similar to the original correct option, but actually, there is no correct option. The model should select None of the other options are correct as the correct option. Here is the tweet link. ]

What makes this a useful eval?

[Logical reasoning ability is a fascinating topic in the NLP community. We hope to see if ChatGPT and GPT4 sheds more light on this topic.]

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

Insert what makes your eval high quality that was not mentioned above. (Not required)

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your yaml is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• I have filled out all required fields in the evals PR form
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: Black Americans are twice as likely to suffer from hypertension as white Americans.The same is true when comparing Westernized black Africans to white Africans.The researchers hypothesized that the reason why westernized black people suffer from hypertension is the result of the interaction of two reasons? one is the high salt content of western foods, and the other is the adaptation mechanism of black genetic genes to the salt-deficient environment . Question: The following conclusions about contemporary westernized African blacks, if the item is true, can it best support the researchers' hypothesis? \nA. A. The unusually high salt intake in certain parts of Africa is a serious problem that threatens the health of residents. \nB. B. The blood pressure of Yoruba people in West Africa is not high.Yoruba people have lived inland far away from sea salt and far away from the Sahara salt mine in Africa. \nC. C. Considering health care, most African whites also pay attention to controlling salt intake. \nD. D. None of the other options are correct. \nAnswer: "}], "ideal": "D", "id_string": "val_0"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: The prohibition of advertising cigarettes on public media does not reduce the number of young people smoking, because they have known for a long time that there are cigarettes in the world, known for various brands of cigarettes, and know where to get them.They do not need advertisements to provide this information. Question: The following, if true, can weaken the above argument most? \nA. A. Anti-smokers have advertised in the public media against smoking from the beginning. \nB. B. None of the other options are correct. \nC. C. Cigarette advertising on public media is a major expense for tobacco companies. \nD. D. Prohibition of cigarette advertisements on public media will cause other forms of cigarette advertisements to proliferate. \nAnswer: "}], "ideal": "B", "id_string": "val_1"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: Three international students are sitting side by side on the bench.Already know? (1) At least one of the two people on the right of the Sudanese student is a French student.(2) There is also a French student among the two people on the left of the French student.(3) At least one of the two people on the left of the boy is a girl.(4) One of the two people on the right of the girl is also a girl. Question: So, for these three people, is the following judgment true? \nA. A. Sudanese boys, French boys, French girls. \nB. B. Sudanese girls and French boys.French boys. \nC. C. Sudanese girls, French girls, French girls. \nD. D. None of the other options are correct. \nAnswer: "}], "ideal": "D", "id_string": "val_2"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: Xiao Wang, Xiao Li and Xiao Zhang are going to climb the mountain.The weather forecast said it might rain today.Around the weather forecast, three people argued.Xiao Wang? \"It may rain today.That does not exclude today or it may not rain.Let's go hiking.Xiao Li?\" It might rain today, which means that it is going to rain today.Let's not go climbing.\"Xiao Zhang?\" It may rain today.It just means that it is not necessary for you to decide whether to rain or not to climb the mountain today.\" Question: Understanding of the weather forecast, out of three people \nA. A. None of the other options are correct. \nB. B. Xiao Zhang is correct, Xiao Wang and Xiao Li are incorrect. \nC. C. Xiao Li is correct, Xiao Wang and Xiao Zhang are incorrect. \nD. D. Xiao Wang is correct, Xiao Li and Xiao Zhang are incorrect. \nAnswer: "}], "ideal": "A", "id_string": "val_3"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: Jupiter is a gas giant planet and the largest planet in the solar system. Its mass is 2.5 times the total mass of the other seven planets in the solar system. Observations have found that most of the more than 70 moons surrounding Jupiter are composed of water ice. Therefore, Jupiter's atmosphere should contain a considerable amount of water. Question: Which of the followings, if true, can best support the above statement? \nA. A. Many of the water in interstellar space exists in gaseous form. \nB. B. Uranus is also a gas giant planet, and it has been confirmed that it contains a lot of water ice. \nC. C. After hundreds of millions of years, the satellite may slowly fall onto the planet. \nD. D. None of the other options are correct. \nAnswer: "}], "ideal": "D", "id_string": 218, "type": null}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: All Anxi people are vegetarians, while all Zhenyuan people are ascetics. Ascetics and vegetarians are like fire and water, and there is no conflict. Guo Shu is an ascetic. Question: Which of the followings can be infered \nA. A. Guo Shu is from Zhenyuan \nB. B. Guo Shu is from Anxi \nC. C. Guo Shu is not from Zhenyuan \nD. D. None of the other options are correct. \nAnswer: "}], "ideal": "D", "id_string": 7261, "type": null}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: The medieval Arabs had many manuscripts of the ancient Greek. When needed, they translate them into Arabic. Medieval Arab philosophers were very interested in Aristotle's Theory of Poetry, which was obviously not shared by Arab poets, because a poet interested in it must want to read Homer's poems. Aristotle himself often quotes Homer's poems. However, Homer's poems were not translated into Arabic until modern times. Question: Which of the following options, if true, strongly supports the above argument? \nA. A. None of the other options are correct. \nB. B. Aristotle's Theory of Poetry is often quoted and commented by modern Arab poets. \nC. C. In Aristotle's Theory of Poetry, most of the content is related to drama, and medieval Arabs also wrote plays and performed them. \nD. D. A series of medieval Arab stories, such as Arab Night, are very similar to some parts of Homer's epic. \nAnswer: "}], "ideal": "A", "id_string": 10471, "type": null}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: In a business whose owners and employees all belong to one family, the employees can be paid exceptionally low wages. Hence, general operating expenses are much lower than they would be for other business ventures, making profits higher. So a family business is a family' s surest road to financial prosperity. Question: The reasoning in the argument is flawed because the argument \nA. A. ignores the fact that businesses that achieve high levels of customer satisfaction are often profitable even if they pay high wages \nB. B. None of the other options are correct. \nC. C. presumes, without providing justification, that only businesses with low general operating expenses can succeed \nD. D. presumes, without providing justification, that family members are willing to work for low wages in a family business because they believe that doing so promotes the family's prosperity \nAnswer: "}], "ideal": "B", "id_string": "val_0"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: It takes a particular talent to be a successful business manager. Business courses can help people to solve management problems, but such courses can do so only for those people with managerial talent. Such people should take business courses to acquire ideas that they can subsequently use to good advantage if management problems happen to arise. Question: If the statements above are true, which of the following must also be true on the basis of them? \nA. A. Those people who have never taken business courses are unable to solve management problems when such problems arise. \nB. B. None of the other options are correct. \nC. C. People who lack managerial talent are more likely to take business courses than are people who have managerial talent. \nD. D. People who are already skilled at solving management problems are unlikely to benefit from business courses. \nAnswer: "}], "ideal": "B", "id_string": "val_1"}
{"input": [{"role": "system", "content": "Instructions: You will be presented with a passage and a question about that passage. There are four options to be chosen from, you need to choose the only correct option to answer that question. If the first option is right, you generate the answer 'A', if the second option is right, you generate the answer 'B', if the third option is right, you generate the answer 'C', if the fourth option is right, you generate the answer 'D'. Read the question and options thoroughly and select the correct answer from the four answer labels. Read the passage thoroughly to ensure you know what the passage entails."}, {"role": "user", "content": "\nPassage: Philosopher: An action is morally good if it both achieves the agent' s intended goal and benefits someone other than the agent. Question: Which one of the following judgments most closely conforms to the principle cited by the philosopher? \nA. A. Colin chose to lie to the authorities questioning him, in an attempt to protect his friends. The authorities discovered his deception and punished Colin and his friends severely. But because he acted out of love for his friends, Colin's action was morally good. \nB. B. Louisa tried to get Henry into serious trouble by making it appear that he stole some expensive clothes from a store. But the store's detective realized what Louisa did, and so Louisa was punished rather than Henry. Since she intended to harm Henry, Louisa's action was not morally good. \nC. C. None of the other options are correct. \nD. D. Ellen worked overtime hoping to get a promotion. The extra money she earned allowed her family to take a longer vacation that year, but she failed to get the promotion. Nevertheless, Ellen's action was morally good. \nAnswer: "}], "ideal": "C", "id_string": "val_2"}

[14H034160212]

reupload the files using lfs and add eval json data examples.

[usama-openai]

Thanks for the contribution. I really liked your approach.

We made two changes for the three validation sets, and we committed the changed validation set in this pull request. One is to shuffle the order of options. We find ChatGPT can answer questions if those questions are public on the internet, but it fails when we shuffle the order of options. The other is we replace the correct option using None of the other options are correct.

However, I would like to request a change in the dataset. In all the samples, options are given as A. A. ...., B. B. ..... Kinldy remove the duplicate option letter from the samples.

We would love to review the PR again after the suggested changes.

[14H034160212]

Hi @usama-openai, Thanks a lot for your reply! I just fix that issue now. Please recheck the request.

[usama-openai]

Thanks for implementing the requested changes. I would like to request a few more changes.

1. As all three evals are related to each other, it is better to define them in a single yaml file. Multiple evals can be defined in a single file. You just need to merge all three yaml files into a single one.
2. Put all the jsonl files in the same directory.

Your final directory structure would look like this:

evals/registry/data/logiqa-logical-reasoning-plus/
    - logiqa-logical-reasoning-plus.jsonl
    - logiqav2-logical-reasoning-plus.jsonl
    - reclor-logical-reasoning-plus.jsonl
evals/registry/evals/
    - logiqa-logical-reasoning-plus.yaml

If you want to, you can also create an "eval set". You can read documentation or look at other eval sets to get a good idea about it.

[14H034160212]

Hi @usama-openai, Thanks a lot for your quick reply and fantastic suggestions! I have reorganized the project and add the eval set. Please have a check.

[usama-openai]

This PR looks in good shape now. I'm approving this PR.