A contrastive dataset for probing commonsense causal reasoning in Large Language Models.
AITIA is a hand-crafted dataset designed to test how well language models genuinely reason about cause and effect, rather than reproducing familiar associations acquired during pretraining. The dataset was written and curated manually by two linguists with expertise in NLP at the University of Pisa, ensuring careful control over meaning, plausibility, and the subtle distinctions between item variants.
The name stands for Assessment of Implausible, Typical and Incongruent causal Associations, and it echoes the Ancient Greek Ξ±αΌ°ΟΞ―Ξ± (aitΓa), meaning "cause" or "reason."
The underlying idea is straightforward: starting from a well-formed cause-and-effect pair, the "cause" side is progressively moved away from the ordinary β first into an equivalent event, then into a highly improbable one, and finally into an event with no causal link at all. A model that truly understands causality should track these shifts consistently.
AITIA is built as an extension of ExpliCa, a dataset of explicit causal and temporal sentence pairs. From ExpliCa we drew 107 causal pairs, retaining those most compatible with our experimental setup.
Each original pair follows a simple causal schema:
- Sentence A β describes a cause
- Sentence B β describes its likely effect
Beginning from each pair, we wrote three additional versions of Sentence A, each holding a different relationship with the same Sentence B. The table below illustrates the four resulting variants, using the original pair "It started raining" (Sentence A) β "The driver turned the wipers on" (Sentence B):
| π·οΈ Variant | π¬ Description | π Causal link with B? | π Example (Sentence A) |
|---|---|---|---|
| π’ Causal (original) | The typical cause taken from ExpliCa | β Yes β plausible and typical | It started raining |
| π΅ Causal-Equivalent | A different event that still brings about B | β Yes β plausible | There was a sudden downpour |
| π£ Causal-Implausible | A fantastical or near-impossible event that would still cause B | A flying whale released water over the city | |
| β« Non-Causal | An event unrelated to B | β No causal link | The clouds turned pink at sunset |
Notice how the Causal-Implausible variant is absurd on its surface, yet β if a whale really did drench the city β the driver reaching for the wipers still makes causal sense. This is intentional: the plausibility of the premise and the validity of the causal link are deliberately pulled apart.
The final dataset comprises 107 quadruples β one Sentence B paired with four Sentence A variants each β for a total of 428 sentences.
By walking each item across a spectrum β from a typical cause, to an equivalent one, to an implausible one, and finally to no cause at all β AITIA makes it possible to assess whether a model can:
- Recognise genuine causal relationships
- Generalise a causal link to a similar event
- Maintain consistency in its causal reasoning even when the premise ceases to be plausible or ordinary
This design helps distinguish models that grasp an abstract notion of causality from those that simply rely on events they have already encountered many times.
AITIA was introduced in our paper (reference forthcoming).
If you use this dataset, please cite both AITIA and the ExpliCa dataset it builds upon:
- ExpliCa β github.com/Unipisa/explica
(BibTeX entries will be added here upon publication.)
Curated at the CoLingLab of University of Pisa. For questions, issues, or collaborations, please open an issue.