Evaluating Text Representations on Lexical Composition

Dependencies

• Python 3
• argparse
• allennlp (0.8.1)

Downloading Data:

Download the pre-trained models using bash download.sh.

The VPC classification and LVC classification tasks need a copy of the BNC corpus. Please download the XML version from here, and update its path in the JSON files.

Once you do, you will need to extract the sentences themselves:

python preprocessing/get_sentences_from_bnc.py \ 
    [/path/to/corpora]/bnc/2554/download/Texts/ \ 
    diagnostic_classifiers/data/vpc_classification/ \ 
    diagnostic_classifiers/data/vpc_classification

Running experiments:

To train all the models for a given task, e.g. NC literality, run:

bash diagnostic_classifiers/experiments/nc_literality/train.sh 

To evaluate:

bash diagnostic_classifiers/experiments/nc_literality/evaluate.sh

To get the predictions for the test set:

bash diagnostic_classifiers/experiments/nc_literality/predict.sh

Adding a new task:

You will need to create a directory under experiments with the JSON files specifying the architecture and hyper-parameters. Each model requires a DatasetReader, Model, and a Predictor. You can use the ones implemented in this repository or implement new ones according to the specific model's needs.

See the AllenNLP tutorial for additional instructions on configuring models.

If you'd like to create new data, follow the preprocessing instructions.

Adding a new representation:

You will need to implement a new TokenIndexer and TokenEmbedder or TextFieldEmbedder. The first takes a sequence of words and returns their IDs, and the second gets the IDs and returns the vectors. Look at the implementations in this repository and in the AllenNLP repository, and read the documentation there.

You will also need to add a JSON file for the task + representation combination and add the command to the train/evaluate/predict bash files.

Citation

Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Vered Shwartz and Ido Dagan. arXiv 2019.