ProbeGrammarRobustness

Source code for our ACL2020 paper: On the Robustness of Language Encoders against Grammatical Errors

Dependencies

Python >= 3.5

Download and install berkeleyparser.

Install python requirments via requirments file: pip install -r requirements.txt

Preparation

Download datasets

The General Language Understanding Evaluation (GLUE) benchmark aims to analyze model ability in natural language understanding. We use some tasks of GLUE as downstream tasks.

You should follow the instructions in this repo to download GLUE benchmark and unpack it to your $data_dir.

The CoNLL-2014 Shared Task: Grammatical Error Correction is where we collect error statistics.

Follow the instructions in this page to download NUCLE Release3.2 and annotated test data.

Remember to change the file path in line 13 and 141 of utils/statistic_all.py to your own path.

Download pre-trained models

For experiments regarding Infersent, you need to download fastText embeddings and the corresponding pre-trained Infersent model.

curl -Lo crawl-300d-2M.vec.zip https://s3-us-west-1.amazonaws.com/fasttext-vectors/crawl-300d-2M.vec.zip
curl -Lo examples/infersent2.pkl https://dl.fbaipublicfiles.com/senteval/infersent/infersent2.pkl

Usage

Downstream task evaluations

The framework in this repo allows evaluating BERT, RoBERTa, and Infersent on MRPC, MNLI, QNLI, and SST-2. We will provide an example of evaluating bert-base-uncased on MRPC dataset.

The follows can be done with bash run_trans_case.sh . But let's elaborate it step by step. The script for Infersent is run_infer_case.sh .

First train or fine-tune models on clean data (${data_dir} indicates where you store data):

python run_transformers.py --mode fine-tune --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

To inject grammatical errors using adversarial attack algorithms, you need to assign the importance score to each token (not necessary for genetic algorithm):

python run_transformers.py --mode score --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

then, run the attack algorithms, --adv_type can be greedy, beam_search or genetic:

python run_transformers.py --mode attack  --adv_type greedy --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC

To inject grammatical errors based on berkeleyparser (the probabilistic case in our paper), you need to first obtain the syntactic parse tree for each sentence in the dataset. Then run:

python generate_error_sent_all.py csv --input_tsv ${data_dir}/MRPC/dev.tsv --parsed_sent1 ${data_dir}/MRPC/parsed_sent1 --parsed_sent2 ${data_dir}/MRPC/parsed_sent2 --output_tsv ${data_dir}/MRPC/mrpc.tsv --rate 0.15

then, test the model under the probabilistic case:

python run_transformers.py --mode attack --adv_type random --random_attack_file ${data_dir}/MRPC/mrpc.tsv --target_model bert --model_name_or_path bert-base-uncased --do_lower_case --data_dir ${data_dir}/MRPC --data_sign MRPC 

Note that our framework is flexible. If you want to test new models, you can simply add a new class in attack_agent.py like what we did ( See attack_agent.py for detials, the new class mainly tells attack algorithms how to construct and forward a new instance with the tested model):

class infersent_enc(object):
    def __init__(self, infersent, config):
        self.infersent = infersent
        self.config = config

    def make_instance(self, text_a, text_b, label, label_map):
        sent1s = [' '.join(text_a)]
        if isinstance(text_b, list):
            sent2s = [' '.join(text_b)]
        else:
            sent2s = [text_b]
        return [sent1s, sent2s, [label]]

    def model_forward(self, model, batch):
        sent1s, sent2s, label_ids = [list(item) for item in batch]
        sent1_tensor = self.infersent.encode(sent1s, tokenize=True)
        sent2_tensor = self.infersent.encode(sent2s, tokenize=True)
        ...
        return logits

Model layer evaluations

Navigate to the probing directory:

cd probing

Run:

bash run_selfattn_test.sh

The input data contains three columns seperated by /t, where the first column indicates the data split tr, va or te; the second column indicates the binary correctness of this example 0, 1; the third column is the sentence. For example:

tr      1       It knows we want to eat a doughnut , not drink it .

BERT masked language model evaluations

We provide some examples for a quick check.

Navigate to the probing directory:

cd probing

Run:

bash run_mlm_dis.sh

To collect the complete data, navigate to the examples directory and run:

python collect_data.py --type *error_type*

where *error_type* can be Prep, ArtOrDet, Wci, Trans, Nn, SVA, Vform.

Acknowledgement

Our framework is developed based on PyTorch implementations of BERT and RoBERTa from PyTorch-Transformers, Infersent from SentEval, and ELMo from AllenNLP and Jiant.

We also borrowed and edited code from the following repos: nlp_adversarial_examples, nmt_grammar_noise, interpret_bert.

We would like to thank the authors of these repos for their efforts.

Citation

If you find our work useful, please cite our ACL2020 paper: On the Robustness of Language Encoders against Grammatical Errors

@inproceedings{yin2020robustnest,
  author = {Yin, Fan and Long, Quanyu and Meng, Tao and Chang, Kai-Wei},
  title = {On the Robustness of Language Encoders against Grammatical Errors},
  booktitle = {ACL},
  year = {2020}
}