Add Semantic Similarity with BERT example #205

MohamadMerchant · 2020-08-15T10:52:43Z

No description provided.

fchollet

Thank you for the PR! This will make a great new example 👍

fchollet · 2020-08-16T00:50:14Z

examples/nlp/semantic_similarity_with_bert.py

+    with strategy.scope():
+        model = build_model()
+    print(f"Strategy: {strategy}")
+except:


The try-except should not be necessary. Why would it fail?

I thought it will fail while training on CPU but it doesn't. So I have removed try-except block as it doesn't seem necessary.

fchollet · 2020-08-19T19:52:53Z

examples/nlp/semantic_similarity_with_bert.py

+Description: Natural Language Inference by Fine tuning BERT model on SNLI Corpus.
+"""
+"""
+## **Introduction**


No need for the bold markers, it's already a title

Removed bold markers.

fchollet · 2020-08-19T19:53:35Z

examples/nlp/semantic_similarity_with_bert.py

+print(f"Total test samples: {valid_df.shape[0]}")
+
+"""
+  Dataset Info:


Please format this block as markdown

fchollet · 2020-08-19T19:54:26Z

examples/nlp/semantic_similarity_with_bert.py

+## Preprocessing
+"""
+
+# we have some nan in our train data, we will simply drop them


Please make sure that all comments start with a capital letter and end with a period

Done with all the comments.

fchollet · 2020-08-19T19:56:31Z

examples/nlp/semantic_similarity_with_bert.py

+
+def build_model():
+    """
+    model inputs:


Docstring format: start with a one-line description. Then "Arguments:" section.

Changed it to comment descriptions.

fchollet · 2020-08-19T20:14:15Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+## Train the Model
+"""
+h = model.fit_generator(


Use full-spelled argument names

Changed all argument/variable names as full-spelled

fchollet · 2020-08-19T20:14:39Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+
+
+class DataGenerator(tf.keras.utils.Sequence):


Please use a more descriptive name than "DataGenerator"

Used BertSemanticDataGenerator.

fchollet · 2020-08-19T20:15:38Z

examples/nlp/semantic_similarity_with_bert.py

+
+"""
+
+"""


"Check results on some example sentence pairs"

Added line.

fchollet · 2020-08-19T20:16:17Z

examples/nlp/semantic_similarity_with_bert.py

+        shape=(MAXLEN,), dtype=tf.int32, name="tt_ids"
+    )
+    # Loading pretrained BERT model
+    bertModel = TFBertModel.from_pretrained(MODEL)


Use lower case camel case for variable names

Used lower_case for all variable names.

fchollet · 2020-08-19T20:16:36Z

examples/nlp/semantic_similarity_with_bert.py

+LR = 3e-5
+np.random.seed(42)
+# we will use base-base-uncased pretrained model
+MODEL = "bert-base-uncased"


"MODEL" is too generic, this is "pretrained_model_name"

removed "MODEL" from configurations and added description comment while loading.

MohamadMerchant · 2020-08-20T09:51:03Z

Thanks for the review @fchollet . I have made all the necessary changes.

fchollet

Thanks for the update!

fchollet · 2020-08-20T21:20:44Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+
+"""
+## Setup


Please mention that transformers should be installed.

Mentioned to install transformers in Setup description.

fchollet · 2020-08-20T21:21:02Z

examples/nlp/semantic_similarity_with_bert.py

+import pandas as pd
+import tensorflow as tf
+import transformers
+from transformers import BertTokenizer, TFBertModel


You don't need this import line, you can just access these objects from transformers.

Removed this line.

fchollet · 2020-08-20T21:23:46Z

examples/nlp/semantic_similarity_with_bert.py

+batch_size = 32
+epochs = 4
+learning_rate = 3e-5
+np.random.seed(42)


It is unclear what this seed is meant to control. It will have no effect over Keras reproducibility. If it is specifically for the data shuffling, prefer passing a seed argument to your Sequence object and use it locally there (via a RandomState object). In general avoid global seeding since it has side effects.

Yes, it is mentioned specifically for data shuffling.
If it's not a problem, I have changed and used static value as seed in RandomState object in Sequence.

fchollet · 2020-08-20T21:26:50Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+Distribution of our training and validation targets.
+"""
+print(train_df.similarity.value_counts())


Use string formatting to print a sentence that makes it clear this is training (resp. validation) data

Added string to separate both training and validation data.

fchollet · 2020-08-20T21:27:08Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+
+# We have some nan in our train data, we will simply drop them.
+print(train_df.isnull().sum())


Use string formatting to print a sentence

Added string.

fchollet · 2020-08-20T21:28:32Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+history = model.fit_generator(
+    train_data,
+    steps_per_epoch=len(train_data) // batch_size,


Since a Sequence has a __len__, this argument should not be necessary (same for validation_steps)

I have experimented with steps_per_epoch but removing it increases training time drastically.

experiment 1: With steps_per_epoch whole training data took around 1 hour for training 4 epochs.
Train Samples : 549661
Train Steps: 536 (549661 // 32) // 32.

experiment 2 : Without steps_per_epoch 100k training samples took around 1 hour to finish only 1 epoch.
Train Samples : 100000
Train Steps: 1560 (100000 // 64)

So I have used it for faster training as without steps_per_epoch was taking a lot of time and increasing batch_size more than 64 causes OOM errors.

len(train_data) is already num_sentences // batch_size.

Here you are "training faster" simply because you are training on 32x fewer batches per epoch (one batch out of 32).

The proper procedure would be to train on all batches at each epoch (no steps_per_epoch arg), and train for fewer epochs.

Since you are only training with 4 epochs, it sounds like your model is too large for your dataset. Consider adding more dropout. Is bert_model supposed to be trainable?

Thanks for the explanation, I got the point.

I have made changes and trained a model on 100k train samples (with comment description) and no steps_per_epoch argument on two epochs with higher dropout(0.3) and it outperformed all the earlier models I used.

I have trained it with setting bert_model trainable to False, and also by stacking some dense layers, but in both the cases the model didn't improve and was stuck at ~35% accuracy. I think huggingface has more large models pretrained on MNLI such as roberta-large-mnli but those are too large models for our dataset.

fchollet · 2020-08-20T21:29:08Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+## Inference on custom sentences
+"""
+labels = ["CONTRADICTION", "ENTAILMENT", "NEUTRAL"]


This is a kind of global constant, so you can move it higher up in the file

Shifted labels to "Configurations" section with a comment description.

MohamadMerchant · 2020-08-21T16:04:56Z

I have made a commit of all the changes except steps_per_epoch as mentioned in the comment. Please let me know what we can do about it.

fchollet

Thanks for the update. It looks good! 👍

The main remaining issue is the training process. Please try feature extraction first, then see how much gains come from fine-tuning.

Note that I have pushed some copyedits. Please pull them first before editing.

fchollet · 2020-08-24T20:02:52Z

examples/nlp/semantic_similarity_with_bert.py

+
+    def on_epoch_end(self):
+        # Shuffle indexes after each epoch if shuffle is set to True.
+        self.indexes = np.arange(len(self.sentence1))


You can move this line inside the if

Moving this line inside if will result in error message when shuffle=False as the validation and test generator objects cannot find self.indexes in this line in __getitem__
indexes = self.indexes[idx * self.batch_size : (idx + 1) * self.batch_size]

Since it is used in __getitem__, you should define self.indexes in __init__.

fchollet · 2020-08-25T02:26:01Z

examples/nlp/semantic_similarity_with_bert.py

+        return len(self.sentence1) // self.batch_size
+
+    def __getitem__(self, idx):
+        # Generates batch of data.


You can be more precise: "Retrieves the batch of index idx."

fchollet · 2020-08-25T02:43:30Z

examples/nlp/semantic_similarity_with_bert.py

+"""
+## Train the Model
+"""
+history = model.fit_generator(train_data, validation_data=valid_data, epochs=epochs,)


A big problem here is that you're not doing fine-tuning properly.

To do fine-tuning, you need 2 steps:

Freeze bert_model and fit the model. This is called "feature extraction": you just reuse the pretrained features without modifying them.

unfreeze bert_model and fit again (with a very small learning rate). This is called fine-tuning.

Step 2 may not be necessary -- for many applications, feature extraction is sufficient.

Also, please note that:

You can pass a generator to fit. fit_generator is legacy.

If you fit from a generator, you should generally use data multiprocessing to get better GPU utilization. See the docs for fit().

I have used only "feature extraction" and trained by freezing bert_model that gave me accuracy around 37%.
After some tuning, I experimented stacking a classification head below freezed bert_model :

stacking 2 dense layers with dropout gave accuracy around 45%.

stacking 1 Bi-LSTM layer with dropout gave accuracy around 69%.

Accuracy of unfreeze bert_model (our current model) is around 87%.

I have gone through this paper yesterday as they have experimented with different datasets and models, I found this result related to our task.

Frozen up to MNLI MNLI-mm 0th 84.2 82.0 9th 82.0 82.4 12th 56.4 57.1

I have changed fit_generator to fit and also used multiprocessing for better GPU utilization.

It makes sense that you get better results by training the entire BERT model, but in this case it makes little sense to load a pretrained model (since you lose its representations almost immediately due to the presence of untrained layers in the model).

Either load the BERT model from scratch (no pretrained weights), or do actual fine-tuning (the two steps I described above). The first option is the simplest.

I got your point for the first time but I was little bit concerned about the accuracy. Now, the model is trained on both the steps "feature extraction" and "fine-tuning" (unfreeze bert_model) that gave quite good results. Thank you for your great advice.
I have also made a change in data generator and used batch_encode_plus that has speed up the training.

fchollet · 2020-08-25T02:43:46Z

examples/nlp/semantic_similarity_with_bert.py

+    batch_size=batch_size,
+    shuffle=False,
+)
+model.evaluate_generator(test_data, verbose=1)


Likewise, just evaluate

fchollet · 2020-08-27T04:15:49Z

examples/nlp/semantic_similarity_with_bert.py

+    validation_data=valid_data,
+    epochs=epochs,
+    use_multiprocessing=True,
+    workers=-1,


Using 4 or 8 causes nondeterministic deadlocks and doesn't start the training.
(-1 uses 8 workers in colab.)

fchollet

Thank you for the update, it looks good to me!

I've made some copyedits, please pull them first.

Everything looks great, so you can add the generated files, now. Thanks for the great contribution!

MohamadMerchant · 2020-08-31T16:47:25Z

Thank you for the great review, It was great contributing to Keras.

Adding semantic similarity example

e46e18d

fchollet reviewed Aug 19, 2020

View reviewed changes

added all the necessary changes

487cccf

fchollet reviewed Aug 20, 2020

View reviewed changes

added new necessary changes

ff28bf4

MohamadMerchant and others added 4 commits August 23, 2020 13:08

removed step_per_epoch

d179aa9

removed steps_per_epoch

0e920ce

reduced no. of epochs

d23d700

Copediting

8cb6e10

fchollet reviewed Aug 25, 2020

View reviewed changes

added mutiprocessing in fit

11ddfac

fchollet reviewed Aug 27, 2020

View reviewed changes

MohamadMerchant and others added 2 commits August 30, 2020 23:58

updated fine-tuning

11ef530

Copyediting

85d3f75

fchollet approved these changes Aug 30, 2020

View reviewed changes

added generated files

0b31627

fchollet merged commit e59be23 into keras-team:master Aug 31, 2020

Add Semantic Similarity with BERT example #205

Add Semantic Similarity with BERT example #205

Conversation

MohamadMerchant commented Aug 15, 2020

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant commented Aug 20, 2020

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant Aug 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant Aug 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant commented Aug 21, 2020

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant Aug 26, 2020 • edited Loading

Choose a reason for hiding this comment

fchollet Aug 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MohamadMerchant Aug 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

MohamadMerchant commented Aug 31, 2020

MohamadMerchant Aug 21, 2020 •

edited

Loading

MohamadMerchant Aug 23, 2020 •

edited

Loading

MohamadMerchant Aug 26, 2020 •

edited

Loading

fchollet Aug 27, 2020 •

edited

Loading

MohamadMerchant Aug 30, 2020 •

edited

Loading