Text Classification with FNet [KerasNLP] #898

abheesht17 · 2022-06-01T19:48:01Z

Dataset: IMDb
Compared results with Transformer model

mattdangerw

Thanks! Looks good. Left some initial comments.

mattdangerw · 2022-06-01T21:06:38Z

guides/keras_nlp/fnet_classification.py

+def preprocess_dataset(dataset):
+    dataset = dataset.map(
+        lambda x: {
+            "sentence": tf.strings.lower(x["sentence"]),


Can't we lowercase inside the tokenizer? Why do that here?

It lowercases the special tokens as well, which I wanted to avoid.

mattdangerw · 2022-06-01T21:09:39Z

guides/keras_nlp/fnet_classification.py

+we have. WordPiece Tokenizer is a subword tokenizer; training it on a corpus gives
+us a vocabulary of subwords. A subword tokenizer is a compromise between word tokenizers
+(word tokenizers have the issue of many OOV tokens), and character tokenizers
+(characters don't really encode meaning like words do). Luckily, TensorFlow Text makes it very


TensorFlow Text makes it very simple to train WordPiece on a corpus, as described in this [guide](link location)

mattdangerw · 2022-06-01T21:10:55Z

guides/keras_nlp/fnet_classification.py

+        vocab_size=vocab_size,
+        # Reserved tokens that must be included in the vocabulary
+        reserved_tokens=reserved_tokens,
+        # Arguments for `text.BertTokenizer`


Same comments as other guide... bert_tokenizer_params={"lower_case"=True} remove Arguments for text.BertTokenizercomment, removelearn_params`.

mattdangerw · 2022-06-01T21:16:28Z

guides/keras_nlp/fnet_classification.py

+## Formatting the Dataset
+
+Next, we'll format our datasets in the form that will be fed to the models.
+We need to add [START] and [END] tokens to the input sentences. We also need


I don't believe you need [START] and [END], and you aren't using them. Please remove.

mattdangerw · 2022-06-01T21:17:12Z

guides/keras_nlp/fnet_classification.py

+original text.
+"""
+
+for element in train_ds.take(1):


element = train_ds.take(1).get_single_element()

mattdangerw · 2022-06-01T21:17:34Z

guides/keras_nlp/fnet_classification.py

+We first need an Embedding layer, i.e., a vector for every token in our input sequence.
+This Embedding layer can be initialised randomly. We also need a Positional
+Embedding layer which encodes the word order in the sequence. The convention is
+to add these two embeddings. KerasNLP has a `TokenAndPositionEmbedding ` layer


full paths to these symbols generally, for hyperlinking

mattdangerw · 2022-06-01T21:21:20Z

guides/keras_nlp/fnet_classification.py

+with datasets that are stored in the TensorFlow format. We will use TFDS to load
+the SST-2 dataset.
+"""
+train_ds, val_ds, test_ds = tfds.load(


We generally try to show downloading and using source file directly. It is more flexible when copying and updating a guide to a new dataset. You can see how to download sst directly in our current guide for KerasNLP

I feel it's okay to use tfds here, which already does the splitting. The hassle I see from data loading is usually not how to switch between tfds and other sources, but how to find sources.

One more thing - let's be more concise in the description, if we choose to use tfds, just say load SST-2 from Tensorflow Datasets (TFDS).

This was a request from @fchollet when I was doing my guide, so maybe we should discuss with him? Personally not particularly opinionated.

Ah okay, let's bring this up in the team chat. I feel since tfds is part of TF ecosystem and still being maintained, we should try using their product.

Removed TFDS. We don't need it for IMDb :)

mattdangerw · 2022-06-01T21:28:23Z

guides/keras_nlp/fnet_classification.py

+
+### Model
+
+In 2017, a paper titled [Attention is All You Need](https://arxiv.org/abs/1706.03762)


I would generally tighten up this section. Examples shouldn't have a ton of offhand comments, we should focus on what is show in this guide. This reads a little too much like a blog currently.

Roughly, we should just say...

BERT, RoBERTa, etc have shown the effectiveness of using transformers to compute a rich embedding for input text.

However transformers are expensive, an ongoing question is how to lower the compute requirements.

In this guide we will focus on FNet, which replace the expensive attention mechanism with a Fourier transform.

We will show how this can speed up training, without significantly degrading performance.

Right. Changes made 👍🏼 . Sorry for the rather verbose introduction! :P

mattdangerw · 2022-06-01T21:29:59Z

guides/keras_nlp/fnet_classification.py

+layer and get comparable results?
+
+A couple of points from the paper stood out:
+1. The authors claim that FNet is 80% faster than BERT on GPUs and 70% faster on TPUs.


Why is our speedup so much less pronounced? Are we including compilation time in the total time? If we grow the model would the speedup become clearer?

Hey, the SST-2 dataset has very short sequences. I tried it with the IMDb dataset (which has longer sequences) and I'm getting a noticeable speed-up :D

mattdangerw · 2022-06-01T21:31:15Z

guides/keras_nlp/fnet_classification.py

+"""
+
+"""
+Let's make a table and compare the two models.


I would state this point a little more clearly--We can see that FNet significantly speeds up our run time, with only a small sacrifice in overall accuracy.

chenmoneygithub

Thanks for the PR! left some comments

chenmoneygithub · 2022-06-02T01:19:13Z

guides/keras_nlp/fnet_classification.py

+
+"""shell
+pip install -q keras-nlp
+pip install -q tfds-nightly


Why are we using nightly?

I used nightly because it has the huggingface:sst dataset. I've removed it now because I am using the IMDb dataset.

chenmoneygithub · 2022-06-02T01:20:04Z

guides/keras_nlp/fnet_classification.py

+from tensorflow_text.tools.wordpiece_vocab import bert_vocab_from_dataset as bert_vocab
+
+"""
+Let's also define our parameters/hyperparameters.


All of these are hypers?

chenmoneygithub · 2022-06-02T01:22:56Z

guides/keras_nlp/fnet_classification.py

+with datasets that are stored in the TensorFlow format. We will use TFDS to load
+the SST-2 dataset.
+"""
+train_ds, val_ds, test_ds = tfds.load(


I feel it's okay to use tfds here, which already does the splitting. The hassle I see from data loading is usually not how to switch between tfds and other sources, but how to find sources.

chenmoneygithub · 2022-06-02T01:23:39Z

guides/keras_nlp/fnet_classification.py

+with datasets that are stored in the TensorFlow format. We will use TFDS to load
+the SST-2 dataset.
+"""
+train_ds, val_ds, test_ds = tfds.load(


One more thing - let's be more concise in the description, if we choose to use tfds, just say load SST-2 from Tensorflow Datasets (TFDS).

chenmoneygithub · 2022-06-02T01:27:40Z

guides/keras_nlp/fnet_classification.py

+
+"""
+Now, let's define the tokenizer. We will use the vocabulary obtained above as
+input to the tokenizers. We will define a maximum sequence length so that


vocabulary is not the "input" to tokenizer. We could say "configure the tokenizer with vocabulary trained above."

chenmoneygithub · 2022-06-14T01:47:53Z

guides/keras_nlp/fnet_classification.py

+"""
+Title: Text Classification using FNet
+Author: [Abheesht Sharma](https://github.com/abheesht17/)
+Date created: 2021/06/01


Ah, man. Looks like I'm still mentally stuck in 2021 😆 . Changed!

chenmoneygithub · 2022-06-14T01:56:30Z

guides/keras_nlp/fnet_classification.py

+"""
+
+"""shell
+ls aclImdb


One question - here you are chaining three ls commands, do all of these prints get shown?

Hmmm, I'll have to check this by generating the .ipynb file. Ideally, it should print all. I'll generate the .iypnb file and let you know.

None of this will get outputted in the rendered example the way things work on keras.io. If you want to do that you would need to actually os.listdir or something

Ah, I see. Will change.

chenmoneygithub · 2022-06-14T02:04:31Z

guides/keras_nlp/fnet_classification.py

+## Building the Model
+
+Now, let's move on to the exciting part - defining our model!
+We first need an embedding layer, i.e., a vector for every token in our input sequence.


a layer that maps every token in input sequence to a vector.

abheesht17

Thanks for the review, @chenmoneygithub!

abheesht17 · 2022-06-14T02:12:37Z

guides/keras_nlp/fnet_classification.py

+"""
+Title: Text Classification using FNet
+Author: [Abheesht Sharma](https://github.com/abheesht17/)
+Date created: 2021/06/01


Ah, man. Looks like I'm still mentally stuck in 2021 😆 . Changed!

abheesht17 · 2022-06-14T02:14:20Z

guides/keras_nlp/fnet_classification.py

+"""
+
+"""shell
+ls aclImdb


Hmmm, I'll have to check this by generating the .ipynb file. Ideally, it should print all. I'll generate the .iypnb file and let you know.

Dockerfile LICENSE Makefile README.md call_for_contributions.md contributor_guide.md examples guides redirects requirements.txt scripts site sources templates theme ~

fchollet

Thanks!

fchollet · 2022-06-16T17:42:36Z

guides/keras_nlp/fnet_classification.py

+from tensorflow import keras
+from tensorflow_text.tools.wordpiece_vocab import bert_vocab_from_dataset as bert_vocab
+
+random.seed(42)


Don't do this, instead use keras.utils.set_random_seed()

fchollet · 2022-06-16T17:42:47Z

guides/keras_nlp/fnet_classification.py

+import os
+
+from tensorflow import keras
+from tensorflow_text.tools.wordpiece_vocab import bert_vocab_from_dataset as bert_vocab


Same as in the other example -- this should be hidden away

fchollet · 2022-06-16T17:43:27Z

guides/keras_nlp/fnet_classification.py

+our labelled `tf.data.Dataset` dataset from text files.
+"""
+
+train_ds = tf.keras.utils.text_dataset_from_directory(


Use from tensorflow import keras so you don't have to use tf.keras everywhere

fchollet · 2022-06-16T17:43:39Z

guides/keras_nlp/fnet_classification.py

+
+"""
+### Tokenizing the Data
+We'll be using the `keras_nlp.tokenizers.WordPieceTokenizer` layer to tokenize


Add line break above

fchollet · 2022-06-16T17:43:55Z

guides/keras_nlp/fnet_classification.py

+
+"""
+Every vocabulary has a few special, reserved tokens. We have four such tokens:
+- `"[PAD]"` - Padding token. Padding tokens are appended to the input sequence length


Add line break before list

fchollet · 2022-06-16T17:44:12Z

guides/keras_nlp/fnet_classification.py

+
+
+"""
+## Formatting the Dataset


Only capitalize the first word in a section title

abheesht17

Thanks for the review, @fchollet! Addressed your comments :)

fchollet

LGTM, thanks for the updates!

fchollet · 2022-06-20T23:16:02Z

examples/nlp/fnet_classification_with_keras_nlp.py

+1. The authors claim that FNet is 80% faster than BERT on GPUs and 70% faster
+on TPUs.
+The reason for this speed-up is two-fold:
+    a. The Fourier Transform layer is unparametrized, it does not have any parameters!


Does this get rendered as a nested list when you generate the website?

fchollet

Thank you for the great contribution! 👍

abheesht17 · 2022-06-29T17:27:50Z

Thank you, @mattdangerw, @fchollet, @chenmoneygithub for the review comments and the approval! :) Have to make a teensy change, don't merge it just yet.

abheesht17 · 2022-06-29T17:47:00Z

Thank you, @mattdangerw, @fchollet, @chenmoneygithub for the review comments and the approval! :) Have to make a teensy change, don't merge it just yet.

Done making the final changes!

fchollet · 2022-06-29T18:44:11Z

Continuous integration / black (pull_request) Failing after 23s — black

Please fix the code formatting.

abheesht17 · 2022-06-29T19:09:50Z

Continuous integration / black (pull_request) Failing after 23s — black

Please fix the code formatting.

(keras_io) abheesht@LAPTOP-M2NKFTLU:~/repos/keras-io$ black examples/nlp/fnet_classification_with_keras_nlp.py
All done! ✨ 🍰 ✨
1 file left unchanged.
(keras_io) abheesht@LAPTOP-M2NKFTLU:~/repos/keras-io$ python3
Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import black
>>> black.__version__
'19.10b0'

On running pip install -r requirements.txt, black 19.10b0 get installed. But looking here, seems like we use black 22.1.0. Anyway, formatted the file with the latest version of black!

fchollet · 2022-06-29T22:13:42Z

The CI error is unrelated. I'll merge. Thank you!

Add rough FNet classifier tutobook

d106d8c

abheesht17 requested review from fchollet and MarkDaoust as code owners June 1, 2022 19:48

Minor doc changes

bf66837

mattdangerw requested changes Jun 1, 2022

View reviewed changes

chenmoneygithub reviewed Jun 2, 2022

View reviewed changes

abheesht17 added 6 commits June 10, 2022 20:53

Address review comments - I

bef06ed

Format

5320271

Use keras text_dataset_from_directory utility

b460836

Minor doc-string change

8be2947

Remove unnecessary imports

eb437ea

Remove SST-2 refs

ba2b01c

chenmoneygithub reviewed Jun 14, 2022

View reviewed changes

abheesht17 commented Jun 14, 2022

View reviewed changes

abheesht17 added 3 commits June 14, 2022 07:46

Address review comments - II

dfbaf9b

Use instead of CODEOWNERS

1beaef3

Dockerfile LICENSE Makefile README.md call_for_contributions.md contributor_guide.md examples guides redirects requirements.txt scripts site sources templates theme ~

Import os

426a23c

fchollet reviewed Jun 16, 2022

View reviewed changes

abheesht17 commented Jun 16, 2022

View reviewed changes

abheesht17 added 3 commits June 16, 2022 23:41

Address review comments - III

fcd15ad

Format

a6fbcea

Shift example from ./guides to ./examples

61fff5b

fchollet reviewed Jun 20, 2022

View reviewed changes

mattdangerw approved these changes Jun 28, 2022

View reviewed changes

abheesht17 added 6 commits June 29, 2022 20:10

Merge branch 'keras-team:master' into fnet-keras-nlp

653efec

Small bug fixes

4143538

Remove pip install cmd

bf93044

Fix 1a, 1b

5cb9cdf

Fix nested list

46708ca

Add generated files

7284c3f

Small changes

0e5f1f3

fchollet approved these changes Jun 29, 2022

View reviewed changes

New generated files

60c0c8d

Final changes

dd53d39

abheesht17 added 3 commits June 30, 2022 00:40

Format with black 22.1.0

43d48dc

Add new generated files

3775e4f

Regenerate

42b8362

fchollet merged commit 0d933fc into keras-team:master Jun 29, 2022


		### Model

		In 2017, a paper titled [Attention is All You Need](https://arxiv.org/abs/1706.03762)

Text Classification with FNet [KerasNLP] #898

Text Classification with FNet [KerasNLP] #898

Conversation

abheesht17 commented Jun 1, 2022 • edited Loading

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenmoneygithub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fchollet left a comment

Choose a reason for hiding this comment

abheesht17 commented Jun 29, 2022 • edited Loading

abheesht17 commented Jun 29, 2022 • edited Loading

fchollet commented Jun 29, 2022

abheesht17 commented Jun 29, 2022

fchollet commented Jun 29, 2022

abheesht17 commented Jun 1, 2022 •

edited

Loading

abheesht17 commented Jun 29, 2022 •

edited

Loading

abheesht17 commented Jun 29, 2022 •

edited

Loading