Overview of issues in torchtext and the plan for revamping

[zhangguanheng66]

Motivation and summary of the current issues in torchtext

Based on the feedback from users, there are several issues existing in torchtext, including

• Several components and functionals are unclear and difficult to adopt. For example, Field class couples tokenizer, vocabulary, split, batching and sampling, padding, and numericalization together. The current Field class works as a "black box", and users are confused about what's going on within the class. Instead, those components should be divided into several basic building blocks. This is more consistent with PyTorch core library, which grants users the freedom to build the models and pipelines with orthogonal components.
• Incompatible with DataLoader and Sampler in torch.utils.data. The current datasets in torchtext are not compatible with PyTorch core library. Some custom modules/functions in torchtext (e.g. Iterator, Batch, splits) should be replaced by the corresponding modules in torch.utils.data.

New datasets in torchtext.experimental.datasets

We have re-written several datasets in torchtext.experimental.datasets which were using the new abstractions. The old version of the datasets are still available in torchtext.datasets and the new datasets are opt-in.

• Sentiment analysis dataset (Re-write IMDB dataset in torchtext.experimental.datasets #651)
  • IMDB
• Language modeling datasets (Re-write language_modeling datasets (PennTreebank, WikiText103, WikiText2) #624), including
  • WikiText2
  • WikiText103
  • PennTreebank

Case study for IMDB dataset

API for new datasets

To load the new datasets, simply call the dataset API, as follow:

from torchtext.experimental.datasets import IMDB
train_dataset, test_dataset = IMDB()

To specify a tokenizer:

from torchtext.data.utils import get_tokenizer
tokenizer = get_tokenizer("spacy")
train_dataset, test_dataset = IMDB(tokenizer=tokenizer)

If you just need the test set (must pass a Vocab object!):

vocab = train_dataset.get_vocab()
test_dataset, = IMDB(tokenizer=tokenizer, vocab=vocab, data_select='test')

Legacy code

The old IMDB dataset is still available in the folder torchtext.datasets. You can use the legacy datasets, as follow:

import torchtext.data as data
TEXT = torchtext.data.Field(lower=True, include_lengths=True, batch_first=True)
LABEL = torchtext.data.Field(sequential=False)
train, test = torchtext.datasets.IMDB.splits(TEXT, LABEL)

Difference

With the old pattern, users have to create a Field object including a specific tokenizer. In the new dataset API, user can pass a custom tokenizer directly to the dataset constructor. A custom tokenizer defines the method to convert a string to a list of tokens

from torchtext.data.utils import get_tokenizer

# Old pattern
TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"))

# New pattern
train_dataset, test_dataset = IMDB(tokenizer=get_tokenizer("spacy"))

In the old dataset, vocab object is associated with Field class, which is not flexible enough to accept a pre-trained vocab object. In the new dataset, the vocab object can be obtained by

vocab = train_dataset.get_vocab()
new_vocab = torchtext.vocab.Vocab(counter=vocab.freqs, max_size=1000, min_freq=10)

and apply to generate other new datasets.

from torchtext.experimental.datasets import WikiText2
train_dataset, test_dataset, valid_dataset = WikiText2(vocab=new_vocab)

The datasets with the new pattern return a tensor of token IDs, instead of tokens in the old pattern. If users would like to retrieve the tokens, simply use the following command:

train_vocab = train_dataset.get_vocab()
# label and text are saved as a tuple
tokens = [train_vocab.itos[id] for id in train_dataset[0][1]]

Unlike the old pattern using BucketIterator.splits, users are encouraged to use torch.utils.data.DataLoader to generate batches of data. You can specify how to batch and pad the samples with a custom function passed to collate_fn. Here is an example to pad sequences with similar lengths and load data through DataLoader. To generate random samples, turn on the shuffle flag in DataLoader. Otherwise, a sequential sampler will be automatically constructed.

# Generate a list of tuples of text length, index, label, text
data_len = [(len(txt), idx, label, txt) for idx, (label, txt) in enumerate(train_dataset)]
data_len.sort() # sort by length and pad sequences with similar lengths

# Generate the pad id
pad_id = train_dataset.get_vocab()['<pad>']

# Generate 8x8 batches
# Pad sequences with similar lengths
import torch
from torch.utils.data import DataLoader
def pad_data(data):
    # Find max length of the mini-batch
    max_len = max(list(zip(*data))[0])
    label_list = list(zip(*data))[2]
    txt_list = list(zip(*data))[3]
    padded_tensors = torch.stack([torch.cat((txt, \
            torch.tensor([pad_id] * (max_len - len(txt))).long())) \
            for txt in txt_list])
    return padded_tensors, label_list

dataloader = DataLoader(data_len, batch_size=8, collate_fn=pad_data)
for idx, (txt, label) in enumerate(dataloader):
    print(idx, txt.size(), label)

Randomly split a dataset into non-overlapping new datasets of given lengths.

from torchtext.experimental.datasets import IMDB
train_dataset, test_dataset = IMDB()
train_subset, valid_subset = torch.utils.data.random_split(train_dataset, [15000, 10000])

Reference:

A few recent issues from OSS users:

• Sorting sentence within a batch is confusing Setting sort_within_batch of pool to True when called in the method create_batches of BucketIterator? #641
• split function is confusing Using Google 1-billion benchmark data on PyTorch #644
• Generate vocab object based on a subset of text file KeyError in vocab if the vocab is built on a subset of the initially read data #642
• Pass a pre-trained vocab object to build a dataset TEXT.build_vocab for two datasets #648
• Load unconstructed text data How to perform common sense reasoning task with GPT-2? #649
• More flexibility to support word vector layers combining TEXT.build_vocab with flair embeddings #650
• More compatible with torch.utils.data.DataLoader How to prefetch data? #660

[zhangguanheng66]

@cpuhrsch @vincentqb @fmassa @soumith @mttk @bentrevett

[bentrevett]

The new dataset API looks like a good start - I think it's a good idea to move away from using Field - but there's a few things that still need to be looked at.

I'm currently experimenting on the text_classification datasets in 0.4 using the following:

def create_dataset(dataset, dataset_args, vocab_args):
    
    #assert datset exists in text_classification
    assert dataset in torchtext.datasets.text_classification.DATASETS
        
    #get dataset
    train_data, _ = getattr(torchtext.datasets.text_classification, dataset)(**dataset_args)
    
    #get unfiltered vocab (default args)
    old_vocab = train_data.get_vocab()
    
    #create new filtered vocabulary (with desired args)
    new_vocab = torchtext.vocab.Vocab(counter = old_vocab.freqs, 
                                      **vocab_args)
    
    #return dataset with new vocabulary
    return getattr(torchtext.datasets.text_classification, dataset)(vocab = new_vocab, 
                                                                    **dataset_args)

An example of use:

train_data, test_data = create_dataset('AG_NEWS',
                                       dataset_args = {'ngrams': 1},
                                       vocab_args = {'max_size': 25_000})

With 0.5, we'll be able to pass a tokenizer to the dataset - which is much needed - but as we can't pass arguments to the initialization of the Vocab, it means we have to tokenize and create a vocabulary then use that vocabulary to construct another Vocab with our desired arguments and then tokenize again. Some way to pass Vocab arguments to the dataset is needed.

The default pad and unk tokens should also be part of the Vocab object and should be an argument as we won't have Field to handle them anymore and they're more related to the vocabulary than the dataset. I see that changing the unk is currently a TODO and even though the Vocab object doesn't actually use the pad token anywhere I think it makes sense for it to be part of the Vocab object - happy to hear if you disagree.

Also, a better collate_fn assuming the data is a list of (label, sequence) tuples is:

def collator(batch, pad_idx):
    
    labels, sequences = zip(*batch)

    #FloatTensor for binary classification, LongTensor for multiclass classification
    labels = torch.FloatTensor(labels)
            
    sequences = torch.nn.utils.rnn.pad_sequence(sequences, 
                                                padding_value = pad_idx)
        
    return labels, sequences

rnn.pad_sequences is faster than manipulating the lists and doing a concatenation then a stack.

Then this can be passed to the iterator with:

import functools

pad_idx = train_data.get_vocab()['<pad>']
batch_size = 64

collate_fn = functools.partial(collator, 
                               pad_idx = pad_idx)

train_iterator = torch.utils.data.DataLoader(train_data, 
                                             shuffle = True, 
                                             batch_size = batch_size,
                                             collate_fn = collate_fn)

The collate function above can also be expanded to provide sequence lengths and masks:

def collator(batch, pad_idx):
    
    labels, sequences = zip(*batch)

    labels = torch.FloatTensor(labels)

    lengths = torch.LongTensor([len(sequence) for sequence in sequences])
    
    sequences = torch.nn.utils.rnn.pad_sequence(sequences, 
                                                padding_value = pad_idx)
    
    masks = (sequences != pad_idx)
    
    return labels, sequences, lengths, masks

This gets back the missing include_lengths ability from the Field.

Apart from those small things - I think you've done a good job!

[zhangguanheng66]

@bentrevett Thanks for your comments and great feedback.
For the vocab part, I think some changes could be made in build_vocab_from_iterator. Then, users build a custom vocab object, which is passed to the dataset interface.

For padding, yes, torch.nn.utils.rnn.pad_sequence is doing the same thing (don't know why it's under the "rnn" category). There are also different ways to pad sequences (like a long Tensor with offsets, a tensor with mask), or even NestedTensor in the future. Those are some interesting blocks to think about.

[zhangguanheng66]

Batch/iterator issue with RawField #266

[zhangguanheng66]

translation datasets are too slow with torchtext.data.Iterator (#668). Need to re-write the datasets and make them compatible with torch.utils.data.DataLoader.

[Darktex]

Maybe it's worth using torch.utils.data.IterableDataset?

[zhangguanheng66]

Yeap. We had discussions about this. Since all the other datasets are using torch.utils.data.Dataset in this release, we decided to keep consistent. Thanks for the advice.

[rpatrik96]

Hi everyone!

I am quite new to torchtext (but a PyTorch user for 2+ years). My first impression about the new dataset API is great, however, there is one thing I cannot accomplish in a clean manner.

I am using the Penn Treebank dataset from torchtext.experimental, like this:

 # query training set to build vocab
vocab = PennTreebank(data_select="train")[0].get_vocab()
# create new filtered vocabulary (with word-level embeddings)
self.vocab = torchtext.vocab.Vocab(counter=vocab.freqs, vectors=torchtext.vocab.GloVe(name='6B', dim=self.embedding_dim))

# create datasets
train_dataset, valid_dataset, test_dataset = PennTreebank(vocab=self.vocab)

The problem is, that the returned dataset cannot provide sequences (if a write a collate_fn to reshape the batch to have a sequence dimension, then I cannot shuffle the data), train_dataset[i] will be a single value.

My solution would be to write a custom dataset and copy data from the one returned by torchtext. Is there a more clean way to do that?

In pseudocode, I need something like this:

train_dataset, valid_dataset, test_dataset = PennTreebank(vocab=self.vocab, seq_len=32)
print(train_dataset[0].shape) # should be 32 X 1

[zhangguanheng66]

Hi everyone!

I am quite new to torchtext (but a PyTorch user for 2+ years). My first impression about the new dataset API is great, however, there is one thing I cannot accomplish in a clean manner.

I am using the Penn Treebank dataset from torchtext.experimental, like this:

 # query training set to build vocab
vocab = PennTreebank(data_select="train")[0].get_vocab()
# create new filtered vocabulary (with word-level embeddings)
self.vocab = torchtext.vocab.Vocab(counter=vocab.freqs, vectors=torchtext.vocab.GloVe(name='6B', dim=self.embedding_dim))

# create datasets
train_dataset, valid_dataset, test_dataset = PennTreebank(vocab=self.vocab)

The problem is, that the returned dataset cannot provide sequences (if a write a collate_fn to reshape the batch to have a sequence dimension, then I cannot shuffle the data), train_dataset[i] will be a single value.

My solution would be to write a custom dataset and copy data from the one returned by torchtext. Is there a more clean way to do that?

In pseudocode, I need something like this:

train_dataset, valid_dataset, test_dataset = PennTreebank(vocab=self.vocab, seq_len=32)
print(train_dataset[0].shape) # should be 32 X 1

Yes. That's the problem for all the three WLM datasets. One way to wrap up the dataset is here

[rpatrik96]

Yes. That's the problem for all the three WLM datasets. One way to wrap up the dataset is here

@zhangguanheng66: Thanks for the help!

[thvasilo]

Is it possible to use this new-style datasets when loading from csv using a TabularDataset?