Recommendation for training on new language ?

[PSanni]

Any recommendations to train or fine-tune model on new language.

1. Does training for new language (E.x. arabic) will work on existing pre-trained models ? Or it has to be from scratch.
2. What is recommended amount of data for new language ?

[baudm]

1. Fine-tuning should work for any language based on the Latin alphabet. If the language uses a different set of characters, you should define a new training charset configuration, e.g. configs/charset/arabic.yaml and use it during training. You should also update charset_test with the same set of characters used for training. The training command should look something like ./train.py charset=arabic, where configs/charset/arabic.yaml contains:

# @package _global_
model:
  charset_train: "..."
  charset_test: "..."

2. I don't have a definite answer for this since it would depend on the quality of your training data, and how similar its distribution is to the test data. In our experiments with real training data, PARSeq starts to perform well after 40k iterations (batch size = 384).

[baudm]

This is an old result for PARSeq. I was comparing the validation word accuracy for models trained exclusively on TextOCR (arbitrary) and its pose-corrected version (horizontal). DDP was used with 2 GPUs, so the effective iteration is the number shown in the x-axis multiplied by 2.

[phamkhactu]

1. Fine-tuning should work for any language based on the Latin alphabet. If the language uses a different set of characters, you should define a new training charset configuration, e.g. configs/charset/arabic.yaml and use it during training. You should also update charset_test with the same set of characters used for training. The training command should look something like ./train.py charset=arabic model.charset_test=<Arabic characters>
2. I don't have a definite answer for this since it would depend on the quality of your training data, and how similar its distribution is to the test data. In our experiments with real training data, PARSeq starts to perform well after 40k iterations (batch size = 384).

@baudm Thank for your great repo. I want to fintune for vietnamese language. Does train it?. And It can, how to prepare Dataset to training.
Many thank for your reponse.

[baudm]

@baudm Thank for your great repo. I want to fintune for vietnamese language. Does train it?. And It can, how to prepare Dataset to training. Many thank for your reponse.

@phamkhactu
Re: finetuning, please refer to my first comment.

For dataset preparation, please refer to clovaai/deep-text-recognition-benchmark on how to convert your image-text pairs into LMDB databases.

[PSanni]

1. Fine-tuning should work for any language based on the Latin alphabet. If the language uses a different set of characters, you should define a new training charset configuration, e.g. configs/charset/arabic.yaml and use it during training. You should also update charset_test with the same set of characters used for training. The training command should look something like ./train.py charset=arabic model.charset_test=<Arabic characters>
2. I don't have a definite answer for this since it would depend on the quality of your training data, and how similar its distribution is to the test data. In our experiments with real training data, PARSeq starts to perform well after 40k iterations (batch size = 384).

When you say quality, does that mean quality of images ? or the term coverage ?

[siddagra]

I am also having an issue with this. When training and validating I have set all character sets in train and test to chinese characters + latin alphanumerics and even created a separate file for yaml.

When I print out model.config while training, it seems to show the charset properly, but then after training when I use the checkpoints to recognise images using read.py it does not output any chinese characters.

Not sure if this is an issue in read.py, train.py, test.py or lmdb dataset itself as the val accuracy is 99.93%. Please guide/help if possible.

[siddagra]

For now I have used a dirty hack and used 94_full charset's symbols to represent Chinese characters. Mapping chinese characters to symbols in lmdb dataset and then back from symbols to Chinese characters during/after inference.

[baudm]

@PSanni

When you say quality, does that mean quality of images ? or the term coverage ?

By dataset quality, I mean dataset size, diversity of samples, accuracy of labels, etc., not quality of images per se.

[baudm]

I am also having an issue with this. When training and validating I have set all character sets in train and test to chinese characters + latin alphanumerics and even created a separate file for yaml.

When I print out model.config while training, it seems to show the charset properly, but then after training when I use the checkpoints to recognise images using read.py it does not output any chinese characters.

Not sure if this is an issue in read.py, train.py, test.py or lmdb dataset itself as the val accuracy is 99.93%. Please guide/help if possible.

@siddagra unless you have a very small and easy val set, val accuracy of 99.93% likely indicates a problem with your training setup.

1. First, you have to make sure that your training dataset is correctly prepared. Open the lmdb archives, query an image and its corresponding label, and check if the image and label are correct and intact.
2. Disable Unicode normalization: data.normalize_unicode=false
3. Probe the SceneTextDataModule instance. You can do it in any script (train.py, read.py, and test.py). You can check the labels returned by LmdbDataset using the train_dataset or val_dataset property of the data module instance. Make sure it returns the expected labels.
4. Check if CharsetAdapter works. Create an instance using your charset, e.g. adapter = CharsetAdapter(charset), then test if it returns the correct output given Chinese text: adapter(some_text).

[siddagra]

Thanks a lot for your help!

I printed out several labels at several places, base.py, dataset.py, while lmdb encoding and decoding, etc.

base.py

 141           pred = self.charset_adapter(pred)
 142           print(pred)

dataset.py

                label = charset_adapter(label)
                print(label)

It seems to be working fine everywhere. The only issue seems to be the read.py itself perhaps.
The argmax output of the variable p (logits from output) are not outputting any index higher than 23, even though the sequence should include characters indices up to 96.

tensor([[ 6,  9, 20,  9,  9,  1,  9,  3,  4,  2,  5, 12, 23,  8,  8, 12, 23, 12,
         23,  8, 21, 23, 13, 19, 13, 23,  8, 12, 12,  8, 12, 23, 23, 23,  8, 21,
          8,  0,  8,  8,  8, 20, 13,  9, 22, 21, 21, 20,  8, 18, 25, 21,  8, 23,
         22, 14, 16, 22, 19, 13, 14, 21, 17, 22, 23, 24, 13, 22, 25, 13, 23, 21,
         12,  9, 23, 23, 13, 23, 22, 23, 23, 22, 13,  9,  8, 23, 23, 23, 24, 23,
         13, 23, 23, 23, 21]], device='cuda:0')

I wanted to get results to report, but read.py is ignoring chinese characters and
It somehow started working now. I think because I disabled unicode normalisation. Thanks!

test.py is giving this error:

dataset.py", line 72, in __del__
    self.env.close()
AttributeError: 'LmdbDataset' object has no attribute 'env'

Also, is it possible to train only the LM model? The dataset I am training on contains limited language of a specific format, but I do not want it to overfit to this format and have poor results otherwise, I was wondering if it was possible to only train it on text data/character sequences itself, instead of images+labels. It may be useful to be able to train LM on larger language (non-image) datasets for other languages with limited image data.

[baudm]

@siddagra

It somehow started working now. I think because I disabled unicode normalisation. Thanks!

test.py is giving this error:

dataset.py", line 72, in __del__
    self.env.close()
AttributeError: 'LmdbDataset' object has no attribute 'env'

You're using old code. Pull the latest and update your dependencies.

Also, is it possible to train only the LM model? The dataset I am training on contains limited language of a specific format, but I do not want it to overfit to this format and have poor results otherwise, I was wondering if it was possible to only train it on text data/character sequences itself, instead of images+labels. It may be useful to be able to train LM on larger language (non-image) datasets for other languages with limited image data.

Sorry, this is not possible with PARSeq since its LM is internal. You can do this with ABINet, but honestly in my opinion, training on raw text has limited utility for STR since it is still primarily a visual recognition problem.

To alleviate the issue with your limited training data, I would suggest using a more extreme augmentation on the images: rotate them by 90, 180, or 270 degrees. You can do this by modifying augment.py or by directly adding the rotations inside the image transforms in module.py.

You may also lower the batch size in order to increase the variance and lessen the bias for each mini-batch. You could also play around with the value of K.

One STR-specific augmentation would be to form new training data by concatenating existing data. I have implemented a simple version of this and it works (but more experimental validation is needed). The algorithm is something like this:

1. Choose a pair of samples.
2. Allocate the image width proportional to the label length. That is, W = 128 * len(A) / (len(A) + len(B)) would be the allocation in pixels for image A.
3. Resize the images to W x 32 and (128 - W) x 32 pixels, then concatenate them side by side.
4. Concatenate the labels.

Lastly, you may try adding the augmentations implemented in straug.

[PSanni]

is it possible to continue training from checkpoints ? if so, is there any pre-trained weights available for fine-tuning ? Would be great if you can write short note on it.

[baudm]

is it possible to continue training from checkpoints ? if so, is there any pre-trained weights available for fine-tuning ? Would be great if you can write short note on it.

#7 (comment)

[baudm]

@PSanni @siddagra @bmusq
As of commit b290950, finetuning is now officially supported. checkpoint parameter of test.py and read.py has been changed accordingly.

Now you can do:

# Finetuning
./train.py pretrained=parseq  # parseq-tiny, etc. See released weights
# Resume from PL checkpoint
./train.py ckpt_path=outputs/parseq/.../last.ckpt

# Use pretrained weights for testing
./test.py pretrained=parseq  # same with read.py
# Or your own trained weights
./test.py outputs/parseq/.../last.ckpt  # same with read.py