Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added earlystopping mechanism #1389

Merged
merged 2 commits into from
Apr 9, 2019
Merged

Added earlystopping mechanism #1389

merged 2 commits into from
Apr 9, 2019

Conversation

antoniovilarinholopes
Copy link
Contributor

Early stopping

A very requested feature for training regularisation to avoid overfitting. Issue #1382 is an example of requesting this simple feature.

This pull request uses the previous codebase for inspiration.

Detailed changes

  • Added option to opts.py regarding the number of validation steps without improving (tolerance);
  • Added earlystopping callable to trainer.py (only for gpu rank==0);
  • Added earlystopping logic: after validation deciding whether or not to stop training; added onmt/utils/earlystopping.py with all the logic. Design was made thinking about future metrics, should be fairly easy to add ROUGE or BLEU in the future.

Adding metrics for earlystopping

  1. Add new scorer that extracts the metric from stats (TODO, add other metrics to stats during validation);
  2. Register new scorer in SCORER_BUILDER.

Running with new options:

  • ... --early_stopping 4 --early_stopping_criteria accuracy ppl : 4 tolerance and use both accuracy and perplexity as earlystopping criteria.

@vince62s
Copy link
Member

vince62s commented Apr 9, 2019

Thanks for this, looks great.
I have not tested it but since it won't break anything, might be good to merge and have users test it.
@francoishernandez @guillaumekln do yo want ot have a look ?

@guillaumekln
Copy link
Contributor

What happens in the case of multi-GPU training? The master worker should somehow send a message to the other workers to terminate their training loop.

@vince62s
Copy link
Member

vince62s commented Apr 9, 2019

Well you are correct, but:
If my understanding is correct, in the current codebase, we perform the validation on each spawn process (whether this is wanted or not is another story) which means that the valid task is the same on easch process with the "gathered" stats. So maybe the easiest way is to remove the test gpu_rank==0 in the callable definition.
Am I correct ?

@antoniovilarinholopes
Copy link
Contributor Author

@guillaumekln you are correct, I forgot sending the other workers a stopping signal.
@vince62s suggestion should solve it, I think.

@antoniovilarinholopes
Copy link
Contributor Author

I added and tested @vince62s suggestion and it works as expected.

@vince62s
Copy link
Member

vince62s commented Apr 9, 2019

ok thanks, merging.

@vince62s vince62s merged commit f7fc40e into OpenNMT:master Apr 9, 2019
rishibommasani added a commit to rishibommasani/OpenNMT-py that referenced this pull request Aug 29, 2019
@rishibommasani
sync (#1)
9e011ff 
* advanced noam with decay and accum scheduler (OpenNMT#1367)

* advanced noam with decay and accum scheduler

* Add phrase_table translation argument (OpenNMT#1370)

* Add phrase_table translation argument

If phrase_table is provided (with replace_unk), it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token.

* Have EnsembleDecoder set attentional property. (OpenNMT#1381)

* More efficient embeddings_to_torch.py (OpenNMT#1372)

* Update embeddings_to_torch.py to be more memory efficient by only loading vectors which are present in the vocab into memory.

* remove dead code and flake8 violations introduced with 57cefb7

* update docs of using Glove embeddings. Fix spelling error

* write attention debug to log file (OpenNMT#1384)

* Better handle Cuda OOM with overflow batches (OpenNMT#1385)

* Added earlystopping mechanism (OpenNMT#1389)

* Added earlystopping mechanism
* Fixed earlystopping multi-gpu stoppage

* check vocab files exist at start of preprocessing (OpenNMT#1396)

* Avoid padding indices in MeanEncoder (OpenNMT#1398)

* We avoid padding while mean pooling
* placed batch dimension first for bmm
* replaced accidentally deleted line

* fix Runtime error in Library tutorial (OpenNMT#1399)

* Check -gpu_ranks option to ensure saving a model (OpenNMT#1407)

* Check -gpu_ranks option to ensure saving a model
* split condition to check -gpu_ranks inconsistency

* add src or tgt min frequency to counter value (OpenNMT#1414)

* fix typo (OpenNMT#1416)

* fix goldscore OpenNMT#1383 (OpenNMT#1423)

* fix OpenNMT#1383

* fix gold score only

* Upgrade Travis to Torch 1.1 (OpenNMT#1426)

* Introduce dropout scheduler (OpenNMT#1421)

* add update_dropout methods approx. everywhere, dropout scheduler
* more meaningful log
* forgot some layers in audio_encoder

* Preprocessing: faster build vocab + multiple weighted datasets (OpenNMT#1413)

* handle multiple training corpora and enable weighting
* move fields vocab building logic in function
* fix device handling MultipleDatasetIterator
* fix multi/yield_raw_batch parameter DatasetLazyIter
* update FAQ.md
* add -pool_factor option
* reduce pool_factor for travis runs

* bump version (OpenNMT#1434)

* make MultipleDatasetIterator only if necessary (OpenNMT#1436)

* Update README.md (OpenNMT#1437)

* small fix multi when common root in data_ids (OpenNMT#1444)

* do not overwrite pt vocab when preprocessing again (OpenNMT#1447)

* trim vocab(s) before saving checkpoint (OpenNMT#1453)

* Using Producer-Consumer for batches (OpenNMT#1450)

* Working queues on multi-GPU on text and audio
* Working quite well, even with dynamic_dict
* Remove explicit garbage collect making some queue hang and other fixes
* fix process not ending
* properly set random seed and fill queues sequentially
* make queues work with distributed training

* [fix] Make queue.put() blocking again (OpenNMT#1455)

Fix OpenNMT#1454 .

* Clarify mixed precision training support (OpenNMT#1458)

Change the wording to avoid confusion. Mixed precision ensures both higher arithmetic throughput and numerical stability, not exactly synonymous to pure half-precision/FP16 training. Also add mentioning of tensor cores since older generation GPUs without tensor cores don't support true mixed precision training.

* Update requirements.opt.txt

* Update requirements.opt.txt

* Change map_location to be 'cpu' (OpenNMT#1461)

* Change map_location to be 'cpu'

If you are on a CPU-only machine, it will give an error otherwise. Model averaging should not require a GPU; moreover, it may be faster to use CPU rather than move all models to the GPU to average them.

* New apex amp API (OpenNMT#1465)

* use new apex amp API
* make apex opt_level as option

* bump 0.9.1 (OpenNMT#1466)

* Do not raise an error for missing validation data (OpenNMT#1467)

* fix incorrect script path in CONTRIBUTING.md (OpenNMT#1470) (OpenNMT#1472)

* Fix a potential IndexError when translating with replace_unk (OpenNMT#1469)

* Fix IndexError which happens with replace_unk, when the argmax of the attention is on the padding instead of a real source token

* add health endpoint to server.py (OpenNMT#1471)

* fix typo

* Minor change in MultiHeadedAttention  documentation (OpenNMT#1479)

* Minor change in documentation

* Optimize AAN transformer and small fixes (OpenNMT#1482)

* Optimize AAN transformer and small fixes
* Make use of FFN layer in AAN an option

* Implementing coverage loss of abisee (2017) (OpenNMT#1464)

* Implementing coverage loss of abisee (2017)
* fix lambda_coverage value

* Video captioning (OpenNMT#1409)

* Add feature extraction tool.
* Update preprocess.
* Add training and translation.
* Adapt transformer for video.
* Add tutorial to docs.
* Add folded val files for easier 'early stop.'
* Add and document transformer.

* ignore batch if over allowed tokens batch, add warning (OpenNMT#1490)

* allow implicit batch_size in translation_server (OpenNMT#1492)

* ensure building sequence mask on same device as lengths (OpenNMT#1494)

* add preprocess_opt in rest server (ZH) (OpenNMT#1493)

* fix build_dataset_iter in train_single (OpenNMT#1499)

* Use functions as preprocess / postprocess in REST server (OpenNMT#1505)

* add preprocess_opt in rest server (ZH)

* add preprocess and postprocess in rest server

* simplify

* fix function name

* fix function name v2

* [fix] remove implicit check in preprocess (OpenNMT#1507)

* [fix] remove implicit check in preprocess

There were some implicit checks on `src_vocab` and `tgt_vocab` in preprocessing.
This was creating some unwanted behavior when loading an existing vocab as a text file.

* fix typo

* add attention_dropout separate from dropout (OpenNMT#1512)

* add attention_dropout separate from dropout

* fix compatibility with models without attention_dropout (OpenNMT#1514)

* pytorch 1.2 compatibility - mask & bool tensor (OpenNMT#1527)

* Fix typo: traget -> target (OpenNMT#1537)

* Tokens batch for translation (OpenNMT#1545)

* wip translate batch tokens
* move logic in translator
@chenqiuyuan
Copy link

Hi, I think this page should be integrated into the tutorial (I mean, the library http://opennmt.net/OpenNMT-py/Library.html). It takes me a lot of time to find this page before wasting a lot of time.

jrvc added a commit to Helsinki-NLP/FoTraNMT that referenced this pull request Jan 13, 2021
@jrvc
Revert "Added earlystopping mechanism (OpenNMT#1389)"
468aaf4 
This reverts commit f7fc40e.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@antoniovilarinholopes @vince62s @guillaumekln @chenqiuyuan