advanced noam with decay and accum scheduler #1367
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@francoishernandez @guillaumekln if you got some time to review, thanks. |
|
|
||
| # Set model in training mode. | ||
| self.model.train() | ||
|
|
||
| def _accum_count(self, step): | ||
| for i in range(len(self.accum_steps)): |
There was a problem hiding this comment.
here instead of looping every time shouldn't we update the accum_count_l list at each change so that all we need to check is step > accum_count_l[0]?
marekstrong
pushed a commit
to marekstrong/OpenNMT-py
that referenced
this pull request
Jun 10, 2019
* advanced noam with decay and accum scheduler Add phrase_table translation argument (OpenNMT#1370) * Add phrase_table translation argument If phrase_table is provided (with replace_unk), it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token. update collected update update policy update auto and conf guide update requirements.txt collect more data update update
rishibommasani
added a commit
to rishibommasani/OpenNMT-py
that referenced
this pull request
Aug 29, 2019
* advanced noam with decay and accum scheduler (OpenNMT#1367) * advanced noam with decay and accum scheduler * Add phrase_table translation argument (OpenNMT#1370) * Add phrase_table translation argument If phrase_table is provided (with replace_unk), it will look up the identified source token and give the corresponding target token. If it is not provided (or the identified source token does not exist in the table), then it will copy the source token. * Have EnsembleDecoder set attentional property. (OpenNMT#1381) * More efficient embeddings_to_torch.py (OpenNMT#1372) * Update embeddings_to_torch.py to be more memory efficient by only loading vectors which are present in the vocab into memory. * remove dead code and flake8 violations introduced with 57cefb7 * update docs of using Glove embeddings. Fix spelling error * write attention debug to log file (OpenNMT#1384) * Better handle Cuda OOM with overflow batches (OpenNMT#1385) * Added earlystopping mechanism (OpenNMT#1389) * Added earlystopping mechanism * Fixed earlystopping multi-gpu stoppage * check vocab files exist at start of preprocessing (OpenNMT#1396) * Avoid padding indices in MeanEncoder (OpenNMT#1398) * We avoid padding while mean pooling * placed batch dimension first for bmm * replaced accidentally deleted line * fix Runtime error in Library tutorial (OpenNMT#1399) * Check -gpu_ranks option to ensure saving a model (OpenNMT#1407) * Check -gpu_ranks option to ensure saving a model * split condition to check -gpu_ranks inconsistency * add src or tgt min frequency to counter value (OpenNMT#1414) * fix typo (OpenNMT#1416) * fix goldscore OpenNMT#1383 (OpenNMT#1423) * fix OpenNMT#1383 * fix gold score only * Upgrade Travis to Torch 1.1 (OpenNMT#1426) * Introduce dropout scheduler (OpenNMT#1421) * add update_dropout methods approx. everywhere, dropout scheduler * more meaningful log * forgot some layers in audio_encoder * Preprocessing: faster build vocab + multiple weighted datasets (OpenNMT#1413) * handle multiple training corpora and enable weighting * move fields vocab building logic in function * fix device handling MultipleDatasetIterator * fix multi/yield_raw_batch parameter DatasetLazyIter * update FAQ.md * add -pool_factor option * reduce pool_factor for travis runs * bump version (OpenNMT#1434) * make MultipleDatasetIterator only if necessary (OpenNMT#1436) * Update README.md (OpenNMT#1437) * small fix multi when common root in data_ids (OpenNMT#1444) * do not overwrite pt vocab when preprocessing again (OpenNMT#1447) * trim vocab(s) before saving checkpoint (OpenNMT#1453) * Using Producer-Consumer for batches (OpenNMT#1450) * Working queues on multi-GPU on text and audio * Working quite well, even with dynamic_dict * Remove explicit garbage collect making some queue hang and other fixes * fix process not ending * properly set random seed and fill queues sequentially * make queues work with distributed training * [fix] Make queue.put() blocking again (OpenNMT#1455) Fix OpenNMT#1454 . * Clarify mixed precision training support (OpenNMT#1458) Change the wording to avoid confusion. Mixed precision ensures both higher arithmetic throughput and numerical stability, not exactly synonymous to pure half-precision/FP16 training. Also add mentioning of tensor cores since older generation GPUs without tensor cores don't support true mixed precision training. * Update requirements.opt.txt * Update requirements.opt.txt * Change map_location to be 'cpu' (OpenNMT#1461) * Change map_location to be 'cpu' If you are on a CPU-only machine, it will give an error otherwise. Model averaging should not require a GPU; moreover, it may be faster to use CPU rather than move all models to the GPU to average them. * New apex amp API (OpenNMT#1465) * use new apex amp API * make apex opt_level as option * bump 0.9.1 (OpenNMT#1466) * Do not raise an error for missing validation data (OpenNMT#1467) * fix incorrect script path in CONTRIBUTING.md (OpenNMT#1470) (OpenNMT#1472) * Fix a potential IndexError when translating with replace_unk (OpenNMT#1469) * Fix IndexError which happens with replace_unk, when the argmax of the attention is on the padding instead of a real source token * add health endpoint to server.py (OpenNMT#1471) * fix typo * Minor change in MultiHeadedAttention documentation (OpenNMT#1479) * Minor change in documentation * Optimize AAN transformer and small fixes (OpenNMT#1482) * Optimize AAN transformer and small fixes * Make use of FFN layer in AAN an option * Implementing coverage loss of abisee (2017) (OpenNMT#1464) * Implementing coverage loss of abisee (2017) * fix lambda_coverage value * Video captioning (OpenNMT#1409) * Add feature extraction tool. * Update preprocess. * Add training and translation. * Adapt transformer for video. * Add tutorial to docs. * Add folded val files for easier 'early stop.' * Add and document transformer. * ignore batch if over allowed tokens batch, add warning (OpenNMT#1490) * allow implicit batch_size in translation_server (OpenNMT#1492) * ensure building sequence mask on same device as lengths (OpenNMT#1494) * add preprocess_opt in rest server (ZH) (OpenNMT#1493) * fix build_dataset_iter in train_single (OpenNMT#1499) * Use functions as preprocess / postprocess in REST server (OpenNMT#1505) * add preprocess_opt in rest server (ZH) * add preprocess and postprocess in rest server * simplify * fix function name * fix function name v2 * [fix] remove implicit check in preprocess (OpenNMT#1507) * [fix] remove implicit check in preprocess There were some implicit checks on `src_vocab` and `tgt_vocab` in preprocessing. This was creating some unwanted behavior when loading an existing vocab as a text file. * fix typo * add attention_dropout separate from dropout (OpenNMT#1512) * add attention_dropout separate from dropout * fix compatibility with models without attention_dropout (OpenNMT#1514) * pytorch 1.2 compatibility - mask & bool tensor (OpenNMT#1527) * Fix typo: traget -> target (OpenNMT#1537) * Tokens batch for translation (OpenNMT#1545) * wip translate batch tokens * move logic in translator
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces 2 things:
noam with decay (decay_method="noamwd")
When training on a lot of GPUs, the number of steps may be reduced (eg 50k) and you may want to decay faster the classic noam curve to a lower value.
accum gradient scheduler
When training on several GPUs, if you want a good performance you need a very high value of accum_count (eg 24 or more) but if you start training with such a huge batch size, it decreases the performance.
accum_count can now take a list of values, eg '4 8 16' which corresponds to a list of steps at which accum_count changes '10000 20000 30000'
Enjoy.