Implement Gradient Accumulation #2
Conversation
|
What's your distribute strategy? |
|
For training my model I use |
|
Does the bug happen on single GPU? |
Your code works for me on CPU and single GPU but not on multi-GPU. |
|
Could you debug my code with |
|
Add a test case for embedding. |
Your code seems to work with I added a test for the |
|
|
||
| if isinstance(grad, tf.IndexedSlices): | ||
| # Not sure why e.g. the Embedding layer requires an additional dimension here | ||
| grad_indices = grad.indices[..., None] if len(grad.indices.shape) < 2 else grad.indices |
| return dataset | ||
|
|
||
| strategy = tf.distribute.MirroredStrategy(test_utils.gpus_for_testing()) | ||
| strategy = tf.distribute.get_strategy() |
There was a problem hiding this comment.
@fsx950223 I noticed that I used the wrong strategy here.
|
It's a good idea to use two-pointers to accumulate gradients. It's better to beautify the code. |
What do you mean? btw: do we need this PR or should I close it? |
|
Since the google cla issue, you should open another PR without my commits. |
2 participants
@fsx950223 want to discuss here?