Get NAN loss after 35k steps

[StevenLOL]

Anyone got NAN ?

[abisee]

@StevenLOL I see this happen sometimes too -- seems to be a very common problem with Tensorflow training in general.

[hate5six]

I've been having this problem and I decreased the learning rate as per various discussions on SO and that seemed to work. After a while I tried increasing it by 0.01 and started getting NaN's again. I've tried restoring the checkpoint and re-running with the lower learning rate, but I'm still seeing NaN. Does this mean my checkpoint is useless?

[bytecraft-ai]

I am also getting NaN. Found out the culprit to be
line #227: log_dists = [tf.log(dist) for dist in final_dists]
in model.py

[hate5six]

@Rahul-Iisc Is your workaround to filter out cases where dist == 0, for example:

log_dists = [tf.log(dist) for dist in final_dists if dist != 0] ?

[bytecraft-ai]

@hate5six I'm still thinking about an appropriate solution. Each dist is a tensor shape (batch_size, extended_vsize), so I am not sure if dist != 0 will work. Also, I want the log_dists length to be the same as final_dists

[bytecraft-ai]

Trying to convert NaNs to 0 for now. Need to further look for the cause of 0s coming up in the distribution. @abisee @StevenLOL reopen the issue?

def _change_nan_to_number(self, tensor, number=1):
    return tf.where(tf.logical_not(tf.is_finite(tensor)), tf.ones_like(tensor) * number, tensor)

log_dists = [tf.log(self._change_nan_to_number(dist)) for dist in final_dists]

UPDATE: This didn't work. Loss ended up 0 instead of NaN.

[bytecraft-ai]

The below change worked for me. Add the below code to def _calc_final_dist(self, vocab_dists, attn_dists). Info is in comments.

      # OOV part of vocab is max_art_oov long. Not all the sequences in a batch will have max_art_oov tokens.
      # That will cause some entries to be 0 in the distribution, which will result in NaN when calulating log_dists
      # Add a very small number to prevent that.

      def add_epsilon(dist, epsilon=sys.float_info.epsilon):
        epsilon_mask = tf.ones_like(dist) * epsilon
        return dist + epsilon_mask

      final_dists = [add_epsilon(dist) for dist in final_dists]
      
      return final_dists

[lizaigaoge550]

final_dists = [tf.clip_by_value(dist,1e-10,1.) for dist in final_dists]

[jamesposhtiger]

@lizaigaoge550 did this work for you?
final_dists = [tf.clip_by_value(dist,1e-10,1.) for dist in final_dists]
Could you let me know which line you put this at?
Many thanks

[lizaigaoge550]

@jamesposhtiger
after finishing the final_dists