Masking broken in v2.2.0

[humcasma]

• [ X] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

• [ X] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

The following (simplified) piece of code used to work in Keras 2.1.4:

from keras.models import Model, Sequential
from keras.layers import Dense, Input, Masking, Activation
from keras.layers.wrappers import TimeDistributed
from keras.layers.recurrent import LSTM

input = Input(shape=(3,2))
hidden = Masking(mask_value=-100)(input)
s = Sequential()
s.add(Dense(5, input_shape=(2,)))
s.add(Activation('elu'))
hidden = TimeDistributed(s)(hidden)
hidden = LSTM(10)(hidden)
m = Model(inputs=input, outputs=hidden)

When upgrading to Keras 2.2.0, it crashes with the following error trace:

  File "/Users/test/anaconda/envs/tensorflow/lib/python3.5/site-packages/keras/engine/base_layer.py", line 460, in __call__
    output = self.call(inputs, **kwargs)
  File "/Users/test/anaconda/envs/tensorflow/lib/python3.5/site-packages/keras/layers/wrappers.py", line 248, in call
    y = self.layer.call(inputs, **kwargs)
  File "/Users/test/anaconda/envs/tensorflow/lib/python3.5/site-packages/keras/engine/network.py", line 573, in call
    output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  File "/Users/test/anaconda/envs/tensorflow/lib/python3.5/site-packages/keras/engine/network.py", line 732, in run_internal_graph
    computed_mask)
  File "/Users/test/anaconda/envs/tensorflow/lib/python3.5/site-packages/keras/engine/base_layer.py", line 622, in compute_mask
    str(mask))
TypeError: Layer dense_1_input does not support masking, but was passed an input_mask: Tensor("time_distributed_1/Reshape_1:0", shape=(?,), dtype=bool)

If importing Keras via tensorflow 1.9, it works:

from tensorflow.python.keras.models import Model, Sequential
from tensorflow.python.keras.layers import Dense, Input, Masking, Activation
from tensorflow.python.keras.layers.wrappers import TimeDistributed
from tensorflow.python.keras.layers.recurrent import LSTM

[yanboliang]

@humcasma This is due to InputLayer(which is inside Sequential and be added automatically) doesn't support masking. I think it should support, so I sent #10794 to fix this. Wish it can help you.

[humcasma]

@yanboliang thanks for the fix, but I believe the problem is not completely solved. With the changes made to InputLayer the problem got solved when layers that support masking are used as the first layer of the Sequential model (like the Dense layer in the example code I provided). However, if we substitute Dense in the example code with, e.g., Conv2D or Flatten, the problem is still there.

[yanboliang]

@humcasma That's because layer Conv2D and Flatten don't support masking yet. Note Masking layer is used to masks a sequence to skip time steps, so it doesn't make sense to use Conv2D after Masking.

[humcasma]

@yanboliang Yes, I know Masking is used to mask time steps, and that is exactly how it is used in the example I originally posted. The Sequential model is applied to each time step with help of TimeDistributed. I believe that with independence of how the Sequential model is built, masking should work. And, indeed, it used to work in Keras 2.1.6.