Attempt to infer the output shape for Deconvolution2D layer

[lukovkin]

I've used the same approach that I've used in 1D Convolution Transpose implementation here - https://github.com/lukovkin/ufcnn-keras/blob/master/models/convolutional_transpose.py#L11-L23.
Was also inspired by the output shape calculation in Caffe - https://github.com/BVLC/caffe/blob/master/src/caffe/layers/deconv_layer.cpp#L14-L21 (we don't account for the dilated case for now AFAIK).

It looks like it passes the test at least for TensorFlow, but I will play with it a bit more.

[yaringal]

cool. let me know when you're happy with it and I'll merge (also note the conflicts)

[lukovkin]

It works for TF for all cases, but it fails for the Theano early, and as I've checked it fails for Theano if I rollback to your latest commit before my changes - 0742daa (see error log below).
Any advice will be appreciated.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:

ValueError: The hardcoded shape for the image stack size (2) isn't the run time shape (3).

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-5-19b390959db5> in <module>()
----> 1 test_deconvolution_2d()

/root/keras/keras/utils/test_utils.py in wrapper(*args, **kwargs)
    114     @functools.wraps(func)
    115     def wrapper(*args, **kwargs):
--> 116         output = func(*args, **kwargs)
    117         if K._BACKEND == 'tensorflow':
    118             K.clear_session()

<ipython-input-4-6d69c97f2378> in test_deconvolution_2d()
     32                                    'subsample': subsample},
     33                            input_shape=(nb_samples, stack_size, nb_row, nb_col),
---> 34                            fixed_batch_size=True)
     35 
     36                 layer_test(convolutional.Deconvolution2D,

/root/keras/keras/utils/test_utils.py in layer_test(layer_cls, kwargs, input_shape, input_dtype, input_data, expected_output, expected_output_dtype, fixed_batch_size)
     76 
     77     expected_output_shape = layer.get_output_shape_for(input_shape)
---> 78     actual_output = model.predict(input_data)
     79     actual_output_shape = actual_output.shape
     80     assert expected_output_shape == actual_output_shape

/root/keras/keras/engine/training.py in predict(self, x, batch_size, verbose)
   1177         f = self.predict_function
   1178         return self._predict_loop(f, ins,
-> 1179                                   batch_size=batch_size, verbose=verbose)
   1180 
   1181     def train_on_batch(self, x, y,

/root/keras/keras/engine/training.py in _predict_loop(self, f, ins, batch_size, verbose)
    876                 ins_batch = slice_X(ins, batch_ids)
    877 
--> 878             batch_outs = f(ins_batch)
    879             if type(batch_outs) != list:
    880                 batch_outs = [batch_outs]

/root/keras/keras/backend/theano_backend.py in __call__(self, inputs)
    616     def __call__(self, inputs):
    617         assert type(inputs) in {list, tuple}
--> 618         return self.function(*inputs)
    619 
    620 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    869                     node=self.fn.nodes[self.fn.position_of_error],
    870                     thunk=thunk,
--> 871                     storage_map=getattr(self.fn, 'storage_map', None))
    872             else:
    873                 # old-style linkers raise their own exceptions

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/gof/link.py in raise_with_op(node, thunk, exc_info, storage_map)
    312         # extra long error message in that case.
    313         pass
--> 314     reraise(exc_type, exc_value, exc_trace)
    315 
    316 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
    683             value = tp()
    684         if value.__traceback__ is not tb:
--> 685             raise value.with_traceback(tb)
    686         raise value
    687 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    857         t0_fn = time.time()
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:
    861             if hasattr(self.fn, 'position_of_error'):

ValueError: The hardcoded shape for the image stack size (2) isn't the run time shape (3).
Apply node that caused the error: ConvOp{('imshp', (2, 10, 6)),('kshp', (3, 3)),('nkern', 2),('bsize', 2),('dx', 1),('dy', 1),('out_mode', 'full'),('unroll_batch', 2),('unroll_kern', 2),('unroll_patch', False),('imshp_logical', (2, 10, 6)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}(input_1, Subtensor{::, ::, ::int64, ::int64}.0)
Toposort index: 4
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(2, 3, 10, 6), (2, 3, 3, 3)]
Inputs strides: [(720, 240, 24, 4), (108, 36, -12, -4)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Add}[(0, 0)](ConvOp{('imshp', (2, 10, 6)),('kshp', (3, 3)),('nkern', 2),('bsize', 2),('dx', 1),('dy', 1),('out_mode', 'full'),('unroll_batch', 2),('unroll_kern', 2),('unroll_patch', False),('imshp_logical', (2, 10, 6)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}.0, Reshape{4}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/root/keras/keras/utils/test_utils.py", line 116, in wrapper
    output = func(*args, **kwargs)
  File "<ipython-input-4-6d69c97f2378>", line 34, in test_deconvolution_2d
    fixed_batch_size=True)
  File "/root/keras/keras/utils/test_utils.py", line 71, in layer_test
    y = layer(x)
  File "/root/keras/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/root/keras/keras/layers/convolutional.py", line 441, in call
    filter_shape=self.W_shape)
  File "/root/keras/keras/backend/theano_backend.py", line 1066, in deconv2d
    conv_out = op(kernel, x, output_shape[2:])
  File "/root/keras/keras/utils/test_utils.py", line 116, in wrapper
    output = func(*args, **kwargs)
  File "<ipython-input-4-6d69c97f2378>", line 34, in test_deconvolution_2d
    fixed_batch_size=True)
  File "/root/keras/keras/utils/test_utils.py", line 71, in layer_test
    y = layer(x)
  File "/root/keras/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/root/keras/keras/layers/convolutional.py", line 441, in call
    filter_shape=self.W_shape)
  File "/root/keras/keras/backend/theano_backend.py", line 1066, in deconv2d
    conv_out = op(kernel, x, output_shape[2:])

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

[yaringal]

weird.. the tests seem to pass here
keras-team#3251
and also on my machine.. make sure you don't have any .pyc files lying around.
Also, is the number of filters correct in the output shape?

[lukovkin]

I think there was a kind of inconsistency here - we receive the filter_shape as a parameter, after that we reshape the kernel itself (shape should change) but leave filter_shape intact and pass it to the AbstractConv2d_gradInputs then.

Probably we could totally remove filter_shape from the parameters, if there are no other reasons to use it.

[lukovkin]

Here I stuck probably the same issue that @yaringal mentioned - if we infer output_shape, it becomes the tuple of Theano variables/expressions, which Theano couldn't accept as an input due to some reasons.
So I try to evaluate them, but here I get another error:

MissingInputError: An input of the graph, used to compute Shape(input_25), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.

I'm not an expert in Theano, so have no idea how to fix it quick.

[lukovkin]

Now it passes TF tests and Theano tests where output_shape is not being inferred. Need to fix evaluation of Theano expressions in output_shape.

[yaringal]

@fchollet any suggestions for the theano part?

[lukovkin]

Passing of K.eval(K.shape(x)) instead of K.shape(x) doesn't help either.

self.output_shape_ = self.get_output_shape_for(K.eval(K.shape(x)))

[yaringal]

@lukovkin my deconv PR has been merged into fchollet/keras. You can open a new PR in fchollet/keras if you want to merge these changes

[fchollet]

A proper solution would be to do shape inference yourself, without relying on Theano. Basically do Theano's job, in Python-land.