Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to infer the output shape for Deconvolution2D layer #4

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

lukovkin
Copy link

I've used the same approach that I've used in 1D Convolution Transpose implementation here - https://github.com/lukovkin/ufcnn-keras/blob/master/models/convolutional_transpose.py#L11-L23.
Was also inspired by the output shape calculation in Caffe - https://github.com/BVLC/caffe/blob/master/src/caffe/layers/deconv_layer.cpp#L14-L21 (we don't account for the dilated case for now AFAIK).

It looks like it passes the test at least for TensorFlow, but I will play with it a bit more.

@yaringal
Copy link
Owner

cool. let me know when you're happy with it and I'll merge (also note the conflicts)

@lukovkin
Copy link
Author

It works for TF for all cases, but it fails for the Theano early, and as I've checked it fails for Theano if I rollback to your latest commit before my changes - 0742daa (see error log below).
Any advice will be appreciated.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:

ValueError: The hardcoded shape for the image stack size (2) isn't the run time shape (3).

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-5-19b390959db5> in <module>()
----> 1 test_deconvolution_2d()

/root/keras/keras/utils/test_utils.py in wrapper(*args, **kwargs)
    114     @functools.wraps(func)
    115     def wrapper(*args, **kwargs):
--> 116         output = func(*args, **kwargs)
    117         if K._BACKEND == 'tensorflow':
    118             K.clear_session()

<ipython-input-4-6d69c97f2378> in test_deconvolution_2d()
     32                                    'subsample': subsample},
     33                            input_shape=(nb_samples, stack_size, nb_row, nb_col),
---> 34                            fixed_batch_size=True)
     35 
     36                 layer_test(convolutional.Deconvolution2D,

/root/keras/keras/utils/test_utils.py in layer_test(layer_cls, kwargs, input_shape, input_dtype, input_data, expected_output, expected_output_dtype, fixed_batch_size)
     76 
     77     expected_output_shape = layer.get_output_shape_for(input_shape)
---> 78     actual_output = model.predict(input_data)
     79     actual_output_shape = actual_output.shape
     80     assert expected_output_shape == actual_output_shape

/root/keras/keras/engine/training.py in predict(self, x, batch_size, verbose)
   1177         f = self.predict_function
   1178         return self._predict_loop(f, ins,
-> 1179                                   batch_size=batch_size, verbose=verbose)
   1180 
   1181     def train_on_batch(self, x, y,

/root/keras/keras/engine/training.py in _predict_loop(self, f, ins, batch_size, verbose)
    876                 ins_batch = slice_X(ins, batch_ids)
    877 
--> 878             batch_outs = f(ins_batch)
    879             if type(batch_outs) != list:
    880                 batch_outs = [batch_outs]

/root/keras/keras/backend/theano_backend.py in __call__(self, inputs)
    616     def __call__(self, inputs):
    617         assert type(inputs) in {list, tuple}
--> 618         return self.function(*inputs)
    619 
    620 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    869                     node=self.fn.nodes[self.fn.position_of_error],
    870                     thunk=thunk,
--> 871                     storage_map=getattr(self.fn, 'storage_map', None))
    872             else:
    873                 # old-style linkers raise their own exceptions

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/gof/link.py in raise_with_op(node, thunk, exc_info, storage_map)
    312         # extra long error message in that case.
    313         pass
--> 314     reraise(exc_type, exc_value, exc_trace)
    315 
    316 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
    683             value = tp()
    684         if value.__traceback__ is not tb:
--> 685             raise value.with_traceback(tb)
    686         raise value
    687 

/root/miniconda2/envs/keras/lib/python3.5/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    857         t0_fn = time.time()
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:
    861             if hasattr(self.fn, 'position_of_error'):

ValueError: The hardcoded shape for the image stack size (2) isn't the run time shape (3).
Apply node that caused the error: ConvOp{('imshp', (2, 10, 6)),('kshp', (3, 3)),('nkern', 2),('bsize', 2),('dx', 1),('dy', 1),('out_mode', 'full'),('unroll_batch', 2),('unroll_kern', 2),('unroll_patch', False),('imshp_logical', (2, 10, 6)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}(input_1, Subtensor{::, ::, ::int64, ::int64}.0)
Toposort index: 4
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(2, 3, 10, 6), (2, 3, 3, 3)]
Inputs strides: [(720, 240, 24, 4), (108, 36, -12, -4)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Add}[(0, 0)](ConvOp{('imshp', (2, 10, 6)),('kshp', (3, 3)),('nkern', 2),('bsize', 2),('dx', 1),('dy', 1),('out_mode', 'full'),('unroll_batch', 2),('unroll_kern', 2),('unroll_patch', False),('imshp_logical', (2, 10, 6)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}.0, Reshape{4}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/root/keras/keras/utils/test_utils.py", line 116, in wrapper
    output = func(*args, **kwargs)
  File "<ipython-input-4-6d69c97f2378>", line 34, in test_deconvolution_2d
    fixed_batch_size=True)
  File "/root/keras/keras/utils/test_utils.py", line 71, in layer_test
    y = layer(x)
  File "/root/keras/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/root/keras/keras/layers/convolutional.py", line 441, in call
    filter_shape=self.W_shape)
  File "/root/keras/keras/backend/theano_backend.py", line 1066, in deconv2d
    conv_out = op(kernel, x, output_shape[2:])
  File "/root/keras/keras/utils/test_utils.py", line 116, in wrapper
    output = func(*args, **kwargs)
  File "<ipython-input-4-6d69c97f2378>", line 34, in test_deconvolution_2d
    fixed_batch_size=True)
  File "/root/keras/keras/utils/test_utils.py", line 71, in layer_test
    y = layer(x)
  File "/root/keras/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/root/keras/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/root/keras/keras/layers/convolutional.py", line 441, in call
    filter_shape=self.W_shape)
  File "/root/keras/keras/backend/theano_backend.py", line 1066, in deconv2d
    conv_out = op(kernel, x, output_shape[2:])

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

@yaringal
Copy link
Owner

weird.. the tests seem to pass here
keras-team#3251
and also on my machine.. make sure you don't have any .pyc files lying around.
Also, is the number of filters correct in the output shape?

Dmitry Lukovkin and others added 6 commits July 24, 2016 21:58
…gal-master

Conflicts:
	keras/layers/convolutional.py
…er and actual kernel shape - evaluate kernel shape in runtime

BUG: Inferred output shape is a tuple of Theano variables/expressions, Theano expects tupe of int - Not fixed!
@@ -1052,7 +1052,15 @@ def deconv2d(x, kernel, output_shape, strides=(1, 1),
kernel = kernel.dimshuffle((1, 0, 2, 3))
th_border_mode = _preprocess_border_mode(border_mode)
np_kernel = kernel.eval()
filter_shape = _preprocess_filter_shape(dim_ordering, filter_shape)
filter_shape = _preprocess_filter_shape(dim_ordering, shape(kernel))
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there was a kind of inconsistency here - we receive the filter_shape as a parameter, after that we reshape the kernel itself (shape should change) but leave filter_shape intact and pass it to the AbstractConv2d_gradInputs then.

Probably we could totally remove filter_shape from the parameters, if there are no other reasons to use it.

for v in output_shape:
try:
v = v.eval()
_v = v.eval()
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I stuck probably the same issue that @yaringal mentioned - if we infer output_shape, it becomes the tuple of Theano variables/expressions, which Theano couldn't accept as an input due to some reasons.
So I try to evaluate them, but here I get another error:

MissingInputError: An input of the graph, used to compute Shape(input_25), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.

I'm not an expert in Theano, so have no idea how to fix it quick.

@lukovkin
Copy link
Author

Now it passes TF tests and Theano tests where output_shape is not being inferred. Need to fix evaluation of Theano expressions in output_shape.

@yaringal
Copy link
Owner

@fchollet any suggestions for the theano part?

@lukovkin
Copy link
Author

Passing of K.eval(K.shape(x)) instead of K.shape(x) doesn't help either.

self.output_shape_ = self.get_output_shape_for(K.eval(K.shape(x)))

@yaringal
Copy link
Owner

yaringal commented Jul 25, 2016

@lukovkin my deconv PR has been merged into fchollet/keras. You can open a new PR in fchollet/keras if you want to merge these changes

@fchollet
Copy link

A proper solution would be to do shape inference yourself, without relying on Theano. Basically do Theano's job, in Python-land.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants