Importing model and visualizing it with Lucid

[ricardobarroslourenco]

I'm trying to open an autoencoder model I've trained myself, on Lucid, and I'm using as reference the notebook Importing a graph into modelzoo.

I'm mostly in doubt on how to use the provided class:

  model_path = 'nasnet_mobile_graphdef_frozen.pb.modelzoo'
  image_shape = [224, 224, 3]
  image_value_range = (0, 1)
  input_name = 'input' 

What should I define as image_shape, image_value_range? For what images I'm considering this? The output of a certain convolutional layer?

Also, for what is defined the input_name?

[tschwabe]

Hi Ricardo,

the input_name refers to the input-node of your Tensorflow-graph. The image-shape, as far as i know, refers to the size of images your model expects. The image_value_range specifies the values, that are possible for every channel in your Picture. For a typical picture, those are between 0 and 255. However, for a CNN you normalize them to be between -1 and 1 or 0 and 1.

When i tried to import my own model into Lucid, it only worked when i removed all constraints for the expected size of the images, exept the number of channels(3).

[ricardobarroslourenco]

Thanks for the highlight @tschwabe . Another question, if I want to run Lucid on a model with more channels (>3), is it possible to use it? I wonder that perhaps I need to constrain it for three channels due to the RGB mapping.

[tschwabe]

Im not exactly sure. I would recommend to remove the constraints from the height and the width and keep it for the channels, thats what i did.

[ricardobarroslourenco]

First question: When you said to remove width and height constraints, and keep channels, would be something like going from [224, 224, 3] to [:, :, 3]?

Second: I've now tested just passing the model_path and input_name arguments, and got this error:
ValueError: Dimensions must be equal, but are 3 and 7 for 'import/conv2d_1/convolution' (op: 'Conv2D') with input shapes: [1,?,?,3], [3,3,7,16]. It is weird, because I've not specified a image_shape argument.

[tschwabe]

1.) Yeah, [None,None,3], to be exactly. But I removed such constraints inside of my model, not in image_shape !

2.) I would assume this Error arises, just like in my case, because the shapes inside your model are fixed and therefore not aligned with image_shape

[ricardobarroslourenco]

Oh, I see. How did you change/edit your trained model? (I'm new to Tensorflow)

[tschwabe]

I trained my model with fixed size of the pictures. Then i constructed a new model with the same architecture, but no constraints, and loaded the wheights from the other model Into the new one..
I guess there is a prettier way to do it..

[ludwigschubert]

image_shape, image_value_range, and input_name are all required even in theory.

In a nutshell, when lucid imports your model's graph, it needs to replace your model's input (usually a tf.placeholder) with it's own parameterization. To construct this input, it needs to know about the following:

• image_shape so the input lucid creates fits onto your model. (See details after this listing.)
• image_value_range so the input lucid creates provides pixel values similar to those your model was trained on. Different architectures use different pre-processing, so we can not guess. Many architectures normalize to between 0 and 1, but some use -1 to 1, yet others expect 0 to 225.
• input_name is the name of the tensor (usually a tf.placeholder) that is used as an input in your original model. We need to know about it so we know where to feed data into your model. Many architectures simply call this input, but different architectures use different names, so we can not guess.

image_shape has an additional layer of complexity, because we try to allow for flexibility in the input size. In models that pool, smaller inputs lead to smaller spatial dimensions in higher layers—which can lead to shape issues when e.g. your pooling layer has a kernel size that is larger than the remaining spatial dimensions.

As a first step, set image_shape to the input size your original model expects, and use param.image(...) with that same size. This should avoid most shape-related issues.

Also, @ricardobarroslourenco, feel free to create colab notebooks that let me reproduce your issue. I can not promise actual support, but if you can make it easier for me to work on your problem… it increases the probability of me doing just that. Thanks for being an early adopter! :-)

[ludwigschubert]

@tschwabe 's extra step should not be necessary when you tell lucid to use the same input size as you use during training.

[ricardobarroslourenco]

@ludwigschubert thanks for the thorough answer. Let me give you some background on my application.

In my project, I'm building a convolutional autoencoder, but the application is meant for remote sensing images I'm parsing from Google Earth Engine. More specifically I'm ingesting several multispectral images (7 channels, each one associated with a spectral band acquired at the satellite sensor) in which I want to replicate cloud texture patterns. We believe that such cloud class we feed to the autoencoder is too broad, and we expect to have a reasonable amount of variance at the hidden layers, even the most compact ones, even after convergence (by class misclassification of a cloud class, or due to cloud classes that are not described in the present framework, but were enclosed into a hyper class due to lack of a more proper label).

The reason I'm willing to use Lucid is to streamline the process of visualizing the embeddings I'm learning because I will probably drive my network architecture development, and parametrization, by the features I'm able to represent, and their quality. Ideally, this embeddings would show textures present in the samples I'm providing and would be subject to analysis by geophysicists on their feasibility as new cloud classes.

Currently, I've been working on a colab notebook which needs some cleanup, and I'll be glad to share it with you 😃

What google account should I share it to?

[ludwigschubert]

Sounds like an exciting application!

My Google account is [EDIT]

[ricardobarroslourenco]

Thanks @ludwigschubert . I've just shared with you a more clean and commented notebook. Thanks a lot for your availability :)