How to make my own channel spritemaps for my TensorFlow model？

[PaulChongPeng]

Does lucid support to make my own TensorFlow model visualizations now？And how to make my own channel spritemaps？
Thanks！

[ludwigschubert]

We will have both soon! There will be a notebook showing how to import a custom model, and a recipe for creating sprites as well.

[vijayvee]

Has this functionality for visualizing custom tensorflow models been added? Thanks!

[ludwigschubert]

@vijayvee we have! Unfortunately documentation is rough because of the many ways people store their models. Read through the linked notebook, and give feedback if the instructions don't work for you. :-)

[vijayvee]

Sure, I shall check it out. Thanks! 👯‍♂️

[vijayvee]

@ludwigschubert I did go through the notebook, it's great! Although it looks like I'm stuck at a point where tensorflow computes an output size that goes negative.

I am trying to visualize kernels in my custom Resnetv2 model with a [224,224,3] input layer.

Right before the global average pooling in the end, where the actual spatial dimensions should be [7,7], the model building code in render_vis receives a tensor with spatial dimensions [5,5]. As a result, while lucid build the model, it errors out before applying an average pool with kernel size [7,7], since the input it receives is an [n,5,5,c] tensor.

Am I messing up with some default padding that's being added to the input before lucid builds the model? Have you encountered this sort of an error before with lucid?

Error:

InvalidArgumentError (see above for traceback): Computed output size would be negative: -1 [input_size: 5, effective_filter_size: 7, stride: 1]
[[Node: import/cnn/average_pooling2d/AvgPool = AvgPoolT=DT_FLOAT, data_format="NHWC", ksize=[1, 7, 7, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[ludwigschubert]

Are you sure your Resnetv2's input size is not 229x229x3?

We are trying to support various input sizes (one effort, for example, is this forget_xy function).
However, if you're actually visualizing a kernel that's pretty deep into your architecture, the pooling can be set up in such a way that you do need to give it (at least) the input size it was built for.

[vijayvee]

Yes, I am sure my model takes an input of size [224,224,3]. I am able to render visualizations if I switch the size of my average pool from 7 to 5, although the rendered visualizations are not super-interpretable. I'm guessing the model being loaded is in some way different from the model stored in the checkpoint 🤔

Anyway, I can dig deeper into that. Is there also a way to create those amazing sprite-maps of the sorts I see here for my custom model? If yes, could you please direct me to it?

[ludwigschubert]

No guarantees, but things you can try:

• Use a bigger input. At some point you should at least get a different error—in all cases I have seen pooling is set up in such a way as to downsample inputs by a constant factor, so a larger input will at some point cause the layer you're concerned about to have a width of 7 rather than just 5.
• Create a colab notebook that reproduces your error and we can take a look.

We have no code we can share for spritesheet creation yet. Conceptually, we created the spritesheets by visualizing each individual neuron/channel in the network—so no magic.

[ludwigschubert]

@vijayvee I realized I left an ambiguity in my wording earlier. Are you specifying an input size of 224x224 for lucid? That is, are you using param.image(224,…)? If you don't specify a custom parameterization, lucid defaults to 128x128. This behaviour could cause the error you are seeing.

[vijayvee]

@ludwigschubert THAT FIXED IT! 💃 I guess it would be neat if you could emphasize somewhere early in the tutorial notebook about passing the custom model input size through param.

Also, I noticed that there could have been a little demo code for exporting a graphdef. I've added a couple comments in the same notebook with a link to some demo code. Thanks!

[JaspervDalen]

@ludwigschubert Is there already some code available to create spritemaps?

[vijayvee]

@JaspervDalen You could check this notebook for something close. This notebook will walk you through creating visualizations for all conv layers in a network, but you could swap this out with any (or all) layer(s) you want to visualize.

Stitching the resulting images into a spritemap should be trivial. Hope this helps.