Submitting a PR for Implementation of Work for Mesh torch.Optimizer

[JulianKnodt]

🚀 Feature

I was wondering if it would be acceptable to submit a PR for an implementation of the paper:
https://rgl.epfl.ch/publications/Nicolet2021Large

This would be an implementation of torch.Optimizer which only operates on meshes, and since there are already a lot of existing tools for meshes in this repo, I thought it would make sense to put it here.

Motivation

Optimization over meshes from differentiable rendering is ill-posed with standard optimizers, as described in the paper, but the proposed one allows for better mesh optimization through both inverse rendering and other means.

Pitch

You should be able to to do any optimizations, similar to using an adam optimizer, but with an additional requirement that you can construct a mesh from the parameters, and there is one parameter per vertex, and it should optimize better.

For example:

opt = opt.MeshUniformAdam(params, mesh_from_params=lambda p: ..., lr=....)

[nikhilaravi]

Hi @JulianKnodt thanks for the suggestion. This definitely sounds like something that could be relevant for PyTorch3D. We'd need to think where this would go in the codebase.

Do you already have an implementation and a demo of using it? Or e.g. do you think we could modify the fit_textured_mesh or other tutorials to use this optimizer instead?

[JulianKnodt]

I was thinking it semantically makes sense under losses, since it's an optimization approach. That being said, I thought the original paper did not release their code, so I implemented my own version of it, but later found that they had released an implementation after I had looked at it. The assumption they make in their implementation is that they optimize over coordinates in the latent space u, whereas I originally thought that it would be in the original vertex space, so the optimizer I implemented reconstructs the latent space every time by recomputing the laplacian and resolving it. I can put up a PR to look at it tmrw, but the original authors have a demo here, and their whole codebase is tiny.
https://github.com/rgl-epfl/large-steps-pytorch/blob/master/Tutorial.ipynb

I would think that modifying one of the tutorials to use this method would be more confusing, but if it's most appropriate I can add a tutorial for this to the repo.

[JulianKnodt]

Sorry about the delay, I've been caught up in other things but will get to this at some point.

[JulianKnodt]

So I've tried out a basic implementation, but it doesn't seem to be very effective, even though it resembles their code release. I suspect this may be due to how gradients are handled in either the solver or the rasterization process. If I had more time, I might pursue this further, but as of now I don't think I'll be able to. If anyone is interested in continuing this the implementation is here. Sorry about that!