3D Style Transfer: Use content/output texture only where needed

I was wondering about the viability of only sampling the content/output texture and updating where it's necessary from the view frame for each iteration.

More specifically to sample the content texture frame and the optimizing texture frame for the given view and then send that to the optimizing model as inputs for the content loss. This would (I think) keep the full size textures away from the backprop step and (I think) would allow for arbitrarily large texture sizes constrained instead by the MeshRenderer size.

It would be amazing to be able to trade some speed and more iterations for really high resolution textures without being constrained by GPU memory.

In the sample_bilinear call, I tried various ways of forcing the gradients to be sparse in the gather_nd function, but no luck so far, seems like it's a common issue people have with word embeddings.

I might just try intercepting the gradients before the gather and update it explicitly into the texture image, keeping track of the sampled indices.