Arbitrary tensor modification tracking

[ffuuugor]

There are many possible tensor modifications that break DP guarantees.
We make a weak attempt to verify the model does not break any conventions (e.g. check for BatchNorm layers), but it's in no way comprehensive.

Is there a way to generalise such modification? Only allow certain operations? Use tensor hooks somehow. No answer yet, but interesting topic to explore

[ffuuugor]

One idea on how to approach this: functional testing.
Static analysis might prove very challenging, but running a batch and then comparing the results with individually-run examples looks easy. We do the same when unit-testing grad samplers, now we just need to package it differently and provide interface for clients