float8 investigate cuda graphs + dynamic scaling leading to memory fragmentations

TODO add more context / a repro, for now just capturing the request from chat

copied from https://github.com/pytorch-labs/float8_experimental/issues/249