[Feature request] Support CPU+GPU mixed execution

[malfet]

Assumption right now, it's only needed when there is not enough GPU memory, but perhaps sometimes it's just faster this way

Right now we only doing tokenization on CPU and inference can run on either CPU or GPU

[mikekgfb]

Already supported in PyTorch, but it's on the user to orchestrate