Refactor python processor to use lib/ preprocessor and model deployment cards

[alec-flowers]

Right now we have duplicated work in our lib/ and in the python examples. We want to standardize on our lib/ and remove some of the bespoke stuff we have written in python.

This means removing the vLLM specific preprocessing and tokenization and switching to the pre-processor in lib/ and then also using the model deployment cards so we don't need to specify worker args in multiple locations. These args should be read from the MDC.

[alec-flowers]

@biswapanda you helped bring over and add python bindings for pre-processor and model deployment cards right? Any tips for @PeaBrane as he starts this work?

[michaelfeil]

Question on this: Is there a chance that you can apply this with the python chat template?
e.g. TGI lets this part run in the python tokenizer https://github.com/huggingface/text-generation-inference/blob/0b28aabb94e89360e762f2ac2945e8ae3f8e5dc0/router/src/server.rs#L1665

https://huggingface.co/docs/transformers/main/en/chat_templating