Inference compile cache script

[philschmid]

What does this PR do?

This PR adds a tools script to easily add LLMs to the public cache. It leverages the optimum-cli behind the scenes and checks if the version matches. There are currently 2 ways to use it

Single compilation:

huggingface-cli login --token hf_xxx # access to cache repo
python tools/cache_model_for_inference.py --hf_model_id "HuggingFaceH4/zephyr-7b-beta" --batch_size 1 --sequence_length 2048 --num_cores 2 --auto_cast_type fp16

File based compilation

python tools/auto_fill_inference_cache.py --config_file test.json

with a file

{
  "openai-community/gpt2": [
    {
      "batch_size": 1,
      "sequence_length": 1024,
      "num_cores": 1,
      "auto_cast_type": "fp16"
    }
  ],
  "meta-llama/Llama-2-7b-chat-hf": [
    {
      "batch_size": 1,
      "sequence_length": 4096,
      "num_cores": 2,
      "auto_cast_type": "fp16"
    },
    {
      "batch_size": 1,
      "sequence_length": 4096,
      "num_cores": 8,
      "auto_cast_type": "fp16"
    }
  ],

Remote file based config

python tools/auto_fill_inference_cache.py --config_file https://huggingface.co/aws-neuron/optimum-neuron-cache/raw/main/inference-cache-config/gpt2.json

The configs can be found in the aws-neuron/optimum-neuron-cache under inference-cache-config

[dacorvo]

Since it partially duplicates auto_fill_neuronx_inference_cache.py, could you transform this into a simple configuration JSON file for you own script ?

[dacorvo]

This will fail everytime you pass a config_file without a model_id if you don't return here or use an elif

[dacorvo]

This is not about the schema. Here you will err because the user did pass a config_file that you already parsed, and you nevertheless expect them to have passed a model_id.

[philschmid]

I think i got what you meant. I made it a hardcode "if elif else" without fall through

[dacorvo]

What I meant is that if the input was a config file, then the script should end immediately after having processed it, and not try to also parse the other arguments.

[dacorvo]

OK, so your last commit. About the schema, naaah, this is overkill.

[dacorvo]

LGTM, thanks !

[philschmid]

Tested with

python tools/auto_fill_inference_cache.py --config_file https://huggingface.co/aws-neuron/optimum-neuron-cache/raw/main/inference-cache-config/gpt2.json