Add option to use hf tokenizer

[RissyRan]

Description

Add option to use hf tokenizer

Test

• provide invalid hf tokenizer, it will failed to find right tokenizer

python JetStream/benchmarks/benchmark_serving.py   \
--tokenizer ~/maxtext/assets/tokenizer.mistral-v3  \
--save-result   \
--save-request-outputs   \
--request-outputs-file-path outputs-scanned-8x22b-hf-4.json   \
--num-prompts 4   \
--max-output-length 1024   \
--dataset openorca --run-eval true --use-hf-tokenizer=True


Using HuggingFace tokenizer: /home/ranran/maxtext/assets/tokenizer.mistral-v3
Traceback (most recent call last):
  File "/home/ranran/venv-maxtext/lib/python3.10/site-packages/transformers/utils/hub.py", line 403, in cached_file
    resolved_file = hf_hub_download(
  File "/home/ranran/venv-maxtext/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/home/ranran/venv-maxtext/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/ranran/maxtext/assets/tokenizer.mistral-v3'. Use `repo_type` argument if needed.

• provide valid hf tokenizer, it will pass

python JetStream/benchmarks/benchmark_serving.py   \
--tokenizer mistralai/Mixtral-8x22B-Instruct-v0.1  \
--save-result   \
--save-request-outputs   \
--request-outputs-file-path outputs-scanned-8x22b-hf-4.json   \
--num-prompts 4   \
--max-output-length 1024   \
--dataset openorca --run-eval true --use-hf-tokenizer=True

Using HuggingFace tokenizer: mistralai/Mixtral-8x22B-Instruct-v0.1
len(sampled_indices)=4
In InputRequest, pass in actual output_length for each sample
The dataset contains 4 samples.
The filtered dataset contains 4 samples.
  0%|                                                                                                                               | 0/4 [00:00<?, ?it/s]

[vipannalla]

LGTM, thanks

[vipannalla]

User needs to accept the MixtralAI (or other model) agreeement on HuggingFace, else this line will error out. Can you add a comment about it for future users?

[RissyRan]

Sure thing!

[RissyRan]

Added comment accordingly.