[Misc]: Throughput/Latency for guided_json with ~100% GPU cache utilization

[jens-create]

Anything you want to discuss about vllm.

Hi,

I am running some benchmarks on the vllm.entrypoints.openai.api_server measuring latency and throughput with different number of concurrent requests.

Specs:

• H100 80GB
• qwen-1.5-14B-chat

I am sending 1000 requests with random prompts of token length 512. These are the results I get (see attached image):

Guided_json

• ~100 running requests
• ~70 generation tokens per second
• ~1700 ms median token time

Non-guided_json

• ~100 running requests
• ~800 generation tokens per second
• ~75 ms median token time (TPOT)

At 10 concurrent request (GPU utlization << 100%

Non-guided_json: ~20 ms median token time
guided_json: ~ 160 ms median token time

Currently the application I am building heavily relies on guided_json, however, to put it in an online setting I would like to ask 1) are the numbers I experience sensible and 2) what can be done to improve performance in the guided_json paradigm?

I am debating whether I should try and prompt my way to structured outputs and thus avoiding constrained decoding.

)

[simon-mo]

Is the JSON schema complex at all, and is it the same each time? The 70 toks/s number is a bit lower than I expected. This can be due to several factor if it's the same schema:

• Currently the logits mask computation is performed on the critical path but it can be moved earlier.
• We currently don't batch the application of logits processors.
• Python overhead in general

I'm interested in fixing the performance here.

[jens-create]

Hi Simon,

The JSON schema is the same at all times, and it is as follows:

"guided_json": {"$defs": {"SearchQuery": {"description": "Search query for the retrieval task.", "properties": {"query_type": {"description": "The type of query most effective for handling the retrieval task.", "title": "Query Type", "type": "string"}, "query": {"description": "A random user's search query.", "title": "Query", "type": "string"}}, "required": ["query_type", "query"], "title": "SearchQuery", "type": "object"}}, "description": "A list of search queries anticipating a user looking for information from a given web page.", "properties": {"queries": {"description": "Brainstormed search queries for the given web page.", "items": {"$ref": "#/$defs/SearchQuery"}, "title": "Queries", "type": "array"}}, "required": ["queries"], "title": "Brainstorm", "type": "object"}

Thanks for looking into this 🫶

[jens-create]

@simon-mo any update on this? 😊

[taoisu]

Facing similar issue here, I have a json with 14 fields, the request stucks forever.

[lithafnium]

My schema only has 2 fields and also has significant latency issues than when using without guided_json. Would love to have this fixed as model performance severely decreases without it.

[simon-mo]

I would suggest trying out setting --guided-decoding-backend lm-format-enforcer (through args) or "guided_decoding_backend": "lm-format-enforcer" as part of the request to see whether it helps. see original PR here: #3868 (cc @noamgat)

[noamgat]

If testing lm-format-enforcer, I highly recommend adding the latest version of it to the image, as there have been performance improvements to the JsonSchemaParser. The next version of vLLM will include them, but until them, do pip install lm-format-enforcer==0.10.1 in the image before testing.

[jarrelscy]

what speeds are you getting @noamgat vs the outlines backend?

[noamgat]

I didn't test on A100/H100s, but on my dev setup (GTX 3090, Mistral7B), for simple schemas, I was getting a less than 2x reduction of tokens/s.

[nullpointer0xffff]

+1, it seems not GPU related, I tested with A100 / V100 GPUs both have similar issue.

Using line profiler, I found this get_guided_decoding_logits_processor call takes 93% time

[nullpointer0xffff]

If testing lm-format-enforcer, I highly recommend adding the latest version of it to the image, as there have been performance improvements to the JsonSchemaParser. The next version of vLLM will include them, but until them, do pip install lm-format-enforcer==0.10.1 in the image before testing.

This get vllm 0.4.1+cu118 requires lm-format-encorcer==0.9.8, requiring to add --no-deps.

Just tested, the speed up is not obvious, probabbly the main bottleneck is still the get_guided_decoding_logits_processor

[nullpointer0xffff]

If testing lm-format-enforcer, I highly recommend adding the latest version of it to the image, as there have been performance improvements to the JsonSchemaParser. The next version of vLLM will include them, but until them, do pip install lm-format-enforcer==0.10.1 in the image before testing.

This get vllm 0.4.1+cu118 requires lm-format-encorcer==0.9.8, requiring to add --no-deps.

Just tested, the speed up is not obvious (25tok/s -> 32 tok/s on V100), probabbly the main bottleneck is still the get_guided_decoding_logits_processor

[nullpointer0xffff]

@noamgat here's a profling when I use lm-format-enforcer 0.10.1.

/lib/python3.10/site-packages/lmformatenforcer/integrations/transformers.py

Function: _build_regular_tokens_list at line 58

 

Line #      Hits         Time  Per Hit   % Time  Line Contents

==============================================================

    58                                           @profile

    59                                           def _build_regular_tokens_list(tokenizer: PreTrainedTokenizerBase) -> List[Tuple[int, str, bool]]:

    60         1  912794903.0    9e+08      9.5      token_0 = tokenizer.encode("0")[-1]

    61         1       8025.0   8025.0      0.0      regular_tokens = []

    62    128257   28050361.0    218.7      0.3      for token_idx in range(len(tokenizer)):

    63    128256   78294452.0    610.5      0.8          if token_idx in tokenizer.all_special_ids:

    64         2        450.0    225.0      0.0              continue

    65                                                   # We prepend token 0 and skip the first letter of the result to get a space if the token is a start word.

    66    128254 5319568501.0  41476.8     55.3          decoded_after_0 = tokenizer.decode([token_0, token_idx])[1:]

    67    128254 3162992335.0  24661.9     32.9          decoded_regular = tokenizer.decode([token_idx])

    68    128254   56427009.0    440.0      0.6          is_word_start_token = len(decoded_after_0) > len(decoded_regular)

    69    128254   61975079.0    483.2      0.6          regular_tokens.append((token_idx, decoded_after_0, is_word_start_token))

70         1        240.0    240.0      0.0      return regular_tokens

The two decoding in for loop seems took most time. Happy to make further test if needed.