Support custom stop function?

[imoneoi]

Consider supporting a function as a stop argument. This will speed up some generation use cases, such as stopping when an answer matches in evaluation.

[gesanqiu]

stop parameter is already supported, you can see the SamplingParams for details.

[creatorrr]

@imoneoi but how would you pass a function? Do you mean to use a DSL like jsonschema or a CFG or something?

But I’d really love to have something like this. Perhaps a simple regex parameter could work?

[creatorrr]

The stop parameter accepts a list of strings but I think what @imoneoi is suggesting is more like stop_when=lambda x: int(x) % 2

@gesanqiu

[imoneoi]

The stop parameter accepts a list of strings but I think what @imoneoi is suggesting is more like stop_when=lambda x: int(x) % 2

@gesanqiu

I think this interface is good, pass RequestOutput as an argument is okay, which would fit most use scenarios of the custom stop function, such as implementing the vanilla stop on a piece of text, parsing JSON, or matching answers.

[zhuohan123]

Added this to our roadmap. However, the stop criteria is on the critical path of the decoding process. We might not be able to support this due to performance issues. Will investigate more.

[nathan-az]

Hey @zhuohan123 just wanted to elaborate in my use case for this. I'd be curious how common it is.

OSS models often struggle with looping. For code examples, I've had luck finetuning them on a specific format, (e.g. backtick surrounds) so when they fail to use an EOS token, often I can stop generation if there's a regex match on a pattern such as:

r"```.+?```"

I manage this in HF using the stopping_criteria and a custom function.

[hmellor]

Closing because early stopping has now been implemented.

[DreamGenX]

@hmellor Hey there! Can you give a pointer? Because I can't find this functionality. Early stopping is beam search specific functionality and does not let you pass custom function.

It might be possible to achieve this with custom logit processors and EOS.

[ZiyiGaoCN]

@hmellor Hey there! Can you give a pointer? Because I can't find this functionality. Early stopping is beam search specific functionality and does not let you pass custom function.

It might be possible to achieve this with custom logit processors and EOS.

Agree. This issue should be reopen since the request feature is not properly resolved.

[hmellor]

Ah I hadn't realised that early_stopping was beam search specific. However, you could use logits_processors to ensure that the stop token is generated when arbitrary conditions are met?

Snippet from #1728 (comment):

def process_token(token_ids, logits):
    logits[6623] = -9999.999 
    logits[4853] = -9999.999
    logits[630] = -9999.999
    logits[985] = -9999.999
    return logits

sampling_params = SamplingParams(
            n=1,
            temperature=temperature,
            top_p=top_p,
            use_beam_search=use_beam_search,
            stop=list(stop),
            max_tokens=max_new_tokens,
            top_k=top_k,
            presence_penalty=presence_penalty,
            frequency_penalty=frequency_penalty,
            best_of=best_of,
            logits_processors=[process_token]
        )

[DreamGenX]

Yes, that's what I meant by:

It might be possible to achieve this with custom logit processors and EOS.

This is the approach I am using now. Specifically I use float("-inf") instead for all non-EOS tokens and non-inf for EOS.

Don't forget to also handle ignore_eos correctly.

[goswamig]

Having a more custom stop function available in vllm will make vllm seamless to integrate with HF models. For e.g. something like HF's transformers Stopping Criteria makes life easy to add custom stop.

This issue might need another visit and may be extend the existing stop_checker

Regarding perf impact implication, I think its better to have this option and let user choose with the assumption that there will be perf impact.