The document of generation seems to wrongly describe the default value of top_p, top_k and temperature

[Kamichanw]

System Info

The newest transformers

Who can help?

@stevhliu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The document of generation says that top_p, top_k and temperature will be set to a specific value.

However, as shown below, the settings of top_p, top_k, and temperature depend on the generation configuration of the model, which is not guaranteed to be consistent with the behavior described in the docstring.

transformers/src/transformers/generation/utils.py

Lines 1035 to 1044 in 052e652

	if generation_config.temperature is not None and generation_config.temperature != 1.0:
	processors.append(TemperatureLogitsWarper(generation_config.temperature))
	if generation_config.top_k is not None and generation_config.top_k != 0:
	processors.append(
	TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep)
	)
	if generation_config.top_p is not None and generation_config.top_p < 1.0:
	processors.append(
	TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep)
	)

Expected behavior

I'd like to know if I'm missing something, and if not, please make the docstring of "generation" and its behavior consistent.

[stevhliu]

Thanks for the feedback! I think we can clarify the docstrings of these parameters to reflect what you're saying. Something along the lines of:

temperature (`float`, *optional*, defaults to 1.0) - The value used to module the next token probabilities. This value is set in a model's `generation_config.json` file. If it isn't set, the default value is 1.0.

[Kamichanw]

Yes, that's make sense