Search results

vLLM run script prefill + decode trace pre-capture to avoid TTFT on first completions being unexpectedly high or stalling

enhancementNew feature or requestNew feature or request

Status: Open.

#56 In tenstorrent/tt-inference-server;

· tstescoTT opened on Dec 12, 2024

Provide example chat template usage

documentationImprovements or additions to documentationImprovements or additions to documentation

enhancementNew feature or requestNew feature or request

Status: Open.

#36 In tenstorrent/tt-inference-server;

· tstescoTT opened on Nov 15, 2024

Docker run support for HF_TOKEN authentication using env var pass in

enhancementNew feature or requestNew feature or request

Status: Open.

#23 In tenstorrent/tt-inference-server;

· tstescoTT opened on Oct 24, 2024

Add status messaging and endpoint to allow for client-side users to reason about model initialization and life cycle.

enhancementNew feature or requestNew feature or request

Status: Open.

#17 In tenstorrent/tt-inference-server;

· tstescoTT opened on Sep 26, 2024

llama model install script support for llama CLI and huggingface hub

enhancementNew feature or requestNew feature or request

Status: Open.

#14 In tenstorrent/tt-inference-server;

· tstescoTT opened on Sep 26, 2024

Capture tt-metal and tt-NN loguru logs in inference server python log files

enhancementNew feature or requestNew feature or request

Status: Open.

#13 In tenstorrent/tt-inference-server;

· tstescoTT opened on Sep 25, 2024