How to retrieve the LLM hidden state?

[ChenShuwei1001]

Hi, I'm using ChatGLM3 as an encoder to encode sentences, and vllm is deployed to speed up the process.

ChatGLM3 contains a transformer encoder to generate hidden state (with shape L x D where L is #.tokens and D is the dimensionality), and a linear 'decoder' to generate next token. I want to use the hidden state of the final input token to represent the sentence. I have carefully read the source code of vllm, but I can't find a clear solution for my requirement, unless drastically revise the code.

Is there any api or configuration that can meet my requirement? Or do you have any suggestion for implementing my requirement?

Best regards.

[ChenShuwei1001]

The encoder will be invoked multiple times during scheduling, and I only want to use the hidden state from the first invoking.

[ChenShuwei1001]

BTW, I've tested vllm on my data with 'generation length=1' (which means only a token is generated), and it processes ~30 samples/second. On the other hand, I also tried the hugging face ChatGLM3 (w/o vllm) without generating any token (by breaking the forward process after encoder hidden states calculated), and the throughput is ~25 samples/second.

When generation length is limited, the difference seems not very obvious, is it normal? And I do try vllm with unlimited generation length, which brings 8x speed up.