Skip to content

Explore vicuna-1.5 16k context models #60

@the-crypt-keeper

Description

@the-crypt-keeper

Currently only transformers runtime can handle the fp16 versions, but vLLM has an open PR to support rope-scaling: vllm-project/vllm#555

Since we run everything else with vLLM it would be good to do apples-to-apples.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions