Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model][VLM] Add Qwen2-VL model support #7905

Merged
merged 44 commits into from
Sep 11, 2024
Merged

[Model][VLM] Add Qwen2-VL model support #7905

merged 44 commits into from
Sep 11, 2024

Commits on Aug 23, 2024

  1. Add support to Qwen2-VL.

    @fyabc
    fyabc committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    0a648b2 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Merge branch 'refs/heads/main' into add_qwen2_vl_new

    @fyabc
    fyabc committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    320df57 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Reformat

    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    7f96df8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refs/heads/main' into add_qwen2_vl_new

    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    fbf2b8b View commit details
    Browse the repository at this point in the history

  3. Update transformers link.

    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    bcaff4f View commit details
    Browse the repository at this point in the history

  4. Bugfix of mrope_input_positions in model_runner.py.

    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    f2185bf View commit details
    Browse the repository at this point in the history

  5. Rename pixel_values_video to pixel_values_videos in qwen2_vl.py. 

    Add Qwen2-VL support in chat_utils.py.
    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    60448cb View commit details
    Browse the repository at this point in the history

  6. Fix the bug of MultiModalInputs.batch() when passing different modali… 

    …ties in a single batch.
    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    71a77b1 View commit details
    Browse the repository at this point in the history

  7. Fix the bug when running OpenAI-compatible API server.

    @fyabc
    fyabc committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    60c4cbd View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Merge branch 'refs/heads/main' into add_qwen2_vl_new

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    e29ff54 View commit details
    Browse the repository at this point in the history

  2. Refactor qwen2_vl.py based on review comments.

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    ddb7138 View commit details
    Browse the repository at this point in the history

  3. reformat

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    14fe12a View commit details
    Browse the repository at this point in the history

  4. reformat

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    89def23 View commit details
    Browse the repository at this point in the history

  5. Fix the bug of model_is_mrope in model_runner.py.

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    e721e60 View commit details
    Browse the repository at this point in the history

  6. fix type hints in qwen2_vl.py

    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    d66d167 View commit details
    Browse the repository at this point in the history

  7. Update mm input processors according to new MultiModalInput.batch() i… 

    …mplementation.
    @fyabc
    fyabc committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    acd85ed View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Merge branch 'refs/heads/main' into add_qwen2_vl_new 

    # Conflicts:
#	vllm/worker/model_runner.py
    @fyabc
    fyabc committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    8d762c6 View commit details
    Browse the repository at this point in the history

  2. Fix SamplerOutput.

    @fyabc
    fyabc committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    87ba5ed View commit details
    Browse the repository at this point in the history

  3. Fix bug of quantization.

    @fyabc
    fyabc committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    cda300a View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2024

  1. Bugfix of type hints in qwen2_vl.py.

    @fyabc
    fyabc committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    da03a3f View commit details
    Browse the repository at this point in the history

  2. reformat.

    @fyabc
    fyabc committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    25fb189 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. Merge branch 'main' into add_qwen2_vl_new

    @ywang96
    ywang96 authored Sep 1, 2024
    Configuration menu
    Copy the full SHA
    d01530d View commit details
    Browse the repository at this point in the history

  2. fix typo from resolving conflict

    @ywang96
    ywang96 authored Sep 1, 2024
    Configuration menu
    Copy the full SHA
    faebfe4 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Merge branch 'refs/heads/main' into add_qwen2_vl_new

    @fyabc
    fyabc committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    e492e53 View commit details
    Browse the repository at this point in the history

  2. Bugfix in qwen2_vl.py.

    @fyabc
    fyabc committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    2e87db7 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Adding xformers implementation

    @fyabc
    fyabc committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    39a1069 View commit details
    Browse the repository at this point in the history

  2. Fix bug of attn_bias in xformers implementation

    @fyabc
    fyabc committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    855c78b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Fix bug in xformers implementation, and add backend check in vision a… 

    …ttention.
    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    091983f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refs/heads/main' into add_qwen2_vl_new 

    # Conflicts:
#	vllm/model_executor/models/__init__.py
    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    b406571 View commit details
    Browse the repository at this point in the history

  3. Bugfix in qwen2_vl.py.

    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    7739588 View commit details
    Browse the repository at this point in the history

  4. Bugfix in qwen2_vl.py.

    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    5bab9ba View commit details
    Browse the repository at this point in the history

  5. reformat.

    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    4587346 View commit details
    Browse the repository at this point in the history

  6. Refactor MRotaryEmbedding.

    @fyabc
    fyabc committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    ffad79f View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Merge branch 'refs/heads/main' into add_qwen2_vl_new

    @fyabc
    fyabc committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    9e7a946 View commit details
    Browse the repository at this point in the history

  2. Add "video" into ModalityStr.

    @fyabc
    fyabc committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    d527417 View commit details
    Browse the repository at this point in the history

  3. Add Qwen2-VL examples.

    @fyabc
    fyabc committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    6f3116c View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Optimizer Qwen2-VL input processor. Update document.

    @fyabc
    fyabc committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    386f302 View commit details
    Browse the repository at this point in the history

  2. Update model notes and requirements-common.txt.

    @fyabc
    fyabc committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    c64c217 View commit details
    Browse the repository at this point in the history

  3. Update model notes.

    @fyabc
    fyabc committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    6bdefd6 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Skip loading model

    @DarkLight1337
    DarkLight1337 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    33dd048 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_qwen2_vl_new

    @DarkLight1337
    DarkLight1337 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    369ce7d View commit details
    Browse the repository at this point in the history

  3. format

    @DarkLight1337
    DarkLight1337 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    282c66a View commit details
    Browse the repository at this point in the history

  4. Increase max_model_len to fit the original image

    @DarkLight1337
    DarkLight1337 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    14ef94d View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into add_qwen2_vl_new

    @DarkLight1337
    DarkLight1337 committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    09b7a4f View commit details
    Browse the repository at this point in the history