Code

[Core] Support loading GGUF model#5191

Code

Merged

mgoin merged 76 commits intovllm-project:mainvllm-project/vllm:mainfrom Isotr0py:ggufIsotr0py/vllm:ggufCopy head branch name to clipboardAug 5, 2024

+4,970-21

Commits

Commits on Jun 2, 2024

• init gguf loading support

  Isotr0py

  committed
  1ffda2e

• add gguf running support

  Isotr0py

  committed
  f3058b1

• Fix numpy warning

  Isotr0py

  committed
  259d5b5

• Merge remote-tracking branch 'upstream/main' into gguf

  Isotr0py

  committed
  0035bdf

• fix gguf load format

  Isotr0py

  committed
  995f98e

• add more example prompts

  Isotr0py

  committed
  d116f2e

• update requirements.txt

  Isotr0py

  committed
  f387f9e

Commits on Jun 3, 2024

• add dequant runtime

  Isotr0py

  committed
  516552a

• remove debug code

  Isotr0py

  committed
  de5950d

Commits on Jun 4, 2024

• format code

  Isotr0py

  committed
  5bda5f0

• update gguf example

  Isotr0py

  committed
  980c018

• Merge branch 'main' into gguf

  Isotr0py

  authored
  f969b36

Commits on Jun 5, 2024

• Merge branch 'vllm-project:main' into gguf

  Isotr0py

  authored
  e99f521

• Fix requirements.txt

  Isotr0py

  committed
  9d36996

• rename ggml -> gguf

  Isotr0py

  committed
  3a18502

• auto detect gguf quant and format

  Isotr0py

  committed
  e194e28

• use autotokenizer to load gguf tokenizer

  Isotr0py

  committed
  164b643

Commits on Jun 6, 2024

• Add runtime dequantization for all layers

  Isotr0py

  committed
  b055fb3

Commits on Jun 18, 2024

• Merge branch 'main' into gguf

  Isotr0py

  authored
  c93c44e

Commits on Jun 19, 2024

• port gguf cuda kernel

  Isotr0py

  committed
  8960270

Commits on Jun 21, 2024

• add qwen2 support and gguf mmq for linear

  Isotr0py

  committed
  1d0c6a4

• remove transformers load_dequant_gguf_tensor

  Isotr0py

  committed
  957faec

Commits on Jun 22, 2024

• reorder gguf weight iterator

  Isotr0py

  committed
  4555cf5

• fix imatrix

  Isotr0py

  committed
  7f7af2b

• fix imatrix

  Isotr0py

  committed
  87078be

• refactor, fix column parallel

  Isotr0py

  committed
  ca39edf

Commits on Jun 24, 2024

• refactor gguf_kernel and remove dmmv

  Isotr0py

  committed
  cf03757

Commits on Jun 29, 2024

• refactor to unmerge weights for gguf

  Isotr0py

  committed
  c2524a8

• revert get_quantization_config

  Isotr0py

  committed
  446c64a

• revert get_quantization_config

  Isotr0py

  committed
  dc43654

• revert qwen2

  Isotr0py

  committed
  2861670

• add quant vocal embeddings

  Isotr0py

  committed
  1622966

• support quantized parallelhead

  Isotr0py

  committed
  c4d4f96

• revert qwen2

  Isotr0py

  committed
  9a99252

Commits on Jul 3, 2024

• Merge remote-tracking branch 'upstream/main' into gguf

  Isotr0py

  committed
  bc1ab48

• rebase gguf support

  Isotr0py

  committed
  3fad5bd

• format code

  Isotr0py

  committed
  409bed3

• format code

  Isotr0py

  committed
  b38bd1d

Commits on Jul 4, 2024

• support qwen2 gguf

  Isotr0py

  committed
  3586f12

• Merge branch 'main' into gguf

  Isotr0py

  authored
  8a56d55

• fix gguf loader

  Isotr0py

  committed
  defe23f

• add gguf test

  Isotr0py

  committed
  6c4300e

• format code

  Isotr0py

  committed
  266447b

• format code

  Isotr0py

  committed
  d5a7e2f

• remove archs<7.0 in cmakelists

  Isotr0py

  committed
  6026e02

• fix a typo

  Isotr0py

  committed
  9dc8794

• format code

  Isotr0py

  committed
  ef9b8a3

• format code

  Isotr0py

  committed
  b708ce6

Commits on Jul 5, 2024

• fix failed model test

  Isotr0py

  committed
  be51a27

Commits on Jul 7, 2024

• Merge branch 'vllm-project:main' into gguf

  Isotr0py

  authored
  1bd7d16

Commits on Jul 10, 2024

• Merge branch 'main' into gguf

  Isotr0py

  authored
  c155f74

• add imatrix and qwen2 test

  Isotr0py

  committed
  e49f96e

Commits on Jul 12, 2024

• reorganize gguf kernel

  Isotr0py

  committed
  af0c051

• exclude gguf copied code

  Isotr0py

  committed
  0ce3961

• refactor to merge weights

  Isotr0py

  committed
  e599b07

• forma code

  Isotr0py

  committed
  25dcc08

• format code

  Isotr0py

  committed
  eed9a23

• import gguf

  Isotr0py

  committed
  6e5330d

Commits on Jul 13, 2024

• import gguf

  Isotr0py

  committed
  e5a61be

• refactor quantized vocal embedding

  Isotr0py

  committed
  64c5375

Commits on Jul 14, 2024

• optimize docs

  Isotr0py

  committed
  86ef2b5

Commits on Jul 17, 2024

• add docs

  Isotr0py

  committed
  7ccfacb

• Merge remote-tracking branch 'upstream/main' into gguf

  Isotr0py

  committed
  28dc7b6

• fix llama embed quant

  Isotr0py

  committed
  1b39fbc

Commits on Jul 18, 2024

• Fix CUDA graph with gguf

  Isotr0py

  committed
  d413f60

Commits on Jul 28, 2024

• Merge remote-tracking branch 'upstream/main' into gguf

  Isotr0py

  committed
  1868a94

• fix quant embeddings

  Isotr0py

  committed
  b4e2f29

Commits on Jul 31, 2024

• Merge branch 'main' into gguf

  mgoin

  authored
  2cc6753

• Fix embedding method and format

  mgoin

  committed
  db54a19

• Cleanup linear comments

  mgoin

  committed
  2549c3e

Commits on Aug 1, 2024

• move gguf to cuda requirements

  Isotr0py

  committed
  0890fa9

• raise error for gguf when tp>1

  Isotr0py

  committed
  5166ac9

Commits on Aug 5, 2024

• Merge branch 'main' into gguf

  mgoin

  authored
  26349db

• Last round of cleanup

  mgoin

  committed
  73da240

• Improve qweight_type size calc

  mgoin

  committed
  1c83d63

• Fix lm head tests

  mgoin

  committed
  1139e7b