Adds method to read the pooling types from model's files #9506

flaviabeo · 2024-10-18T14:19:03Z

This adds a method to load the pooling config file from sentence transformer models like sentence-transformers/all-MiniLM-L12-v2.

The pooling types added can be found at the sentence-transformers Pooling

FIX #9388 (link existing issues this PR will resolve)

cc: @maxdebayser

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Adding or changing kernels

Each custom kernel needs a schema and one or more implementations to be registered with PyTorch.

Make sure custom ops are registered following PyTorch guidelines: Custom C++ and CUDA Operators and The Custom Operators Manual
Custom operations that return Tensors require meta-functions. Meta-functions should be implemented and registered in python so that dynamic dims can be handled automatically. See above documents for a description of meta-functions.
Use torch.libary.opcheck() to test the function registration and meta-function for any registered ops. See tests/kernels for examples.
When changing the C++ signature of an existing op, the schema must be updated to reflect the changes.
If a new custom type is needed, see the following document: Custom Class Support in PT2.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

github-actions · 2024-10-18T14:19:16Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

robertgshaw2-neuralmagic · 2024-10-21T16:33:55Z

Thanks! This is great! Can you make sure to add some integration tests?

vllm/model_executor/models/bert.py

DarkLight1337 · 2024-10-22T06:23:47Z

vllm/model_executor/layers/pooler.py

+    MEAN = 3
+
+
+class PoolingConfig():


Maybe make this a dataclass?

DarkLight1337 · 2024-10-22T06:25:51Z

vllm/transformers_utils/config.py

+def get_pooling_config(model, revision='main'):
+    """
+    This function gets the pooling and normalize 
+    config from the model.


Can we rename this so it's more obvious that this only applies to sentence-transformers models? We currently have embedding models that are not from sentence-transformers, so it's rather confusing why only BERT accepts PoolingConfig.

Sorry about the force push!! I made a rebase with main to solve the conflicts.

maxdebayser · 2024-10-23T12:58:11Z

tests/model_executor/test_model_load_with_params.py

+                     dtype=torch.half if QUANTIZATION == "gptq" else "auto",
+                     max_model_len=MAX_MODEL_LEN) as model:
+        output = model.encode("Write a short story about a robot that"
+                              " dreams for the first time.\n")


Here we need to add a verification to make sure that the pooling layer is configured correctly

maxdebayser · 2024-10-23T13:00:36Z

vllm/engine/llm_engine.py

-            model_config.max_model_len,
-            load_config.download_dir,
-            load_config.load_format,
+            "mm_processor_kwargs=%s, pooling_config=%s)", VLLM_VERSION,


It looks like the linter did more than it should have here. Ideally a commit should not change code lines that are unrelated.

DarkLight1337 · 2024-10-23T15:32:32Z

vllm/engine/llm_engine.py

-            model_config.rope_scaling,
-            model_config.rope_theta,
-            model_config.tokenizer_revision,
-            model_config.trust_remote_code,
-            model_config.dtype,
-            model_config.max_model_len,
-            load_config.download_dir,
-            load_config.load_format,
+            "mm_processor_kwargs=%s, "
+            "pooling_config_type=%s, normalize=%s)", VLLM_VERSION,
+            model_config.model, speculative_config, model_config.tokenizer,
+            model_config.skip_tokenizer_init, model_config.tokenizer_mode,
+            model_config.revision, model_config.override_neuron_config,
+            model_config.rope_scaling, model_config.rope_theta,
+            model_config.tokenizer_revision, model_config.trust_remote_code,
+            model_config.dtype, model_config.max_model_len,
+            load_config.download_dir, load_config.load_format,
            parallel_config.tensor_parallel_size,
            parallel_config.pipeline_parallel_size,
            parallel_config.disable_custom_all_reduce,
-            model_config.quantization,
-            model_config.enforce_eager,
-            cache_config.cache_dtype,
-            model_config.quantization_param_path,
-            device_config.device,
-            decoding_config,
-            observability_config,
-            model_config.seed,
-            model_config.served_model_name,
+            model_config.quantization, model_config.enforce_eager,
+            cache_config.cache_dtype, model_config.quantization_param_path,
+            device_config.device, decoding_config, observability_config,
+            model_config.seed, model_config.served_model_name,
            scheduler_config.num_scheduler_steps,
            scheduler_config.chunked_prefill_enabled,
            scheduler_config.multi_step_stream_outputs,
            cache_config.enable_prefix_caching,
-            model_config.use_async_output_proc,
-            use_cached_outputs,
-            model_config.mm_processor_kwargs,
-        )
+            model_config.use_async_output_proc, use_cached_outputs,
+            model_config.mm_processor_kwargs, 
+            model_config.pooling_config.pooling_type, 
+            model_config.pooling_config.normalize)


Let's keep this at one argument per line.

vllm/model_executor/layers/pooler.py

DarkLight1337 · 2024-10-23T15:33:58Z

vllm/model_executor/model_loader/loader.py

+    return build_model(model_class,
+                       model_config.hf_config,
+                       cache_config=cache_config,
+                       quant_config=_get_quantization_config(
+                           model_config, load_config),
+                       lora_config=lora_config,
+                       multimodal_config=model_config.multimodal_config,
+                       scheduler_config=scheduler_config,
+                       pooling_config=model_config.pooling_config)


I think the old indentation is easier to read.

vllm/config.py

vllm/transformers_utils/tokenizer_group/__init__.py

maxdebayser · 2024-10-24T12:19:05Z

vllm/config.py

@@ -1835,6 +1851,9 @@ def _get_and_verify_max_len(
                raise ValueError(
                    f"{msg} To allow overriding this maximum, set "
                    "the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN=1")
+
+    if bert_config:


Can we put this at line 1821 like this?

if bert_config and "max_seq_lenght" in bert_config: derived_max_model_len = bert_config["max_seq_length"]

Otherwise it will disable the user override logic.

maxdebayser · 2024-10-24T14:26:16Z

vllm/config.py

+        self.bert_config = self._get_bert_config()
+        self.do_lower_case = self._get_bert_tokenization_config()


Suggested change

self.bert_config = self._get_bert_config()

self.do_lower_case = self._get_bert_tokenization_config()

self.bert_config, self.do_lower_case = self._get_bert_config()

vllm/config.py

maxdebayser · 2024-10-24T14:30:10Z

vllm/transformers_utils/config.py

+    - dict: A dictionary containing the configuration parameters 
+    for the Sentence Transformer BERT model.
+    """
+    bert_dict = get_hf_file_to_dict("sentence_bert_config.json", model,


"sentence_bert_config.json" is one of several possible names: https://github.com/UKPLab/sentence-transformers/blob/f286d9f210824d6ea1563e789f49894b19c24f0e/sentence_transformers/models/Transformer.py#L421C13-L421C24

Thank you! Will be adding these.

maxdebayser · 2024-10-24T14:31:24Z

vllm/transformers_utils/config.py

+    return None
+
+
+def get_sentence_transformer_bert_config(model, revision='main'):


Maybe rename to get_sentence_transformer_tokenizer_config

maxdebayser · 2024-10-24T14:35:07Z

vllm/transformers_utils/tokenizer_group/__init__.py

+    if model_config.do_lower_case is not None:
+        init_kwargs["do_lower_case"] = model_config.do_lower_case
+


Suggested change

if model_config.do_lower_case is not None:

init_kwargs["do_lower_case"] = model_config.do_lower_case

if model_config.bert_config is not None and "do_lower_case" in model_config.bert_config:

init_kwargs["do_lower_case"] = model_config.bert_config["do_lower_case"]

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

mergify · 2024-10-29T23:03:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. @flaviabeo please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337 · 2024-10-30T13:12:22Z

Sorry, can you hold this for a bit? I'm trying to get #9697 merged which will definitely introduce merge conflicts. The CI is failing on main branch right now so we are all waiting for force merge.

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

mergify · 2024-10-30T16:34:51Z

This pull request has merge conflicts that must be resolved before it can be
merged. @flaviabeo please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

maxdebayser · 2024-10-31T23:28:56Z

In vllm/engine/arg_utils.py the MEAN type is missing in the set of acceptable values for the command line arg:

        parser.add_argument(
            '--pooling-type',
            choices=['LAST', 'ALL', 'CLS', 'STEP'],
            default=None,

In before the rebase you had a version where this list was generated automatically from the Enum type. Can you add this again?

maxdebayser · 2024-10-31T23:06:41Z

vllm/config.py

@@ -107,6 +108,8 @@ class ModelConfig:
            can not be gathered from the vllm arguments.
        config_format: The config format which shall be loaded.
            Defaults to 'auto' which defaults to 'hf'.
+        bert_config: tokenizationconfiguration dictionary for a given 


bert_config is not actually a parameter of the __init__ method. The documentation can be moved close to the initialization of self.bert_config or removed. Otherwise the docstring will be wrong.

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

vllm/config.py

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

maxdebayser

LGTM

flaviabeo force-pushed the main branch 3 times, most recently from 6b01faf to 0d6123a Compare October 21, 2024 15:31

flaviabeo marked this pull request as ready for review October 21, 2024 16:32

flaviabeo requested review from WoosukKwon, zhuohan123, youkaichao, alexm-neuralmagic, comaniac and njhill as code owners October 21, 2024 16:32

DarkLight1337 reviewed Oct 22, 2024

View reviewed changes

flaviabeo requested review from mgoin, simon-mo, tlrmchlsmth, ywang96 and robertgshaw2-neuralmagic as code owners October 22, 2024 13:59

flaviabeo force-pushed the main branch from 7260d79 to c6af5d5 Compare October 22, 2024 14:05

flaviabeo requested a review from DarkLight1337 October 23, 2024 12:28

maxdebayser reviewed Oct 23, 2024

View reviewed changes

DarkLight1337 reviewed Oct 23, 2024

View reviewed changes

vllm/model_executor/layers/pooler.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Oct 23, 2024

View reviewed changes

maxdebayser reviewed Oct 24, 2024

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved

maxdebayser reviewed Oct 24, 2024

View reviewed changes

flaviabeo added 5 commits October 28, 2024 18:33

Format linting

ae73f4b

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

wip

02195c8

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

test

37167ff

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

Fixes and Exception for not supported pooling types

3d36a8c

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

Fixing lint

fbcd540

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

flaviabeo force-pushed the main branch from 44390a1 to 83eec19 Compare October 29, 2024 16:46

Fixing linting

c268d89

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

flaviabeo force-pushed the main branch from 83eec19 to c268d89 Compare October 29, 2024 16:52

mergify bot added the needs-rebase label Oct 29, 2024

Merge branch 'upstream_main'

5627d2f

mergify bot removed the needs-rebase label Oct 30, 2024

flaviabeo requested review from maxdebayser and DarkLight1337 October 30, 2024 13:11

Lint

0da7979

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

mergify bot added the needs-rebase label Oct 30, 2024

flaviabeo added a commit to flaviabeo/vllm that referenced this pull request Oct 31, 2024

Apply changes from vllm-project#9506 at main

3d29a18

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

Merge branch 'upstream_main'

4531c33

mergify bot removed the needs-rebase label Oct 31, 2024

flaviabeo force-pushed the main branch from 36fee51 to e1cc7e2 Compare October 31, 2024 21:18

Fix merge conflicts

8df9d63

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

flaviabeo force-pushed the main branch from e1cc7e2 to 8df9d63 Compare October 31, 2024 21:19

maxdebayser reviewed Oct 31, 2024

View reviewed changes

Review changes requested

4ac1f20

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

DarkLight1337 reviewed Nov 1, 2024

View reviewed changes

vllm/config.py Outdated Show resolved Hide resolved

Simplify pooler config init

ff9705b

Signed-off-by: Flavia Beo <flavia.beo@ibm.com>

maxdebayser approved these changes Nov 1, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds method to read the pooling types from model's files #9506

Adds method to read the pooling types from model's files #9506

flaviabeo commented Oct 18, 2024 •

edited

Loading

github-actions bot commented Oct 18, 2024

robertgshaw2-neuralmagic commented Oct 21, 2024

DarkLight1337 Oct 22, 2024

DarkLight1337 Oct 22, 2024 •

edited

Loading

flaviabeo Oct 22, 2024

maxdebayser Oct 23, 2024

maxdebayser Oct 23, 2024

DarkLight1337 Oct 23, 2024

DarkLight1337 Oct 23, 2024 •

edited

Loading

maxdebayser Oct 24, 2024

maxdebayser Oct 24, 2024

maxdebayser Oct 24, 2024

flaviabeo Oct 24, 2024

maxdebayser Oct 24, 2024

maxdebayser Oct 24, 2024

mergify bot commented Oct 29, 2024

DarkLight1337 commented Oct 30, 2024 •

edited

Loading

mergify bot commented Oct 30, 2024

maxdebayser commented Oct 31, 2024

maxdebayser Oct 31, 2024

maxdebayser left a comment

		self.bert_config = self._get_bert_config()
		self.do_lower_case = self._get_bert_tokenization_config()

	self.bert_config = self._get_bert_config()
	self.do_lower_case = self._get_bert_tokenization_config()
	self.bert_config, self.do_lower_case = self._get_bert_config()

		return None


		def get_sentence_transformer_bert_config(model, revision='main'):

		if model_config.do_lower_case is not None:
		init_kwargs["do_lower_case"] = model_config.do_lower_case

Adds method to read the pooling types from model's files #9506

Are you sure you want to change the base?

Adds method to read the pooling types from model's files #9506

Conversation

flaviabeo commented Oct 18, 2024 • edited Loading

PR Title and Classification

Code Quality

Adding or changing kernels

Notes for Large Changes

What to Expect for the Reviews

Thank You

github-actions bot commented Oct 18, 2024

robertgshaw2-neuralmagic commented Oct 21, 2024

Choose a reason for hiding this comment

DarkLight1337 Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DarkLight1337 Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Oct 29, 2024

DarkLight1337 commented Oct 30, 2024 • edited Loading

mergify bot commented Oct 30, 2024

maxdebayser commented Oct 31, 2024

Choose a reason for hiding this comment

maxdebayser left a comment

Choose a reason for hiding this comment

flaviabeo commented Oct 18, 2024 •

edited

Loading

DarkLight1337 Oct 22, 2024 •

edited

Loading

DarkLight1337 Oct 23, 2024 •

edited

Loading

DarkLight1337 commented Oct 30, 2024 •

edited

Loading