We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent d5387a3 commit d05e8d4Copy full SHA for d05e8d4
docs/source/quantization/bnb.rst
@@ -15,11 +15,10 @@ Below are the steps to utilize BitsAndBytes with vLLM.
15
16
vLLM reads the model's config file and supports both in-flight quantization and pre-quantized checkpoint.
17
18
-Read quantized checkpoint.
19
-
20
You can find bitsandbytes quantized models on https://huggingface.co/models?other=bitsandbytes.
21
And usually, these repositories have a config.json file that includes a quantization_config section.
22
+Read quantized checkpoint.
23
--------------------------
24
25
.. code-block:: python
0 commit comments