update vector search docs #18779
Open
update vector search docs #18779
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ti-chi-bot
bot
added
missing-translation-status
ti-chi-bot
bot
removed
the
missing-translation-status
ti-chi-bot
bot
added
size/L
Oreoxmt
approved these changes
Sep 3, 2024
[LGTM Timeline notifier]Timeline:
|
ti-chi-bot
bot
added
size/XL
qiancai
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/approve cancel |
ti-chi-bot
bot
added
the
needs-rebase
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
qiancai
added
v8.4
Oreoxmt
reviewed
Oct 10, 2024
|
|
||
| ## Prerequisites | ||
|
|
||
| To complete this tutorial, you need: | ||
|
|
||
| - [Python 3.8 or higher](https://www.python.org/downloads/) installed. | ||
| - [Git](https://git-scm.com/downloads) installed. | ||
| - A TiDB Cloud Serverless cluster. Follow [creating a TiDB Cloud Serverless cluster](/tidb-cloud/create-tidb-cluster-serverless.md) to create your own TiDB Cloud cluster if you don't have one. | ||
| - A TiDB cluster. |
There was a problem hiding this comment.
Should we specify that v8.4.0 or higher is required for TiDB Self-Managed clusters?
There was a problem hiding this comment.
@Oreoxmt Good idea. How about we add that in L45?
| > | ||
| > This section is only applicable to [TiDB Serverless](https://docs.pingcap.com/tidbcloud/select-cluster-tier#tidb-serverless) clusters. | ||
|
|
||
| Define a 3-dimensional vector column and optimize it with a [vector search index](https://docs.pingcap.com/tidbcloud/vector-search-index) (HNSW index). |
There was a problem hiding this comment.
Suggested change
| Define a 3-dimensional vector column and optimize it with a [vector search index](https://docs.pingcap.com/tidbcloud/vector-search-index) (HNSW index). | |
| Define a 3-dimensional vector column and optimize it with a [vector search index](/vector-search-index.md) (HNSW index). |
| # For example: TIDB_DATABASE_URL="mysql+pymysql://root@127.0.0.1:4000/test" | ||
| ``` | ||
|
|
||
| If you are running TiDB on your local machine, `HOST` is `127.0.0.1` by default. The initial `PASSWORD` is empty, so if you are starting the cluster for the first time, you can omit this field. |
There was a problem hiding this comment.
Suggested change
| If you are running TiDB on your local machine, `HOST` is `127.0.0.1` by default. The initial `PASSWORD` is empty, so if you are starting the cluster for the first time, you can omit this field. | |
| If you are running TiDB on your local machine, `<HOST>` is `127.0.0.1` by default. The initial `<PASSWORD>` is empty, so if you are starting the cluster for the first time, you can omit this field. |
|
|
||
| If you are running TiDB on your local machine, `HOST` is `127.0.0.1` by default. The initial `PASSWORD` is empty, so if you are starting the cluster for the first time, you can omit this field. | ||
|
|
||
| The following are descriptions for each parameter: |
There was a problem hiding this comment.
Suggested change
| The following are descriptions for each parameter: | |
| The following are descriptions for each placeholder: |
There was a problem hiding this comment.
"parameter" here matches "connection parameters" in L124
Comment on lines
135
to
139
| - `<HOST>`: The host of the TiDB cluster. | ||
| - `<PORT>`: The port of the TiDB cluster. | ||
| - `<USER>`: The username to connect to the TiDB cluster. | ||
| - `<PASSWORD>`: The password to connect to the TiDB cluster. | ||
| - `<DATABASE>`: The name of the database you want to connect to. |
There was a problem hiding this comment.
Suggested change
| - `<HOST>`: The host of the TiDB cluster. | |
| - `<PORT>`: The port of the TiDB cluster. | |
| - `<USER>`: The username to connect to the TiDB cluster. | |
| - `<PASSWORD>`: The password to connect to the TiDB cluster. | |
| - `<DATABASE>`: The name of the database you want to connect to. | |
| - `<USER>`: The username to connect to the TiDB cluster. | |
| - `<PASSWORD>`: The password to connect to the TiDB cluster. | |
| - `<HOST>`: The host of the TiDB cluster. | |
| - `<PORT>`: The port of the TiDB cluster. | |
| - `<DATABASE>`: The name of the database you want to connect to. |
Oreoxmt
reviewed
Oct 10, 2024
Signed-off-by: JaySon-Huang <tshent@qq.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Co-authored-by: Aolin <aolinz@outlook.com>
Oreoxmt
reviewed
Oct 16, 2024
qiancai
commented
Oct 17, 2024
qiancai
commented
Oct 17, 2024
qiancai
commented
Oct 17, 2024
github-actions bot
pushed a commit
to qiancai/pingcap-docsite-preview
that referenced
this pull request
Oct 17, 2024
github-actions bot
pushed a commit
to qiancai/pingcap-docsite-preview
that referenced
this pull request
Oct 17, 2024
github-actions bot
pushed a commit
to qiancai/pingcap-docsite-preview
that referenced
this pull request
Oct 17, 2024
qiancai
commented
Oct 17, 2024
TOC.md
Outdated
There was a problem hiding this comment.
Oreoxmt
reviewed
Oct 17, 2024
| - ORM Libraries | ||
| - [SQLAlchemy](/vector-search-integrate-with-sqlalchemy.md) | ||
| - [peewee](/vector-search-integrate-with-peewee.md) | ||
| - [Django ORM](/vector-search-integrate-with-django-orm.md) |
There was a problem hiding this comment.
Suggested change
| - [Django ORM](/vector-search-integrate-with-django-orm.md) | |
| - [Django](/vector-search-integrate-with-django-orm.md) |
hfxsd
reviewed
Oct 17, 2024
|
|
||
| ## Create the HNSW vector index | ||
|
|
||
| [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy (> 98% in typical cases). |
There was a problem hiding this comment.
Suggested change
| [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy (> 98% in typical cases). | |
| [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) is one of the most popular vector indexing algorithms. The HNSW index provides good performance with relatively high accuracy, up to 98% in specific cases. |
| - Cosine Distance: `((VEC_COSINE_DISTANCE(embedding)))` | ||
| - L2 Distance: `((VEC_L2_DISTANCE(embedding)))` | ||
|
|
||
| The vector index can only be created for fixed-dimensional vector columns, such as a column defined as `VECTOR(3)`. It cannot be created for mixed-dimensional vector columns (such as a column defined as `VECTOR`) because vector distances can only be calculated between vectors with the same dimensions. |
There was a problem hiding this comment.
Suggested change
| The vector index can only be created for fixed-dimensional vector columns, such as a column defined as `VECTOR(3)`. It cannot be created for mixed-dimensional vector columns (such as a column defined as `VECTOR`) because vector distances can only be calculated between vectors with the same dimensions. | |
| The vector index can only be created for fixed-dimensional vector columns, such as a column defined as `VECTOR(3)`. It cannot be created for mixed-dimensional vector columns (such as a column defined as `VECTOR`) because vector distances can only be calculated between vectors with the same dimension. |
| LIMIT 10 | ||
| ``` | ||
|
|
||
| You must use the same distance metric as you have defined when creating the vector index if you want to utilize the index in vector search. |
There was a problem hiding this comment.
这一句中文没有，而且意思和 L77 似乎相同？好像可以删掉。
Suggested change
| You must use the same distance metric as you have defined when creating the vector index if you want to utilize the index in vector search. |
|
|
||
| Explanation of some important fields: | ||
|
|
||
| - `vector_index.load.total`: The total duration of loading index. This field could be larger than actual query time because multiple vector indexes may be loaded in parallel. |
There was a problem hiding this comment.
Suggested change
| - `vector_index.load.total`: The total duration of loading index. This field could be larger than actual query time because multiple vector indexes may be loaded in parallel. | |
| - `vector_index.load.total`: The total duration of loading index. This field might be larger than the actual query time because multiple vector indexes might be loaded in parallel. |
| - `vector_index.load.from_s3`: Number of indexes loaded from S3. | ||
| - `vector_index.load.from_disk`: Number of indexes loaded from disk. The index was already downloaded from S3 previously. | ||
| - `vector_index.load.from_cache`: Number of indexes loaded from cache. The index was already downloaded from S3 previously. | ||
| - `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there are heavy I/O operations when searching through the index. This field could be larger than actual query time because multiple vector indexes might be searched in parallel. |
There was a problem hiding this comment.
Suggested change
| - `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there are heavy I/O operations when searching through the index. This field could be larger than actual query time because multiple vector indexes might be searched in parallel. | |
| - `vector_index.search.total`: The total duration of searching in the index. Large latency usually means the index is cold (never accessed before, or accessed long ago) so that there are heavy I/O operations when searching through the index. This field might be larger than the actual query time because multiple vector indexes might be searched in parallel. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First-time contributors' checklist
What is changed, added or deleted? (Required)
This PR moves 15 vector search docs from the tidb-cloud folder to the vector-search folder to so they can be reused by TiDB self-managed docs.
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?