Skip to content

Commit e1944b1

Browse files
committed
Update for dense_vector support
Signed-off-by: Andre Kurait <akurait@amazon.com>
1 parent 399c0ba commit e1944b1

File tree

6 files changed

+123
-313
lines changed

6 files changed

+123
-313
lines changed

_data/migration-scenarios.yml

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,12 @@ scenarios:
2121
url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/"
2222
- title: "Handling breaking changes in field types"
2323
url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/"
24-
- title: "Flattened type handling"
25-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-flattened-type/"
26-
- title: "Field Data Type Deprecation - string"
27-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-string-type-deprecation/"
28-
- title: "dense_vector to knn_vector conversion"
29-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/"
24+
- title: "Transform flattened to flat_object"
25+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-flattened-flat-object/"
26+
- title: "Transform string to text/keyword"
27+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-string-text-keyword/"
28+
- title: "Transform dense_vector to knn_vector"
29+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/"
3030
- title: "Backfill"
3131
url: "/migration-assistant/migration-phases/backfill/"
3232
- title: "Teardown"
@@ -54,12 +54,12 @@ scenarios:
5454
url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/"
5555
- title: "Handling breaking changes in field types"
5656
url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/"
57-
- title: "Flattened type handling"
58-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-flattened-type/"
59-
- title: "Field Data Type Deprecation - string"
60-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-string-type-deprecation/"
61-
- title: "dense_vector to knn_vector conversion"
62-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/"
57+
- title: "Transform flattened to flat_object"
58+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-flattened-flat-object/"
59+
- title: "Transform string to text/keyword"
60+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-string-text-keyword/"
61+
- title: "Transform dense_vector to knn_vector"
62+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/"
6363
- title: "Replay captured traffic"
6464
url: "/migration-assistant/migration-phases/replay-captured-traffic/"
6565
- title: "Reroute traffic from the Capture Proxy to the target"
@@ -93,12 +93,12 @@ scenarios:
9393
url: "/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/"
9494
- title: "Handling breaking changes in field types"
9595
url: "/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/"
96-
- title: "Flattened type handling"
97-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-flattened-type/"
98-
- title: "Field Data Type Deprecation - string"
99-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-string-type-deprecation/"
100-
- title: "dense_vector to knn_vector conversion"
101-
url: "/migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/"
96+
- title: "Transform flattened to flat_object"
97+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-flattened-flat-object/"
98+
- title: "Transform string to text/keyword"
99+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-string-text-keyword/"
100+
- title: "Transform dense_vector to knn_vector"
101+
url: "/migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/"
102102
- title: "Backfill"
103103
url: "/migration-assistant/migration-phases/backfill/"
104104
- title: "Replay captured traffic"

_migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes.md

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -73,14 +73,6 @@ The following script demonstrates how to perform common field type conversions,
7373
* Replacing the deprecated `string` type with `text`.
7474
* Converting `flattened` to `flat_object` and removing the `index` property if present.
7575

76-
{: .note }
77-
For migrations from Elasticsearch 1.x-5.x, the `string` to `text`/`keyword` conversion is handled automatically by the Migration Assistant. See [Field Data Type Deprecation - string]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-string-type-deprecation/) for more details.
78-
79-
{: .note }
80-
For migrations from Elasticsearch 7.x, the `dense_vector` to `knn_vector` conversion is handled automatically by the Migration Assistant. See [dense_vector to knn_vector conversion]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/) for more details.
81-
82-
{: .note }
83-
For OpenSearch 2.7+, the `flattened` to `flat_object` conversion is handled automatically by the Migration Assistant. See [Flattened type handling]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-flattened-type/) for more details.
8476

8577
```javascript
8678
function main(context) {

_migration-assistant/migration-phases/migrate-metadata/index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -177,9 +177,9 @@ Metadata migration requires modifying data from the source to the target version
177177
For guidance on handling specific field type compatibility issues, see:
178178
- [Transform type mappings]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-type-mapping-deprecation/) - Handle deprecated mapping types from Elasticsearch 6.x
179179
- [Transform field types]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/) - Configure custom field type transformations
180-
- [Flattened type handling]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-flattened-type/) - Automatic conversion of flattened to flat_object for OpenSearch 2.7+
181-
- [Field Data Type Deprecation - string]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-string-type-deprecation/) - Automatic conversion of string to text/keyword for Elasticsearch 1.x-5.x
182-
- [dense_vector to knn_vector conversion]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/) - Automatic conversion of dense_vector to knn_vector for Elasticsearch 7.x
180+
- [Transform flattened to flat_object]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/transform-flattened-flat-object/) - Automatic transformation of flattened to flat_object
181+
- [Transform string to text/keyword]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/transform-string-text-keyword/) - Automatic transformation of string to text/keyword
182+
- [Transform dense_vector to knn_vector]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/) - Automatic transformation of dense_vector to knn_vector
183183

184184
#### Deprecation of Mapping Types
185185

Lines changed: 33 additions & 144 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,43 @@
11
---
22
layout: default
3-
title: dense_vector to knn_vector conversion
3+
title: Transform dense_vector to knn_vector
44
nav_order: 5
55
parent: Migrate metadata
66
grand_parent: Migration phases
7-
permalink: /migration-assistant/migration-phases/migrate-metadata/handling-dense-vector-conversion/
7+
permalink: /migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/
88
---
99

10-
# dense_vector to knn_vector conversion
10+
# Transform dense_vector to knn_vector
1111

12-
Convert mapping type dense_vector to OpenSearch knn_vector.
12+
Convert field data type dense_vector to OpenSearch knn_vector.
1313

14-
This guide explains how the Migration Assistant automatically handles the conversion of Elasticsearch's `dense_vector` field type to OpenSearch's `knn_vector` field type during migration.
14+
This guide explains how the Migration Assistant automatically handles the transformation of Elasticsearch's `dense_vector` field type to OpenSearch's `knn_vector` field type during migration.
1515

1616
## Overview
1717

1818
The `dense_vector` field type was introduced in Elasticsearch 7.x for storing dense vectors used in machine learning and similarity search applications. When migrating from Elasticsearch 7.x to OpenSearch, the Migration Assistant automatically converts `dense_vector` fields to OpenSearch's equivalent `knn_vector` type.
1919

20-
This conversion includes mapping the vector configuration parameters and enabling the necessary OpenSearch k-NN plugin settings.
20+
This transformation includes mapping the vector configuration parameters and enabling the necessary OpenSearch k-NN plugin settings.
2121

2222
## Compatibility
2323

24-
The `dense_vector` to `knn_vector` conversion applies to:
25-
- **Source clusters**: Elasticsearch 7.x with dense_vector fields
26-
- **Target clusters**: OpenSearch (any version)
24+
The `dense_vector` to `knn_vector` transformation applies to:
25+
- **Source clusters**: Elasticsearch 7.x+
26+
- **Target clusters**: OpenSearch 1.x+
2727
- **Automatic conversion**: No configuration required
2828

2929
## Automatic conversion logic
3030

3131
The Migration Assistant performs the following transformations when converting `dense_vector` to `knn_vector`:
3232

33-
### Field type conversion
33+
### Field type transformation
3434
- Changes `type: "dense_vector"` to `type: "knn_vector"`
3535
- Maps `dims` parameter to `dimension`
3636
- Converts similarity metrics to OpenSearch space types
3737
- Configures HNSW algorithm with Lucene engine
3838

3939
### Similarity mapping
40-
The conversion maps Elasticsearch similarity functions to OpenSearch space types:
40+
The transformation maps Elasticsearch similarity functions to OpenSearch space types:
4141
- `cosine``cosinesimil`
4242
- `dot_product``innerproduct`
4343
- `l2` (default) → `l2`
@@ -54,21 +54,24 @@ During the migration process, you'll see this transformation in the output:
5454
```
5555
Transformations:
5656
dense_vector to knn_vector:
57-
Convert mapping type dense_vector to OpenSearch knn_vector
57+
Convert field data type dense_vector to OpenSearch knn_vector
5858
```
5959

6060
## Example transformations
6161

62-
### Basic dense_vector conversion
63-
64-
<table>
65-
<tr>
66-
<th>Source Field Type</th>
67-
<th>Target Field Type</th>
68-
</tr>
69-
<tr>
70-
<td>
71-
<pre><code class="language-json">{
62+
### Basic dense_vector transformation
63+
64+
<table style="border-collapse: collapse; border: 1px solid #ddd;">
65+
<thead>
66+
<tr>
67+
<th style="border: 1px solid #ddd; padding: 8px;">Source Field Type</th>
68+
<th style="border: 1px solid #ddd; padding: 8px;">Target Field Type</th>
69+
</tr>
70+
</thead>
71+
<tbody>
72+
<tr>
73+
<td style="border: 1px solid #ddd; padding: 8px;">
74+
<pre><code>{
7275
"properties": {
7376
"embedding": {
7477
"type": "dense_vector",
@@ -77,9 +80,9 @@ Transformations:
7780
}
7881
}
7982
}</code></pre>
80-
</td>
81-
<td>
82-
<pre><code class="language-json">{
83+
</td>
84+
<td style="border: 1px solid #ddd; padding: 8px;">
85+
<pre><code>{
8386
"properties": {
8487
"embedding": {
8588
"type": "knn_vector",
@@ -97,130 +100,16 @@ Transformations:
97100
}
98101
}
99102
}</code></pre>
100-
</td>
101-
</tr>
102-
</table>
103-
104-
### Advanced configuration with index options
105-
106-
<table>
107-
<tr>
108-
<th>Source Field Type</th>
109-
<th>Target Field Type</th>
110-
</tr>
111-
<tr>
112-
<td>
113-
<pre><code class="language-json">{
114-
"properties": {
115-
"vector_field": {
116-
"type": "dense_vector",
117-
"dims": 256,
118-
"similarity": "dot_product",
119-
"index_options": {
120-
"m": 16,
121-
"ef_construction": 200
122-
}
123-
}
124-
}
125-
}</code></pre>
126-
</td>
127-
<td>
128-
<pre><code class="language-json">{
129-
"properties": {
130-
"vector_field": {
131-
"type": "knn_vector",
132-
"dimension": 256,
133-
"method": {
134-
"name": "hnsw",
135-
"engine": "lucene",
136-
"space_type": "innerproduct",
137-
"parameters": {
138-
"encoder": {
139-
"name": "sq"
140-
},
141-
"m": 16,
142-
"ef_construction": 200
143-
}
144-
}
145-
}
146-
}
147-
}</code></pre>
148-
</td>
149-
</tr>
150-
</table>
151-
152-
### Multiple vector fields
153-
154-
<table>
155-
<tr>
156-
<th>Source Field Type</th>
157-
<th>Target Field Type</th>
158-
</tr>
159-
<tr>
160-
<td>
161-
<pre><code class="language-json">{
162-
"properties": {
163-
"title_embedding": {
164-
"type": "dense_vector",
165-
"dims": 384,
166-
"similarity": "cosine"
167-
},
168-
"content_embedding": {
169-
"type": "dense_vector",
170-
"dims": 768,
171-
"similarity": "l2"
172-
},
173-
"title": {
174-
"type": "text"
175-
}
176-
}
177-
}</code></pre>
178-
</td>
179-
<td>
180-
<pre><code class="language-json">{
181-
"properties": {
182-
"title_embedding": {
183-
"type": "knn_vector",
184-
"dimension": 384,
185-
"method": {
186-
"name": "hnsw",
187-
"engine": "lucene",
188-
"space_type": "cosinesimil",
189-
"parameters": {
190-
"encoder": {
191-
"name": "sq"
192-
}
193-
}
194-
}
195-
},
196-
"content_embedding": {
197-
"type": "knn_vector",
198-
"dimension": 768,
199-
"method": {
200-
"name": "hnsw",
201-
"engine": "lucene",
202-
"space_type": "l2",
203-
"parameters": {
204-
"encoder": {
205-
"name": "sq"
206-
}
207-
}
208-
}
209-
},
210-
"title": {
211-
"type": "text"
212-
}
213-
}
214-
}</code></pre>
215-
</td>
216-
</tr>
103+
</td>
104+
</tr>
105+
</tbody>
217106
</table>
218107

219108
## Configuration details
220109

221110
### HNSW algorithm parameters
222111

223-
The conversion automatically configures the HNSW (Hierarchical Navigable Small World) algorithm with:
112+
The transformation automatically configures the HNSW (Hierarchical Navigable Small World) algorithm with:
224113
- **Engine**: `lucene` (OpenSearch's default)
225114
- **Encoder**: `sq` (scalar quantization for memory efficiency)
226115
- **Method**: `hnsw` (approximate nearest neighbor search)
@@ -247,7 +136,7 @@ When any `dense_vector` fields are converted, the following index setting is aut
247136

248137
### Migrating from Elasticsearch 7.x
249138

250-
The Migration Assistant automatically converts all `dense_vector` fields during metadata migration. The k-NN plugin must be installed and enabled on the target OpenSearch cluster.
139+
The Migration Assistant automatically transforms all `dense_vector` fields during metadata migration. The k-NN plugin must be installed and enabled on the target OpenSearch cluster.
251140

252141
### Query compatibility
253142

0 commit comments

Comments
 (0)