|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: Transform dense_vector fields to knn_vector |
| 4 | +nav_order: 5 |
| 5 | +parent: Migrate metadata |
| 6 | +grand_parent: Migration phases |
| 7 | +permalink: /migration-assistant/migration-phases/migrate-metadata/transform-dense-vector-knn-vector/ |
| 8 | +--- |
| 9 | + |
| 10 | +# Transform dense_vector fields to knn_vector |
| 11 | + |
| 12 | + |
| 13 | +This guide explains how Migration Assistant automatically handles the transformation of Elasticsearch's `dense_vector` field type to OpenSearch's `knn_vector` field type during migration. |
| 14 | + |
| 15 | +## Overview |
| 16 | + |
| 17 | +The `dense_vector` field type was introduced in Elasticsearch 7.x for storing dense vectors used in machine learning and similarity search applications. When migrating from Elasticsearch 7.x to OpenSearch, Migration Assistant automatically converts `dense_vector` fields to OpenSearch's equivalent `knn_vector` type. |
| 18 | + |
| 19 | +This transformation includes mapping the vector configuration parameters and enabling the necessary OpenSearch k-NN plugin settings. |
| 20 | + |
| 21 | +To determine whether an Elasticsearch cluster uses `dense_vector` field types, make a call to your source cluster's `GET /_mapping` API. In the migration console, run `console clusters curl source_cluster "/_mapping"`. If you see `"type":"dense_vector"`, then this transformation is applicable and these fields will be automatically transformed during migration. |
| 22 | + |
| 23 | +## Compatibility |
| 24 | + |
| 25 | +The `dense_vector` to `knn_vector` transformation applies to: |
| 26 | +- **Source clusters**: Elasticsearch 7.x+ |
| 27 | +- **Target clusters**: OpenSearch 1.x+ |
| 28 | +- **Automatic conversion**: No configuration required |
| 29 | + |
| 30 | +## Automatic conversion logic |
| 31 | + |
| 32 | +Migration Assistant performs the following transformations when converting `dense_vector` to `knn_vector` fields. |
| 33 | + |
| 34 | +### Field type transformation |
| 35 | +- Changes `type: "dense_vector"` to `type: "knn_vector"` |
| 36 | +- Maps `dims` parameter to `dimension` |
| 37 | +- Converts similarity metrics to OpenSearch space types |
| 38 | +- Configures the Hierarchical Navigable Small World (HNSW) algorithm with the Lucene engine |
| 39 | + |
| 40 | +### Similarity mapping |
| 41 | +The transformation maps Elasticsearch similarity functions to OpenSearch space types: |
| 42 | +- `cosine` → `cosinesimil` |
| 43 | +- `dot_product` → `innerproduct` |
| 44 | +- `l2` (default) → `l2` |
| 45 | + |
| 46 | +### Index settings |
| 47 | +When `dense_vector` fields are converted, Migration Assistant automatically performs the following operations: |
| 48 | +- Enables the k-NN plugin by setting `index.knn: true` |
| 49 | +- Ensures proper index configuration for vector search |
| 50 | + |
| 51 | +## Migration output |
| 52 | + |
| 53 | +During the migration process, you'll see this transformation in the output: |
| 54 | + |
| 55 | +``` |
| 56 | +Transformations: |
| 57 | + dense_vector to knn_vector: |
| 58 | + Convert field data type dense_vector to OpenSearch knn_vector |
| 59 | +``` |
| 60 | + |
| 61 | +## Transformation behavior |
| 62 | + |
| 63 | +<table style="border-collapse: collapse; border: 1px solid #ddd;"> |
| 64 | + <thead> |
| 65 | + <tr> |
| 66 | + <th style="border: 1px solid #ddd; padding: 8px;">Source field type</th> |
| 67 | + <th style="border: 1px solid #ddd; padding: 8px;">Target field type</th> |
| 68 | + </tr> |
| 69 | + </thead> |
| 70 | + <tbody> |
| 71 | + <tr> |
| 72 | + <td style="border: 1px solid #ddd; padding: 8px;"> |
| 73 | + <pre><code>{ |
| 74 | + "properties": { |
| 75 | + "embedding": { |
| 76 | + "type": "dense_vector", |
| 77 | + "dims": 128, |
| 78 | + "similarity": "cosine" |
| 79 | + } |
| 80 | + } |
| 81 | +}</code></pre> |
| 82 | + </td> |
| 83 | + <td style="border: 1px solid #ddd; padding: 8px;"> |
| 84 | + <pre><code>{ |
| 85 | + "properties": { |
| 86 | + "embedding": { |
| 87 | + "type": "knn_vector", |
| 88 | + "dimension": 128, |
| 89 | + "method": { |
| 90 | + "name": "hnsw", |
| 91 | + "engine": "lucene", |
| 92 | + "space_type": "cosinesimil", |
| 93 | + "parameters": { |
| 94 | + "encoder": { |
| 95 | + "name": "sq" |
| 96 | + } |
| 97 | + } |
| 98 | + } |
| 99 | + } |
| 100 | + } |
| 101 | +}</code></pre> |
| 102 | + </td> |
| 103 | + </tr> |
| 104 | + </tbody> |
| 105 | +</table> |
| 106 | + |
| 107 | +### HNSW algorithm parameters |
| 108 | + |
| 109 | +The transformation automatically configures the HNSW algorithm with the following options: |
| 110 | +- `engine`: `lucene` (OpenSearch default) |
| 111 | +- `encoder`: `sq` (scalar quantization for memory efficiency) |
| 112 | +- `method`: `hnsw` (approximate nearest neighbor search) |
| 113 | + |
| 114 | +### Index options mapping |
| 115 | + |
| 116 | +Elasticsearch `index_options` are mapped to OpenSearch HNSW parameters: |
| 117 | +- `m` → `m` (maximum number of connections per node) |
| 118 | +- `ef_construction` → `ef_construction` (size of dynamic candidate list) |
| 119 | + |
| 120 | +### Index settings |
| 121 | + |
| 122 | +When any `dense_vector` fields are converted, the following index setting is automatically added: |
| 123 | + |
| 124 | +```json |
| 125 | +{ |
| 126 | + "settings": { |
| 127 | + "index.knn": true |
| 128 | + } |
| 129 | +} |
| 130 | +``` |
| 131 | + |
| 132 | +## Behavior differences |
| 133 | + |
| 134 | +Migration Assistant automatically transforms all `dense_vector` fields during metadata migration. The k-NN plugin must be installed and enabled on the target OpenSearch cluster. Note: Most OpenSearch distributions include the k-NN plugin in which case no action is needed. |
| 135 | + |
| 136 | +### Query compatibility |
| 137 | + |
| 138 | +After migration, vector search queries need to be updated: |
| 139 | +- Elasticsearch uses `script_score` queries with vector functions. |
| 140 | +- OpenSearch uses native `knn` query syntax. |
| 141 | + |
| 142 | +**Elasticsearch query example**: |
| 143 | +```json |
| 144 | +{ |
| 145 | + "query": { |
| 146 | + "script_score": { |
| 147 | + "query": {"match_all": {}}, |
| 148 | + "script": { |
| 149 | + "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0", |
| 150 | + "params": {"query_vector": [0.1, 0.2, 0.3]} |
| 151 | + } |
| 152 | + } |
| 153 | + } |
| 154 | +} |
| 155 | +``` |
| 156 | + |
| 157 | +**OpenSearch query example**: |
| 158 | +```json |
| 159 | +{ |
| 160 | + "query": { |
| 161 | + "knn": { |
| 162 | + "embedding": { |
| 163 | + "vector": [0.1, 0.2, 0.3], |
| 164 | + "k": 10 |
| 165 | + } |
| 166 | + } |
| 167 | + } |
| 168 | +} |
| 169 | +``` |
| 170 | + |
| 171 | +## Troubleshooting |
| 172 | + |
| 173 | +If you encounter issues with `dense_vector` conversion: |
| 174 | + |
| 175 | +1. **Verify the k-NN plugin** -- Ensure the k-NN plugin is installed and enabled on your target OpenSearch cluster: |
| 176 | + ```bash |
| 177 | + GET /_cat/plugins |
| 178 | + ``` |
| 179 | + |
| 180 | +2. **Check migration logs** -- Review the detailed migration logs for any warnings or errors: |
| 181 | + ```bash |
| 182 | + tail /shared-logs-output/migration-console-default/*/metadata/*.log |
| 183 | + ``` |
| 184 | + |
| 185 | +3. **Validate mappings** -- After migration, verify that the field types have been correctly converted: |
| 186 | + ```bash |
| 187 | + GET /your-index/_mapping |
| 188 | + ``` |
| 189 | + |
| 190 | +4. **Test vector search** -- Verify that vector search functionality works with sample queries: |
| 191 | + ```bash |
| 192 | + POST /your-index/_search |
| 193 | + { |
| 194 | + "query": { |
| 195 | + "knn": { |
| 196 | + "embedding": { |
| 197 | + "vector": [0.1, 0.2, 0.3], |
| 198 | + "k": 5 |
| 199 | + } |
| 200 | + } |
| 201 | + } |
| 202 | + } |
| 203 | + ``` |
| 204 | + |
| 205 | +5. **Monitor performance** -- Vector search performance may differ between Elasticsearch and OpenSearch. Monitor query performance and adjust HNSW parameters if needed. |
| 206 | + |
| 207 | +## Related documentation |
| 208 | + |
| 209 | +- [Transform field types documentation]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/migrate-metadata/handling-field-type-breaking-changes/) -- Configure custom field type transformations. |
| 210 | +- [k-NN documentation]({{site.url}}{{site.baseurl}}/vector-search/vector-search-techniques/approximate-knn/) -- Approximate k-NN search documentation. |
0 commit comments