[GRPC] Documentation for 3.2 GRPC GA (Bulk and k-NN Search)

.github/vale/styles/OpenSearch/HeadingCapitalization.yml

@@ -11,4 +11,5 @@ exceptions:
   - k    # ignores lowercase k-NN
   - '[A-Z]{2,}' # ignores all acronyms
   - '([A-Z][a-z0-9]+){2,}' # ignores all camel case words
-  - '\b\w+\s+API'
+  - '\b\w+\s+API'
+  - 'gRPC' # ignores gRPC acronym

.github/vale/styles/Vocab/OpenSearch/Words/accept.txt

@@ -44,6 +44,7 @@ Gantt
 [Gg]eospatial
 [Gg]eotile
 gibibyte
+gRPC
 [Hh]ashmap
 [Hh]ostname
 [Hh]yperparameters

_api-reference/grpc-apis/index.md

@@ -12,67 +12,110 @@ redirect_from:
 **Introduced 3.0**
 {: .label .label-purple }
 
-This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/16787).    
-{: .warning}
+**Bulk and k-NN search generally available 3.2**
+{: .label .label-green }
 
-The OpenSearch gRPC plugin provides an alternative, high-performance transport layer using [gRPC](https://grpc.io/) for communication with OpenSearch. It uses protocol buffers over gRPC for lower overhead and faster serialization. This reduces overhead, speeds up serialization, and improves request-side latency, based on initial benchmarking results.
+Starting with OpenSearch version 3.2, the gRPC [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/bulk/) and [k-NN search queries]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/knn/) are generally available. These use [protobuf version 0.6.0](https://github.com/opensearch-project/opensearch-protobufs/releases/tag/0.6.0). However, expect updates to the protobuf structure as the feature matures in upcoming versions. Other gRPC search functionality remains experimental and not recommended for production use. For updates on the progress of these features or to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/16787).
+{: .note}
 
-The primary goal of the gRPC plugin is to:
+The OpenSearch gRPC functionality provides an alternative, high-performance transport layer using [gRPC](https://grpc.io/) for communication with OpenSearch. It uses protocol buffers over gRPC for lower overhead and faster serialization. This reduces overhead, speeds up serialization, and improves request-side latency, based on initial benchmarking results.
 
-* Offer a **binary-encoded** alternative to HTTP/REST-based communication.  
-* **Improve performance** for bulk workloads and large-scale ingestion scenarios.  
+The primary goal of gRPC support is to:
+
+* Offer a **binary-encoded** alternative to HTTP/REST-based communication.
+* **Improve performance** for bulk workloads and large-scale ingestion scenarios.
 * **Enable more efficient client integrations** across languages, like Java, Go, and Python, using native gRPC stubs.
 
-## Enabling the plugin
+## Performance benefits
 
-To enable the gRPC plugin (`transport-grpc`) in OpenSearch: 
-1. Install the `transport-grpc` plugin. For more information, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/).  
+Using gRPC APIs provides several advantages over HTTP APIs:
 
-1. Add the following settings to `opensearch.yml`:
-    ```yaml
-    aux.transport.types: [experimental-transport-grpc]
-    aux.transport.experimental-transport-grpc.port: '9400-9500' // optional
-    ```
-    {% include copy.html %}
+- **Reduced latency**: Binary protocol buffers eliminate JSON parsing overhead.
+- **Higher throughput**: More efficient network utilization for high-frequency queries.
+- **Lower CPU usage**: Reduced serialization and deserialization costs.
+- **Type safety**: Protocol buffer schemas provide compile-time validation.
+- **Smaller payload sizes**: Binary encoding reduces network traffic.
 
-    Alternatively, configure a secure transport protocol using the following settings:
-    ```yaml
-    aux.transport.types: [experimental-secure-transport-grpc]
-    aux.transport.experimental-transport-grpc.port: '9400-9500' // optional
-    ```
-    {% include copy.html %}
+## Enabling gRPC APIs
 
-1. Configure additional settings if needed (see [Advanced gRPC settings](#advanced-grpc-settings)):
-    ```yaml
-    grpc.host: localhost
-    grpc.publish_host: 10.74.124.163
-    grpc.bind_host: 0.0.0.0
-    ```
-    {% include copy.html %}
+The `transport-grpc` module is included by default with OpenSearch installations. To enable it, add the following settings to `opensearch.yml`:
 
+```yaml
+aux.transport.types: [experimental-transport-grpc]
+aux.transport.experimental-transport-grpc.port: '9400-9500' // optional
+```
+{% include copy.html %}
 
-## Advanced gRPC settings
+Alternatively, configure a secure transport protocol using the following settings:
 
-OpenSearch supports the following advanced network settings for gRPC communication:
+```yaml
+aux.transport.types: [experimental-secure-transport-grpc]
+aux.transport.experimental-transport-grpc.port: '9400-9500' // optional
+```
+{% include copy.html %}
 
-- `grpc.host` (Static, list): Sets the address of an OpenSearch node for gRPC communication. The `grpc.host` setting is a combination of `grpc.bind_host` and `grpc.publish_host` if they are the same value. An alternative to `grpc.host` is to configure `grpc.bind_host` and `grpc.publish_host` separately, as needed. 
+Configure additional settings if needed (see [Advanced gRPC settings](#advanced-grpc-settings)):
 
-- `grpc.bind_host` (Static, list): Specifies an address or addresses to which an OpenSearch node binds to listen for incoming gRPC connections. 
+```yaml
+grpc.host: localhost
+grpc.publish_host: 10.74.124.163
+grpc.bind_host: 0.0.0.0
+```
+{% include copy.html %}
 
-- `grpc.publish_host` (Static, list): Specifies an address or addresses that an OpenSearch node publishes to other nodes for gRPC communication.
+## Advanced gRPC settings
 
-These settings are similar to the [HTTP Network settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/network-settings/#advanced-http-settings).
+OpenSearch supports the following advanced settings for gRPC communication. These settings can be configured in `opensearch.yml`.
+
+| Setting name | Description | Example value | Default value |
+| :---- | :---- | :---- | :---- |
+| `grpc.publish_port` | The external port number that this node uses to publish itself to peers for gRPC transport. | `9400` | `-1` (disabled) |
+| `grpc.host` | A list of addresses the gRPC server will bind to. | `["0.0.0.0"]` | `[]` |
+| `grpc.bind_host` | A list of addresses to bind the gRPC server to. Can be distinct from publish hosts. | `["0.0.0.0", "::"]` | Value of `grpc.host` |
+| `grpc.publish_host` | A list of hostnames or IPs published to peers for client connections. | `["thisnode.example.com"]` | Value of `grpc.host` |
+| `grpc.netty.worker_count` | The number of Netty worker threads for the gRPC server. Controls concurrency and parallelism. | `2` | Number of processors |
+| `grpc.netty.max_concurrent_connection_calls` | The maximum number of simultaneous in-flight requests allowed per client connection. | `200` | `100` |
+| `grpc.netty.max_connection_age` | The maximum age a connection can reach before being gracefully closed. Supports time units like `ms`, `s`, or `m`. See [Time units]({{site.url}}{{site.baseurl}}/api-reference/common-parameters/#time-units). | `500ms` | Not set (no limit) |
+| `grpc.netty.max_connection_idle` | The maximum duration for which a connection can be idle before being closed. Supports time units like `ms`, `s`, or `m`. See [Time units]({{site.url}}{{site.baseurl}}/api-reference/common-parameters/#time-units). | `2m` | Not set (no limit) |
+| `grpc.netty.keepalive_timeout` | The amount of time to wait for `keepalive` ping acknowledgment before closing the connection. Supports [time units]({{site.url}}{{site.baseurl}}/api-reference/common-parameters/#time-units). | `1s` | Not set |
+| `grpc.netty.max_msg_size` | The maximum inbound message size for gRPC requests. Supports units like `b`, `kb`, `mb`, or `gb`. See [Supported units]({{site.url}}{{site.baseurl}}/api-reference/units/). | `10mb` or `10485760` | `10mb` |
+
+### Example configuration
+
+The following is an example of a complete gRPC configuration in `opensearch.yml`:
+
+```yaml
+# Basic gRPC transport configuration
+aux.transport.types: [transport-grpc]
+aux.transport.transport-grpc.port: '9400-9500'
+
+# Advanced gRPC settings
+grpc.host: ["0.0.0.0"]
+grpc.bind_host: ["0.0.0.0", "::"]
+grpc.publish_host: ["thisnode.example.com"]
+grpc.publish_port: 9400
+grpc.netty.worker_count: 4
+grpc.netty.max_concurrent_connection_calls: 200
+grpc.netty.max_connection_age: 500ms
+grpc.netty.max_connection_idle: 2m
+grpc.netty.keepalive_timeout: 1s
+grpc.netty.max_msg_size: 10mb
+```
+{% include copy.html %}
+
+These settings are similar to the [HTTP Network settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/network-settings/#advanced-http-settings) but specifically apply to gRPC communication.
 
 ## Using gRPC APIs
 
 To submit gRPC requests, you must have a set of protobufs on the client side. You can obtain the protobufs in the following ways:
 
-- **Raw protobufs**: Download the raw protobuf schema from the [OpenSearch Protobufs GitHub repository (v0.3.0)](https://github.com/opensearch-project/opensearch-protobufs). You can then generate client-side code using the protocol buffer compilers for the [supported languages](https://grpc.io/docs/languages/). 
-- **Java client-side programs only**: Download the `opensearch-protobufs` jar from the [Maven Central repository](https://repo1.maven.org/maven2/org/opensearch/protobufs/0.3.0).
+- **Raw protobufs**: Download the raw protobuf schema from the [OpenSearch Protobufs GitHub repository (v0.6.0)](https://github.com/opensearch-project/opensearch-protobufs/releases/tag/0.6.0). You can then generate client-side code using the protocol buffer compilers for the [supported languages](https://grpc.io/docs/languages/).
+- **Java client-side programs only**: Download the `opensearch-protobufs` jar from the [Maven Central repository](https://repo1.maven.org/maven2/org/opensearch/protobufs/0.6.0).
 
 ## Supported APIs
 
-This feature is currently under development and supports the following APIs:
+The following gRPC APIs are supported:
 
-- [Bulk]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/bulk/)
+- [Bulk]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/bulk/) **Generally available 3.2**
 - [Search]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/) (for select query types)
+- [k-NN]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/knn/) (k-NN search queries) **Generally available 3.2**

_api-reference/grpc-apis/knn.md

@@ -0,0 +1,143 @@
+---
+layout: default
+title: k-NN (gRPC)
+parent: gRPC APIs
+nav_order: 30
+---
+
+# k-NN (gRPC) API
+**Introduced 3.2**
+{: .label .label-purple }
+
+
+The gRPC k-NN API is generally available starting with OpenSearch 3.2. However, expect updates to the protobuf structure as the feature matures in upcoming versions.
+
+The gRPC k-NN API provides an efficient, binary-encoded interface for performing k-nearest neighbor searches using protocol buffers over gRPC. The k-NN plugin offers a specific search query type for vector similarity searches. This API offers superior performance compared to the traditional HTTP-based approach, making it ideal for large-scale machine learning and vector database applications.
+
+For information about HTTP-based k-NN queries, see [k-NN query]({{site.url}}{{site.baseurl}}/query-dsl/specialized/k-nn/).
+
+## Prerequisite
+
+To submit gRPC requests, you must have a set of protobufs on the client side. For ways to obtain the protobufs, see [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#using-grpc-apis).
+
+## gRPC service and method
+
+gRPC k-NN APIs reside in the [`SearchService`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/search_service.proto#L22), the same service used for general search operations.
+
+You can submit k-NN search requests by invoking the [`Search`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/services/search_service.proto#L23) gRPC method within the `SearchService`, using a [`KnnQuery`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L1058) within the search request. The method takes a [`SearchRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/search.proto#L18) and returns a [`SearchResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/search.proto#L317).
+
+The gRPC implementation uses the same underlying k-NN functionality as the HTTP API while providing improved performance through protocol buffer serialization.
+
+## KnnQuery fields
+
+The gRPC k-NN API uses the [`KnnQuery`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L1058) message within a [`QueryContainer`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L341) for k-NN searches. The `KnnQuery` message accepts the following fields.
+
+| Field | Protobuf type | Description |
+| :---- | :---- | :---- |
+| `field` | `string` | The vector field against which to run the search query. Required. |
+| `vector` | `repeated float` | A query vector. Must have the same number of dimensions as the vector field. Optional. |
+| `k` | `int32` | The number of nearest neighbors to return as top hits. Optional. |
+| `min_score` | `float` | The minimum similarity score required for a neighbor to be considered a hit. Optional. |
+| `max_distance` | `float` | The maximum physical distance in vector space required for a neighbor to be considered a hit. Optional. |
+| `filter` | [`QueryContainer`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L341) | Filters for the k-NN search query. See [Filter limitations](#filter-limitations). Optional. |
+| `boost` | `float` | A boost value used to increase or decrease relevance scores. Default is 1.0. Optional. |
+| `underscore_name` | `string` | A query name for query tagging (JSON key: `_name`). Optional. |
+| `method_parameters` | [`ObjectMap`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L76) | Algorithm-specific parameters (for example, `ef_search` or `nprobes`). Optional. |
+| `rescore` | [`KnnQueryRescore`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/common.proto#L1107) | A rescoring configuration for improved accuracy. Available in versions later than 2.17. Optional. |
+| `expand_nested_docs` | `bool` | When `true`, retrieves scores for all nested field documents within each parent document. Used with nested queries. Optional. |
+
+## Example request
+
+The following example shows a gRPC search request with a k-NN query. It searches for the 10 most similar vectors to the query vector `[0.1, 0.2, 0.3, 0.4]` in the `my_vector` field of the `vector_index` index:
+
+```json
+{
+  "index": ["vector_index"],
+  "request_body": {
+    "query": {
+      "knn": {
+        "field": "my_vector",
+        "vector": [0.1, 0.2, 0.3, 0.4],
+        "k": 10
+      }
+    },
+    "size": 10
+  }
+}
+```
+{% include copy.html %}
+
+## Java gRPC client example
+
+The following is a basic example of using the gRPC k-NN API (the actual implementation depends on your gRPC client setup):
+
+```java
+import org.opensearch.protobufs.*;
+import io.grpc.ManagedChannel;
+import io.grpc.ManagedChannelBuilder;
+
+public class KnnGrpcClient {
+    public static void main(String[] args) {
+        ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 9400)
+                .usePlaintext()
+                .build();
+
+        // Create a gRPC stub for search operations
+        SearchServiceGrpc.SearchServiceBlockingStub searchStub =
+            SearchServiceGrpc.newBlockingStub(channel);
+
+        // Build a k-NN query using protocol buffers
+        QueryContainer knnQuery = QueryContainer.newBuilder()
+            .setKnn(KnnQuery.newBuilder()
+                .setField("my_vector")
+                .addAllVector(Arrays.asList(0.1f, 0.2f, 0.3f, 0.4f))
+                .setK(10)
+                .build())
+            .build();
+
+        // Create the search request
+        SearchRequest request = SearchRequest.newBuilder()
+            .addIndex("vector_index")
+            .setRequestBody(SearchRequestBody.newBuilder()
+                .setQuery(knnQuery)
+                .setSize(10)
+                .build())
+            .build();
+
+        // Execute the search
+        SearchResponse response = searchStub.search(request);
+
+        System.out.println("Found " + response.getResponseBody().getHits().getHitsCount() + " results");
+
+        channel.shutdown();
+    }
+}
+```
+{% include copy.html %}
+
+## Response fields
+
+k-NN search requests return the same [`SearchResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/0.6.0/protos/schemas/search.proto#L317) structure as regular search operations. For information about response fields, see [Search (gRPC) response fields]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/#response-fields).
+
+The response includes the standard search metadata (`took`, `timed_out`, and `shards`) and a `hits` array containing the k-NN documents with their similarity scores.
+
+## Filter limitations
+
+The gRPC k-NN API has limited support for the `filter` clause compared to the HTTP API. For the current list of supported query types in gRPC, see the [Search API QueryContainer documentation]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/#querycontainer-fields) and [Supported queries]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/#supported-queries).
+
+For complex filtering requirements, consider using the HTTP k-NN API, simplifying your filter logic, or waiting for the next version of k-NN gRPC.
+
+
+
+## Related APIs
+
+- [Search (gRPC)]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/) - General gRPC search functionality
+- [Bulk (gRPC)]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/bulk/) - Bulk operations using gRPC
+- [k-NN queries]({{site.url}}{{site.baseurl}}/query-dsl/specialized/k-nn/) - HTTP-based k-NN query documentation
+
+## Next steps
+
+- Learn more about [vector search in OpenSearch]({{site.url}}{{site.baseurl}}/search-plugins/knn/index/).
+- Explore [k-NN index settings]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index/).
+- Review [performance tuning for k-NN]({{site.url}}{{site.baseurl}}/search-plugins/knn/performance-tuning/).
+- Read about [gRPC configuration]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#enabling-grpc-apis).