docs: Add docs for new types HalfVector, SparseVector and Bit (#293)

Author: marcelomendoncasoares

docs/06-concepts/02-models.md

@@ -18,7 +18,7 @@ fields:
   employees: List<Employee>
 ```
 
-Supported types are [bool](https://api.dart.dev/dart-core/bool-class.html), [int](https://api.dart.dev/dart-core/int-class.html), [double](https://api.dart.dev/dart-core/double-class.html), [String](https://api.dart.dev/dart-core/String-class.html), [Duration](https://api.dart.dev/dart-core/Duration-class.html), [DateTime](https://api.dart.dev/dart-core/DateTime-class.html), [ByteData](https://api.dart.dev/dart-typed_data/ByteData-class.html), [UuidValue](https://pub.dev/documentation/uuid/latest/uuid_value/UuidValue-class.html), [Uri](https://api.dart.dev/dart-core/Uri-class.html), [BigInt](https://api.dart.dev/dart-core/BigInt-class.html), [Vector](#vector-fields) and other serializable [classes](#class), [exceptions](#exception) and [enums](#enum). You can also use [List](https://api.dart.dev/dart-core/List-class.html)s, [Map](https://api.dart.dev/dart-core/Map-class.html)s and [Set](https://api.dart.dev/dart-core/Set-class.html)s of the supported types, just make sure to specify the types. All supported types can also be used inside [Record](https://api.dart.dev/dart-core/Record-class.html)s. Null safety is supported. Once your classes are generated, you can use them as parameters or return types to endpoint methods.
+Supported types are [bool](https://api.dart.dev/dart-core/bool-class.html), [int](https://api.dart.dev/dart-core/int-class.html), [double](https://api.dart.dev/dart-core/double-class.html), [String](https://api.dart.dev/dart-core/String-class.html), [Duration](https://api.dart.dev/dart-core/Duration-class.html), [DateTime](https://api.dart.dev/dart-core/DateTime-class.html), [ByteData](https://api.dart.dev/dart-typed_data/ByteData-class.html), [UuidValue](https://pub.dev/documentation/uuid/latest/uuid_value/UuidValue-class.html), [Uri](https://api.dart.dev/dart-core/Uri-class.html), [BigInt](https://api.dart.dev/dart-core/BigInt-class.html), [Vector](#vector), [HalfVector](#halfvector), [SparseVector](#sparsevector), [Bit](#bit) and other serializable [classes](#class), [exceptions](#exception) and [enums](#enum). You can also use [List](https://api.dart.dev/dart-core/List-class.html)s, [Map](https://api.dart.dev/dart-core/Map-class.html)s and [Set](https://api.dart.dev/dart-core/Set-class.html)s of the supported types, just make sure to specify the types. All supported types can also be used inside [Record](https://api.dart.dev/dart-core/Record-class.html)s. Null safety is supported. Once your classes are generated, you can use them as parameters or return types to endpoint methods.
 
 ### Limiting visibility of a generated class
 
@@ -135,7 +135,25 @@ fields:
 
 ## Vector fields
 
-The `Vector` type is used for storing high-dimensional vectors, which are specially useful for similarity search operations.
+Vector types are used for storing high-dimensional vectors, which are especially useful for similarity search operations.
+
+When specifying vector types, the dimension is required between parentheses (e.g., `Vector(1536)`). Common dimensions include:
+
+- 1536 (OpenAI embeddings)
+- 768 (many sentence transformers)
+- 384 (smaller models)
+
+All vector types support specialized distance operations for similarity search and filtering. See the [Vector distance operators](database/filter#vector-distance-operators) section for details.
+
+To ensure optimal performance with vector similarity searches, consider creating specialized vector indexes on your vector fields. See the [Vector indexes](database/indexing#vector-indexes) section for more details.
+
+:::info
+The usage of Vector fields requires the pgvector PostgreSQL extension to be installed, which comes by default on new Serverpod projects. To upgrade an existing project, see the [Upgrading to pgvector support](../upgrading/upgrade-to-pgvector) guide.
+:::
+
+### Vector
+
+The `Vector` type stores full-precision floating-point vectors for general-purpose embeddings.
 
 ```yaml
 class: Document
@@ -151,17 +169,44 @@ fields:
   embedding: Vector(1536)
 ```
 
-The number in parentheses specifies the vector dimensions. Common dimensions include:
+### HalfVector
 
-- 1536 (OpenAI embeddings)
-- 768 (many sentence transformers)
-- 384 (smaller models)
+The `HalfVector` type uses half-precision (16-bit) floating-point numbers, providing memory savings with acceptable precision loss for several applications.
+
+```yaml
+class: Document
+table: document
+fields:
+  content: String
+  ### Half-precision embedding for memory efficiency
+  embedding: HalfVector(1536)
+```
 
-Vector fields support specialized distance operations for similarity search and filtering. See the [Vector distance operators](database/filter#vector-distance-operators) section for details.
+### SparseVector
 
-:::info
-The usage of Vector fields requires the pgvector PostgreSQL extension to be installed, which comes by default on new Serverpod projects. To upgrade an existing project, see the [Upgrading to pgvector support](../upgrading/upgrade-to-pgvector) guide.
-:::
+The `SparseVector` type efficiently stores sparse vectors where most values are zero, which is ideal for high-dimensional data with few non-zero elements.
+
+```yaml
+class: Document
+table: document
+fields:
+  content: String
+  ### Sparse vector for keyword-based embeddings
+  keywords: SparseVector(10000)
+```
+
+### Bit
+
+The `Bit` type stores binary vectors where each element is 0 or 1, offering maximum memory efficiency for binary embeddings.
+
+```yaml
+class: Document
+table: document
+fields:
+  content: String
+  ### Binary vector for semantic hashing
+  hash: Bit(256)
+```
 
 ## Generated code
 

docs/06-concepts/06-database/04-indexing.md

@@ -65,10 +65,10 @@ If no type is specified the default is `btree`. All [PostgreSQL index types](htt
 
 ### Vector indexes
 
-To enhance the performance of vector similarity search, it is possible to create specialized vector indexes on `Vector` fields. Serverpod supports both `HNSW` and `IVFFLAT` index types with full parameter specification.
+To enhance the performance of vector similarity search, it is possible to create specialized vector indexes on vector fields (`Vector`, `HalfVector`, `SparseVector`, `Bit`). Serverpod supports both `hnsw` and `ivfflat` index types with full parameter specification.
 
 :::info
-Each vector index can only be created on a single `Vector` field. It is not possible to create a vector index on multiple fields of any kind.
+Each vector index can only be created on a single vector field. It is not possible to create a vector index on multiple fields of any kind.
 :::
 
 #### HNSW indexes
@@ -81,6 +81,8 @@ table: document
 fields:
   content: String
   embedding: Vector(1536)
+  keywords: SparseVector(10000)
+  hash: Bit(256)
 indexes:
   document_embedding_hnsw_idx:
     fields: embedding
@@ -89,11 +91,26 @@ indexes:
     parameters:
       m: 16
       ef_construction: 64
+  document_keywords_idx:
+    fields: keywords
+    type: hnsw
+    distanceFunction: innerProduct
+    parameters:
+      m: 16
+      ef_construction: 64
+  document_hash_idx:
+    fields: hash
+    type: hnsw
+    distanceFunction: hamming
+    parameters:
+      m: 16
+      ef_construction: 64
 ```
 
 Available HNSW parameters:
-- `m`: Maximum number of bi-directional links for each node (default: 16)
-- `efConstruction`: Size of the dynamic candidate list (default: 64)
+
+- `m`: Maximum number of bidirectional links for each node (default: 16)
+- `ef_construction`: Size of the dynamic candidate list (default: 64)
 
 #### IVFFLAT indexes
 
@@ -115,6 +132,7 @@ indexes:
 ```
 
 Available IVFFLAT parameters:
+
 - `lists`: Number of inverted lists (default: 100)
 
 #### Distance functions
@@ -127,6 +145,14 @@ Supported distance functions for vector indexes (`distanceFunction` parameter):
 | `innerProduct`    | Inner product                 | When vectors are normalized  |
 | `cosine`          | Cosine distance               | Text embeddings              |
 | `l1`              | Manhattan or taxicab distance | Sparse/high-dimensional data |
+| `hamming`         | Hamming distance              | Binary vectors (Bit type)    |
+| `jaccard`         | Jaccard distance              | Binary vectors (Bit type)    |
+
+Different vector types have specific limitations when creating indexes:
+
+- **SparseVector**: Can only use HNSW indexes (IVFFLAT is not supported).
+- **HalfVector**: When using IVFFLAT indexes, the L1 distance function is not supported.
+- **Bit**: Only supports `hamming` (default) and `jaccard` distance functions.
 
 :::tip
 If more than one distance function is going to be frequently used on the same vector field, consider creating one index for each distance function to ensure optimal performance.

docs/06-concepts/06-database/06-filter.md

@@ -211,18 +211,25 @@ In the example we fetch all users that has a name that starts with A _or_ B.
 
 ### Vector distance operators
 
-Vector fields support specialized distance operations for similarity search. Available vector distance operations:
+All vector field types support specialized distance operations for similarity search. Available vector distance operations:
 
+**Vector, HalfVector, and SparseVector fields:**
 - `distanceL2` - Euclidean (L2) distance.
 - `distanceInnerProduct` - Inner product distance.
 - `distanceCosine` - Cosine distance.
 - `distanceL1` - Manhattan or taxicab (L1) distance.
 
+**Bit vector fields:**
+- `distanceHamming` - Hamming distance.
+- `distanceJaccard` - Jaccard distance.
+
 You can use vector distance operations with numeric comparisons for filtering and ordering:
 
 ```dart
 // The vector to compare against
 var queryVector = Vector([0.1, 0.2, 0.3, ...]);
+var sparseQuery = SparseVector([0.0, 1.0, 0.0, 2.5, ...]);
+var binaryQuery = Bit([1, 0, 1, 1, 0, ...]);
 
 // Find top documents similar to a query vector
 var similarDocs = await Document.db.find(
@@ -232,6 +239,22 @@ var similarDocs = await Document.db.find(
   limit: 10,
 );
 
+// Search using sparse vectors
+var keywordMatches = await Document.db.find(
+  session,
+  where: (t) => t.keywords.distanceInnerProduct(sparseQuery) < 0.3,
+  orderBy: (t) => t.keywords.distanceInnerProduct(sparseQuery),
+  limit: 5,
+);
+
+// Search using binary vectors with Hamming distance
+var binaryMatches = await Document.db.find(
+  session,
+  where: (t) => t.hash.distanceHamming(binaryQuery) < 10,
+  orderBy: (t) => t.hash.distanceHamming(binaryQuery),
+  limit: 5,
+);
+
 // Filter by distance range
 var mediumSimilarity = await Document.db.find(
   session,