Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Couchbase as an online store #4637

Merged
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,25 @@ test-python-universal-singlestore-online:
-k "test_retrieve_online_documents" \
sdk/python/tests/integration/online_store/test_universal_online.py

test-python-universal-couchbase-online:
PYTHONPATH='.' \
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.couchbase_repo_configuration \
PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.couchbase \
python -m pytest -n 8 --integration \
-k "not test_universal_cli and \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to confirm which ones to skip here?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just skip what other third party providers do

not test_go_feature_server and \
not test_feature_logging and \
not test_reorder_columns and \
not test_logged_features_validation and \
not test_lambda_materialization_consistency and \
not test_offline_write and \
not test_push_features_to_offline_store and \
not gcs_registry and \
not s3_registry and \
not test_universal_types and \
not test_snowflake" \
sdk/python/tests

test-python-universal:
python -m pytest -n 8 --integration sdk/python/tests

Expand Down
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@
* [Remote](reference/online-stores/remote.md)
* [PostgreSQL (contrib)](reference/online-stores/postgres.md)
* [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md)
* [Couchbase (contrib)](reference/online-stores/couchbase.md)
* [MySQL (contrib)](reference/online-stores/mysql.md)
* [Hazelcast (contrib)](reference/online-stores/hazelcast.md)
* [ScyllaDB (contrib)](reference/online-stores/scylladb.md)
Expand Down
5 changes: 5 additions & 0 deletions docs/reference/online-stores/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
[cassandra.md](cassandra.md)
{% endcontent-ref %}

{% content-ref url="couchbase.md" %}
[couchbase.md](cassandra.md)
Copy link
Member

@franciscojavierarceo franciscojavierarceo Nov 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says Cassandra, is that a typo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good catch, thanks. Fixed: d1ae5d9

{% endcontent-ref %}

{% content-ref url="mysql.md" %}
[mysql.md](mysql.md)
{% endcontent-ref %}
Expand All @@ -60,6 +64,7 @@ Please see [Online Store](../../getting-started/components/online-store.md) for

{% content-ref url="remote.md" %}
[remote.md](remote.md)
{% endcontent-ref %}

{% content-ref url="singlestore.md" %}
[singlestore.md](singlestore.md)
Expand Down
78 changes: 78 additions & 0 deletions docs/reference/online-stores/couchbase.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Couchbase Online Store
> NOTE:
> This is a community-contributed online store that is in alpha development. It is not officially supported by the Feast project.

## Description
The [Couchbase](https://www.couchbase.com/) online store provides support for materializing feature values into a Couchbase Operational cluster for serving online features in real-time.

* Only the latest feature values are persisted
* Features are stored in a document-oriented format

The data model for using Couchbase as an online store follows a document format:
* Document ID: `{project}:{table_name}:{entity_key_hex}:{feature_name}`
* Document Content:
* `metadata`:
* `event_ts` (ISO formatted timestamp)
* `created_ts` (ISO formatted timestamp)
* `feature_name` (String)
* `value` (Base64 encoded protobuf binary)


## Getting started
In order to use this online store, you'll need to run `pip install 'feast[couchbase]'`. You can then get started with the command `feast init REPO_NAME -t couchbase`.

To get started with Couchbase Capella Operational:
1. [Create a Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier)
2. [Deploy an Operational cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started)
3. [Create a bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket)
- This can be named anything, but must correspond to the bucket described in the `feature_store.yaml` configuration file.
4. [Create cluster access credentials](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials)
- These credentials should have full access to the bucket created in step 3.
5. [Configure allowed IP addresses](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html)
- You must allow the IP address of the machine running Feast.

## Example
{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: couchbase
connection_string: couchbase://127.0.0.1 # Couchbase connection string, copied from 'Connect' page in Couchbase Capella console
user: Administrator # Couchbase username from access credentials
password: password # Couchbase password from access credentials
bucket_name: feast # Couchbase bucket name, defaults to feast
kv_port: 11210 # Couchbase key-value port, defaults to 11210. Required if custom ports are used.
entity_key_serialization_version: 2
```
{% endcode %}

The full set of configuration options is available in `CouchbaseOnlineStoreConfig`.


## Functionality Matrix
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Couchbase online store.

| | Couchbase |
| :-------------------------------------------------------- | :-------- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | yes |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).

Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
feast.infra.online\_stores.contrib.couchbase\_online\_store package
===================================================================

Submodules
----------

feast.infra.online\_stores.contrib.couchbase\_online\_store.couchbase module
----------------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.couchbase_online_store.couchbase
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: feast.infra.online_stores.contrib.couchbase_online_store
:members:
:undoc-members:
:show-inheritance:
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Subpackages
:maxdepth: 4

feast.infra.online_stores.contrib.cassandra_online_store
feast.infra.online_stores.contrib.couchbase_online_store
feast.infra.online_stores.contrib.hazelcast_online_store
feast.infra.online_stores.contrib.hbase_online_store
feast.infra.online_stores.contrib.ikv_online_store
Expand All @@ -24,6 +25,14 @@ feast.infra.online\_stores.contrib.cassandra\_repo\_configuration module
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.couchbase\_repo\_configuration module
------------------------------------------------------------------------

.. automodule:: feast.infra.online_stores.contrib.couchbase_repo_configuration
:members:
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.contrib.elasticsearch module
-------------------------------------------------------

Expand Down
8 changes: 8 additions & 0 deletions sdk/python/docs/source/feast.infra.online_stores.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,14 @@ feast.infra.online\_stores.sqlite module
:undoc-members:
:show-inheritance:

feast.infra.online\_stores.vector\_store module
-----------------------------------------------

.. automodule:: feast.infra.online_stores.vector_store
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

Expand Down
14 changes: 11 additions & 3 deletions sdk/python/docs/source/feast.permissions.client.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ feast.permissions.client.auth\_client\_manager module
:undoc-members:
:show-inheritance:

feast.permissions.client.auth\_client\_manager\_factory module
--------------------------------------------------------------
feast.permissions.client.client\_auth\_token module
---------------------------------------------------

.. automodule:: feast.permissions.client.auth_client_manager_factory
.. automodule:: feast.permissions.client.client_auth_token
:members:
:undoc-members:
:show-inheritance:
Expand All @@ -44,6 +44,14 @@ feast.permissions.client.http\_auth\_requests\_wrapper module
:undoc-members:
:show-inheritance:

feast.permissions.client.intra\_comm\_authentication\_client\_manager module
----------------------------------------------------------------------------

.. automodule:: feast.permissions.client.intra_comm_authentication_client_manager
:members:
:undoc-members:
:show-inheritance:

feast.permissions.client.kubernetes\_auth\_client\_manager module
-----------------------------------------------------------------

Expand Down
16 changes: 16 additions & 0 deletions sdk/python/docs/source/feast.protos.feast.core.rst
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,22 @@ feast.protos.feast.core.Policy\_pb2\_grpc module
:undoc-members:
:show-inheritance:

feast.protos.feast.core.Project\_pb2 module
-------------------------------------------

.. automodule:: feast.protos.feast.core.Project_pb2
:members:
:undoc-members:
:show-inheritance:

feast.protos.feast.core.Project\_pb2\_grpc module
-------------------------------------------------

.. automodule:: feast.protos.feast.core.Project_pb2_grpc
:members:
:undoc-members:
:show-inheritance:

feast.protos.feast.core.Registry\_pb2 module
--------------------------------------------

Expand Down
32 changes: 32 additions & 0 deletions sdk/python/docs/source/feast.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ feast.aggregation module
:undoc-members:
:show-inheritance:

feast.arrow\_error\_handler module
----------------------------------

.. automodule:: feast.arrow_error_handler
:members:
:undoc-members:
:show-inheritance:

feast.base\_feature\_view module
--------------------------------

Expand Down Expand Up @@ -196,6 +204,14 @@ feast.flags\_helper module
:undoc-members:
:show-inheritance:

feast.grpc\_error\_interceptor module
-------------------------------------

.. automodule:: feast.grpc_error_interceptor
:members:
:undoc-members:
:show-inheritance:

feast.importer module
---------------------

Expand Down Expand Up @@ -244,6 +260,14 @@ feast.online\_response module
:undoc-members:
:show-inheritance:

feast.project module
--------------------

.. automodule:: feast.project
:members:
:undoc-members:
:show-inheritance:

feast.project\_metadata module
------------------------------

Expand Down Expand Up @@ -292,6 +316,14 @@ feast.repo\_operations module
:undoc-members:
:show-inheritance:

feast.rest\_error\_handler module
---------------------------------

.. automodule:: feast.rest_error_handler
:members:
:undoc-members:
:show-inheritance:

feast.saved\_dataset module
---------------------------

Expand Down
Loading
Loading