Skip to content

Commit

Permalink
docs: Functionality matrix for online stores (feast-dev#3007)
Browse files Browse the repository at this point in the history 
* Cleanup online store docs

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

* Add overview page

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

* Add Snowflake online store to RTD

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

* Online store docs

Signed-off-by: Felix Wang <wangfelix98@gmail.com>

* add overiew to README in online store

Signed-off-by: Danny Chiao <danny@tecton.ai>

Signed-off-by: Felix Wang <wangfelix98@gmail.com>
Signed-off-by: Danny Chiao <danny@tecton.ai>
Co-authored-by: Danny Chiao <danny@tecton.ai>
  • Loading branch information
@felixwang9817 @adchia
felixwang9817 and adchia authored Sep 2, 2022
1 parent 41cb476 commit 219d04a
Show file tree
Hide file tree
Showing 13 changed files with 279 additions and 14 deletions.
1 change: 1 addition & 0 deletions docs/SUMMARY.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
* [Trino (contrib)](reference/offline-stores/trino.md)
* [Azure Synapse + Azure SQL (contrib)](reference/offline-stores/mssql.md)
* [Online stores](reference/online-stores/README.md)
* [Overview](reference/online-stores/overview.md)
* [SQLite](reference/online-stores/sqlite.md)
* [Snowflake](reference/online-stores/snowflake.md)
* [Redis](reference/online-stores/redis.md)
Expand Down
13 changes: 8 additions & 5 deletions docs/getting-started/architecture-and-components/online-store.md
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
# Online store

The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the `materialize` command.
Feast uses online stores to serve features at low latency.
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command.

The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.
The storage schema of features within the online store mirrors that of the original data source.
One key difference is that for each [entity key](../concepts/entity.md), only the latest feature values are stored.
No historical values are stored.

Example batch data source
Here is an example batch data source:

![](../../.gitbook/assets/image%20%286%29.png)

Once the above data source is materialized into Feast \(using `feast materialize`\), the feature values will be stored as follows:
Once the above data source is materialized into Feast (using `feast materialize`), the feature values will be stored as follows:

![](../../.gitbook/assets/image%20%285%29.png)

Features can also be written to the online store via [push sources](../../reference/data-sources/push.md)
Features can also be written directly to the online store via [push sources](../../reference/data-sources/push.md) .
4 changes: 4 additions & 0 deletions docs/reference/online-stores/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

Please see [Online Store](../../getting-started/architecture-and-components/online-store.md) for an explanation of online stores.

{% content-ref url="overview.md" %}
[overview.md](overview.md)
{% endcontent-ref %}

{% content-ref url="sqlite.md" %}
[sqlite.md](sqlite.md)
{% endcontent-ref %}
Expand Down
29 changes: 28 additions & 1 deletion docs/reference/online-stores/cassandra.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,34 @@ online_store:
```
{% endcode %}

The full set of configuration options is available in [CassandraOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.cassandra_online_store.cassandra_online_store.CassandraOnlineStoreConfig).
For a full explanation of configuration options please look at file
`sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md`.

Storage specifications can be found at `docs/specs/online_store_format.md`.
Storage specifications can be found at `docs/specs/online_store_format.md`.

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Cassandra online store.

| | Cassandra |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | yes |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
28 changes: 27 additions & 1 deletion docs/reference/online-stores/datastore.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,30 @@ online_store:
```
{% endcode %}
Configuration options are available [here](https://rtd.feast.dev/en/latest/#feast.repo_config.DatastoreOnlineStoreConfig).
The full set of configuration options is available in [DatastoreOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.datastore.DatastoreOnlineStoreConfig).
## Functionality Matrix
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Datastore online store.
| | Datastore |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
28 changes: 27 additions & 1 deletion docs/reference/online-stores/dynamodb.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ online_store:
```
{% endcode %}
Configuration options are available [here](https://github.com/feast-dev/feast/blob/17bfa6118d6658d2bff53d7de8e2ccef5681714d/sdk/python/feast/infra/online_stores/dynamodb.py#L36).
The full set of configuration options is available in [DynamoDBOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.dynamodb.DynamoDBOnlineStoreConfig).
## Permissions
Expand Down Expand Up @@ -53,3 +53,29 @@ The following inline policy can be used to grant Feast the necessary permissions
```

Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps [here](https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum-add-role.html).

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the DynamoDB online store.

| | DynamoDB |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
54 changes: 54 additions & 0 deletions docs/reference/online-stores/overview.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Overview

## Functionality

Here are the methods exposed by the `OnlineStore` interface, along with the core functionality supported by the method:
* `online_write_batch`: write feature values to the online store
* `online_read`: read feature values from the online store
* `update`: update infrastructure (e.g. tables) in the online store
* `teardown`: teardown infrastructure (e.g. tables) in the online store
* `plan`: generate a plan of infrastructure changes based on feature repo changes

There is also additional functionality not properly captured by these interface methods:
* support for on-demand transforms
* readable by Python SDK
* readable by Java
* readable by Go
* support for entityless feature views
* support for concurrent writing to the same key
* support for ttl (time to live) at retrieval
* support for deleting expired data

Finally, there are multiple data models for storing the features in the online store. For example, features could be:
* collocated by feature view
* collocated by feature service
* collocated by entity key

See this [issue](https://github.com/feast-dev/feast/issues/2254) for a discussion around the tradeoffs of each of these data models.

## Functionality Matrix

There are currently five core online store implementations: `SqliteOnlineStore`, `RedisOnlineStore`, `DynamoDBOnlineStore`, `SnowflakeOnlineStore`, and `DatastoreOnlineStore`.
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, and `CassandraOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
Details for each specific online store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md).

Below is a matrix indicating which online stores support what functionality.

| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | Cassandra |
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes |
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Java | no | yes | no | no | no | no | no | no |
| readable by Go | yes | yes | no | no | no | no | no | no |
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes |
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no |
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no |
| support for deleting expired data | no | yes | no | no | no | no | no | no |
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes |
| collocated by feature service | no | no | no | no | no | no | no | no |
| collocated by entity key | no | yes | no | no | no | no | no | no |
28 changes: 27 additions & 1 deletion docs/reference/online-stores/postgres.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,30 @@ online_store:
```
{% endcode %}
Configuration options are available [here](https://rtd.feast.dev/en/latest/feast.infra.utils.postgres.html#module-feast.infra.utils.postgres.postgres_config).
The full set of configuration options is available in [PostgreSQLOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.postgres.PostgreSQLOnlineStoreConfig).
## Functionality Matrix
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Postgres online store.
| | Postgres |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
34 changes: 30 additions & 4 deletions docs/reference/online-stores/redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

The [Redis](https://redis.io) online store provides support for materializing feature values into Redis.

* Both Redis and Redis Cluster are supported
* Both Redis and Redis Cluster are supported.
* The data model used to store feature values in Redis is described in more detail [here](../../specs/online\_store\_format.md).

## Examples

Connecting to a single Redis instance
Connecting to a single Redis instance:

{% code title="feature_store.yaml" %}
```yaml
Expand All @@ -22,7 +22,7 @@ online_store:
```
{% endcode %}
Connecting to a Redis Cluster with SSL enabled and password authentication
Connecting to a Redis Cluster with SSL enabled and password authentication:
{% code title="feature_store.yaml" %}
```yaml
Expand All @@ -36,4 +36,30 @@ online_store:
```
{% endcode %}
Configuration options are available [here](https://rtd.feast.dev/en/master/#feast.infra.online\_stores.redis.RedisOnlineStoreConfig).
The full set of configuration options is available in [RedisOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.redis.RedisOnlineStoreConfig).
## Functionality Matrix
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Redis online store.
| | Redis |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | yes |
| readable by Go | yes |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | yes |
| support for ttl (time to live) at retrieval | yes |
| support for deleting expired data | yes |
| collocated by feature view | no |
| collocated by feature service | no |
| collocated by entity key | yes |
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
28 changes: 28 additions & 0 deletions docs/reference/online-stores/snowflake.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,31 @@ online_store:
database: SNOWFLAKE_DATABASE
```
{% endcode %}
The full set of configuration options is available in [SnowflakeOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.snowflake.SnowflakeOnlineStoreConfig).
## Functionality Matrix
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the Snowflake online store.
| | Snowflake |
| :-------------------------------------------------------- | :-- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | no |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | yes |
| collocated by feature service | no |
| collocated by entity key | no |
To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
Loading

0 comments on commit 219d04a

Please sign in to comment.