Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions spiceaidocs/docs/reference/spicepod/datasets.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Datasets"
sidebar_label: "Datasets"
title: 'Datasets'
sidebar_label: 'Datasets'
description: 'Datasets YAML reference'
---

Expand All @@ -11,6 +11,7 @@ A Spicepod can contain one or more datasets referenced by relative path, or defi
Inline example:

`spicepod.yaml`

```yaml
datasets:
- from: spice.ai/eth/beacon/eigenlayer
Expand All @@ -27,6 +28,7 @@ datasets:
```

`spicepod.yaml`

```yaml
datasets:
- from: databricks:spiceai.datasets.specific_table
Expand All @@ -45,12 +47,14 @@ datasets:
Relative path example:

`spicepod.yaml`

```yaml
datasets:
- from: datasets/eth_recent_transactions
```

`datasets/eth_recent_transactions/dataset.yaml`

```yaml
from: spiceai:spice.ai/eth.recent_transactions
name: eth_recent_transactions
Expand All @@ -75,6 +79,7 @@ Where:
- `<source>`: The source of the dataset

Currently supported sources:

- `spiceai`
- `dremio`
- `databricks`
Expand Down Expand Up @@ -108,25 +113,25 @@ Enable or disable acceleration, defaults to `true`.

The acceleration engine to use, defaults to `arrow`. The following engines are supported:

- `arrow` - Accelerated in-memory backed by Apache Arrow DataTables.
- `duckdb` - Accelerated by an embedded DuckDB database.
- `postgres` - Accelerated by an embedded DuckDB database.
- `arrow` - Accelerated in-memory backed by Apache Arrow DataTables.
- `duckdb` - Accelerated by an embedded DuckDB database.
- `postgres` - Accelerated by an embedded DuckDB database.

## `acceleration.mode`

Optional. The mode of acceleration. The following values are supported:

- `memory` - Store acceleration data in-memory.
- `file` - Store acceleration data in a file.
- `memory` - Store acceleration data in-memory.
- `file` - Store acceleration data in a file.

`mode` is currently only supported for the `duckdb` engine.

## `acceleration.refresh_mode`

Optional. How to refresh the dataset. The following values are supported:

- `full` - Refresh the entire dataset.
- `append` - Append new data to the dataset.
- `full` - Refresh the entire dataset.
- `append` - Append new data to the dataset.

## `acceleration.refresh_interval`

Expand Down
24 changes: 24 additions & 0 deletions spiceaidocs/docs/secret-stores/env/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: 'Environment Secret Store'
sidebar_label: 'Environment Secret Store'
sidebar_position: 1
description: 'Environment Variables Secret Store Documentation'
---

The `env` store type enables Spice to read secrets from environment variables. Environment variables should be formatted `SPICE_SECRET_<secret-name>_<secret-value-key>`.

All variables with the same prefix `SPICE_SECRET_<secret-name>` are combined into a single secret. This allows grouping of related secret values under a single secret name.

## Example

```yaml
secrets:
store: env
```

Setting `spiceai` secret with spice.ai API key in `key` secret value:

```bash
SPICE_SECRET_SPICEAI_KEY="343533|**************" \
spice run
```
10 changes: 10 additions & 0 deletions spiceaidocs/docs/secret-stores/file/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
title: 'File Secret Store'
sidebar_label: 'File Secret Store'
sidebar_position: 2
description: 'File Secret Store Documentation'
---

The default secret store is the `file` secret store which stores secrets in a file located at `~/.spice/auth`.

The Spice CLI offers the `spice login` command to streamline credential storage using the file secret store. When using `spice login`, credentials are automatically stored in the `~/.spice/auth` file under `spiceai`.
80 changes: 8 additions & 72 deletions spiceaidocs/docs/secret-stores/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,84 +5,20 @@ description: ''
sidebar_position: 8
---

A secret store is a location where `secret` objects are stored, which are used to store sensitive data, like a passwords, a tokens, or keys.
A Secret Store is a location where `secret` objects are stored, used to store sensitive data, like passwords, tokens, secret keys.

Spice.ai supports four types of secret stores: `file`, `env`, `kubernetes` and `keyring`. The type of secret store is specified in the `store` field of the `secrets` section in the Spicepod manifest.
Spice supports multiple types of secret stores: `file`, `env`, `kubernetes` and `keyring`. The type of secret store is specified in the `store` field of the `secrets` section in the Spicepod manifest.

### File Secret Store

Default secret store uses a file located at `~/.spice/auth`.
The Spice.ai CLI offers the `spice login` command to streamline credential storage. When logging into Spice.ai, it automatically saves credentials in the secret store file under `spiceai` secret.

### Environment Secret Store

The `env` store type allows Spice.ai to read secrets from environment variables. The environment variables should be formatted like `SPICE_SECRET_<secret-name>_<secret-value-key>`.

All variables with the same prefix `SPICE_SECRET_<secret-name>` are combined into a single secret. This allows you to group related secret values under a single secret name.

**Example**
## Example

```yaml
secrets:
store: env
```

Setting `spiceai` secret with spice.ai API key in `key` secret value:

```bash
SPICE_SECRET_SPICEAI_KEY="343533|**************" \
spice run
```

### Kubernetes Secret Store

The `kubernetes` store type allows Spice.ai to read Kubernetes secrets.
## Secret Stores

```yaml
secrets:
store: kubernetes
```

Note: This method requires the Kubernetes service account, which is running the Spice.ai pod, to have extended roles for secrets API access. Make sure to configure this service account with the necessary permissions to read secrets from the Kubernetes API.

Example of Kubernetes role configuration for a custom service account:

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spiced-account-role
rules:
- apiGroups: ['']
resources: ['secrets']
verbs: ['get']
```

### Keyring Secret Store

The `keyring` store allows Spice.ai to access secrets from the secure store of the host operating system:

- On Linux, it uses the secret-service and kernel keyutils.
- On macOS, it uses the keychain.
- On Windows, it uses the credential manager.

The keyring store will read entries name to be formatted as `spice_secret_<secret-name>`, and entry account or user should be set to `spiced`.

Note: secret values required to be stored as JSON strings for compatibility with Spice.ai secret objects, as the keyring store supports only string values.

**Example**

For setting `spiceai` api key secret using macOS keychain, create new keychain entry, with following JSON string value

```
"{ key: "<your spice.ai app api key>" }"
```

<img src="/img/secrets-keychain-example.png" alt="" width="800" />

Then set `store` field of the `secrets` section in the Spicepod manifest:

```yaml
secrets:
store: keyring
```
- [Environment Secret Store](env/index.md)
- [File Secret Store](file/index.md)
- [Kubernetes Secret Store](kubernetes/index.md)
- [Keyring Secret Store](keyring/index.md)
33 changes: 33 additions & 0 deletions spiceaidocs/docs/secret-stores/keyring/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: 'Keyring Secret Store'
sidebar_label: 'Keyring Secret Store'
sidebar_position: 4
description: 'Keyring Secret Store Documentation'
---

The `keyring` store enables Spice to access secrets from the secure/credential store of the host operating system:

- Linux: The secret-service and kernel keyutils.
- macOS: The keychain.
- Windows: The Credential Manager.

The Keyring Store will read entries for names formatted as `spice_secret_<secret-name>` and where the entry account or user is set to `spiced`.

Note: For compatibility with Spice secret objects, secret values are required to be stored as JSON strings, as the keyring store only supports string values.

## Example

For setting `spiceai` api key secret using macOS keychain, create new keychain entry, with following JSON string value

```json
"{ key: "<your spice.ai app api key>" }"
```

<img src="/img/secrets-keychain-example.png" alt="" width="800" />

Then set `store` field of the `secrets` section in the Spicepod manifest:

```yaml
secrets:
store: keyring
```
30 changes: 30 additions & 0 deletions spiceaidocs/docs/secret-stores/kubernetes/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: 'Kubernetes Secret Store'
sidebar_label: 'Kubernetes Secret Store'
sidebar_position: 3
description: 'Kubernetes Secret Store Documentation'
---

The `kubernetes` store enables Spice to read Kubernetes secrets.

## Example

```yaml
secrets:
store: kubernetes
```

Note: This method requires the Kubernetes service account, which is running the `spiced` pod, to have extended roles for secrets API access. Make sure to configure this service account with the necessary permissions to read secrets from the Kubernetes API.

Example of Kubernetes role configuration for a custom service account:

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spiced-account-role
rules:
- apiGroups: ['']
resources: ['secrets']
verbs: ['get']
```