Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 31 additions & 35 deletions docs-website/router/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1091,53 +1091,49 @@ storage_providers:

### Storage Provider Types

#### File System Storage Provider
Storage providers are configured as arrays, so environment variables use indexed notation: `STORAGE_PROVIDER_{TYPE}_{INDEX}_{FIELD}`. Indexes must start from `0` and be sequential. If an index is skipped, all indexes after the gap are ignored. See [Storage Providers](/router/storage-providers) for more details.

This provider allows you to load files from the local file system. This is particularly useful for the MCP server to load GraphQL operations.
#### S3 Storage Provider

| Environment Variable | YAML | Required | Description | Default Value |
| ------------------------ | ---- | --------------------------------------------- | ------------------------------------------------------------- | ------------- |
| STORAGE_PROVIDER_FS_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_FS_PATH | path | <Icon icon="square-check" iconType="solid" /> | The path on the file system where files should be loaded from | |
For storing operations and configs in S3-compatible storage.

<Note>
The file system storage provider is **currently only supported by the MCP
feature**. It cannot be used for persisted operations or execution
configurations.
</Note>
| Environment Variable | YAML | Required | Description | Default Value |
| --------------------------------- | ---------- | --------------------------------------------- | ----------------------------------------------------------- | ------------- |
| STORAGE_PROVIDER_S3_\{n\}_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_S3_\{n\}_ENDPOINT | endpoint | <Icon icon="square-check" iconType="solid" /> | The endpoint of the S3 service | |
| STORAGE_PROVIDER_S3_\{n\}_ACCESS_KEY| access_key | <Icon icon="square-check" iconType="solid" /> | The access key for the S3 service | |
| STORAGE_PROVIDER_S3_\{n\}_SECRET_KEY| secret_key | <Icon icon="square-check" iconType="solid" /> | The secret key for the S3 service | |
| STORAGE_PROVIDER_S3_\{n\}_BUCKET | bucket | <Icon icon="square-check" iconType="solid" /> | The bucket where objects will be stored | |
| STORAGE_PROVIDER_S3_\{n\}_REGION | region | <Icon icon="square" /> | The region of the S3 bucket | |
| STORAGE_PROVIDER_S3_\{n\}_SECURE | secure | <Icon icon="square" /> | Whether to use HTTPS for S3 connections | false |

#### Redis Storage Provider
#### CDN Storage Provider

Used for distributed caching like automatic persisted queries.
For connecting to the Cosmo CDN.

| Environment Variable | YAML | Required | Description | Default Value |
| -------------------------------------- | --------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------- |
| STORAGE_PROVIDER_REDIS_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_REDIS_URLS | urls | <Icon icon="square-check" iconType="solid" /> | List of Redis URLs to connect to. If cluster_enabled is true, these are the seeds to discover the cluster | |
| STORAGE_PROVIDER_REDIS_CLUSTER_ENABLED | cluster_enabled | <Icon icon="square" /> | Whether to use the Redis Cluster client | false |
| Environment Variable | YAML | Required | Description | Default Value |
| ------------------------------ | ---- | --------------------------------------------- | ----------------------------------------------------------- | --------------------------------- |
| STORAGE_PROVIDER_CDN_\{n\}_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_CDN_\{n\}_URL | url | <Icon icon="square" /> | The URL of the CDN | https://cosmo-cdn.wundergraph.com |

#### S3 Storage Provider
#### Redis Storage Provider

For storing operations and configs in S3-compatible storage.
Used for distributed caching like automatic persisted queries.

| Environment Variable | YAML | Required | Description | Default Value |
| -------------------- | ---------- | --------------------------------------------- | ----------------------------------------------------------- | ------------- |
| | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| | endpoint | <Icon icon="square-check" iconType="solid" /> | The endpoint of the S3 service | |
| | access_key | <Icon icon="square-check" iconType="solid" /> | The access key for the S3 service | |
| | secret_key | <Icon icon="square-check" iconType="solid" /> | The secret key for the S3 service | |
| | bucket | <Icon icon="square-check" iconType="solid" /> | The bucket where objects will be stored | |
| | region | <Icon icon="square" /> | The region of the S3 bucket | |
| | secure | <Icon icon="square" /> | Whether to use HTTPS for S3 connections | true |
| Environment Variable | YAML | Required | Description | Default Value |
| --------------------------------------- | --------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------- |
| STORAGE_PROVIDER_REDIS_\{n\}_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_REDIS_\{n\}_URLS | urls | <Icon icon="square-check" iconType="solid" /> | List of Redis URLs to connect to. If cluster_enabled is true, these are the seeds to discover the cluster | |
| STORAGE_PROVIDER_REDIS_\{n\}_CLUSTER_ENABLED | cluster_enabled | <Icon icon="square" /> | Whether to use the Redis Cluster client | false |

#### CDN Provider
#### File System Storage Provider

For connecting to the Cosmo CDN.
This provider allows you to load files from the local file system.

| Environment Variable | YAML | Required | Description | Default Value |
| -------------------- | ---- | --------------------------------------------- | ----------------------------------------------------------- | --------------------------------- |
| | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| | url | <Icon icon="square-check" iconType="solid" /> | The URL of the CDN | https://cosmo-cdn.wundergraph.com |
| Environment Variable | YAML | Required | Description | Default Value |
| --------------------------- | ---- | --------------------------------------------- | ------------------------------------------------------------- | ------------- |
| STORAGE_PROVIDER_FS_\{n\}_ID | id | <Icon icon="square-check" iconType="solid" /> | The ID of the provider to reference in other configurations | |
| STORAGE_PROVIDER_FS_\{n\}_PATH | path | <Icon icon="square-check" iconType="solid" /> | The path on the file system where files should be loaded from | |

## Persisted Operations

Expand Down
4 changes: 4 additions & 0 deletions docs-website/router/storage-providers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,10 @@ Once a provider is defined, you reference it by `provider_id` in the configurati
* [**Execution config**](/router/configuration#execution-config-options) — load the router execution configuration from S3 instead of the Cosmo CDN.
* [**Persisted operations**](/router/persisted-queries/persisted-operations#using-a-custom-storage-provider) — load individual persisted operations or the PQL manifest from S3.

## Configuration via environment variables

Storage providers can also be configured entirely through environment variables using indexed notation. This is useful in containerized deployments or when secrets should not live in config files. See the [configuration reference](/router/configuration#storage-providers) for the full list of environment variables for each provider type.

## Best Practices

* Create different S3 credentials for READ and WRITE to reduce the attack surface.
38 changes: 19 additions & 19 deletions router/pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -902,10 +902,10 @@ type SubgraphErrorPropagationConfiguration struct {
}

type StorageProviders struct {
S3 []S3StorageProvider `yaml:"s3,omitempty"`
CDN []CDNStorageProvider `yaml:"cdn,omitempty"`
Redis []RedisStorageProvider `yaml:"redis,omitempty"`
FileSystem []FileSystemStorageProvider `yaml:"file_system,omitempty"`
S3 []S3StorageProvider `yaml:"s3,omitempty" envPrefix:"S3_"`
CDN []CDNStorageProvider `yaml:"cdn,omitempty" envPrefix:"CDN_"`
Redis []RedisStorageProvider `yaml:"redis,omitempty" envPrefix:"REDIS_"`
FileSystem []FileSystemStorageProvider `yaml:"file_system,omitempty" envPrefix:"FS_"`
}

type PersistedOperationsStorageConfig struct {
Expand All @@ -919,29 +919,29 @@ type AutomaticPersistedQueriesStorageConfig struct {
}

type S3StorageProvider struct {
ID string `yaml:"id,omitempty"`
Endpoint string `yaml:"endpoint,omitempty"`
AccessKey string `yaml:"access_key,omitempty"`
SecretKey string `yaml:"secret_key,omitempty"`
Bucket string `yaml:"bucket,omitempty"`
Region string `yaml:"region,omitempty"`
Secure bool `yaml:"secure,omitempty"`
ID string `yaml:"id,omitempty" env:"ID"`
Endpoint string `yaml:"endpoint,omitempty" env:"ENDPOINT"`
AccessKey string `yaml:"access_key,omitempty" env:"ACCESS_KEY"`
SecretKey string `yaml:"secret_key,omitempty" env:"SECRET_KEY"`
Bucket string `yaml:"bucket,omitempty" env:"BUCKET"`
Region string `yaml:"region,omitempty" env:"REGION"`
Secure bool `yaml:"secure,omitempty" env:"SECURE"`
}

type CDNStorageProvider struct {
ID string `yaml:"id,omitempty"`
URL string `yaml:"url,omitempty" envDefault:"https://cosmo-cdn.wundergraph.com"`
ID string `yaml:"id,omitempty" env:"ID"`
URL string `yaml:"url,omitempty" env:"URL" envDefault:"https://cosmo-cdn.wundergraph.com"`
}

type FileSystemStorageProvider struct {
ID string `yaml:"id,omitempty" env:"STORAGE_PROVIDER_FS_ID"`
Path string `yaml:"path,omitempty" env:"STORAGE_PROVIDER_FS_PATH"`
ID string `yaml:"id,omitempty" env:"ID"`
Path string `yaml:"path,omitempty" env:"PATH"`
}

type RedisStorageProvider struct {
ID string `yaml:"id,omitempty" env:"STORAGE_PROVIDER_REDIS_ID"`
URLs []string `yaml:"urls,omitempty" env:"STORAGE_PROVIDER_REDIS_URLS"`
ClusterEnabled bool `yaml:"cluster_enabled,omitempty" envDefault:"false" env:"STORAGE_PROVIDER_REDIS_CLUSTER_ENABLED"`
ID string `yaml:"id,omitempty" env:"ID"`
URLs []string `yaml:"urls,omitempty" env:"URLS"`
ClusterEnabled bool `yaml:"cluster_enabled,omitempty" env:"CLUSTER_ENABLED" envDefault:"false"`
}

type PersistedOperationsCDNProvider struct {
Expand Down Expand Up @@ -1237,7 +1237,7 @@ type Config struct {

SubgraphErrorPropagation SubgraphErrorPropagationConfiguration `yaml:"subgraph_error_propagation" envPrefix:"SUBGRAPH_ERROR_PROPAGATION_"`

StorageProviders StorageProviders `yaml:"storage_providers"`
StorageProviders StorageProviders `yaml:"storage_providers" envPrefix:"STORAGE_PROVIDER_"`
ExecutionConfig ExecutionConfig `yaml:"execution_config"`
PersistedOperationsConfig PersistedOperationsConfig `yaml:"persisted_operations" envPrefix:"PERSISTED_OPERATIONS_"`
AutomaticPersistedQueries AutomaticPersistedQueriesConfig `yaml:"automatic_persisted_queries"`
Expand Down
122 changes: 122 additions & 0 deletions router/pkg/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -595,6 +595,128 @@ execution_config:
})
}

func TestS3StorageProviderFromEnv(t *testing.T) {
// First S3 provider
t.Setenv("STORAGE_PROVIDER_S3_0_ID", "s3-one")
t.Setenv("STORAGE_PROVIDER_S3_0_ENDPOINT", "s3-one.example.com:9000")
t.Setenv("STORAGE_PROVIDER_S3_0_BUCKET", "bucket-one")
t.Setenv("STORAGE_PROVIDER_S3_0_ACCESS_KEY", "accessKey1")
t.Setenv("STORAGE_PROVIDER_S3_0_SECRET_KEY", "secretKey1")
t.Setenv("STORAGE_PROVIDER_S3_0_REGION", "us-east-1")
t.Setenv("STORAGE_PROVIDER_S3_0_SECURE", "true")

// Second S3 provider
t.Setenv("STORAGE_PROVIDER_S3_1_ID", "s3-two")
t.Setenv("STORAGE_PROVIDER_S3_1_ENDPOINT", "s3-two.example.com:9000")
t.Setenv("STORAGE_PROVIDER_S3_1_BUCKET", "bucket-two")
t.Setenv("STORAGE_PROVIDER_S3_1_ACCESS_KEY", "accessKey2")
t.Setenv("STORAGE_PROVIDER_S3_1_SECRET_KEY", "secretKey2")
t.Setenv("STORAGE_PROVIDER_S3_1_REGION", "eu-west-1")
t.Setenv("STORAGE_PROVIDER_S3_1_SECURE", "false")

f := createTempFileFromFixture(t, `
version: "1"
`)
cfg, err := LoadConfig([]string{f})
require.NoError(t, err)

require.Len(t, cfg.Config.StorageProviders.S3, 2)

s3One := cfg.Config.StorageProviders.S3[0]
require.Equal(t, "s3-one", s3One.ID)
require.Equal(t, "s3-one.example.com:9000", s3One.Endpoint)
require.Equal(t, "bucket-one", s3One.Bucket)
require.Equal(t, "accessKey1", s3One.AccessKey)
require.Equal(t, "secretKey1", s3One.SecretKey)
require.Equal(t, "us-east-1", s3One.Region)
require.True(t, s3One.Secure)

s3Two := cfg.Config.StorageProviders.S3[1]
require.Equal(t, "s3-two", s3Two.ID)
require.Equal(t, "s3-two.example.com:9000", s3Two.Endpoint)
require.Equal(t, "bucket-two", s3Two.Bucket)
require.Equal(t, "accessKey2", s3Two.AccessKey)
require.Equal(t, "secretKey2", s3Two.SecretKey)
require.Equal(t, "eu-west-1", s3Two.Region)
require.False(t, s3Two.Secure)
}

func TestCDNStorageProviderFromEnv(t *testing.T) {
t.Setenv("STORAGE_PROVIDER_CDN_0_ID", "cdn-one")
t.Setenv("STORAGE_PROVIDER_CDN_0_URL", "https://cdn-one.example.com")

t.Setenv("STORAGE_PROVIDER_CDN_1_ID", "cdn-two")
t.Setenv("STORAGE_PROVIDER_CDN_1_URL", "https://cdn-two.example.com")

f := createTempFileFromFixture(t, `
version: "1"
`)
cfg, err := LoadConfig([]string{f})
require.NoError(t, err)

require.Len(t, cfg.Config.StorageProviders.CDN, 2)

cdnOne := cfg.Config.StorageProviders.CDN[0]
require.Equal(t, "cdn-one", cdnOne.ID)
require.Equal(t, "https://cdn-one.example.com", cdnOne.URL)

cdnTwo := cfg.Config.StorageProviders.CDN[1]
require.Equal(t, "cdn-two", cdnTwo.ID)
require.Equal(t, "https://cdn-two.example.com", cdnTwo.URL)
}

func TestRedisStorageProviderFromEnv(t *testing.T) {
t.Setenv("STORAGE_PROVIDER_REDIS_0_ID", "redis-one")
t.Setenv("STORAGE_PROVIDER_REDIS_0_URLS", "redis://localhost:6379,redis://localhost:6380")
t.Setenv("STORAGE_PROVIDER_REDIS_0_CLUSTER_ENABLED", "true")

t.Setenv("STORAGE_PROVIDER_REDIS_1_ID", "redis-two")
t.Setenv("STORAGE_PROVIDER_REDIS_1_URLS", "redis://localhost:7379")
t.Setenv("STORAGE_PROVIDER_REDIS_1_CLUSTER_ENABLED", "false")

f := createTempFileFromFixture(t, `
version: "1"
`)
cfg, err := LoadConfig([]string{f})
require.NoError(t, err)

require.Len(t, cfg.Config.StorageProviders.Redis, 2)

redisOne := cfg.Config.StorageProviders.Redis[0]
require.Equal(t, "redis-one", redisOne.ID)
require.Equal(t, []string{"redis://localhost:6379", "redis://localhost:6380"}, redisOne.URLs)
require.True(t, redisOne.ClusterEnabled)

redisTwo := cfg.Config.StorageProviders.Redis[1]
require.Equal(t, "redis-two", redisTwo.ID)
require.Equal(t, []string{"redis://localhost:7379"}, redisTwo.URLs)
require.False(t, redisTwo.ClusterEnabled)
}

func TestFileSystemStorageProviderFromEnv(t *testing.T) {
t.Setenv("STORAGE_PROVIDER_FS_0_ID", "fs-one")
t.Setenv("STORAGE_PROVIDER_FS_0_PATH", "/data/configs")

t.Setenv("STORAGE_PROVIDER_FS_1_ID", "fs-two")
t.Setenv("STORAGE_PROVIDER_FS_1_PATH", "/data/backups")

f := createTempFileFromFixture(t, `
version: "1"
`)
cfg, err := LoadConfig([]string{f})
require.NoError(t, err)

require.Len(t, cfg.Config.StorageProviders.FileSystem, 2)

fsOne := cfg.Config.StorageProviders.FileSystem[0]
require.Equal(t, "fs-one", fsOne.ID)
require.Equal(t, "/data/configs", fsOne.Path)

fsTwo := cfg.Config.StorageProviders.FileSystem[1]
require.Equal(t, "fs-two", fsTwo.ID)
require.Equal(t, "/data/backups", fsTwo.Path)
}

func TestInvalidExecutionConfig(t *testing.T) {
t.Parallel()

Expand Down
Loading