Remove support schema flags, only use config file. (#2221)

* Remove support schema flags, only use config file.

Also rename the schema config file flag to something sane.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Update docs to use the schema file.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Deprecate not remove the config flag.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Make integration tests pass?

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Remove support schema flags, only use config file.

Also rename the schema config file flag to something sane.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Update docs to use the schema file.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Deprecate not remove the config flag.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Update docs/configuration/schema-config-reference.md

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed schema config doc

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixes after rebase

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Remove duplicated entry from CHANGELOG

Signed-off-by: Marco Pracucci <marco@pracucci.com>

Co-authored-by: Marco Pracucci <marco@pracucci.com>

Author: gouthamve

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## master / unreleased
 
+* [CHANGE] Removed support for flags to configure schema. Further, the flag for specifying the config file (`-config-yaml`) has been deprecated. Please use `schema-config-file`. See https://cortexmetrics.io/docs/configuration/schema-configuration/ for more details on how to configure the schema using the YAML file. #2221
 * [CHANGE] The frontend http server will now send 502 in case of deadline exceeded and 499 if the user requested cancellation. #2156
 * [CHANGE] Config file changed to remove top level `config_store` field in favor of a nested `configdb` field. #2125
 * [CHANGE] We now enforce queries to be up to `-querier.max-query-into-future` into the future (defaults to 10m). #1929

docs/configuration/schema-config-reference.md

@@ -0,0 +1,127 @@
+---
+title: "Schema Configuration"
+linkTitle: "Schema Configuration"
+weight: 1
+slug: schema-configuration
+---
+
+Cortex uses a NoSQL Store to store its index and optionally an Object store to store its chunks. Cortex has overtime evolved its schema to be more optimal and better fit the use cases and query patterns that arose. 
+
+Currently there are 9 schemas that are used in production but we recommend running with `v9` schema when possible. You can move from one schema to another if a new schema fits your purpose better, but you still need to configure Cortex to make sure it can read the old data in the old schemas.
+
+You can configure the schemas using a YAML config file, that you can point to using the `-schema-config-file` flag. It has the following YAML spec:
+
+```yaml
+configs: []<period_config>
+```
+
+Where `period_config` is
+```
+# In YYYY-MM-DD format, for example: 2020-03-01.
+from: <string>
+# The index client to use, valid options: aws-dynamo, bigtable, bigtable-hashed, cassandra, boltdb.
+store: <string>
+# The object client to use. If none is specified, `store` is used for storing chunks as well. Valid options: s3, aws-dynamo, bigtable, bigtable-hashed, gcs, cassandra, filesystem.
+object_store: <string>
+# The schema version to use. Valid ones are v1, v2, v3,... v6, v9, v10, v11. Recommended for production: v9.
+schema: <string>
+index: <periodic_table_config>
+chunks: <periodic_table_config>
+```
+
+Where `periodic_table_config` is
+```
+# The prefix to use for the tables.
+prefix: <string>
+# We typically run Cortex with new tables every week to keep the index size low and to make retention easier. This sets the period at which new tables are created and used. Typically 168h (1week).
+period: <duration>
+# The tags that can be set on the dynamo table.
+tags: <map[string]string>
+```
+
+Now an example of this file (also something recommended when starting out) is:
+```
+configs:
+  - from: "2020-03-01" # Or typically a week before the Cortex cluster was created.
+    schema: v9
+    index:
+      period: 168h
+      prefix: cortex_index_
+    # Chunks section is optional and required only if you're not using a
+    # separate object store.
+    chunks:
+      period: 168h
+      prefix: cortex_chunks
+    store: aws-dynamo/bigtable-hashed/cassandra/boltdb
+    object_store: <above options>/s3/gcs/azure/filesystem
+```
+
+An example of an advanced schema file with a lot of changes:
+```
+configs:
+  # Starting from 2018-08-23 Cortex should store chunks and indexes
+  # on Google BigTable using weekly periodic tables. The chunks table
+  # names will be prefixed with "dev_chunks_", while index tables will be
+  # prefixed with "dev_index_".
+  - from: "2018-08-23"
+    schema: v9
+    chunks:
+        period: 168h0m0s
+        prefix: dev_chunks_
+    index:
+        period: 168h0m0s
+        prefix: dev_index_
+    store: gcp-columnkey
+
+  # Starting 2018-02-13 we moved from BigTable to GCS for storing the chunks.
+  - from: "2019-02-13"
+    schema: v9
+    chunks:
+        period: 168h
+        prefix: dev_chunks_
+    index:
+        period: 168h
+        prefix: dev_index_
+    object_store: gcs
+    store: gcp-columnkey
+
+  # Starting 2019-02-24 we moved our index from bigtable-columnkey to bigtable-hashed
+  # which improves the distribution of keys.
+  - from: "2019-02-24"
+    schema: v9
+    chunks:
+        period: 168h
+        prefix: dev_chunks_
+    index:
+        period: 168h
+        prefix: dev_index_
+    object_store: gcs
+    store: bigtable-hashed
+
+  # Starting 2019-03-05 we moved from v9 schema to v10 schema. 
+  - from: "2019-03-05"
+    schema: v10
+    chunks:
+        period: 168h
+        prefix: dev_chunks_
+    index:
+        period: 168h
+        prefix: dev_index_
+    object_store: gcs
+    store: bigtable-hashed
+```
+
+Note how we started out with v9 and just Bigtable, but later migrated to GCS as the object store, finally moving to v10. This is a complex schema file showing several changes changes over the time, while a typical schema config file usually has just one or two schema versions.
+
+### Migrating from flags to schema file
+
+Legacy versions of Cortex did support the ability to configure schema via flags. If you are still using flags, you need to migrate your configuration from flags to the config file.
+
+If you're using:
+
+* `chunk.storage-client`: then set the corresponding `object_store` field correctly in the schema file.
+* `dynamodb.daily-buckets-from`: then set the corresponding `from` date with `v2` schema.
+* `dynamodb.base64-buckets-from`: then set the corresponding `from` date with `v3` schema.
+* `dynamodb.v{4,5,6,9}-schema-from`: then set the corresponding `from` date with schema `v{4,5,6,9}`
+* `bigtable.column-key-from`: then set the corresponding `from` date and use the `store` as `bigtable-columnkey`.
+* `dynamodb.use-periodic-tables`: then set the right `index` and `chunk` fields with corresponding values from `dynamodb.periodic-table.{prefix, period, tag}` and `dynamodb.chunk-table.{prefix, period, tag}` flags. Note that the default period is 7 days, so please set the `period` as `168h` in the config file if none is set in the flags.

docs/guides/aws-specific.md

@@ -51,10 +51,8 @@ example set of command-line parameters from a fairly modest install:
    -metrics.url=http://prometheus.monitoring.svc.cluster.local./api/prom/
    -metrics.target-queue-length=100000
    -dynamodb.url=dynamodb://us-east-1/
-   -dynamodb.use-periodic-tables=true
+   -schema-config-file=/etc/schema.yaml
 
-   -dynamodb.periodic-table.prefix=cortex_index_
-   -dynamodb.periodic-table.from=2019-05-02
    -dynamodb.periodic-table.write-throughput=1000
    -dynamodb.periodic-table.write-throughput.scale.enabled=true
    -dynamodb.periodic-table.write-throughput.scale.min-capacity=200
@@ -64,8 +62,6 @@ example set of command-line parameters from a fairly modest install:
    -dynamodb.periodic-table.read-throughput=300
    -dynamodb.periodic-table.tag=product_area=cortex
 
-   -dynamodb.chunk-table.from=2019-05-02
-   -dynamodb.chunk-table.prefix=cortex_data_
    -dynamodb.chunk-table.write-throughput=800
    -dynamodb.chunk-table.write-throughput.scale.enabled=true
    -dynamodb.chunk-table.write-throughput.scale.min-capacity=200

docs/guides/running.md

@@ -55,22 +55,7 @@ redundancy or less, depending on your risk appetite.
 
 ### Schema
 
-#### Schema periodic table
-
-The periodic table from argument (`-dynamodb.periodic-table.from=<date>` if
-using command line flags, the `from` field for the first schema entry if using
-YAML) should be set to the date the oldest metrics you will be sending to
-Cortex. Generally that means set it to the date you are first deploying this
-instance. If you use an example date from years ago table-manager will create
-hundreds of tables. You can also avoid creating too many tables by setting a
-reasonable retention in the table-manager
-(`-table-manager.retention-period=<duration>`).
-
-#### Schema version
-
-Choose schema version 9 in most cases; version 10 if you expect
-hundreds of thousands of timeseries under a single name.  Anything
-older than v9 is much less efficient.
+See [schema config file docs](../configuration/schema-config-reference.md).
 
 ### Chunk encoding
 

integration/backward_compatibility_test.go

@@ -31,13 +31,18 @@ func TestBackwardCompatibilityWithChunksStorage(t *testing.T) {
 	consul := e2edb.NewConsul()
 	require.NoError(t, s.StartAndWaitReady(dynamo, consul))
 
+	flagsForOldImage := mergeFlags(ChunksStorageFlags, map[string]string{
+		"-schema-config-file": "",
+		"-config-yaml":        ChunksStorageFlags["-schema-config-file"],
+	})
+
 	// Start Cortex components (ingester running on previous version).
 	require.NoError(t, writeFileToSharedDir(s, cortexSchemaConfigFile, []byte(cortexSchemaConfigYaml)))
-	tableManager := e2ecortex.NewTableManager("table-manager", ChunksStorageFlags, previousVersionImage)
+	tableManager := e2ecortex.NewTableManager("table-manager", flagsForOldImage, previousVersionImage)
 	// Old table-manager doesn't expose a readiness probe, so we just check if the / returns 404
 	tableManager.SetReadinessProbe(e2e.NewReadinessProbe(tableManager.HTTPPort(), "/", 404))
-	ingester1 := e2ecortex.NewIngester("ingester-1", consul.NetworkHTTPEndpoint(), ChunksStorageFlags, "")
-	distributor := e2ecortex.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), ChunksStorageFlags, "")
+	ingester1 := e2ecortex.NewIngester("ingester-1", consul.NetworkHTTPEndpoint(), flagsForOldImage, "")
+	distributor := e2ecortex.NewDistributor("distributor", consul.NetworkHTTPEndpoint(), flagsForOldImage, "")
 	// Old ring didn't have /ready probe, use /ring instead.
 	distributor.SetReadinessProbe(e2e.NewReadinessProbe(distributor.HTTPPort(), "/ring", 200))
 	require.NoError(t, s.StartAndWaitReady(distributor, ingester1, tableManager))
@@ -72,7 +77,11 @@ func TestBackwardCompatibilityWithChunksStorage(t *testing.T) {
 
 	// Query the new ingester both with the old and the new querier.
 	for _, image := range []string{previousVersionImage, ""} {
-		querier := e2ecortex.NewQuerier("querier", consul.NetworkHTTPEndpoint(), ChunksStorageFlags, image)
+		flags := ChunksStorageFlags
+		if image == previousVersionImage {
+			flags = flagsForOldImage
+		}
+		querier := e2ecortex.NewQuerier("querier", consul.NetworkHTTPEndpoint(), flags, image)
 		require.NoError(t, s.StartAndWaitReady(querier))
 
 		// Wait until the querier has updated the ring.

integration/configs.go

@@ -90,7 +90,7 @@ tsdb:
 	ChunksStorageFlags = map[string]string{
 		"-dynamodb.url":                   fmt.Sprintf("dynamodb://u:p@%s-dynamodb.:8000", networkName),
 		"-dynamodb.poll-interval":         "1m",
-		"-config-yaml":                    filepath.Join(e2e.ContainerSharedDir, cortexSchemaConfigFile),
+		"-schema-config-file":             filepath.Join(e2e.ContainerSharedDir, cortexSchemaConfigFile),
 		"-table-manager.retention-period": "168h",
 	}
 

integration/e2e/util.go

@@ -30,6 +30,12 @@ func MergeFlags(inputs ...map[string]string) map[string]string {
 		}
 	}
 
+	for k, v := range output {
+		if v == "" {
+			delete(output, k)
+		}
+	}
+
 	return output
 }
 

k8s/ingester-dep.yaml

@@ -45,16 +45,8 @@ spec:
         - -ingester.claim-on-rollout=true
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -s3.url=s3://abc:123@s3.default.svc.cluster.local:4569
-        - -dynamodb.original-table-name=cortex
         - -dynamodb.url=dynamodb://user:pass@dynamodb.default.svc.cluster.local:8000
-        - -dynamodb.periodic-table.prefix=cortex_weekly_
-        - -dynamodb.periodic-table.from=2017-01-06
-        - -dynamodb.daily-buckets-from=2017-01-10
-        - -dynamodb.base64-buckets-from=2017-01-17
-        - -dynamodb.v4-schema-from=2017-02-05
-        - -dynamodb.v5-schema-from=2017-02-22
-        - -dynamodb.v6-schema-from=2017-03-19
-        - -dynamodb.chunk-table.from=2017-04-17
+        - -schema-config-file=/etc/cortex/schema.yaml
         - -memcached.hostname=memcached.default.svc.cluster.local
         - -memcached.timeout=100ms
         - -memcached.service=memcached
@@ -66,3 +58,10 @@ spec:
             port: 80
           initialDelaySeconds: 15
           timeoutSeconds: 1
+        volumeMounts:
+        - name: config-volume
+          mountPath: /etc/cortex
+      volumes:
+        - name: config-volume
+          configMap:
+            name: schema-config

k8s/querier-dep.yaml

@@ -22,20 +22,19 @@ spec:
         - -server.http-listen-port=80
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -s3.url=s3://abc:123@s3.default.svc.cluster.local:4569
-        - -querier.frontend-address=query-frontend.default.svc.cluster.local:9095
-        - -dynamodb.original-table-name=cortex
         - -dynamodb.url=dynamodb://user:pass@dynamodb.default.svc.cluster.local:8000
-        - -dynamodb.periodic-table.prefix=cortex_weekly_
-        - -dynamodb.periodic-table.from=2017-01-06
-        - -dynamodb.daily-buckets-from=2017-01-10
-        - -dynamodb.base64-buckets-from=2017-01-17
-        - -dynamodb.v4-schema-from=2017-02-05
-        - -dynamodb.v5-schema-from=2017-02-22
-        - -dynamodb.v6-schema-from=2017-03-19
-        - -dynamodb.chunk-table.from=2017-04-17
+        - -schema-config-file=/etc/cortex/schema.yaml
+        - -querier.frontend-address=query-frontend.default.svc.cluster.local:9095
         - -memcached.hostname=memcached.default.svc.cluster.local
         - -memcached.timeout=100ms
         - -memcached.service=memcached
         - -distributor.replication-factor=1
         ports:
         - containerPort: 80
+        volumeMounts:
+        - name: config-volume
+          mountPath: /etc/cortex
+      volumes:
+        - name: config-volume
+          configMap:
+            name: schema-config

k8s/ruler-dep.yaml

@@ -25,19 +25,18 @@ spec:
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/api/prom/alertmanager/
         - -consul.hostname=consul.default.svc.cluster.local:8500
         - -s3.url=s3://abc:123@default.svc.cluster.local:4569/s3
-        - -dynamodb.original-table-name=cortex
         - -dynamodb.url=dynamodb://user:pass@dynamodb.default.svc.cluster.local:8000
-        - -dynamodb.periodic-table.prefix=cortex_weekly_
-        - -dynamodb.periodic-table.from=2017-01-06
-        - -dynamodb.daily-buckets-from=2017-01-10
-        - -dynamodb.base64-buckets-from=2017-01-17
-        - -dynamodb.v4-schema-from=2017-02-05
-        - -dynamodb.v5-schema-from=2017-02-22
-        - -dynamodb.v6-schema-from=2017-03-19
-        - -dynamodb.chunk-table.from=2017-04-17
+        - -schema-config-file=/etc/cortex/schema.yaml
         - -memcached.hostname=memcached.default.svc.cluster.local
         - -memcached.timeout=100ms
         - -memcached.service=memcached
         - -distributor.replication-factor=1
         ports:
         - containerPort: 80
+        volumeMounts:
+        - name: config-volume
+          mountPath: /etc/cortex
+      volumes:
+        - name: config-volume
+          configMap:
+            name: schema-config