Merge remote-tracking branch 'upstream/master' into am-distributors

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Author: Ganesh Vernekar

CHANGELOG.md

@@ -4,12 +4,14 @@
 
 * [CHANGE] Ingester: don't update internal "last updated" timestamp of TSDB if tenant only sends invalid samples. This affects how "idle" time is computed. #3727
 * [CHANGE] Require explicit flag `-<prefix>.tls-enabled` to enable TLS in GRPC clients. Previously it was enough to specify a TLS flag to enable TLS validation. #3156
-* [FEATURE] Adds support to S3 server side encryption using KMS. Deprecated `-<prefix>.s3.sse-encryption`, you should use the following CLI flags that have been added. #3651
+* [FEATURE] Adds support to S3 server side encryption using KMS. Deprecated `-<prefix>.s3.sse-encryption`, you should use the following CLI flags that have been added. #3651 #3810
   - `-<prefix>.s3.sse.type`
   - `-<prefix>.s3.sse.kms-key-id`
   - `-<prefix>.s3.sse.kms-encryption-context`
 * [FEATURE] Querier: Enable `@ <timestamp>` modifier in PromQL using the new `-querier.at-modifier-enabled` flag. #3744
-* [FEATURE] Alertmanager: It now shards the `/api/v1/alerts` API using the ring when sharding is enabled.
+* [FEATURE] Overrides Exporter: Add `overrides-exporter` module for exposing per-tenant resource limit overrides as metrics. It is not included in `all` target, and must be explicitly enabled. #3785
+* [FEATURE] Experimental thanosconvert: introduce an experimental tool `thanosconvert` to migrate Thanos block metadata to Cortex metadata. #3770
+* [FEATURE] Alertmanager: It now shards the `/api/v1/alerts` API using the ring when sharding is enabled. #3671
   * Added `NewMaxBytesHandler` in the utils package for limiting the size of http request body.
   * New flags added for communication between alertmanagers:
     * `-alertmanager.max-recv-msg-size`
@@ -20,6 +22,14 @@
     * `-alertmanager.alertmanager-client.tls-ca-path`
     * `-alertmanager.alertmanager-client.tls-server-name`
     * `-alertmanager.alertmanager-client.tls-insecure-skip-verify`
+* [ENHANCEMENT] Ruler: Add TLS and explicit basis authentication configuration options for the HTTP client the ruler uses to communicate with the alertmanager. #3752
+  * `-ruler.alertmanager-client.basic-auth-username`: Configure the basic authentication username used by the client. Takes precedent over a URL configured username.
+  * `-ruler.alertmanager-client.basic-auth-password`: Configure the basic authentication password used by the client. Takes precedent over a URL configured password.
+  * `-ruler.alertmanager-client.tls-ca-path`: File path to the CA file.
+  * `-ruler.alertmanager-client.tls-cert-path`: File path to the TLS certificate.
+  * `-ruler.alertmanager-client.tls-insecure-skip-verify`: Boolean to disable verifying the certificate.
+  * `-ruler.alertmanager-client.tls-key-path`: File path to the TLS key certificate.
+  * `-ruler.alertmanager-client.tls-server-name`: Expected name on the TLS certificate.
 * [ENHANCEMENT] Ingester: exposed metric `cortex_ingester_oldest_unshipped_block_timestamp_seconds`, tracking the unix timestamp of the oldest TSDB block not shipped to the storage yet. #3705
 * [ENHANCEMENT] Prometheus upgraded. #3739
   * Avoid unnecessary `runtime.GC()` during compactions.
@@ -29,8 +39,27 @@
 * [ENHANCEMENT] Distributor: Enable downstream projects to wrap distributor push function and access the deserialized write requests berfore/after they are pushed. #3755
 * [ENHANCEMENT] Add flag `-<prefix>.tls-server-name` to require a specific server name instead of the hostname on the certificate. #3156
 * [ENHANCEMENT] Alertmanager: Remove a tenant's alertmanager instead of pausing it as we determine it is no longer needed. #3722
+* [ENHANCEMENT] Blocks storage: added more configuration options to S3 client. #3775
+  * `-blocks-storage.s3.tls-handshake-timeout`: Maximum time to wait for a TLS handshake. 0 means no limit.
+  * `-blocks-storage.s3.expect-continue-timeout`: The time to wait for a server's first response headers after fully writing the request headers if the request has an Expect header. 0 to send the request body immediately.
+  * `-blocks-storage.s3.max-idle-connections`: Maximum number of idle (keep-alive) connections across all hosts. 0 means no limit.
+  * `-blocks-storage.s3.max-idle-connections-per-host`: Maximum number of idle (keep-alive) connections to keep per-host. If 0, a built-in default value is used.
+  * `-blocks-storage.s3.max-connections-per-host`: Maximum number of connections per host. 0 means no limit.
+* [ENHANCEMENT] Ingester: when tenant's TSDB is closed, Ingester now removes pushed metrics-metadata from memory, and removes metadata (`cortex_ingester_memory_metadata`, `cortex_ingester_memory_metadata_created_total`, `cortex_ingester_memory_metadata_removed_total`) and validation metrics (`cortex_discarded_samples_total`, `cortex_discarded_metadata_total`). #3782
+* [ENHANCEMENT] Distributor: cleanup metrics for inactive tenants. #3784
+* [ENHANCEMENT] Ingester: Have ingester to re-emit following TSDB metrics. #3800
+  * `cortex_ingester_tsdb_blocks_loaded`
+  * `cortex_ingester_tsdb_reloads_total`
+  * `cortex_ingester_tsdb_reloads_failures_total`
+  * `cortex_ingester_tsdb_symbol_table_size_bytes`
+  * `cortex_ingester_tsdb_storage_blocks_bytes`
+  * `cortex_ingester_tsdb_time_retentions_total`
+* [BUGFIX] Cortex: Fixed issue where fatal errors and various log messages where not logged. #3778
 * [BUGFIX] HA Tracker: don't track as error in the `cortex_kv_request_duration_seconds` metric a CAS operation intentionally aborted. #3745
 * [BUGFIX] Querier / ruler: do not log "error removing stale clients" if the ring is empty. #3761
+* [BUGFIX] Store-gateway: fixed a panic caused by a race condition when the index-header lazy loading is enabled. #3775 #3789
+* [BUGFIX] Compactor: fixed "could not guess file size" log when uploading blocks deletion marks to the global location. #3807
+* [BUGFIX] Prevent panic at start if the http_prefix setting doesn't have a valid value. #3796
 
 ## 1.7.0 in progress
 

Makefile

@@ -181,6 +181,10 @@ lint:
 		./pkg/frontend/... \
 		./pkg/querier/tenantfederation/... \
 		./pkg/querier/queryrange/...
+	# Ensure packages that no longer use a global logger don't reintroduce it
+	faillint -paths "github.com/cortexproject/cortex/pkg/util/log.{Logger}" \
+		./pkg/ingester/... \
+		./pkg/flusher/...
 
 	# Validate Kubernetes spec files. Requires:
 	# https://kubeval.instrumenta.dev
@@ -223,6 +227,7 @@ doc: clean-doc
 	go run ./tools/doc-generator ./docs/blocks-storage/querier.template              > ./docs/blocks-storage/querier.md
 	embedmd -w docs/operations/requests-mirroring-to-secondary-cluster.md
 	embedmd -w docs/configuration/single-process-config.md
+	embedmd -w docs/guides/overrides-exporter.md
 
 endif
 

README.md

@@ -154,9 +154,9 @@ in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/ove
 
 ### Grafana Cloud
 
-To use Cortex as part of Grafana Cloud, sign up for [Grafana Cloud](https://grafana.com/cloud)
-by clicking "Log In" in the top right and then "Sign Up Now".  Cortex is included
-as part of the Starter and Basic Hosted Grafana plans.
+The Cortex project was started by Tom Wilkie (Grafana Labs' VP Product) and Julius Volz (Prometheus' co-founder) in June 2016. Employing 6 out of 8 maintainers for Cortex enables [Grafana Labs](https://grafana.com/) to offer Cortex-as-a-service with exceptional performance and reliability. As the creators of [Grafana](https://grafana.com/oss/grafana/), [Loki](https://grafana.com/oss/loki/), and [Tempo](https://grafana.com/oss/tempo/), Grafana Labs can offer you the most wholistic Observability-as-a-Service stack out there.
+
+For further information see Grafana Cloud [documentation](https://grafana.com/docs/grafana-cloud/), [tutorials](https://grafana.com/tutorials/), [webinars](https://grafana.com/videos/), and [KubeCon talks](https://grafana.com/categories/cortex/). Get started today and [sign up here](https://grafana.com/products/cloud/).
 
 ### Amazon Managed Service for Prometheus (AMP)
 

cmd/blocksconvert/main.go

@@ -13,7 +13,7 @@ import (
 	"github.com/weaveworks/common/signals"
 
 	"github.com/cortexproject/cortex/pkg/cortex"
-	"github.com/cortexproject/cortex/pkg/util"
+	util_log "github.com/cortexproject/cortex/pkg/util/log"
 	"github.com/cortexproject/cortex/pkg/util/services"
 	"github.com/cortexproject/cortex/tools/blocksconvert"
 	"github.com/cortexproject/cortex/tools/blocksconvert/builder"
@@ -44,12 +44,12 @@ func main() {
 	cfg.ServerConfig.RegisterFlags(flag.CommandLine)
 	flag.Parse()
 
-	util.InitLogger(&cfg.ServerConfig)
+	util_log.InitLogger(&cfg.ServerConfig)
 
 	cortex.DisableSignalHandling(&cfg.ServerConfig)
 	serv, err := server.New(cfg.ServerConfig)
 	if err != nil {
-		level.Error(util.Logger).Log("msg", "Unable to initialize server", "err", err.Error())
+		level.Error(util_log.Logger).Log("msg", "Unable to initialize server", "err", err.Error())
 		os.Exit(1)
 	}
 
@@ -60,19 +60,19 @@ func main() {
 	var targetService services.Service
 	switch cfg.Target {
 	case "scanner":
-		targetService, err = scanner.NewScanner(cfg.ScannerConfig, cfg.SharedConfig, util.Logger, registry)
+		targetService, err = scanner.NewScanner(cfg.ScannerConfig, cfg.SharedConfig, util_log.Logger, registry)
 	case "builder":
-		targetService, err = builder.NewBuilder(cfg.BuilderConfig, cfg.SharedConfig, util.Logger, registry)
+		targetService, err = builder.NewBuilder(cfg.BuilderConfig, cfg.SharedConfig, util_log.Logger, registry)
 	case "scheduler":
-		targetService, err = scheduler.NewScheduler(cfg.SchedulerConfig, cfg.SharedConfig, util.Logger, registry, serv.HTTP, serv.GRPC)
+		targetService, err = scheduler.NewScheduler(cfg.SchedulerConfig, cfg.SharedConfig, util_log.Logger, registry, serv.HTTP, serv.GRPC)
 	case "cleaner":
-		targetService, err = cleaner.NewCleaner(cfg.CleanerConfig, cfg.SharedConfig, util.Logger, registry)
+		targetService, err = cleaner.NewCleaner(cfg.CleanerConfig, cfg.SharedConfig, util_log.Logger, registry)
 	default:
 		err = fmt.Errorf("unknown target")
 	}
 
 	if err != nil {
-		level.Error(util.Logger).Log("msg", "failed to initialize", "err", err)
+		level.Error(util_log.Logger).Log("msg", "failed to initialize", "err", err)
 		os.Exit(1)
 	}
 
@@ -88,7 +88,7 @@ func main() {
 		err = services.StartManagerAndAwaitHealthy(context.Background(), servManager)
 	}
 	if err != nil {
-		level.Error(util.Logger).Log("msg", "Unable to start", "err", err.Error())
+		level.Error(util_log.Logger).Log("msg", "Unable to start", "err", err.Error())
 		os.Exit(1)
 	}
 
@@ -101,7 +101,7 @@ func main() {
 
 	// We only wait for target service. If any other service fails, listener will stop it (via manager)
 	if err := targetService.AwaitTerminated(context.Background()); err != nil {
-		level.Error(util.Logger).Log("msg", cfg.Target+" failed", "err", targetService.FailureCase())
+		level.Error(util_log.Logger).Log("msg", cfg.Target+" failed", "err", targetService.FailureCase())
 		os.Exit(1)
 	}
 }

cmd/cortex/main.go

@@ -22,7 +22,7 @@ import (
 	"github.com/cortexproject/cortex/pkg/cortex"
 	"github.com/cortexproject/cortex/pkg/util"
 	"github.com/cortexproject/cortex/pkg/util/flagext"
-	"github.com/cortexproject/cortex/pkg/util/log"
+	util_log "github.com/cortexproject/cortex/pkg/util/log"
 )
 
 // Version is set via build flag -ldflags -X main.Version
@@ -118,7 +118,7 @@ func main() {
 
 	// Validate the config once both the config file has been loaded
 	// and CLI flags parsed.
-	err = cfg.Validate(util.Logger)
+	err = cfg.Validate(util_log.Logger)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "error validating config: %v\n", err)
 		if !testMode {
@@ -137,7 +137,7 @@ func main() {
 		runtime.SetMutexProfileFraction(mutexProfileFraction)
 	}
 
-	util.InitLogger(&cfg.Server)
+	util_log.InitLogger(&cfg.Server)
 
 	// Allocate a block of memory to alter GC behaviour. See https://github.com/golang/go/issues/23044
 	ballast := make([]byte, ballastBytes)
@@ -154,7 +154,7 @@ func main() {
 
 		// Setting the environment variable JAEGER_AGENT_HOST enables tracing.
 		if trace, err := tracing.NewFromEnv(name); err != nil {
-			level.Error(util.Logger).Log("msg", "Failed to setup tracing", "err", err.Error())
+			level.Error(util_log.Logger).Log("msg", "Failed to setup tracing", "err", err.Error())
 		} else {
 			defer trace.Close()
 		}
@@ -164,7 +164,7 @@ func main() {
 	rand.Seed(time.Now().UnixNano())
 
 	t, err := cortex.New(cfg)
-	log.CheckFatal("initializing cortex", err)
+	util_log.CheckFatal("initializing cortex", err)
 
 	if printModules {
 		allDeps := t.ModuleManager.DependenciesForModule(cortex.All)
@@ -185,12 +185,12 @@ func main() {
 		return
 	}
 
-	level.Info(util.Logger).Log("msg", "Starting Cortex", "version", version.Info())
+	level.Info(util_log.Logger).Log("msg", "Starting Cortex", "version", version.Info())
 
 	err = t.Run()
 
 	runtime.KeepAlive(ballast)
-	log.CheckFatal("running cortex", err)
+	util_log.CheckFatal("running cortex", err)
 }
 
 // Parse -config.file and -config.expand-env option via separate flag set, to avoid polluting default one and calling flag.Parse on it twice.

cmd/query-tee/main.go

@@ -9,7 +9,7 @@ import (
 	"github.com/weaveworks/common/logging"
 	"github.com/weaveworks/common/server"
 
-	"github.com/cortexproject/cortex/pkg/util"
+	util_log "github.com/cortexproject/cortex/pkg/util/log"
 	"github.com/cortexproject/cortex/tools/querytee"
 )
 
@@ -29,7 +29,7 @@ func main() {
 	cfg.ProxyConfig.RegisterFlags(flag.CommandLine)
 	flag.Parse()
 
-	util.InitLogger(&server.Config{
+	util_log.InitLogger(&server.Config{
 		LogLevel: cfg.LogLevel,
 	})
 
@@ -39,19 +39,19 @@ func main() {
 
 	i := querytee.NewInstrumentationServer(cfg.ServerMetricsPort, registry)
 	if err := i.Start(); err != nil {
-		level.Error(util.Logger).Log("msg", "Unable to start instrumentation server", "err", err.Error())
+		level.Error(util_log.Logger).Log("msg", "Unable to start instrumentation server", "err", err.Error())
 		os.Exit(1)
 	}
 
 	// Run the proxy.
-	proxy, err := querytee.NewProxy(cfg.ProxyConfig, util.Logger, cortexReadRoutes(cfg), registry)
+	proxy, err := querytee.NewProxy(cfg.ProxyConfig, util_log.Logger, cortexReadRoutes(cfg), registry)
 	if err != nil {
-		level.Error(util.Logger).Log("msg", "Unable to initialize the proxy", "err", err.Error())
+		level.Error(util_log.Logger).Log("msg", "Unable to initialize the proxy", "err", err.Error())
 		os.Exit(1)
 	}
 
 	if err := proxy.Start(); err != nil {
-		level.Error(util.Logger).Log("msg", "Unable to start the proxy", "err", err.Error())
+		level.Error(util_log.Logger).Log("msg", "Unable to start the proxy", "err", err.Error())
 		os.Exit(1)
 	}
 

cmd/test-exporter/main.go

@@ -11,7 +11,6 @@ import (
 	"github.com/weaveworks/common/tracing"
 
 	"github.com/cortexproject/cortex/pkg/testexporter/correctness"
-	"github.com/cortexproject/cortex/pkg/util"
 	"github.com/cortexproject/cortex/pkg/util/flagext"
 	"github.com/cortexproject/cortex/pkg/util/log"
 )
@@ -28,7 +27,7 @@ func main() {
 	flagext.RegisterFlags(&serverConfig, &runnerConfig)
 	flag.Parse()
 
-	util.InitLogger(&serverConfig)
+	log.InitLogger(&serverConfig)
 
 	// Setting the environment variable JAEGER_AGENT_HOST enables tracing
 	if trace, err := tracing.NewFromEnv("test-exporter"); err != nil {

cmd/thanosconvert/Dockerfile

@@ -0,0 +1,9 @@
+FROM       alpine:3.12
+RUN        apk add --no-cache ca-certificates
+COPY       thanosconvert /
+ENTRYPOINT ["/thanosconvert"]
+
+ARG revision
+LABEL org.opencontainers.image.title="thanosconvert" \
+      org.opencontainers.image.source="https://github.com/cortexproject/cortex/tree/master/tools/thanosconvert" \
+      org.opencontainers.image.revision="${revision}"

cmd/thanosconvert/main.go

@@ -0,0 +1,86 @@
+package main
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"strings"
+
+	"github.com/weaveworks/common/logging"
+	"gopkg.in/yaml.v2"
+
+	"github.com/cortexproject/cortex/pkg/storage/bucket"
+	"github.com/cortexproject/cortex/pkg/util/log"
+	"github.com/cortexproject/cortex/tools/thanosconvert"
+)
+
+func main() {
+	var (
+		configFilename string
+		dryRun         bool
+		cfg            bucket.Config
+	)
+
+	logfmt, loglvl := logging.Format{}, logging.Level{}
+	logfmt.RegisterFlags(flag.CommandLine)
+	loglvl.RegisterFlags(flag.CommandLine)
+	cfg.RegisterFlags(flag.CommandLine)
+	flag.StringVar(&configFilename, "config", "", "Path to bucket config YAML")
+	flag.BoolVar(&dryRun, "dry-run", false, "Don't make changes; only report what needs to be done")
+	flag.Usage = func() {
+		fmt.Fprintf(flag.CommandLine.Output(), "%s is a tool to convert block metadata from Thanos to Cortex.\nPlease see %s for instructions on how to run it.\n\n", os.Args[0], "https://cortexmetrics.io/docs/blocks-storage/migrate-storage-from-thanos-and-prometheus/")
+		fmt.Fprintf(flag.CommandLine.Output(), "Flags:\n")
+		flag.PrintDefaults()
+	}
+	flag.Parse()
+
+	logger, err := log.NewPrometheusLogger(loglvl, logfmt)
+	if err != nil {
+		fatal("failed to create logger: %v", err)
+	}
+
+	if configFilename != "" {
+		buf, err := ioutil.ReadFile(configFilename)
+		if err != nil {
+			fatal("failed to load config file from %s: %v", configFilename, err)
+		}
+		err = yaml.UnmarshalStrict(buf, &cfg)
+		if err != nil {
+			fatal("failed to parse config file: %v", err)
+		}
+	}
+
+	if err := cfg.Validate(); err != nil {
+		fatal("bucket config is invalid: %v", err)
+	}
+
+	ctx := context.Background()
+
+	converter, err := thanosconvert.NewThanosBlockConverter(ctx, cfg, dryRun, logger)
+	if err != nil {
+		fatal("couldn't initilize converter: %v", err)
+	}
+
+	iterCtx := context.Background()
+	results, err := converter.Run(iterCtx)
+
+	fmt.Println("Results:")
+	for user, res := range results {
+		fmt.Printf("User %s:\n", user)
+		fmt.Printf("  Converted %d:\n  %s", len(res.ConvertedBlocks), strings.Join(res.ConvertedBlocks, ","))
+		fmt.Printf("  Unchanged %d:\n  %s", len(res.UnchangedBlocks), strings.Join(res.UnchangedBlocks, ","))
+		fmt.Printf("  Failed %d:\n  %s", len(res.FailedBlocks), strings.Join(res.FailedBlocks, ","))
+	}
+
+	if err != nil {
+		fatal("converter failed: %v", err)
+	}
+
+}
+
+func fatal(msg string, args ...interface{}) {
+	fmt.Fprintf(os.Stderr, msg+"\n", args...)
+	os.Exit(1)
+}

docs/_index.md

@@ -153,9 +153,9 @@ in the [Weave Cloud documentation](https://www.weave.works/docs/cloud/latest/ove
 
 ### Grafana Cloud
 
-To use Cortex as part of Grafana Cloud, sign up for [Grafana Cloud](https://grafana.com/cloud)
-by clicking "Log In" in the top right and then "Sign Up Now".  Cortex is included
-as part of the Starter and Basic Hosted Grafana plans.
+The Cortex project was started by Tom Wilkie (Grafana Labs' VP Product) and Julius Volz (Prometheus' co-founder) in June 2016. Employing 6 out of 8 maintainers for Cortex enables [Grafana Labs](https://grafana.com/) to offer Cortex-as-a-service with exceptional performance and reliability. As the creators of [Grafana](https://grafana.com/oss/grafana/), [Loki](https://grafana.com/oss/loki/), and [Tempo](https://grafana.com/oss/tempo/), Grafana Labs can offer you the most wholistic Observability-as-a-Service stack out there.
+
+For further information see Grafana Cloud [documentation](https://grafana.com/docs/grafana-cloud/), [tutorials](https://grafana.com/tutorials/), [webinars](https://grafana.com/videos/), and [KubeCon talks](https://grafana.com/categories/cortex/). Get started today and [sign up here](https://grafana.com/products/cloud/).
 
 ### Amazon Managed Service for Prometheus (AMP)
 [Amazon Managed Service for Prometheus (AMP)](https://aws.amazon.com/prometheus/) is a Prometheus-compatible monitoring service that makes it easy to monitor containerized applications at scale. It is a highly available, secure, and managed monitoring for your containers. Get started [here](https://console.aws.amazon.com/prometheus/home). To learn more about the AMP, reference our [documentation](https://docs.aws.amazon.com/prometheus/latest/userguide/what-is-Amazon-Managed-Service-Prometheus.html) and [Getting Started with AMP blog](https://aws.amazon.com/blogs/mt/getting-started-amazon-managed-service-for-prometheus/).