Loki 2.4.0 `read` target periodically not returning expected logs.

[slim-bean]

We found an issue reported by the loki-canary spot check where logs were not being returned periodically.

The logs are being received and stored, however for brief periods the spot check does not receive them.

It looks to be caused by:

loki/pkg/loki/modules.go

Lines 370 to 399 in 38ff85f

	if loki_storage.UsingBoltdbShipper(t.Cfg.SchemaConfig.Configs) {
	boltdbShipperMinIngesterQueryStoreDuration := boltdbShipperMinIngesterQueryStoreDuration(t.Cfg)
	switch true {
	case t.Cfg.isModuleEnabled(Querier), t.Cfg.isModuleEnabled(Ruler):
	// Do not use the AsyncStore if the querier is configured with QueryStoreOnly set to true
	if t.Cfg.Querier.QueryStoreOnly {
	break
	}
	// Use AsyncStore to query both ingesters local store and chunk store for store queries.
	// Only queriers should use the AsyncStore, it should never be used in ingesters.
	chunkStore = loki_storage.NewAsyncStore(chunkStore, t.ingesterQuerier,
	calculateAsyncStoreQueryIngestersWithin(t.Cfg.Querier.QueryIngestersWithin, boltdbShipperMinIngesterQueryStoreDuration),
	)
	case t.Cfg.isModuleEnabled(All):
	// We want ingester to also query the store when using boltdb-shipper but only when running with target All.
	// We do not want to use AsyncStore otherwise it would start spiraling around doing queries over and over again to the ingesters and store.
	// ToDo: See if we can avoid doing this when not running loki in clustered mode.
	t.Cfg.Ingester.QueryStore = true
	boltdbShipperConfigIdx := loki_storage.ActivePeriodConfig(t.Cfg.SchemaConfig.Configs)
	if t.Cfg.SchemaConfig.Configs[boltdbShipperConfigIdx].IndexType != shipper.BoltDBShipperType {
	boltdbShipperConfigIdx++
	}
	mlb, err := calculateMaxLookBack(t.Cfg.SchemaConfig.Configs[boltdbShipperConfigIdx], t.Cfg.Ingester.QueryStoreMaxLookBackPeriod,
	boltdbShipperMinIngesterQueryStoreDuration)
	if err != nil {
	return nil, err
	}
	t.Cfg.Ingester.QueryStoreMaxLookBackPeriod = mlb
	}
	}

Where we didn't include a check for the new Read target when setting up the boltdb shipper store.

We are testing a fix, will cut a 2.4.1 tomorrow.