Test memory allocations for overwhelmed Tailer.

[jeschkies]

This draft is investigating the memory consumption of the Tailer when the client cannot keep up.

Run tests with

go test -mod=vendor ./pkg/ingester/ -run ^Test_Allocations$ -benchmem -memprofile memprofile.out -cpuprofile cpuprofile.out

View profile with

go tool pprof -source_path=$(pwd) -http :9403 memprofile.out

The Source view is interesting:

github.com/grafana/loki/pkg/ingester.(*tailer).send

/home/karsten/src/loki/pkg/ingester/tailer.go

  Total:           0   198.01MB (flat, cum)  8.17%
    119            .          .           		} 
    120            .          .           		t.dropStream(stream) 
    121            .          .           		return 
    122            .          .           	} 
    123            .          .            
    124            .   198.01MB           	streams := t.processStream(stream, lbs) 
    125            .          .           	if len(streams) == 0 { 
    126            .          .           		return 
    127            .          .           	} 
    128            .          .           	for _, s := range streams { 
    129            .          .           		select { 

Ideally there would be little memory consumed.

[jeschkies]

We would expect to hit this more often.

[jeschkies]

Addressed by https://github.com/grafana/loki/pull/6063/files#diff-54ef13ddfe48f1b98a46994a1d6a11aaad7a9983c26f30e6e40052facbefecc0R132-R136

[dannykopping]

FWIW, I still think this is necessary given #6063 doesn't contain any tests