Re-license and export pkg/ingester WAL code to be used in Promtail's WAL

[thepalbi]

What this PR does / why we need it:

In PRs mentioned in #8197 , we are adding WAL support for Promtail. Since under-the-cover structure of the WAL will be much like Loki's, we need to re-use some bits from Loki's code:

• WALRecord and all of it's encode/decode methods exported (https://github.com/grafana/loki/blob/main/pkg/ingester/encoding.go#L35)
• resettingPool, also exporter (

  loki/pkg/ingester/wal.go

  Line 202 in 855d2b7

  func newRecordPool() *resettingPool {

  )

This two bits have the default Loki license applies (AGPL), which is not compatible with Promtail's , ALv2. This PR moves them to a new package, and re-licenses it.

Also, some other bits are needed which due to its package pkg/util , fall under ALv2 so we are ok with them:

• NewWALReader (

  loki/pkg/util/wal/reader.go

  Line 11 in 855d2b7

  func NewWalReader(dir string, startSegment int) (*wlog.Reader, io.Closer, error) {

  )
• These encoding bits, under pkg/util/encoding (https://github.com/grafana/loki/blob/855d2b74bc075341044130303d879a3a23fa2f64/pkg/util/encoding/encoding.go)

They are also shown in this commit in of the related PRs.

Also, since the new package is called wal, the main WALRecord struct has been renamed to wal.Record to avoid repetition in the name itself.

Which issue(s) this PR fixes:
Required for #8197

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[thepalbi]

I'd like to, as part of this PR, rename WALRecord to Record, since due to the new package name, it's imported reference will be wal.Record. Also the linter doesn't seem to like having wal.WALRecord, but we can omit it if not.

pkg/ingester/wal/encoding.go:40:6: exported: type name will be used as wal.WALRecord by other packages, and that stutters; consider calling this Record (revive)
type WALRecord struct {
     ^

What do you think @cstyan ?

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[thepalbi]

I'd like to, as part of this PR, rename WALRecord to Record, since due to the new package name, it's imported reference will be wal.Record. Also the linter doesn't seem to like having wal.WALRecord, but we can omit it if not.

pkg/ingester/wal/encoding.go:40:6: exported: type name will be used as wal.WALRecord by other packages, and that stutters; consider calling this Record (revive)
type WALRecord struct {
     ^

What do you think @cstyan ?

Agreed on yes

[cstyan]

I think we don't need this package global here? it was just used in the actual ingester package, so we can create one with NewRecordPool there if we really need a global

[thepalbi]

Yhea, just found the dependency why I created that in that file:

• It was used in encoding_test, so created one as global there for the tests themselves to use
• Used in Record#Reset. Inverted the dependency, and moved that to the pool itself, since it was the only caller of reset

Also, created one as global in pkg/ingester/wal.go since it's used a bunch there

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[cstyan]

lets just clean up the imports ordering for consistency's sake

after that it looks like you'll need to merge in main or rebase but after that we're good to go IMO

[thepalbi]

@cstyan all import orders should be fixed now 😀

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[thepalbi]

@cstyan this PR should be good to go right?