panic: unaligned 64-bit atomic operation on 32-bit arm host

[abferm]

Describe the bug
Panic from ingester on 32-bit arm hosts due to misaligned 64-bit atomic operation.

To Reproduce
Steps to reproduce the behavior:

1. Started grafana/loki:2.6.1-arm docker image
2. Started feeding logs with promtail

Expected behavior
No panic

Environment:

• Infrastructure: balena OS on 32-bit arm device
• Deployment tool: balena

Screenshots, Promtail config, or terminal output

 panic: unaligned 64-bit atomic operation
 
 goroutine 507 [running]:
 runtime/internal/atomic.panicUnaligned()
  /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
 runtime/internal/atomic.Xadd64(0x4cc88cc, 0x1)
  /usr/local/go/src/runtime/internal/atomic/atomic_arm.s:256 +0x14
 github.com/grafana/loki/pkg/logqlmodel/stats.(*Context).AddHeadChunkLines(...)
  /src/loki/pkg/logqlmodel/stats/context.go:225
 github.com/grafana/loki/pkg/chunkenc.(*unorderedHeadBlock).forEntries.func1(0x44a7dd0)
  /src/loki/pkg/chunkenc/unordered.go:169 +0x54
 github.com/grafana/loki/pkg/chunkenc.(*unorderedHeadBlock).forEntries(0x48b37c0, {0x1f85088, 0x4446400}, 0x0, 0x0, 0x7fffffffffffffff, 0x4b15ccc)
  /src/loki/pkg/chunkenc/unordered.go:194 +0x2a8
 github.com/grafana/loki/pkg/chunkenc.(*unorderedHeadBlock).Serialise(0x48b37c0, {0x76c918a8, 0x2e26858})
  /src/loki/pkg/chunkenc/unordered.go:335 +0x184
 github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut(0x4904810)
  /src/loki/pkg/chunkenc/memchunk.go:744 +0x130
 github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Close(0x4904810)
  /src/loki/pkg/chunkenc/memchunk.go:692 +0x1c
 github.com/grafana/loki/pkg/ingester.(*Ingester).closeChunk(0x481f180, 0x49e0200, {0x1f7f080, 0x4b84b24})
  /src/loki/pkg/ingester/flush.go:340 +0x74
 github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks(0x481f180, {0x1f850a8, 0x49e02c0}, 0x1e84eee75f66f89d, {0x48b36c0, 0x2, 0x2}, {0x449fe58, 0x1, 0x2}, ...)
  /src/loki/pkg/ingester/flush.go:290 +0x31c
 github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries(0x481f180, {0x4ad6328, 0x4}, 0x1e84eee75f66f89d, 0x1)
  /src/loki/pkg/ingester/flush.go:172 +0x1c8
 github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop(0x481f180, 0x1d)
  /src/loki/pkg/ingester/flush.go:144 +0x2d4
 created by github.com/grafana/loki/pkg/ingester.(*Ingester).InitFlushQueues
  /src/loki/pkg/ingester/flush.go:47 +0x4c

[frepkovsky]

I think it is the same problem as discussed here #5388 but it has already been closed without solution...

[abferm]

This does appear to be the same issue. I thought I searched the closed issues, but obviously I didn't look hard enough.

[gtaws]

as commented on the linked issue, can someone PLEASE REMOVE loki-linux-arm.zip, as it's obviously broken now with no resolution.

[con-cis]

for me image: grafana/loki:2.4.1 with image: grafana/grafana-oss:9.5.2 is the last working config on linux/arm/v7. Hope this is getting fixed as long as there is no grafana alloy build for Raspberry Pi

[jizhuozhi]

I just dealt with a similar problem and found out during my search that Loki also has a similar problem, which comes from field misalignment (struct generated from proto file). The reason is that a bool field suddenly appears among several int64 fields, which will be aligned to 64 bits on 64-bit system, but aligned to 32 bits on 32-bit architecture, causing all int64 fields in the chunk to start at 32 bits.

type Store struct {
	// The total of chunk reference fetched from index.
	TotalChunksRef int64 `protobuf:"varint,1,opt,name=totalChunksRef,proto3" json:"totalChunksRef"`
	// Total number of chunks fetched.
	TotalChunksDownloaded int64 `protobuf:"varint,2,opt,name=totalChunksDownloaded,proto3" json:"totalChunksDownloaded"`
	// Time spent fetching chunks in nanoseconds.
	ChunksDownloadTime int64 `protobuf:"varint,3,opt,name=chunksDownloadTime,proto3" json:"chunksDownloadTime"`
	// Whether the query referenced structured metadata
	QueryReferencedStructured bool  `protobuf:"varint,13,opt,name=queryReferencedStructured,proto3" json:"queryReferencedStructuredMetadata"`
	Chunk                     Chunk `protobuf:"bytes,4,opt,name=chunk,proto3" json:"chunk"`
	// Time spent fetching chunk refs from index.
	ChunkRefsFetchTime int64 `protobuf:"varint,5,opt,name=chunkRefsFetchTime,proto3" json:"chunkRefsFetchTime"`
	// Time spent being blocked on congestion control.
	CongestionControlLatency int64 `protobuf:"varint,6,opt,name=congestionControlLatency,proto3" json:"congestionControlLatency"`
	// Total number of lines filtered by pipeline wrapper.
	PipelineWrapperFilteredLines int64 `protobuf:"varint,7,opt,name=pipelineWrapperFilteredLines,proto3" json:"pipelineWrapperFilteredLines"`
}

It seems an unexpected unordered field QueryReferencedStructured, reorder it in proto file may resolve this problem