[pkg/stanza][receiver/filelog] panic: assignment to entry in nil map when upgrading from v0.71.0 to v0.91.0

[andrzej-stencel]

Component(s)

pkg/stanza, receiver/filelog

What happened?

Description

When using the Filelog receiver with File Storage extension, upgrading the collector from v0.71.0 to v0.91.0 results in a panic.

Steps to Reproduce

1. Make sure the file storage is empty by removing the file receiver_filelog_panic if it exists: rm ./receiver_filelog_panic
2. Run otelcol v0.71.0 with the below configuration, make sure the Filelog receiver reads the logs.log file and writes the metadata to the file storage. Stop the collector.
3. Run otelcol v0.91.0 with the same configuration.

Expected Result

Collector v0.91.0 picks up the data from file storage and continues to watch the logs file.

Actual Result

Collector v0.91.0 crashes with a panic, see below for the logs output.

Collector version

v0.91.0

Environment information

No response

OpenTelemetry Collector configuration

exporters:
  logging:
extensions:
  file_storage:
    directory: .
receivers:
  filelog/panic:
    include:
    - logs.log
    start_at: beginning
    storage: file_storage
service:
  extensions:
  - file_storage
  pipelines:
    logs:
      exporters:
      - logging
      receivers:
      - filelog/panic

Log output

$ ./otc/bin/otelcol-contrib-0.91.0-darwin_arm64 --config 1229-otc-stanza-panic.yaml      
2023-12-29T14:10:08.499+0100    info    service@v0.91.0/telemetry.go:86 Setting up own telemetry...
2023-12-29T14:10:08.499+0100    info    service@v0.91.0/telemetry.go:203        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
2023-12-29T14:10:08.499+0100    info    exporter@v0.91.0/exporter.go:275        Deprecated component. Will be removed in future releases.      {"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-12-29T14:10:08.499+0100    info    service@v0.91.0/service.go:145  Starting otelcol-contrib...     {"Version": "0.91.0", "NumCPU": 10}
2023-12-29T14:10:08.499+0100    info    extensions/extensions.go:34     Starting extensions...
2023-12-29T14:10:08.499+0100    info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "file_storage"}
2023-12-29T14:10:08.499+0100    info    extensions/extensions.go:52     Extension started.      {"kind": "extension", "name": "file_storage"}
2023-12-29T14:10:08.499+0100    info    adapter/receiver.go:45  Starting stanza receiver        {"kind": "receiver", "name": "filelog/panic", "data_type": "logs"}
2023-12-29T14:10:08.499+0100    info    fileconsumer/file.go:64 Resuming from previously known offset(s). 'start_at' setting is not applicable.{"kind": "receiver", "name": "filelog/panic", "data_type": "logs", "component": "fileconsumer"}
2023-12-29T14:10:08.499+0100    info    service@v0.91.0/service.go:171  Everything is ready. Begin running and processing data.
panic: assignment to entry in nil map

goroutine 108 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Factory).NewReaderFromMetadata(0x14002b30110, 0x14002ce2008, 0x14002b3cb10)
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/internal/reader/factory.go:112 +0x648
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).newReader(0x14002b300f0, 0x14002ce2008, 0x14002cd4048)
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:263 +0x38c
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeReaders(0x14002b300f0, {0x14002cd0000, 0x1, 0x0?})
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:234 +0x11c
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume(0x14002b300f0, {0x10cde9998?, 0x14002b11ef0}, {0x14002cd0000, 0x1, 0x1})
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:168 +0x134
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).poll(0x14002b300f0, {0x10cde9998, 0x14002b11ef0})
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:150 +0x2f0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller.func1()
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:118 +0xb0
created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller in goroutine 1
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.91.0/fileconsumer/file.go:106 +0xa8

Additional context

It seems that running v0.81.0 between v0.71.0 and v0.91.0 fixes the issue.

From a quick look, it seems that this could be fixed by the checkpoint package verifying that the reader.Metadata structure is properly initialized before returning it in the Load function: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.91.0/pkg/stanza/fileconsumer/internal/checkpoint/checkpoint.go#L68?

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski
• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[andrzej-stencel]

Here's the smallest change that seems to fix the issue: #30236
@djaglowski please let me know what you think.