Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure Blob Receiver cannot read non-JSON logs #34997

Open
stuart23 opened this issue Sep 4, 2024 · 2 comments
Open

Azure Blob Receiver cannot read non-JSON logs #34997

stuart23 opened this issue Sep 4, 2024 · 2 comments
Labels
bug Something isn't working needs triage New item requiring triage receiver/azureblob

Comments

@stuart23
Copy link
Contributor

stuart23 commented Sep 4, 2024

Component(s)

receiver/azureblob

What happened?

Description

Receiver fails to unmarshal logs that are not in json format.

Steps to Reproduce

Deploy this collector and subscribe to blobs that are not in json format.

Expected Result

We can still parse these logs, although they will be missing a lot of data.

Actual Result

"failed to unmarshal logs: ReadObjectCB: expect { or n, but found...+125 more"

Collector version

v0.107.0

Environment information

Environment

OS: AKS linux

OpenTelemetry Collector configuration

config:
  extensions:
    health_check: {}

  receivers:
    azureblob:
      connection_string: ${env:STORAGE_CONNECTION_STRING}
      logs:
        container_name: "airflow-logs"
      event_hub:
        endpoint: ${env:EVENTHUB_CONNECTION_STRING}

  processors:
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25

  exporters:
    debug:
      verbosity: detailed

  service:
    pipelines:
      metrics: null
      traces: null
      logs:
        receivers:
        - azureblob
        processors:
        - memory_limiter
        exporters:
        - debug

    extensions:
    - health_check

Log output

The unmarshalling error message does not make it to the logs, although it should

Additional context

No response

@stuart23 stuart23 added bug Something isn't working needs triage New item requiring triage labels Sep 4, 2024
Copy link
Contributor

github-actions bot commented Sep 4, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@hgaol
Copy link
Contributor

hgaol commented Sep 11, 2024

Hi @stuart23 , based on source code, it can only unmarshal logs in plog JSON format, which is expected. What's the format of your raw data?

logs, err := b.logsUnmarshaler.UnmarshalLogs(json)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage New item requiring triage receiver/azureblob
Projects
None yet
Development

No branches or pull requests

2 participants