Skip to content

Conversation

edmocosta
Copy link
Contributor

What does this PR do?

Adds the Logstash exporter as an OTel component. This exporter only works with Beats OTel receivers.

Why is it important?

It allows beats receiver to send data (ECS) to Logstash.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Disruptive User Impact

No impact on existing users / configurations.

How to test this PR locally

Use something like the following as otel.yml, and run elastic-agent otel --config otel.yml
This will run the beat receiver, which will produce one event.
The event should be sent to the local Logstash instance you specify in the logstash section.
To run Logstash: bin/logstash -e "input { beats { port => 5044 } } output { stdout { } }"

receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: benchmark
          enabled: true
          count: 1
          message: "The quick brown fox jumped over the lazy dog"
    output:
      otelconsumer:
    logging:
      level: info
      selectors:
        - '*'
    path:
      home: /tmp/<state dir>

exporters:
  logstash:
    hosts: ["localhost:5044"]

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver]
      exporters: [logstash]

Related issues

Questions to ask yourself

  • How are we going to support this in production?
  • How are we going to measure its adoption?
  • How are we going to debug this?
  • What are the metrics I should take care of?
  • ...

Copy link
Contributor

mergify bot commented Sep 24, 2025

This pull request does not have a backport label. Could you fix it @edmocosta? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@edmocosta edmocosta added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team enhancement New feature or request labels Sep 24, 2025
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 24, 2025

💔 Build Failed

Failed CI Steps

History

cc @edmocosta

@edmocosta edmocosta marked this pull request as ready for review September 25, 2025 11:42
@edmocosta edmocosta requested a review from a team as a code owner September 25, 2025 11:42
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@edmocosta edmocosta requested a review from cmacknz September 25, 2025 11:43
Copy link
Contributor

@swiatekm swiatekm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Is the plan for this exporter to always live in the beats codebase, rather than opentelemetry-collector-components?

@edmocosta
Copy link
Contributor Author

LGTM. Is the plan for this exporter to always live in the beats codebase, rather than opentelemetry-collector-components?

Yes, it will live in the beats repository, and there's no short-term plans for moving it into the opentelemetry-collector-components. It was made to only work with beats receivers, and is highly coupled to the beats's codebase.

Copy link

@kaisecheng kaisecheng merged commit ad29588 into elastic:main Sep 25, 2025
23 checks passed
v1v added a commit that referenced this pull request Sep 26, 2025
* upstream: (505 commits)
  Update journald tests now that Filebeat supports watching folders (#10131)
  [deploy/kubernetes]: add info about hostPID for Universal Profiling (#10173)
  Fall back to process runtime if otel runtime is unsupported (#10087)
  Conditionall check for ms_tls13kdf build tag (#10160)
  [docs][edot] add entry for profiles (#10163)
  edot/docs: add support for profiles (#10146)
  Add Logstash exporter (#10137)
  Add back publish to serverless. (#10159)
  Improve Integration test documentation (#10155)
  Fix multiarch service image push from main to serverless (#10129)
  Forward migrate action to endpoint (#9801)
  Comment out check for ms_tls13kdf tag for FIPS-capable binaries (#10148)
  [otel] add receivers: apache, iis, mysql, postgresql, sqlserver v0.135.0 (#9344)
  Add k8sevents receiver in kube-stack (#10086)
  feat: emit system resource metrics for EDOT subprocess (#10003)
  [AutoOps] Configure OTel Exporter to Send Maximum-sized Batches (#10126)
  keep enrollment token when replacing data with signed (#10115)
  Revert "Publish `elastic-agent-service` container directly to serverless from main (#9583)" (#10127)
  Add agent_policy_id and policy_revision_idx to checkin requests (#9931)
  remove resource/k8s processor and use k8sattributes processor for service attributes (#10108)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants