Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add compressed file collection support to filelog receiver #2328

Closed
djaglowski opened this issue Feb 11, 2021 · 15 comments
Closed

Add compressed file collection support to filelog receiver #2328

djaglowski opened this issue Feb 11, 2021 · 15 comments
Assignees
@bacherfl
Labels
good first issue Good for newcomers help wanted Extra attention is needed receiver/filelog spec:logs Stale
Milestone
Logs Support, Pha...

Comments

@djaglowski
Copy link
Member

No description provided.

@djaglowski djaglowski mentioned this issue Feb 11, 2021
Closed
18 tasks
@tigrannajaryan tigrannajaryan added this to the Basic Logs Support milestone Mar 3, 2021
@droonee
Copy link

droonee commented Mar 24, 2022

Are there any updates around collection of compressed .gz files and processing that data to be readable?

@djaglowski
Copy link
Member Author

This is definitely a feature that needs to be added, but the priority has been on stabilizing the log data model and reworking the internals of this library to reflect the latest data model.

My best guess, barring a contribution from someone else, is that this will be completed in the next quarter.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 8, 2022

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

@github-actions github-actions bot added the Stale label Nov 8, 2022
@SaxyPandaBear
Copy link

Any update on this? Would love to see compressed file log support

@atoulme atoulme added good first issue Good for newcomers and removed Stale labels Jan 25, 2023
@atoulme
Copy link
Contributor

atoulme commented Jan 25, 2023

Removing stale and inviting contributions for this issue. @SaxyPandaBear please feel free to help and send a patch if you'd like.

@github-actions
Copy link
Contributor

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Mar 27, 2023
@dmitryax dmitryax added help wanted Extra attention is needed and removed Stale labels Apr 10, 2023
@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Jun 12, 2023
@github-actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 11, 2023
@atoulme
Copy link
Contributor

atoulme commented May 11, 2024

I find myself needing this again. Reopening.

@atoulme atoulme reopened this May 11, 2024
@bacherfl
Copy link
Contributor

bacherfl commented Jun 4, 2024

If this issue is still available, I would like to work on this. CC @evan-bradley

@djaglowski
Copy link
Member Author

@bacherfl, thank you. I'm curious to see how the implementation could look. Feel free to bring a rough proposal here first if you have a clear idea.

@bacherfl bacherfl mentioned this issue Jun 6, 2024
Merged
@bacherfl
Copy link
Contributor

bacherfl commented Jun 6, 2024

@bacherfl, thank you. I'm curious to see how the implementation could look. Feel free to bring a rough proposal here first if you have a clear idea.

Hi @djaglowski, I was looking into this yesterday and tried out a couple of approaches - I now have a (very rough) proof of concept that seems to do the trick: #33406 - Feel free to have a look and see if this makes sense to you - I'm sure there's a lot that can be improved or done differently so I appreciate any feedback :)

djaglowski added a commit that referenced this issue Jun 14, 2024
@bacherfl @djaglowski
[pkg/stanza] support gzip compressed log files for file log receiver (#…
01efffd 
…33406)

**Description:** This PR adds support for reading gzip compressed log
files for the file log receiver. This is done by, if enabled via the
`gzip_file_suffix` parameter, creating a `gzip.Reader` on top of the
file handle of a compressed file.

**Link to tracking Issue:** #2328

**Testing:** Added unit tests for the new functionality. Manually tested
using the following configuration for the filelog receiver:

```
  filelog:
    include: [ ./simple.log*.gz ]
    start_at: beginning
    gzip_file_suffix: ".gz"
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.sev
```

**Documentation:** Added documentation in the readme of the file log
receiver

---------

Signed-off-by: Florian Bacher <florian.bacher@dynatrace.com>
Co-authored-by: Dan Jaglowski <jaglows3@gmail.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Aug 6, 2024
@djaglowski
Copy link
Member Author

Closing this based on #33406. We may want to support additional compression formats in the future but these can be separate issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bacherfl bacherfl

Labels
good first issue Good for newcomers help wanted Extra attention is needed receiver/filelog spec:logs Stale
Projects
None yet
Milestone
Logs Support, Phase 2
Development

No branches or pull requests
7 participants
@atoulme @bacherfl @tigrannajaryan @djaglowski @dmitryax @SaxyPandaBear @droonee