Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/filelog] Add docs for offset tracking #30914

Merged
merged 3 commits into from
Feb 1, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions receiver/filelogreceiver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,4 +153,43 @@ The above configuration will read logs from the "simple.log" file. Some examples
2023-06-20 12:50:00 DEBUG This is a test debug message
```

## Fault tolerance, recovery and proper offset tracking
djaglowski marked this conversation as resolved.
Show resolved Hide resolved

File offset tracking is quite crucial to ensure the quality of this receiver.
djaglowski marked this conversation as resolved.
Show resolved Hide resolved
Filelog receiver has no native support for file offset tracking however this can be achieved by using
the proper `extentions`.
djaglowski marked this conversation as resolved.
Show resolved Hide resolved

Filelog receiver's persistence can be covered by the usage of the following:
- [filestorage](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage) extension,
to ensure that Collector's restarts do not affect the log collection and offset tracking.
- [exporterhelper persistent-queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#persistent-queue),
to ensure that Collector's restarts do not affect the delivery of the already collected logs.
djaglowski marked this conversation as resolved.
Show resolved Hide resolved

A complete example is provided bellow:

```yaml
receivers:
filelog:
include: [/var/log/busybox/simple.log]
storage: file_storage/filelogreceiver

extensions:
file_storage/filelogreceiver:
directory: /var/lib/otelcol/file_storage/receiver
file_storage/otlpoutput:
directory: /var/lib/otelcol/file_storage/output

service:
extensions: [file_storage/filelogreceiver, file_storage/otlpoutput]
pipelines:
logs:
receivers: [filelog]
exporters: [otlp/custom]
processors: []

exporters:
otlp/custom:
endpoint: http://0.0.0.0:4242
sending_queue:
storage: file_storage/otlpoutput
```
Loading