Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(decoding): Implement chunked GELF decoding #20859

Open
wants to merge 60 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
de3e0c0
feat(gelf): Initial implementation of chunked gelf
jorgehermo9 Jul 4, 2024
c6e3204
feat(gelf): improve error handling when decoding
jorgehermo9 Jul 4, 2024
5a1b49f
feat(gelf): improve error handling when decoding
jorgehermo9 Jul 4, 2024
b85235e
test(gelf): Add more tests for chunked gelf
jorgehermo9 Jul 5, 2024
baccaba
feat(gelf): derive clone for chunked gelf decoder
jorgehermo9 Jul 5, 2024
148e09b
docs(gelf): Improve doc comments
jorgehermo9 Jul 5, 2024
530f243
feat(gelf): improve mutex poisioning handling
jorgehermo9 Jul 5, 2024
2bc10a7
feat(gelf): improve mutex poisioning handling
jorgehermo9 Jul 5, 2024
94b1cf6
style: fix Cargo.toml stlye
jorgehermo9 Jul 5, 2024
d405e6c
style: fix Cargo.toml stlye
jorgehermo9 Jul 5, 2024
9b69264
chore: remove comment
jorgehermo9 Jul 5, 2024
bfc332e
fix(gelf): Fix partial read of buffer
jorgehermo9 Jul 6, 2024
0bc3376
test(gelf): Refactor some tests
jorgehermo9 Jul 9, 2024
7f633b0
style: fix log style
jorgehermo9 Jul 9, 2024
103b933
style: fix log style
jorgehermo9 Jul 9, 2024
6d4f8c7
test(socket): Add chunked gelf tests for socket source
jorgehermo9 Jul 10, 2024
2fda018
docs(gelf): Improve doc comments
jorgehermo9 Jul 10, 2024
0862c10
docs(gelf): Improve doc comments
jorgehermo9 Jul 11, 2024
a5b1121
test(socket): Add chunked gelf tests for unix socket source
jorgehermo9 Jul 11, 2024
a5757c6
chore: remove comments
jorgehermo9 Jul 15, 2024
33bda36
docs: add changelog
jorgehermo9 Jul 15, 2024
aaa9dc6
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 Jul 15, 2024
7cff6a7
chore: Remove comments
jorgehermo9 Jul 15, 2024
119e122
chore: Remove comments
jorgehermo9 Jul 15, 2024
1f2751b
docs(framing): Add chunked gelf framing documentation
jorgehermo9 Jul 16, 2024
6f8a214
fix: clippy lints
jorgehermo9 Jul 16, 2024
d5b753f
docs: Fix documentation grammar
jorgehermo9 Jul 17, 2024
1b7bfe1
fix: clippy lints
jorgehermo9 Jul 22, 2024
770f1e3
fix: Add 'unchunked' to spellcheck list
jorgehermo9 Jul 22, 2024
7a03832
Update lib/codecs/src/decoding/framing/chunked_gelf.rs
jorgehermo9 Jul 27, 2024
49540d5
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 Jul 27, 2024
8c66ea2
docs(chunked gelf): Remove default value for chunked gelf options in …
jorgehermo9 Jul 28, 2024
09fb7bc
feat(decoding): Return concrete error types in chunked gelf
jorgehermo9 Jul 28, 2024
67d9992
test: add assert to matches! macros
jorgehermo9 Aug 4, 2024
e9e6563
Merge with master
jorgehermo9 Aug 4, 2024
2adaeb5
feat: change timeout_millis u64 to timeout_secs f64
jorgehermo9 Aug 4, 2024
049c8b6
style: use tracing macros as the style guides dictates
jorgehermo9 Aug 4, 2024
0b41a8a
test: fix broken test
jorgehermo9 Aug 4, 2024
d050b3e
style: modify tracing logs usage according to style guide
jorgehermo9 Aug 4, 2024
8b74992
test: fix broken udp socket test
jorgehermo9 Aug 4, 2024
8331964
test: fix broken chunked gelf framing test
jorgehermo9 Aug 4, 2024
c70a645
fix: clippy lints
jorgehermo9 Aug 5, 2024
ccdc4c0
Merge with master
jorgehermo9 Sep 1, 2024
12fb1e2
refactor: cleanup some code
jorgehermo9 Sep 1, 2024
e76a750
refactor: cleanup
jorgehermo9 Sep 1, 2024
5c8d38c
docs: update components docs
jorgehermo9 Sep 3, 2024
81b1079
Merge remote-tracking branch 'origin/master' into feature/chunked-gelf
jszwedko Oct 10, 2024
6c66ff0
fix: remove malformed comment
jorgehermo9 Oct 10, 2024
72414d5
fix: default chunked gelf options
jorgehermo9 Oct 10, 2024
b97de92
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 Oct 18, 2024
85edb00
feat: default pending_messages_limit to None
jorgehermo9 Oct 18, 2024
b214fdb
feat: remove skip_serialization for timeout_secs
jorgehermo9 Oct 18, 2024
f1e5996
Update lib/codecs/src/decoding/framing/chunked_gelf.rs
jorgehermo9 Oct 18, 2024
77ec8f5
feat: change duplicated chunk log error to debug
jorgehermo9 Oct 18, 2024
2ab9bf3
test: add shuffle messages test
jorgehermo9 Oct 18, 2024
ef3f0a7
fix: clippy lint
jorgehermo9 Oct 18, 2024
308146b
docs: update components docs
jorgehermo9 Oct 18, 2024
16f8228
docs: improve chunked gelf decoder option documentation
jorgehermo9 Oct 18, 2024
e3f8245
docs: improve chunked gelf decoder option documentation
jorgehermo9 Oct 18, 2024
cecfa8c
docs: update components docs
jorgehermo9 Oct 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ serde_json = { version = "1.0.120", default-features = false, features = ["raw_v
serde = { version = "1.0.204", default-features = false, features = ["alloc", "derive", "rc"] }
toml = { version = "0.8.14", default-features = false, features = ["display", "parse"] }
vrl = { version = "0.16.1", features = ["arbitrary", "cli", "test", "test_framework"] }
tokio = { version = "1.38.0", default-features = false, features = ["full"] }
Copy link
Contributor Author

@jorgehermo9 jorgehermo9 Jul 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to use tokio inside the lib/codecs crate, in order to implement gelf decoding timeouts with tokio tasks, so I added the dependency as a workspace one. If there is any problem with this, we may find another solution


[dependencies]
pin-project.workspace = true
Expand All @@ -160,7 +161,7 @@ loki-logproto = { path = "lib/loki-logproto", optional = true }
async-stream = { version = "0.3.5", default-features = false }
async-trait = { version = "0.1.81", default-features = false }
futures = { version = "0.3.30", default-features = false, features = ["compat", "io-compat"], package = "futures" }
tokio = { version = "1.38.0", default-features = false, features = ["full"] }
tokio.workspace = true
tokio-openssl = { version = "0.6.4", default-features = false }
tokio-stream = { version = "0.1.15", default-features = false, features = ["net", "sync", "time"] }
tokio-util = { version = "0.7", default-features = false, features = ["io", "time"] }
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/20769-chunked_gelf_decoding.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Allows for chunked gelf decoding in message-based sources, such as UDP sockets or unix datagram sockets. Implementation is based on [Graylog's documentation](https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html#GELFviaUDP).

This framing method can be configured via the `framing.method = "chunked_gelf"` option in the source configuration.

authors: jorgehermo9
2 changes: 2 additions & 0 deletions lib/codecs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ smallvec = { version = "1", default-features = false, features = ["union"] }
snafu = { version = "0.7.5", default-features = false, features = ["futures"] }
syslog_loose = { version = "0.21", default-features = false, optional = true }
tokio-util = { version = "0.7", default-features = false, features = ["codec"] }
tokio.workspace = true
tracing = { version = "0.1", default-features = false }
vrl.workspace = true
vector-common = { path = "../vector-common", default-features = false }
Expand All @@ -45,6 +46,7 @@ similar-asserts = "1.5.0"
vector-core = { path = "../vector-core", default-features = false, features = ["vrl", "test"] }
uuid = { version = "1", default-features = false, features = ["serde", "v4"] }
rstest = "0.21.0"
tracing-test = "0.2.5"
jorgehermo9 marked this conversation as resolved.
Show resolved Hide resolved
vrl.workspace = true

[features]
Expand Down
Loading
Loading