Add support for Socket Logging Handler #23128

JozefDropco · 2022-01-24T07:33:07Z

Resolves #23127

geoand · 2022-01-24T08:12:15Z

Hi,

Thanks for this!

I think the changes are more complicated than necessary. Personally I would prefer something like: https://github.com/quarkusio/quarkus/compare/main...geoand:%2323127?expand=1.
Mind updating the PR along those lines?

Furthermore, we will need to have the commits squashed.

JozefDropco · 2022-01-24T08:36:13Z

Hi @geoand
thanks I will update it. give me some time and I will update it

geoand · 2022-01-24T08:42:19Z

No rush

geoand · 2022-01-24T13:36:46Z

core/runtime/src/main/java/io/quarkus/runtime/logging/SocketConfig.java

+     * The log message formatter. json or pattern is supported
+     */
+    @ConfigItem(defaultValue = "pattern")
+    String formatter;


Why is this added vs what SysLogConfig?

This is added as the output where you are sending messages can require formatted output differently than the Pattern. In our case we have logstash which has input tcp and json_lines. We need custom attributes/fields to add to the json formatted log meesage. If you have some other suggestion how to achieve it - I am happy to modify it.

@gsmet WDYT about this? Does this use case warrant doing something different than what our SysLog handling does?

Initially I thought I will use logging-json extension however this extension is limited to console logging. Maybe it would be more beneficial to extend that support instead and keep socket logging clean

Yeah, I think that's a better approach

HI @geoand, @gsmet ,
I have reworked the code completely. Now socket handler in core has just pattern and logging-json was extended to support all handlers not just console as it was before. Now file, console, syslog, socket and also named handlers can log as json. Previous implementation of consoleFormatter was opiniated and if JSON was added as extension basic console but also all named handlers were forced to log as JSON. I kept the original behavior but personally I am not fan of it. With "json=false" you can set console to use PatternFormatter and some named console handlers to use JSON. But you cannot use console as JSON and some named console handler to use Pattern. Its just because implementation should be backward compatible.
If users add logging-json extension

all handlers will start log to JSON (even file,syslog,socket) . Not sure if its ok?

Additional fields can be defined as quarkus.log.socket.json.additional-field.myownfield.value=testingvalue

Keyoverrides was exposed too

I changed integration test of logging-json as previous one was not doing anything.

Thanks a lot @JozefDropco.

I'll have a look soon

I somehow missed this PR but I wonder if the work I did in #25505 isn't somehow related.
I probably need to have another look at it.

geoand · 2022-01-28T08:53:19Z

@JozefDropco we can't remove build items as this PR does as that could break other extensions

JozefDropco · 2022-01-29T17:27:55Z

Hi @geoand ,
ok what do you suggest than? keep original extension as it was and introduce new build item + new extension which will be using the new code?

geoand · 2022-01-30T17:01:33Z

I'll have to find some spare time to look into it and let you know.

Thanks for your work on this and for your patience

geoand · 2022-01-31T07:38:43Z

Lets say this PR didn't exist, what is your end goal here?
I am trying to understand what the reason is for all the changes, as they are certainly not all necessary for implementing #23127 (as my https://github.com/quarkusio/quarkus/compare/main...geoand:%2323127?expand=1).

Can you please state your end goal(s) clearly?

Thanks

JozefDropco · 2022-01-31T12:55:33Z

HI @geoand,
our goal is to be able to log into logstash under TCP with SSL enabled. We need to structure the messages as JSON with some custom fields. Index in logstash is not managed by us.

I am able to log to kibana with logback without any issues but this is true for the application. But not for the server. Quarkus is using JUL (jboss implementation with MDC support). but we wouldnt like to write our application dependeing on server directly. We would like to use SLF4J with logback. When I was trying I was unable to force quarkus to use our impelmentation instead jboss-logmanager in DEV mode. When I build it as uber-jar it was working correctly. Is there a specific reason why it wasnt working?

geoand · 2022-02-02T15:17:02Z

So in order to make things absolutely clear, what you need is support for the Socket logging handler along with JSON support, correct?

If so, then I think we'll need to figure out some changes to allow JSON support in other handlers as well

JozefDropco · 2022-02-06T17:14:30Z

Hi, honestly the ConsoleFormatter build item is quite hardcoded and I don't see any reason why somebody would need to log as json in console unless this output is not sent via filebeat somewhere else. Nevertheless I was able to hack it. In application I use slf4j with logback (with tcp appender). I use Command to start server from main method where I reset handlers of JUL and bind it to slf4j. Now all logs are flowing through logback to logstash. Anyways I am opened to implement even more straightforward solution if you are opened to accept it Kind regards, Jozef

geoand · 2022-02-11T10:02:45Z

I agree with you that currently the json support is very hard coded and we should probably make it possible to have json support across all handlers.

The reason to log to json in the console is because in Kubernetes environments applications usually just log to the console and have the platform ship the logs wherever is necessary.

gastaldi · 2022-02-23T20:45:16Z

core/deployment/src/main/java/io/quarkus/deployment/builditem/LogConsoleFormatBuildItem.java

- * console logging formatting configuration and use the formatter provided instead. If multiple formatters
- * are enabled at run time, a warning message is printed and only one is used.
- */
-public final class LogConsoleFormatBuildItem extends MultiBuildItem {


👎🏻 Removing this build item will break other extensions

gsmet · 2023-02-02T11:19:38Z

I'm going to close this. We made several changes in how we can support JSON in other logging transports (which also led to this PR conflicting a lot) so it's probably a good idea to revisit the issue with a fresh mind and make sure we agree on the approach and on what's needed.

Thanks for the effort anyway. Feel free to ping me to have a discussion about how to pursue this. I will be happy to provide the infrastructure so that the implementation is easier on your side.

Jozef Dropco added 2 commits January 23, 2022 18:53

logging socket

ea82928

logging to socket

9e4910a

This comment has been minimized.

Sign in to view

quarkus-bot bot added the area/dependencies Pull requests that update a dependency file label Jan 24, 2022

JozefDropco changed the title ~~https://github.com/quarkusio/quarkus/issues/23127~~ Extension: Loggging into UDP/TCP Socket Jan 24, 2022

package corrected

8370758

geoand changed the title ~~Extension: Loggging into UDP/TCP Socket~~ Add support for Socket Logging Handler Jan 24, 2022

Jozef Dropco added 2 commits January 24, 2022 14:28

logging into socket quarkusio#23127

dcd3805

Merge remote-tracking branch 'origin/main' into main

2d9bb4d

quarkus-bot bot added area/core area/logging labels Jan 24, 2022

geoand reviewed Jan 24, 2022

View reviewed changes

Jozef Dropco and others added 2 commits January 26, 2022 20:45

json support for all handlers not just for console

c5dd234

Merge branch 'main' into main

e2c68dd

gastaldi requested changes Feb 23, 2022

View reviewed changes

Felk mentioned this pull request Mar 8, 2022

Allow removal of stacktrace newlines in logging #6262

Open

gsmet closed this Feb 2, 2023

quarkus-bot bot added the triage/invalid This doesn't seem right label Feb 2, 2023

troosan mentioned this pull request Sep 12, 2024

Add support for Socket Logging Handler with basic ECS format logging #43232

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Socket Logging Handler #23128

Add support for Socket Logging Handler #23128

JozefDropco commented Jan 24, 2022 •

edited by geoand

Loading

This comment has been minimized.

geoand commented Jan 24, 2022 •

edited

Loading

JozefDropco commented Jan 24, 2022

geoand commented Jan 24, 2022

geoand Jan 24, 2022

JozefDropco Jan 24, 2022 •

edited

Loading

geoand Jan 24, 2022

JozefDropco Jan 24, 2022

geoand Jan 26, 2022

JozefDropco Jan 26, 2022 •

edited

Loading

geoand Jan 27, 2022

gsmet Jul 19, 2022

geoand commented Jan 28, 2022

JozefDropco commented Jan 29, 2022

geoand commented Jan 30, 2022

geoand commented Jan 31, 2022

JozefDropco commented Jan 31, 2022

geoand commented Feb 2, 2022

JozefDropco commented Feb 6, 2022 via email •

edited

Loading

geoand commented Feb 11, 2022

gastaldi Feb 23, 2022

gsmet commented Feb 2, 2023

Add support for Socket Logging Handler #23128

Add support for Socket Logging Handler #23128

Conversation

JozefDropco commented Jan 24, 2022 • edited by geoand Loading

This comment has been minimized.

geoand commented Jan 24, 2022 • edited Loading

JozefDropco commented Jan 24, 2022

geoand commented Jan 24, 2022

geoand Jan 24, 2022

Choose a reason for hiding this comment

JozefDropco Jan 24, 2022 • edited Loading

Choose a reason for hiding this comment

geoand Jan 24, 2022

Choose a reason for hiding this comment

JozefDropco Jan 24, 2022

Choose a reason for hiding this comment

geoand Jan 26, 2022

Choose a reason for hiding this comment

JozefDropco Jan 26, 2022 • edited Loading

Choose a reason for hiding this comment

geoand Jan 27, 2022

Choose a reason for hiding this comment

gsmet Jul 19, 2022

Choose a reason for hiding this comment

geoand commented Jan 28, 2022

JozefDropco commented Jan 29, 2022

geoand commented Jan 30, 2022

geoand commented Jan 31, 2022

JozefDropco commented Jan 31, 2022

geoand commented Feb 2, 2022

JozefDropco commented Feb 6, 2022 via email • edited Loading

geoand commented Feb 11, 2022

gastaldi Feb 23, 2022

Choose a reason for hiding this comment

gsmet commented Feb 2, 2023

JozefDropco commented Jan 24, 2022 •

edited by geoand

Loading

geoand commented Jan 24, 2022 •

edited

Loading

JozefDropco Jan 24, 2022 •

edited

Loading

JozefDropco Jan 26, 2022 •

edited

Loading

JozefDropco commented Feb 6, 2022 via email •

edited

Loading