Newlines in cloudwatch logs cause trunction

[lsanders]

I've noticed that any cloudwatch log messages with newlines seem to truncate. Googling suggests this is a syslog constraint. I haven't dug into this plugin or the graylog plugin framework, but I'm wondering if it's possible to create the input with something that supports newlines?

Any pointers would be greatly appreciated. I'm also happy to take a crack at this if it's something that would be considered for an enhancement.

[jalogisch]

@lsanders

could you please share some sample messages that you have issues with. Syslog does not include new lines, or at least no RFC. Only a draft contains newlines as valid. (AFAIK).

When you share some of your messages we can look at, maybe you hit a bug.

[danotorrey]

@lsanders I just want to confirm, is this happening when using the AWS Logs input type in Graylog?
Also echoing @jalogisch's request for sample logs. Even a screen shot of the log message within CloudWatch would help lots.

[lsanders]

Hey guys,

I see what's happening now. Yep, we are using AWS Logs and we created a common logger to add some more structure to our messages as well as some serializing objects to JSON. It decorates the log message with tab delimited fields. So, while looking for our JSON objects, it appeared to truncate because I was filtering on one of the other structured fields (e.g. a tenant_id tacked on by our logger). So, from my view it looked truncated but the next line was raw, without the tenant_id field).

Now that I realize that, we could probably have probably ensured each newline was decorated and had the functionality we were looking for. However those would all be separate messages in graylog which is a little harder to follow.

In case anyone else runs into something like this, the workaround we went with was to replace newlines with a token and then use a graylog pipeline processor rule as a graylog decorator to create a formated_message field that looks a lot better.

Thanks for looking into this and sorry for the noise.

[danotorrey]

@lsanders Thanks for the info. I am glad you were able to identify a workaround. Was it purely the presence of the newline characters that caused the issue? Do you know if the message was being split into multiple messages on the CloudWatch side? I ask, because I am curious if there is anything we can fix on the Graylog side to help with this in the future.