Some filebeat modules incorrectly parse timestamps

[jsoriano]

There are still some filebeat modules with similar issues to the ones fixed in #13308, probably caused by #12253. Timestamps without timezone are parsed as UTC and then converted to a different timezone, what is incorrect, they should be parsed directly as the final timezone.

#13874 has been created to earlier detect unexpected changes in timestamps when non-UTC timezones are used. We could identify some modules where this seems to be still happening:

• logstash (plain at least, also reported in Fix filebeat system module timezone parsing #13308 (comment), fix in Fix timezone parsing of logstash module ingest pipelines #13890)

• Cisco (asa and ftd) Fix timezone parsing of Cisco module ingest pipelines #13893

• Cisco (ios doesn't have a date parser) Fix timezone parsing of Cisco module ingest pipelines #13893

• iptables Fix timezone parsing in iptables, mssql and panw modules #13926

• mssql Fix timezone parsing in iptables, mssql and panw modules #13926

• panw (also reported in PANW incorrectly parse the timezone to UTC when Timezone in the event #13867) Fix timezone parsing in iptables, mssql and panw modules #13926

• Rabbitmq (Fix timezone parsing in Rabbitmq logs #13879)

• Consider removing event.timezone from events that didn't need it (see Set non-UTC timezone for filebeat modules testing #13874 (review)) Remove event.timezone from events from some json logs #13918

• Add if: "ctx.event.timezone == null" condition to pipelines in date processors where another processor exists with the opposite condition. (Fix conditions for date processors without timezones #13883)

[atoom]

Hi,

I can confirm that timezone conversion for Logstash plain logs is an issue with Filebeat 7.3.2. I checked the generated ingest pipeline and I can resolve the issue by refactoring the date processing to look the same way as the Kafka module. The fix for me was to do the following in the ingest pipeline:

• Do not use the ISO8601 format shorthand but instead use an explicit format string (yyyy-MM-dd'T'HH:mm:ss,SSS)
• Do not chain the two date processor but instead use the logstash.log.timestamp as source field and @timestamp as target field in both processors

Not sure if my fix is the preferred way to do it though :)

/Andreas

[jsoriano]

Hey @atoom, yes, this is the fix we are applying in modules with similar issues, we'll do the same in logstash.