Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: incorrect handling of json_v2 timestamp_path #10618

Merged
merged 2 commits into from
Feb 15, 2022
Merged

fix: incorrect handling of json_v2 timestamp_path #10618

merged 2 commits into from
Feb 15, 2022

Conversation

mark-rushakoff
Copy link
Contributor

@mark-rushakoff mark-rushakoff commented Feb 9, 2022 •
edited by sspaink
Loading

resolves: #10606

When json_v2 specifies a top-level timestamp_path, and the timestamp is
a string, the code before this change was passing the raw gjson value,
which included the surrounding quotes, resulting in a
challenging-to-decode error:

parsing time "\"2022-02-09T19:29:04Z\"" as "2006-01-02T15:04:05Z07:00": cannot parse "\"2022-02-09T19:29:04Z\"" as "2006"

Now, detect if the value is a string, and if so, pass the inner string
content.

Required for all PRs:

@mark-rushakoff
fix: incorrect handling of json_v2 timestamp_path
5c1f8f9 
When json_v2 specifies a top-level timestamp_path, and the timestamp is
a string, the code before this change was passing the raw gjson value,
which included the surrounding quotes, resulting in a
challenging-to-decode error:

parsing time "\"2022-02-09T19:29:04Z\"" as "2006-01-02T15:04:05Z07:00": cannot parse "\"2022-02-09T19:29:04Z\"" as "2006"

Now, detect if the value is a string, and if so, pass the inner string
content.
@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label Feb 9, 2022
@sspaink
Copy link
Contributor

sspaink commented Feb 10, 2022

Thank you for fixing this!

@sspaink sspaink added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Feb 10, 2022
@sspaink sspaink mentioned this pull request Feb 10, 2022
Closed
Hipska
Hipska suggested changes Feb 11, 2022
View reviewed changes
Copy link
Contributor

@Hipska Hipska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also wondering why the tests for object_timestamp are now succeeding? They seem to be doing very similar thing as what was done for timestamp_path before?

plugins/parsers/json_v2/parser.go Outdated Show resolved Hide resolved
@mark-rushakoff
Use stringified version of timestamp value
6e5031a 
Rather than switching based on the reported type of the gjson.Result,
just pass result.String() to internal.ParseTimestamp.
@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_i386.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz
static_linux_amd64.tar.gz

@sspaink sspaink mentioned this pull request Feb 14, 2022
Open
@Hipska Hipska mentioned this pull request Feb 15, 2022
Closed
@sspaink sspaink merged commit 25e8b45 into master Feb 15, 2022
@sspaink sspaink deleted the fix-jsonv2-timestamp_path-string branch February 15, 2022 15:08
@andrewreid
Copy link

I was directed here after conversation on Slack — this PR fixed a problem I was having parsing a date in format 2022-02-13T04:30:00.0000000Z, where it was basically being ignored silently and time.Now() being subbed-in instead. Thanks @sspaink @Hipska for your assistance.

powersj pushed a commit that referenced this pull request Feb 16, 2022
@mark-rushakoff @powersj
fix: incorrect handling of json_v2 timestamp_path (#10618)
94543b3 
(cherry picked from commit 25e8b45)
phemmer added a commit to phemmer/telegraf that referenced this pull request Feb 18, 2022
@phemmer
Merge remote-tracking branch 'origin/master' into postgres
e87744d 
* origin/master: (211 commits)
  feat: update configs (influxdata#10676)
  feat[elastic output]: add elastic pipeline flags (influxdata#10505)
  Update changelog
  fix: ensure folders do not get loaded more than once (influxdata#10551)
  docs: update VMWare doc links (influxdata#10663)
  fix: prometheusremotewrite wrong timestamp unit (influxdata#10547)
  feat: update configs (influxdata#10662)
  fix: add graylog toml tags (influxdata#10660)
  feat: add socks5 proxy support for kafka output plugin (influxdata#8192)
  docs: override reported OpenSearch version (influxdata#10586)
  feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#10659)
  fix: bump all go.opentelemetry.io dependencies (influxdata#10647)
  feat: collection offset implementation (influxdata#10545)
  chore: update go to 1.17.7 (influxdata#10658)
  fix: check for nil client before closing in amqp (influxdata#10635)
  fix: timestamp change during execution of json_v2 parser. (influxdata#10657)
  fix: bump github.com/signalfx/golib/v3 from 3.3.38 to 3.3.43 (influxdata#10652)
  fix: bump github.com/aliyun/alibaba-cloud-sdk-go (influxdata#10653)
  fix: incorrect handling of json_v2 timestamp_path (influxdata#10618)
  feat: gather additional stats from memcached (influxdata#10641)
  feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#10649)
  fix: Revert deprecation of http_listener_v2 (influxdata#10648)
  fix: bump github.com/denisenkom/go-mssqldb from 0.10.0 to 0.12.0 (influxdata#10503)
  fix: bump github.com/gopcua/opcua from 0.2.3 to 0.3.1 (influxdata#10626)
  fix: use current time as ecs timestamp (influxdata#10636)
  fix: bump github.com/nats-io/nats-server/v2 from 2.6.5 to 2.7.2 (influxdata#10638)
  chore: add -race flag to go tests (influxdata#10629)
  feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#10631)
  fix: license doc outdated causing CI failure (influxdata#10630)
  fix: bump k8s.io/client-go from 0.22.2 to 0.23.3 (influxdata#10589)
  feat: Implemented support for reading raw values, added tests and doc (influxdata#6501)
  fix: Improve parser tests by using go-cmp/cmp (influxdata#10497)
  feat(mongodb): add FsTotalSize and FsUsedSize informations (influxdata#10625)
  docs: update quay docs for auth (influxdata#10612)
  chore: allow downgrade of go version in windows script (influxdata#10614)
  chore: update CI go to 1.17.6 (influxdata#10611)
  feat: update etc/telegraf.conf and etc/telegraf_windows.conf (influxdata#10600)
  fix(inputs.opcua): add more data to error log (influxdata#10465)
  fix: bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.6.0 to 1.13.0 (influxdata#10601)
  refactor: use early return pattern (influxdata#10591)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sspaink sspaink sspaink approved these changes

@Hipska Hipska Hipska approved these changes

Assignees
No one assigned
Labels
fix pr to fix corresponding bug ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

timestamp_format in json_v2 broken in 1.21.3
4 participants
@mark-rushakoff @sspaink @andrewreid @Hipska