Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Unix epoch timestamp support for JSON parser #4633

Merged
merged 5 commits into from
Sep 7, 2018
Merged

Added Unix epoch timestamp support for JSON parser #4633

merged 5 commits into from
Sep 7, 2018

Conversation

drenizg
Copy link
Contributor

@drenizg drenizg commented Sep 4, 2018

closes: #4607

If parsing a Unix epoch timestamp in seconds, e.g. 1536092344.1, json_time_format config must be set to "unix" (case insensitive); corresponding JSON value can have a decimal part and can be a string or a number JSON representation.
If value is in number representation, it'll be treated as a double precision float, and could have some precision loss. If value is in string representation, there'll be no precision loss up to nanosecond precision. Decimal positions beyond that will be dropped.
If parsing a Unix epoch timestamp in milliseconds, e.g. 1536092344100, this config must be set to "unix_ms" (case insensitive); corresponding JSON value must be a (long) integer, and can be in number or string JSON representation.

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

…t and doc updated.

If parsing a Unix epoch timestamp, e.g. 1536092344000, "json_time_format" config must be set to "Unix"; corresponding JSON value must be in milliseconds and unquoted.
@danielnelson
Copy link
Contributor

Thanks for the pr, can you rename the special value "unix" or make it case insensitive to keep style with the rest of the config? It should handle only seconds, not ms, but could support a decimal value like 1536095663.123. Since you won't be able to parse a time in ms you could add a "unix_ms" variation for your use case.

Added "unix_ms" variation to handle epoch in milliseconds.

Both values are now case insensitive.

Docs and tests updated accordingly.
@drenizg
Copy link
Contributor Author

drenizg commented Sep 4, 2018

Hi @danielnelson, thanks for the feedback! I just commited the requested changes.

Copy link
Contributor

@danielnelson danielnelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add support for string fields before we merge? More details in my comments.

plugins/parsers/json/parser.go Outdated Show resolved Hide resolved
plugins/parsers/json/parser.go Outdated Show resolved Hide resolved
… no decimal precision loss.

For decimal unix timestamps coming as numeric JSON fields, there can be precision loss due to `float64` conversion at `JSONFlattener`.

Docs and tests updated accordingly.
@danielnelson
Copy link
Contributor

One last thing, now that we have quite a bit of code for parsing the timestamps I think we should extract it into a new function.

@drenizg
Copy link
Contributor Author

drenizg commented Sep 7, 2018

Agreed. Just commited the change. Now there's a new function called parseUnixTimestamp with all the parsing logic.

@glinton glinton added this to the 1.8.0 milestone Sep 7, 2018
@glinton glinton merged commit cd4c4e7 into influxdata:master Sep 7, 2018
@Thidjs
Copy link

Thidjs commented Sep 7, 2018

I'm breaking my head over this. When I use the "unix" time_format, i get the following error;
parsing time "1536303994" as "unix": cannot parse "1536303994" as "unix"
Why can't it parse this timestamp? And more important how to fix ;-)

@drenizg drenizg deleted the parsers/json/unixEpoch branch September 7, 2018 14:06
@drenizg
Copy link
Contributor Author

drenizg commented Sep 7, 2018

Hi @Thidjs, are you sure you're using a version with this change in it? If using a nightly build, it could be that it was packaged before the change was merged.

@Thidjs
Copy link

Thidjs commented Sep 8, 2018

@drenizg I tested again today with the latest nightly build and it seems to be working now. Thanx!

rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018
otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
dupondje pushed a commit to dupondje/telegraf that referenced this pull request Apr 22, 2019
athoune pushed a commit to bearstech/telegraf that referenced this pull request Apr 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

JSON parser, UNIX time format for timestamp field
4 participants