NaN converted to null instead of parse error

[spinningarrow]

Test:

echo '{"x": NaN}' | jq .

Expected:

parse error

Actual:

{
  "x": null
}

If I try to parse the same JSON in Node or in Chrome, I get a JSON syntax error. Shouldn't jq have the same behaviour?

[nicowilliams]

printf 'Nan\ninf\n-inf\n' | jq isnan,.
true
null
false
1.7976931348623157e+308
false
-1.7976931348623157e+308

These are artifacts of jv_strtod(), which we could argue is doing the right thing.

Things we can do:

• leave it as is and call this a jq-specific JSON extension
• fix it in jv_strtod()
• fix it in jv_parse*() but not in jv_strtod(), the idea being that tonumber should handle these forms, but fromjson should not

Comments?

[wtlangford]

As a jq-specific JSON extension, I kinda like it, though if we were going to formally support any JSON extensions, I think allowing a trailing comma in objects/arrays would be wonderful...
Otherwise, I like the third option better than the second. I don't want to remove NaN/inf support from tonumber.

We also want to consider that any changes we make to this will break someone's workflow, unless we give jq strict/nonstrict JSON parser modes and default it to nonstrict.

@spinningarrow is the current behavior a problem for your workflow, or just surprising?

[nicowilliams]

@wtlangford I've been wanting to write a fixjson program that as statelessly as possible applies a bunch of optional rules to JSON-ish-but-not-JSON texts on input and outputs proper JSON on stdout.

I'm not sure that such a tool belongs in jq, but then again, I don't object to having support for a variety of JSON-ish formats in jq, so I guess we could integrate it.

[nicowilliams]

The options I'd like for fixjson:

• allow ident-like keys to not be quoted
• support trailing commas
• support missing commas in various places when lexically obvious (e.g., [0 1 2])
• strip out comments
• anything else you might think of (strings with unsecaped ASCII control chars?!)

[spinningarrow]

@wtlangford I came across this issue when I received some JSON from an API and suspected it was malformed. I then ran it through jq to make sure, and it succeeded. After trying to figure out what other problem could be causing my app to fail, I decided to try to parse the JSON in node. That's when I realised that I can't actually use jq for checking if a JSON-ish response is valid.

So in that sense, yes it is surprising. I expected jq to follow the spec. I'd be okay with the exceptions were documented, though then does "command-line JSON processor" become a little bit misleading? How closely does jq want to follow the spec?

I guess a strict/nonstrict mode might be a solution.

[benoit-intrw]

I encounter the same kind of problem today and didn't know that jq cannot be used to validate JSON.

The case is the following:

$ echo '{"dummy": 000}' | jq .
{
  "abc": 0
}

Expected this:

$ echo '{"abc": 000}' | python -m json.tool
Expecting , delimiter: line 1 column 10 (char 9)

Sorry if this is not directly related to the current issue.

[nicowilliams]

@benoit-intrw It's not related to this issue. Please open a new one.

[benoit-intrw]

Done #1404 sorry for the noise here.

[nicowilliams]

We should probably have a command-line switch for strictness.

[spinningarrow]

I would suggest strictness be the default, since non-strict is probably unexpected. It will be a breaking change though.

[karlb]

Irritatingly, the generated null is not even a real null:

$ echo '{"x": null}' | jq '.x == null'
true
$ echo '{"x": NaN}' | jq '.x == null'
false

[pkoppstein]

The "generated null" only occurs when jq is asked to emit nan as a JSON value. That is, jq knows that it is only permitted to emit valid JSON, or a stream of valid JSON texts, so it emits null instead rather than failing.

By the way, if you want to test whether a value is nan, use isnan:

$ echo '{"x": NaN}' | jq '.x | isnan'
true