Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Strict mode for JSON parsing, contributed by @marten-voorberg. #2437

Merged
merged 6 commits into from
Jul 30, 2023
Merged

Strict mode for JSON parsing, contributed by @marten-voorberg. #2437

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9ce668e
Strict mode for JSON parsing (#2323)
marten-voorberg May 30, 2023
d2795b9
Merge branch 'master' into strictness2.
eamonnmcmanus May 31, 2023
2a2b640
Strictness follow-up (#2408)
Marcono1234 Jun 27, 2023
f00f357
Merge with `main`.
eamonnmcmanus Jul 17, 2023
077cbce
`strictness2` fixes & improvements (#2456)
Marcono1234 Jul 30, 2023
da61221
Merge branch 'main' into strictness2
eamonnmcmanus Jul 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Troubleshooting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ For example, let's assume you want to deserialize the following JSON data:
}
```

This will fail with an exception similar to this one: `MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 5 column 4 path $.languages[2]`
This will fail with an exception similar to this one: `MalformedJsonException: Use JsonReader.setStrictness(Strictness.LENIENT) to accept malformed JSON at line 5 column 4 path $.languages[2]`
The problem here is the trailing comma (`,`) after `"French"`, trailing commas are not allowed by the JSON specification. The location information "line 5 column 4" points to the `]` in the JSON data (with some slight inaccuracies) because Gson expected another value after `,` instead of the closing `]`. The JSONPath `$.languages[2]` in the exception message also points there: `$.` refers to the root object, `languages` refers to its member of that name and `[2]` refers to the (missing) third value in the JSON array value of that member (numbering starts at 0, so it is `[2]` instead of `[3]`).
The proper solution here is to fix the malformed JSON data.

Expand All @@ -149,7 +149,7 @@ To spot syntax errors in the JSON data easily you can open it in an editor with

**Solution:** See [`Gson` class documentation](https://www.javadoc.io/doc/com.google.code.gson/gson/latest/com.google.gson/com/google/gson/Gson.html#default-lenient) section "Lenient JSON handling"
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved

Note: Even in non-lenient mode Gson deviates slightly from the JSON specification, see [`JsonReader.setLenient`](https://www.javadoc.io/doc/com.google.code.gson/gson/latest/com.google.gson/com/google/gson/stream/JsonReader.html#setLenient(boolean)) for more details.
Note: Even in non-lenient mode Gson deviates slightly from the JSON specification, see [`JsonReader.setStrictness`](https://www.javadoc.io/doc/com.google.code.gson/gson/latest/com.google.gson/com/google/gson/stream/JsonReader.html#setStrictness(Strictness)) for more details.
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved

## <a id="unexpected-json-structure"></a> `IllegalStateException`: "Expected ... but was ..."

Expand Down
123 changes: 87 additions & 36 deletions gson/src/main/java/com/google/gson/Gson.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,14 @@
* <p>See the <a href="https://github.com/google/gson/blob/main/UserGuide.md">Gson User Guide</a>
* for a more complete set of examples.</p>
*
* <h2 id="default-lenient">Lenient JSON handling</h2>
* <h2 id="default-lenient">JSON Strictness handling</h2>
* For legacy reasons most of the {@code Gson} methods allow JSON data which does not
* comply with the JSON specification, regardless of whether {@link GsonBuilder#setLenient()}
* is used or not. If this behavior is not desired, the following workarounds can be used:
* comply with the JSON specification when no explicit {@linkplain Strictness strictness} is set (the default).
* To specify the strictness of a {@code Gson} instance, you should set it through
* {@link GsonBuilder#setStrictness(Strictness)}.
*
* <p>For older Gson versions, which don't have the strictness mode API, the following
* workarounds can be used:
*
* <h3>Serialization</h3>
* <ol>
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -140,7 +144,8 @@
*/
public final class Gson {
static final boolean DEFAULT_JSON_NON_EXECUTABLE = false;
static final boolean DEFAULT_LENIENT = false;
// Strictness of `null` is the legacy mode where some Gson APIs are always lenient
static final Strictness DEFAULT_STRICTNESS = null;
static final FormattingStyle DEFAULT_FORMATTING_STYLE = FormattingStyle.COMPACT;
static final boolean DEFAULT_ESCAPE_HTML = true;
static final boolean DEFAULT_SERIALIZE_NULLS = false;
Expand Down Expand Up @@ -184,7 +189,7 @@ public final class Gson {
final boolean generateNonExecutableJson;
final boolean htmlSafe;
final FormattingStyle formattingStyle;
final boolean lenient;
final Strictness strictness;
final boolean serializeSpecialFloatingPointValues;
final boolean useJdkUnsafe;
final String datePattern;
Expand Down Expand Up @@ -231,13 +236,15 @@ public final class Gson {
* <li>By default, Gson excludes <code>transient</code> or <code>static</code> fields from
* consideration for serialization and deserialization. You can change this behavior through
* {@link GsonBuilder#excludeFieldsWithModifiers(int...)}.</li>
* <li>No explicit strictness is set. You can change this by calling
* {@link GsonBuilder#setStrictness(Strictness)}.</li>
* </ul>
*/
public Gson() {
this(Excluder.DEFAULT, DEFAULT_FIELD_NAMING_STRATEGY,
Collections.<Type, InstanceCreator<?>>emptyMap(), DEFAULT_SERIALIZE_NULLS,
DEFAULT_COMPLEX_MAP_KEYS, DEFAULT_JSON_NON_EXECUTABLE, DEFAULT_ESCAPE_HTML,
DEFAULT_FORMATTING_STYLE, DEFAULT_LENIENT, DEFAULT_SPECIALIZE_FLOAT_VALUES,
DEFAULT_FORMATTING_STYLE, DEFAULT_STRICTNESS, DEFAULT_SPECIALIZE_FLOAT_VALUES,
DEFAULT_USE_JDK_UNSAFE,
LongSerializationPolicy.DEFAULT, DEFAULT_DATE_PATTERN, DateFormat.DEFAULT, DateFormat.DEFAULT,
Collections.<TypeAdapterFactory>emptyList(), Collections.<TypeAdapterFactory>emptyList(),
Expand All @@ -248,7 +255,7 @@ public Gson() {
Gson(Excluder excluder, FieldNamingStrategy fieldNamingStrategy,
Map<Type, InstanceCreator<?>> instanceCreators, boolean serializeNulls,
boolean complexMapKeySerialization, boolean generateNonExecutableGson, boolean htmlSafe,
FormattingStyle formattingStyle, boolean lenient, boolean serializeSpecialFloatingPointValues,
FormattingStyle formattingStyle, Strictness strictness, boolean serializeSpecialFloatingPointValues,
boolean useJdkUnsafe,
LongSerializationPolicy longSerializationPolicy, String datePattern, int dateStyle,
int timeStyle, List<TypeAdapterFactory> builderFactories,
Expand All @@ -265,7 +272,7 @@ public Gson() {
this.generateNonExecutableJson = generateNonExecutableGson;
this.htmlSafe = htmlSafe;
this.formattingStyle = formattingStyle;
this.lenient = lenient;
this.strictness = strictness;
this.serializeSpecialFloatingPointValues = serializeSpecialFloatingPointValues;
this.useJdkUnsafe = useJdkUnsafe;
this.longSerializationPolicy = longSerializationPolicy;
Expand Down Expand Up @@ -802,7 +809,7 @@ public void toJson(Object src, Appendable writer) throws JsonIOException {
* <pre>
* Type typeOfSrc = new TypeToken&lt;Collection&lt;Foo&gt;&gt;(){}.getType();
* </pre>
* @param writer Writer to which the JSON representation of src needs to be written.
* @param writer Writer to which the JSON representation of src needs to be written
* @throws JsonIOException if there was a problem writing to the writer
* @since 1.2
*
Expand All @@ -822,24 +829,38 @@ public void toJson(Object src, Type typeOfSrc, Appendable writer) throws JsonIOE
* Writes the JSON representation of {@code src} of type {@code typeOfSrc} to
* {@code writer}.
*
* <p>The JSON data is written in {@linkplain JsonWriter#setLenient(boolean) lenient mode},
* regardless of the lenient mode setting of the provided writer. The lenient mode setting
* of the writer is restored once this method returns.
* <p>If the {@code Gson} instance has an {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness setting},
* this setting will be used for writing the JSON regardless of the {@linkplain JsonWriter#getStrictness() strictness}
* of the provided {@link JsonWriter}. For legacy reasons, if the {@code Gson} instance has no explicit strictness setting
* and the writer does not have the strictness {@link Strictness#STRICT}, the JSON will be written in {@link Strictness#LENIENT}
* mode.<br>
* Note that in all cases the old strictness setting of the writer will be restored when this method returns.
*
* <p>The 'HTML-safe' and 'serialize {@code null}' settings of this {@code Gson} instance
* (configured by the {@link GsonBuilder}) are applied, and the original settings of the
* writer are restored once this method returns.
*
* @param src the object for which JSON representation is to be created
* @param typeOfSrc the type of the object to be written
* @param writer Writer to which the JSON representation of src needs to be written
*
* @throws JsonIOException if there was a problem writing to the writer
*/
public void toJson(Object src, Type typeOfSrc, JsonWriter writer) throws JsonIOException {
@SuppressWarnings("unchecked")
TypeAdapter<Object> adapter = (TypeAdapter<Object>) getAdapter(TypeToken.get(typeOfSrc));
boolean oldLenient = writer.isLenient();
writer.setLenient(true);

Strictness oldStrictness = writer.getStrictness();
if (this.strictness != null) {
writer.setStrictness(this.strictness);
} else if (writer.getStrictness() != Strictness.STRICT) {
writer.setStrictness(Strictness.LENIENT);
}

boolean oldHtmlSafe = writer.isHtmlSafe();
writer.setHtmlSafe(htmlSafe);
boolean oldSerializeNulls = writer.getSerializeNulls();

writer.setHtmlSafe(htmlSafe);
writer.setSerializeNulls(serializeNulls);
try {
adapter.write(writer, src);
Expand All @@ -848,7 +869,7 @@ public void toJson(Object src, Type typeOfSrc, JsonWriter writer) throws JsonIOE
} catch (AssertionError e) {
throw new AssertionError("AssertionError (GSON " + GsonBuildConfig.VERSION + "): " + e.getMessage(), e);
} finally {
writer.setLenient(oldLenient);
writer.setStrictness(oldStrictness);
writer.setHtmlSafe(oldHtmlSafe);
writer.setSerializeNulls(oldSerializeNulls);
}
Expand Down Expand Up @@ -892,7 +913,10 @@ public void toJson(JsonElement jsonElement, Appendable writer) throws JsonIOExce
* <li>{@link GsonBuilder#disableHtmlEscaping()}</li>
* <li>{@link GsonBuilder#generateNonExecutableJson()}</li>
* <li>{@link GsonBuilder#serializeNulls()}</li>
* <li>{@link GsonBuilder#setLenient()}</li>
* <li>{@link GsonBuilder#setStrictness(Strictness)}. If no
* {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness has been set} the created
* writer will have a strictness of {@link Strictness#LEGACY_STRICT}. Otherwise, this strictness of
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved
* the {@code Gson} instance will be used for the created writer.</li>
* <li>{@link GsonBuilder#setPrettyPrinting()}</li>
* <li>{@link GsonBuilder#setFormattingStyle(FormattingStyle)}</li>
* </ul>
Expand All @@ -904,7 +928,7 @@ public JsonWriter newJsonWriter(Writer writer) throws IOException {
JsonWriter jsonWriter = new JsonWriter(writer);
jsonWriter.setFormattingStyle(formattingStyle);
jsonWriter.setHtmlSafe(htmlSafe);
jsonWriter.setLenient(lenient);
jsonWriter.setStrictness(strictness == null ? Strictness.LEGACY_STRICT : strictness);
jsonWriter.setSerializeNulls(serializeNulls);
return jsonWriter;
}
Expand All @@ -914,43 +938,58 @@ public JsonWriter newJsonWriter(Writer writer) throws IOException {
*
* <p>The following settings are considered:
* <ul>
* <li>{@link GsonBuilder#setLenient()}</li>
* <li>{@link GsonBuilder#setStrictness(Strictness)}. If no
* {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness has been set} the created
* reader will have a strictness of {@link Strictness#LEGACY_STRICT}. Otherwise, this strictness of
eamonnmcmanus marked this conversation as resolved.
Show resolved Hide resolved
* the {@code Gson} instance will be used for the created reader.</li>
* </ul>
*/
public JsonReader newJsonReader(Reader reader) {
JsonReader jsonReader = new JsonReader(reader);
jsonReader.setLenient(lenient);
jsonReader.setStrictness(strictness == null ? Strictness.LEGACY_STRICT : strictness);
return jsonReader;
}

/**
* Writes the JSON for {@code jsonElement} to {@code writer}.
*
* <p>The JSON data is written in {@linkplain JsonWriter#setLenient(boolean) lenient mode},
* regardless of the lenient mode setting of the provided writer. The lenient mode setting
* of the writer is restored once this method returns.
* <p>If the {@code Gson} instance has an {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness setting},
* this setting will be used for writing the JSON regardless of the {@linkplain JsonWriter#getStrictness() strictness}
* of the provided {@link JsonWriter}. For legacy reasons, if the {@code Gson} instance has no explicit strictness setting
* and the writer does not have the strictness {@link Strictness#STRICT}, the JSON will be written in {@link Strictness#LENIENT}
* mode.<br>
* Note that in all cases the old strictness setting of the writer will be restored when this method returns.
*
* <p>The 'HTML-safe' and 'serialize {@code null}' settings of this {@code Gson} instance
* (configured by the {@link GsonBuilder}) are applied, and the original settings of the
* writer are restored once this method returns.
*
* @param jsonElement the JSON element to be written
* @param writer the JSON writer to which the provided element will be written
* @throws JsonIOException if there was a problem writing to the writer
*/
public void toJson(JsonElement jsonElement, JsonWriter writer) throws JsonIOException {
boolean oldLenient = writer.isLenient();
writer.setLenient(true);
Strictness oldStrictness = writer.getStrictness();
boolean oldHtmlSafe = writer.isHtmlSafe();
writer.setHtmlSafe(htmlSafe);
boolean oldSerializeNulls = writer.getSerializeNulls();

writer.setHtmlSafe(htmlSafe);
writer.setSerializeNulls(serializeNulls);

if (this.strictness != null) {
writer.setStrictness(this.strictness);
} else if (writer.getStrictness() != Strictness.STRICT) {
writer.setStrictness(Strictness.LENIENT);
}

try {
Streams.write(jsonElement, writer);
} catch (IOException e) {
throw new JsonIOException(e);
} catch (AssertionError e) {
throw new AssertionError("AssertionError (GSON " + GsonBuildConfig.VERSION + "): " + e.getMessage(), e);
} finally {
writer.setLenient(oldLenient);
writer.setStrictness(oldStrictness);
writer.setHtmlSafe(oldHtmlSafe);
writer.setSerializeNulls(oldSerializeNulls);
}
Expand Down Expand Up @@ -1169,9 +1208,12 @@ private static void assertFullConsumption(Object obj, JsonReader reader) {
* <p>Unlike the other {@code fromJson} methods, no exception is thrown if the JSON data has
* multiple top-level JSON elements, or if there is trailing data.
*
* <p>The JSON data is parsed in {@linkplain JsonReader#setLenient(boolean) lenient mode},
* regardless of the lenient mode setting of the provided reader. The lenient mode setting
* of the reader is restored once this method returns.
* <p>If the {@code Gson} instance has an {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness setting},
* this setting will be used for reading the JSON regardless of the {@linkplain JsonReader#getStrictness() strictness}
* of the provided {@link JsonReader}. For legacy reasons, if the {@code Gson} instance has no explicit strictness setting
* and the reader does not have the strictness {@link Strictness#STRICT}, the JSON will be written in {@link Strictness#LENIENT}
* mode.<br>
* Note that in all cases the old strictness setting of the reader will be restored when this method returns.
*
* @param <T> the type of the desired object
* @param reader the reader whose next JSON value should be deserialized
Expand All @@ -1198,9 +1240,12 @@ public <T> T fromJson(JsonReader reader, Type typeOfT) throws JsonIOException, J
* <p>Unlike the other {@code fromJson} methods, no exception is thrown if the JSON data has
* multiple top-level JSON elements, or if there is trailing data.
*
* <p>The JSON data is parsed in {@linkplain JsonReader#setLenient(boolean) lenient mode},
* regardless of the lenient mode setting of the provided reader. The lenient mode setting
* of the reader is restored once this method returns.
* <p>If the {@code Gson} instance has an {@linkplain GsonBuilder#setStrictness(Strictness) explicit strictness setting},
* this setting will be used for reading the JSON regardless of the {@linkplain JsonReader#getStrictness() strictness}
* of the provided {@link JsonReader}. For legacy reasons, if the {@code Gson} instance has no explicit strictness setting
* and the reader does not have the strictness {@link Strictness#STRICT}, the JSON will be written in {@link Strictness#LENIENT}
* mode.<br>
* Note that in all cases the old strictness setting of the reader will be restored when this method returns.
*
* @param <T> the type of the desired object
* @param reader the reader whose next JSON value should be deserialized
Expand All @@ -1220,8 +1265,14 @@ public <T> T fromJson(JsonReader reader, Type typeOfT) throws JsonIOException, J
*/
public <T> T fromJson(JsonReader reader, TypeToken<T> typeOfT) throws JsonIOException, JsonSyntaxException {
boolean isEmpty = true;
boolean oldLenient = reader.isLenient();
reader.setLenient(true);
Strictness oldStrictness = reader.getStrictness();

if (this.strictness != null) {
reader.setStrictness(this.strictness);
} else if (reader.getStrictness() != Strictness.STRICT) {
reader.setStrictness(Strictness.LENIENT);
}

try {
JsonToken unused = reader.peek();
isEmpty = false;
Expand All @@ -1244,7 +1295,7 @@ public <T> T fromJson(JsonReader reader, TypeToken<T> typeOfT) throws JsonIOExce
} catch (AssertionError e) {
throw new AssertionError("AssertionError (GSON " + GsonBuildConfig.VERSION + "): " + e.getMessage(), e);
} finally {
reader.setLenient(oldLenient);
reader.setStrictness(oldStrictness);
}
}

Expand Down
Loading