Skip to content

Use string to represent BigDecimal response#11716

Merged
Jackie-Jiang merged 1 commit intoapache:masterfrom
Jackie-Jiang:fix_big_decimal_response
Oct 2, 2023
Merged

Use string to represent BigDecimal response#11716
Jackie-Jiang merged 1 commit intoapache:masterfrom
Jackie-Jiang:fix_big_decimal_response

Conversation

@Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented Sep 30, 2023

It is best practice to store BigDecimal as string in JSON to prevent losing precision.
Currently we directly serialize BigDecimal into number (without quote) in the query response, but JsonAsyncHttpPinotClientTransport will parse it into double and lose precision. Instead of fixing the client, fixing the BigDecimal JSON storage is the recommended solution since JSON number is not designed to preserve precision, and a lot of clients might run into the same problem.

Incompatible

In the JSON response, BigDecimal numbers will be stored as string (with double quotes) instead of number (no double quotes)

@Jackie-Jiang
Copy link
Contributor Author

This issue was introduced in #8503 where responses from aggregation are stored as string, responses from other queries are stored as BigDecimal; #11453 changed it to always store as BigDecimal

@codecov-commenter
Copy link

codecov-commenter commented Sep 30, 2023

Codecov Report

Merging #11716 (02f1dc4) into master (82140f1) will increase coverage by 0.01%.
Report is 2 commits behind head on master.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master   #11716      +/-   ##
============================================
+ Coverage     63.08%   63.09%   +0.01%     
- Complexity     1117     1118       +1     
============================================
  Files          2342     2342              
  Lines        125797   125803       +6     
  Branches      19336    19336              
============================================
+ Hits          79358    79379      +21     
+ Misses        40785    40775      -10     
+ Partials       5654     5649       -5     
Flag Coverage Δ
integration <0.01% <0.00%> (ø)
integration1 <0.01% <0.00%> (ø)
integration2 0.00% <0.00%> (ø)
java-11 63.03% <100.00%> (-0.02%) ⬇️
java-17 62.95% <100.00%> (+0.03%) ⬆️
java-20 62.93% <100.00%> (+0.02%) ⬆️
temurin 63.09% <100.00%> (+0.01%) ⬆️
unittests 63.09% <100.00%> (+0.01%) ⬆️
unittests1 67.25% <100.00%> (+0.02%) ⬆️
unittests2 14.44% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...java/org/apache/pinot/common/utils/DataSchema.java 62.09% <100.00%> (+0.13%) ⬆️

... and 15 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Jackie-Jiang Jackie-Jiang force-pushed the fix_big_decimal_response branch 2 times, most recently from b063fa6 to b706b09 Compare September 30, 2023 08:35
@Jackie-Jiang Jackie-Jiang force-pushed the fix_big_decimal_response branch from b706b09 to 02f1dc4 Compare September 30, 2023 08:39
@Jackie-Jiang
Copy link
Contributor Author

Refactored QueryRunnerTestBase result comparison logic to properly handle the ordering

Copy link
Contributor

@walterddr walterddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

}
}

protected static void sortRows(List<Object[]> rows) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might need to expose sortRows and typeCompatibleFuzzyCompare to integration test b/c some tests are failing b/c of the ordering doesn't match between H2 and Pinot

@Jackie-Jiang Jackie-Jiang merged commit 38acdc6 into apache:master Oct 2, 2023
@Jackie-Jiang Jackie-Jiang deleted the fix_big_decimal_response branch October 2, 2023 20:31
@Jackie-Jiang Jackie-Jiang added the incompatible Indicate PR that introduces backward incompatibility label Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix documentation incompatible Indicate PR that introduces backward incompatibility pinot-client refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants