Open
Description
- Existing string to decimal cast uses parse_string_to_decimal_native
- parse_string_to_decimal_native does not have support for e-notation
- parse_string_to_decimal_native does rounding at scale, not truncate
- parse_decimal an existing method has e-notation support and use elsewhere like arrow-csv, arrow-json.
- parse_decimal tests with scale 0, fails as shown below
assert_eq!(
parse_decimal::<Decimal128Type>("123.45", 38, 0)?,
123_i128
);
assertion `left == right` failed
Left: 12345
Right: 123
- existing test using parse_decimal native works
assert_eq!(
parse_string_to_decimal_native::<Decimal128Type>("123.45", 0)?,
123_i128
);
//works fine..
- pars_decimal truncates as defualt behavior, parse_string_to_decimal_native does rounding, hence read from json, or csv of a decimal has truncated value but result of a cast operation has rounded value.
assert_eq!(
parse_decimal::<Decimal128Type>("123.4567891", 38, 5)?,
12345679_i128
);
assertion `left == right` failed
left: 12345678
right: 12345679
vs
assert_eq!(
parse_string_to_decimal_native::<Decimal128Type>("123.4567891", 5)?,
12345679_i128
);
// works fine..
This issue was raised as part of investigation to fix this [datafusion issue] (apache/datafusion#10315)
if parse_decimal adds rounding logic and fix the issue with scale=0, parse decimal can be used for string to decimal cast, thereby getting the scientific notation along with it.
parse_decimal internally calls parse_e_notation, so rounding logic needs to be added there too, so that
"12345e-5" with scale 4, yields 0.1235
"1265E-4" with scale 3 yields .127
Describe the solution you'd like
Describe alternatives you've considered
Additional context