Skip to content

add rounding logic and scale zero fix in parse_decimal to match parse_string_to_decimal_native behavior #7355

Open
@himadripal

Description

@himadripal
  • Existing string to decimal cast uses parse_string_to_decimal_native
  • parse_string_to_decimal_native does not have support for e-notation
  • parse_string_to_decimal_native does rounding at scale, not truncate
  • parse_decimal an existing method has e-notation support and use elsewhere like arrow-csv, arrow-json.
  • parse_decimal tests with scale 0, fails as shown below
assert_eq!(
            parse_decimal::<Decimal128Type>("123.45", 38, 0)?,
            123_i128
        );

assertion `left == right` failed
Left:  12345
Right: 123
  • existing test using parse_decimal native works
assert_eq!(
           parse_string_to_decimal_native::<Decimal128Type>("123.45", 0)?,
           123_i128
       );
//works fine..
  • pars_decimal truncates as defualt behavior, parse_string_to_decimal_native does rounding, hence read from json, or csv of a decimal has truncated value but result of a cast operation has rounded value.
assert_eq!(
            parse_decimal::<Decimal128Type>("123.4567891", 38, 5)?,
            12345679_i128
        );

assertion `left == right` failed
  left: 12345678
 right: 12345679

vs

assert_eq!(
            parse_string_to_decimal_native::<Decimal128Type>("123.4567891", 5)?,
            12345679_i128
        );

// works fine..

This issue was raised as part of investigation to fix this [datafusion issue] (apache/datafusion#10315)

if parse_decimal adds rounding logic and fix the issue with scale=0, parse decimal can be used for string to decimal cast, thereby getting the scientific notation along with it.

parse_decimal internally calls parse_e_notation, so rounding logic needs to be added there too, so that

 "12345e-5" with scale 4, yields  0.1235
  "1265E-4" with scale 3 yields  .127

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions