Skip to content

BigDecimal#div doesn't round correctly #328

Closed
@tompng

Description

@tompng
BigDecimal.mode(BigDecimal::ROUND_MODE, BigDecimal::ROUND_UP)

BigDecimal('1.' + '0' * 40 + '1').div(2, 40)
#=> 0.5000000000000000000000000000000000000001e0 OK
BigDecimal('1.' + '0' * 40 + '1').div(2, 30)
#=> 0.500000000000000000000000000001e0 OK
BigDecimal('1.' + '0' * 40 + '1').div(2, 20)
#=> 0.5e0 (Should be 0.50000000000000000001e0)
BigDecimal('1.' + '0' * 40 + '1').div(2, 10)
#=> 0.5e0 (Should be 0.5000000001e0)
BigDecimal.mode(BigDecimal::ROUND_MODE, BigDecimal::ROUND_HALF_DOWN)

BigDecimal('1.1111').div(2, 4) #=> 0.5555e0 OK
BigDecimal('1.111101').div(2, 4) #=> 0.5556e0 OK
BigDecimal('1.1111' + '0' * 10 + '1').div(2, 4) #=> 0.5556e0 OK
BigDecimal('1.1111' + '0' * 20 + '1').div(2, 4) #=> 0.5555e0 (Should be 0.5556e0)
BigDecimal.mode(BigDecimal::ROUND_MODE, BigDecimal::ROUND_HALF_DOWN)

BigDecimal(5).div(BigDecimal('9' * 4), 4) #=> 0.5001e-3 OK
BigDecimal(5).div(BigDecimal('9' * 5), 5) #=> 0.50001e-4 OK
BigDecimal(5).div(BigDecimal('9' * 6), 6) #=> 0.500001e-5 OK
BigDecimal(5).div(BigDecimal('9' * 7), 7) #=> 0.5e-6 (Should be 0.5000001e-6)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions