proposal: math/decimal: add decimal128 implementation

[rin01]

Preamble

In financial computation, base-10 computation is mandatory.
Discussion can be found in the proposal proposal: math/big: Decimal #12127 for big.Decimal.

A numeric type in the standard library usable for financial and commercial applications would be a big plus.

Existing solutions

For the moment, there exist 14 package that implement base-10 numbers.
https://godoc.org/?q=decimal

They all seem experimental, except https://github.com/shopspring/decimal which is used in production.

Its Decimal type is defined as:

type Decimal struct {
    value *big.Int
    exp int32
}

So, it is a arbitrary-precision type. But this proposal is about fixed-size decimal type.

Need to retrieve currency values stored as decimal in SQL databases

Base-10 numbers are primarily used for financial computation, for dealing with currency values.

These values are already stored as base-10 fixed-point datatypes (NUMERIC, DECIMAL, MONEY) in SQL databases (MySQL, MS SQL Server, Oracle, etc), but working with them in Go is not so easy.

Even if a big.Decimal type would eventually be added, the API of package math/big is more complex that what is needed for most financial or commercial applications.

A simpler data type, fixed-size 128bits floating-point decimal would be an optimal trade-off.

In the field of decimal data type, the work of Mike Cowlishaw is paramount:
https://en.wikipedia.org/wiki/Mike_Cowlishaw
http://speleotrove.com/decimal/

Note that the Java implementation of java.math.BigDecimal is based on the work of M. Cowlishaw:
http://www.drdobbs.com/jvm/fixed-floating-and-exact-computation-wit/184405721

Financial and commercial applications' requirements

The usual range for monetary values can be estimated as follows.
Assets under management for a very large multinational insurance company is:

$265,507,000,000

This figure is the sum of amounts that can have 8 digits after decimal point, because this precision is needed when working with foreign currency conversion.

This means that for a large company, a decimal data type must cope with figures like:

100,000,000,000.00000000, that is, 20 significant digits.

A decimal128 fixed-size data type provides 34 significant digits, which can easily store these figures.
We can even store USA national debt of $18,000,000,000,000 in it with full precision.

When doing financial computation, intermediate calculation can require some more digits, and 34 significant digits of decimal128 can easily provide them for most real cases of financial computation.

Smaller datatype like decimal 64bits only has 16 significant digits, which is not enough for financial computation.
It is always possible to save them with the proper precision in SQL database tables, to spare storage, but to make arithmetic on them, it is safer to always use a decimal 128bits.

Package for prototyping

So far, there exists no implementation of fixed-size decimal type, so I wrote this package to experiment with it.
It is a thin wrapper around the decNumber package of Mike Cowlishaw written in C.
Only the decQuad type has been implemented, as it is a 128bits fixed-size data type.

decnum.Quad is a floating-point base-10 number with 34 significant digits, and size is 128 bits.
Working with them is almost as easy as working with float64:
- decnum.Quad is a value, and not a pointer.
- all arithmetic is done on values, not pointers.
- computation errors are tracked by denum.Context.

https://github.com/rin01/decnum
https://godoc.org/github.com/rin01/decnum

This package tries to demonstrate that decnum.Quad is easy to work with, and is a proposal for the base of an API.
A complete package would implement all functions described in this document:
http://speleotrove.com/decimal/dnfloat.html
The only difference is that values are passed by value instead of pointers.

[minux]

Why does this need to be in the standard library? I think it makes a perfectly fine go-gettable package.

[rin01]

Because decimal is a fundamental data type for everyday work, everytime you make an application that deals with money. All SQL databases implement such type natively for a reason, and it is quite unfortunate that there is no easy data type in Go in which we can store them.

If this type is in a third-party package, there can be a multiplication of choices, as is already the case for arbitrary-precision decimal, most of which are unfortunately experimental.
If it is in standard library, there will be no choosing, just one reference package with one API, and users and other packages can rely on it and be confident it works well.

[adg]

There are certainly advantages to having such a package in the standard library, but there are also drawbacks. Regardless, to put a new package in the standard library now we would want to first produce a complete implementation outside the standard library. Once we put something in the standard library we must support it for the lifetime of Go 1.x, so we need to get it right.

[minux]

The reason for fixed precision decimal type seems to be that you want to pass it by value, but different projects might have different requirement on precision, thus I think if the package is to be generally useful, the precision must be configurable, but that would conflict with the desire to pass decimals by value. I still think it's best to live outside of the standard library and available as a go generate template package so that people can generate decimal packages of different sizes and include in their projects. And in that case, it's probably best to be branded the package as an arbitrary but fixed precision fixed-point package. If you think outside the field of financial computations, fixed-point types of various sizes are all useful (e.g. in DSP applications).

[rin01]

The main reason to pass by value is simplicity.

In my opinion, a decimal128 type is as fundamental as float64, and we should be able to work with the former as easily as with the latter, in similar way.
And when you declare a float64, it is a value, not a pointer.

People are happy with float64, and there is not much discussion about its 64bits fixed-sizeness or 15 significant digits precision.
It is just that float64is "good enough" for a lot of applications, and really simple to use.
For more stringent requirements, it is now possible to use big.Float instead of float64, at the price of some more complexity.

The purpose of this proposal is essentially to have a fixed-size decimal128 type in the language, that is "good enough", like float64is.
And if this default is not enough, people can use a more specialized arbitrary-precision decimal package, as discussed in [proposal: math/big: Decimal #12127].

That being said, there is no hurry to include decimal128 in Go.
I just think that this issue is a good place to receive suggestions on this topic, and see if there is sufficient support and need for such data type...

[griesemer]

If Go can become viable or even good language for business applications if we had a good standard solution for fixed-size decimal numbers, we should go forward with it. It's a chicken-and-egg problem, but we can lay the egg without the chicken, and see if it hatches. I'm all for a good Decimal implementation, but as @adg already said, let's prototype it externally, as part of the proposal.

Issue #12127 is already discussing some of this, and I think the conclusion is that a Decimal128 type would be sufficient for all practical commercial computations.

[As an aside: I can also appreciate that one would want to have a built-in type decimal128. But that is something that should be considered only after we have gained an extremely good understanding of what decimal128 actually is. For instance, it would require (in my mind) that all the essential operations on decimal128 could be reduced to the basic arithmetic operations, comparisons, and likely some "helper" package, say decmath (like we have math now supporting float64). Most of the time, I would expect a user to be able to work with decimal128 without the need to resort to decmath. Is that possible? I don't know. It depends on the properties of decimal128; e.g., is it a decimal floating-point or fixed point format. If it's the latter, I suspect there's additional functionality needed to deal with precision etc. There's likely many other issues I am blissfully unaware of. ]

It looks like http://www.dec.usc.es/arith16/papers/paper-107.pdf might be a good starting point for a concrete design.

[rin01]

The package decnum now contains the functions needed for basic work with money.

https://github.com/rin01/decnum
https://godoc.org/github.com/rin01/decnum

The API seems good to me, and easy to use. If you want to play with it and tell me your opinion.

I've had a look at various implementations of 128bits decimal floating point.
It is really a huge work to write such code, and I wonder if it is a good idea to duplicate all this effort in Go.
The simplest solution may be to just take the C decNumber library of Mike Cowlishaw, which is very much free of bugs, and make a wrapper around it, like decnum package, after all.
And it is already compliant with all the standards.

Besides, most financial applications works heavily with databases, reading records, working on them, writing back the result to disk. So, the overhead of cgo calls is negligible, compared to IO operations.
And most applications that deals with money, like webstores or reservation sites, don't make tons of calculations.
They just want to easily read money values from databases and work on them.

@griesemer ultimately, there should be no problem to include a built-in decimal128 type in Go.
It should run with a fixed rounding mode, ROUND_HALF_EVEN.
Also, a decmath package will provide functions such as
FromString(string) (decimal128, error)
which must return error, because there is no Context which can track errors.
For arithmetic operations like +, -, etc, the result will be NaN in case of error.

gcc already includes Mike Cowlishaw's library, and proposes a built-in floating-point decimal type:
https://gcc.gnu.org/onlinedocs/gcc/Decimal-Float.html

[bradfitz]

That API (https://godoc.org/github.com/rin01/decnum) is pretty gross. cgo and C style has polluted it everywhere. Is there a pure Go version or at least one with Go style available?

[rin01]

That's right, I just changed it now. It should be a little bit better.

[minux]

I don't like the API that every function has to carry a Context around. I think the API of math/big.Float better where the rounding mode and etc are bundled with the Float itself.

[rin01]

Well, the purpose of this decimal 128bits type is to have a type as easy to work with as possible.
So, it is inherently subject to some trade-offs.

In particular, precision is fixed at 34 significant digits and cannot be changed.

Context is primarily used to track errors that occurs during operations.
There are often long series of operations, and it is good to avoid cluttering the code with error detection lines.
It is better to perform all operations and check for error in the end, with context.Error() method.
That's why this Contextobject must be passed to all operations.

The big.Float methods raise a panic(ErrNaN) in case of error.
So the alternative for decimal 128bits package API would be to panic the same way ?...

Besides, decimal 128bits is passed by value, like float64, and methods return the result as value.

If the API of this package is changed to embed rounding information in the type, and then passing it by pointer, it will become the big.Decimal package discussed in #12127.
But this proposal is to have something a little bit "lighter", simpler, good-enough for most users...

In Python, Context is passed invisibly but implicitly.
https://www.python.org/dev/peps/pep-0327/#use-of-context

In Java BigDecimal, a Context or RoundingMode is passed as argument to methods.
http://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html

In C decNumber, Context is passed explicitly to each function.
Context is the argument called set:
http://speleotrove.com/decimal/dnfloat.html

[rin01]

@minux I have thought about that context problem again.
You are right, it is not good to have this context everywhere.
It would be good if the expression:

r = (a+b)/(c-d)

could be expressed as:

r = a.Add(b).Div(c.Add(d))

if err = r.Error(); err != nil {
    log.Fatal(err)
}

I will rework the whole API, in the following weeks.
Thank you for the comment.