Skip to content

Loss of precision in header values for f64 doubles #448

@dmopalmer

Description

@dmopalmer

When a floating-point header value is added to a FITS file, the code uses 9 digits of precision for both f32 and f64 values. This loses a lot of useful precision for the f64 value. (f64::digits is 5. f64::EPSILON is 2.2e-16).

My particular use case is storing times as POSIX-epoch timestamps (seconds since 1970 Jan 1) to millisecond accuracy. Current times are 10 digits for the seconds part ( 1_775_631_217 ) and f64 has a granularity of about a quarter microsecond for these values. (i.e. two unequal f64 values can be formatted as the same string if the string has 16 digits in it, but two values that format to the same 17 digits are equal).

f64::digits == 15 is probably a reasonable number of digits for floats.

Pull request will be incoming shortly. It is part of the same use case as the pending pull requests from @zec .

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions