Bitwise math for 54+-bit numbers is broken

[p5pRT]

Migrated from rt.perl.org#67432 (status was 'open')

Searchable as RT67432$

[p5pRT]

From beeman@alumni.utexas.net

This is a bug report for perl from beeman@​alumni.utexas.net,
generated with the help of perlbug 1.35 running under perl v5.8.5.

---

A curious bug when doing bitwise math on a 64-bit linux version of perl. I've seen this with 5.8.5, 5.8.7, and 5.10.0. I'll let the debugger illustrate the problem​:

  DB<9> $foo = 2**53

  DB<10> printf("0x%x", $foo)
0x20000000000000
  DB<11> printf("0x%x", $foo-1)
0x1fffffffffffff

Looks good, as it seems to for all smaller numbers. But when we move up to 54-bit​:

  DB<1> $foo = 2**54

  DB<2> printf("0x%x", $foo)
0x40000000000000
  DB<3> printf("0x%x", $foo-1)
0x40000000000000
  DB<4> printf("0x%x", $foo-5)
0x3ffffffffffffc
  DB<5> printf("0x%x", $foo-4)
0x3ffffffffffffc
  DB<6> printf("0x%x", $foo-3)
0x3ffffffffffffc
  DB<7> printf("0x%x", $foo-2)
0x3ffffffffffffe

Some very strange results. Similar problems with larger numbers​:

  DB<<31>> $foo = 2**56

  DB<<32>> printf("0x%x",$foo-1)
0x100000000000000
  DB<<33>> printf("0x%x",$foo-7)
0xfffffffffffff8
  DB<<34>> printf("0x%x",$foo-2)
0x100000000000000
  DB<<35>> printf("0x%x",$foo-3)
0x100000000000000
  DB<<36>> printf("0x%x",$foo-4)
0x100000000000000
  DB<<37>> printf("0x%x",$foo-5)
0xfffffffffffff8

---

---

Flags​:
  category=core
  severity=high

---

Site configuration information for perl v5.8.5​:

Configured by tmnguye3 at Mon Nov 1 16​:32​:56 PST 2004.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration​:
  Platform​:
  osname=linux, osvers=2.6.5-7.97-smp, archname=x86_64-linux
  uname='linux plxb0325 2.6.5-7.97-smp #1 smp fri jul 2 14​:21​:59 utc 2004 x86_64 x86_64 x86_64 gnulinux '
  config_args='-Uusemymalloc -Dprefix=/usr/intel/pkgs/perl/5.8.5 -Dcc=/usr/intel/pkgs/gcc/3.4.2/bin/gcc -Doptimize=none -Dloclibpth=/usr/intel/00r1/lib -Dlocincpth=/usr/intel/00r1/include -Duse64bitall -Uinstallusrbinperl -Duselargefiles -Dperladmin=tmnguye3@​sedona.intel.com -Uinstallusrbinperl -des'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=define use64bitall=define uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ccflags ='-fno-strict-aliasing -pipe -I/usr/intel/00r1/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize=' ',
  cppflags='-fno-strict-aliasing -pipe -I/usr/intel/00r1/include'
  ccversion='', gccversion='3.4.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ldflags =' -L/usr/intel/00r1/lib'
  libpth=/usr/intel/00r1/lib /lib /usr/lib /usr/local/lib
  libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.3.3'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/intel/00r1/lib'

Locally applied patches​:
 

---

@​INC for perl v5.8.5​:
  /usr/intel/pkgs/perl/5.8.5/lib/5.8.5/x86_64-linux
  /usr/intel/pkgs/perl/5.8.5/lib/5.8.5
  /usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5/x86_64-linux
  /usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5
  /usr/intel/pkgs/perl/5.8.5/lib/site_perl
  .

---

Environment for perl v5.8.5​:
  HOME=/fs19/a/bcstrong
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LC_ALL=C
  LC_COLLATE=C
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=.​:/fs19/a/bcstrong/bin​:/nfs/site/proj/dpg/arch/perfhome/bin​:/nfs/site/proj/dpg/arch/bin​:/nfs/pdx/proj/dpg/arch/scripts​:/p/hsw/rtl/proj_bin​:/p/hsw/valid/bin​:/usr/intel/bin​:/usr/bin/X11​:/usr/bin​:/bin​:/usr/lib/java/jre/bin​:/usr/sbin​:/usr/X11R6/bin​:/opt/gnome/bin​:/opt/kde3/bin​:/usr/local/bin​:/p/hsw/rtl/dbin​:/p/hsw/valid/bin​:/p/hsw/valid/av/bin​:/p/hsw/rtl/models/ucode/utils/bin
  PERL_BADLANG (unset)
  SHELL=/usr/intel/00r1.1/bin/tcsh

[p5pRT]

From zefram@fysh.org

beeman@​alumni.utexas.net (via RT) wrote​:

DB<1> $foo = 2**54
DB<4> printf("0x%x", $foo-5)
0x3ffffffffffffc

This is floating-point arithmetic. You have IEEE double-precision
floating point, which has 1+52 bits of significand. In the range
[2^52, 2^53), with exponent 52, all integers can be represented, but no
fractional numbers. In the range [2^53, 2^54), with exponent 53, only
even integers can be represented, so the odd result of the subtraction
gets rounded. The floating-point number gets converted perfectly to
integer for the %x, because it's well within the native integer range.

You get floating-point arithmetic because of the **, which is always
done in floating point. If you construct $foo as 1<<54, you get the same
numerical value, but as a native integer. The subtraction then happens
in integer arithmetic and yields an exact result. You can also get $foo
in integer form by doing "$foo=int($foo);" before the subtraction.

Another way to influence this behaviour, which might constitute a
workaround, is to recompile using "long double" (instead of "double")
for the NV type. On the Intel architecture that you're using this gives
a 1+63-bit significand, which is just enough to be able to represent
all integers in the 64-bit range.

Getting back to the original problem, the use of floating point for the
subtraction is somewhat contrary to perlnumber(1), which says

# Arithmetic operators
# The binary operators "+" "-" "*" "/" "%" "==" "!=" ">" "<" ">="
# "<=" and the unary operators "-" "abs" and "--" will attempt to
# convert arguments to integers. If both conversions are possible
# without loss of precision, and the operation can be performed
# without loss of precision then the integer result is used.

I've noticed some other places that fail to live up to this description,
such as non-integer modulus (though this one now has an accurate
description in perlop(1)).

-zefram

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

beeman@​alumni.utexas.net (via RT) wrote​:

# New Ticket Created by beeman@​alumni.utexas.net
# Please include the string​: [perl #67432]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=67432 >

This is a bug report for perl from beeman@​alumni.utexas.net,
generated with the help of perlbug 1.35 running under perl v5.8.5.

-----------------------------------------------------------------
A curious bug when doing bitwise math on a 64-bit linux version of perl. I've seen this with 5.8.5, 5.8.7, and 5.10.0. I'll let the debugger illustrate the problem​:

DB<9> $foo = 2**53

DB<10> printf("0x%x", $foo)
0x20000000000000
DB<11> printf("0x%x", $foo-1)
0x1fffffffffffff

Looks good, as it seems to for all smaller numbers. But when we move up to 54-bit​:

DB<1> $foo = 2**54

DB<2> printf("0x%x", $foo)
0x40000000000000
DB<3> printf("0x%x", $foo-1)
0x40000000000000
DB<4> printf("0x%x", $foo-5)
0x3ffffffffffffc
DB<5> printf("0x%x", $foo-4)
0x3ffffffffffffc
DB<6> printf("0x%x", $foo-3)
0x3ffffffffffffc
DB<7> printf("0x%x", $foo-2)
0x3ffffffffffffe

Some very strange results. Similar problems with larger numbers​:

DB<<31>> $foo = 2**56

DB<<32>> printf("0x%x",$foo-1)
0x100000000000000
DB<<33>> printf("0x%x",$foo-7)
0xfffffffffffff8
DB<<34>> printf("0x%x",$foo-2)
0x100000000000000
DB<<35>> printf("0x%x",$foo-3)
0x100000000000000
DB<<36>> printf("0x%x",$foo-4)
0x100000000000000
DB<<37>> printf("0x%x",$foo-5)
0xfffffffffffff8

-----------------------------------------------------------------
---
Flags​:
category=core
severity=high
---
Site configuration information for perl v5.8.5​:

Configured by tmnguye3 at Mon Nov 1 16​:32​:56 PST 2004.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration​:
Platform​:
osname=linux, osvers=2.6.5-7.97-smp, archname=x86_64-linux
uname='linux plxb0325 2.6.5-7.97-smp #1 smp fri jul 2 14​:21​:59 utc 2004 x86_64 x86_64 x86_64 gnulinux '
config_args='-Uusemymalloc -Dprefix=/usr/intel/pkgs/perl/5.8.5 -Dcc=/usr/intel/pkgs/gcc/3.4.2/bin/gcc -Doptimize=none -Dloclibpth=/usr/intel/00r1/lib -Dlocincpth=/usr/intel/00r1/include -Duse64bitall -Uinstallusrbinperl -Duselargefiles -Dperladmin=tmnguye3@​sedona.intel.com -Uinstallusrbinperl -des'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=define uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ccflags ='-fno-strict-aliasing -pipe -I/usr/intel/00r1/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize=' ',
cppflags='-fno-strict-aliasing -pipe -I/usr/intel/00r1/include'
ccversion='', gccversion='3.4.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ldflags =' -L/usr/intel/00r1/lib'
libpth=/usr/intel/00r1/lib /lib /usr/lib /usr/local/lib
libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.3.3'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/intel/00r1/lib'

Locally applied patches​:

---
@​INC for perl v5.8.5​:
/usr/intel/pkgs/perl/5.8.5/lib/5.8.5/x86_64-linux
/usr/intel/pkgs/perl/5.8.5/lib/5.8.5
/usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5/x86_64-linux
/usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5
/usr/intel/pkgs/perl/5.8.5/lib/site_perl
.

---
Environment for perl v5.8.5​:
HOME=/fs19/a/bcstrong
LANG=en_US.UTF-8
LANGUAGE (unset)
LC_ALL=C
LC_COLLATE=C
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=.​:/fs19/a/bcstrong/bin​:/nfs/site/proj/dpg/arch/perfhome/bin​:/nfs/site/proj/dpg/arch/bin​:/nfs/pdx/proj/dpg/arch/scripts​:/p/hsw/rtl/proj_bin​:/p/hsw/valid/bin​:/usr/intel/bin​:/usr/bin/X11​:/usr/bin​:/bin​:/usr/lib/java/jre/bin​:/usr/sbin​:/usr/X11R6/bin​:/opt/gnome/bin​:/opt/kde3/bin​:/usr/local/bin​:/p/hsw/rtl/dbin​:/p/hsw/valid/bin​:/p/hsw/valid/av/bin​:/p/hsw/rtl/models/ucode/utils/bin
PERL_BADLANG (unset)
SHELL=/usr/intel/00r1.1/bin/tcsh

I tried this on 5.10 on an x86 running Ubuntu, and it did not fail

[p5pRT]

From beeman@alumni.utexas.net

Strange. I just tried it on another 5.10.0 build, using
cygwin-thread-multi-64int, and got the same (buggy) result.

On Sun, Jul 12, 2009 at 8​:54 PM, Karl Williamson <public@​khwilliamson.com>wrote​:

beeman@​alumni.utexas.net (via RT) wrote​:

# New Ticket Created by beeman@​alumni.utexas.net # Please include the
string​: [perl #67432]
# in the subject line of all future correspondence about this issue. #
<URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=67432 >

This is a bug report for perl from beeman@​alumni.utexas.net,
generated with the help of perlbug 1.35 running under perl v5.8.5.

-----------------------------------------------------------------
A curious bug when doing bitwise math on a 64-bit linux version of perl.
I've seen this with 5.8.5, 5.8.7, and 5.10.0. I'll let the debugger
illustrate the problem​:

DB<9> $foo = 2**53

DB<10> printf("0x%x", $foo)
0x20000000000000
DB<11> printf("0x%x", $foo-1)
0x1fffffffffffff

Looks good, as it seems to for all smaller numbers. But when we move up
to 54-bit​:

DB<1> $foo = 2**54

DB<2> printf("0x%x", $foo)
0x40000000000000
DB<3> printf("0x%x", $foo-1)
0x40000000000000
DB<4> printf("0x%x", $foo-5)
0x3ffffffffffffc
DB<5> printf("0x%x", $foo-4)
0x3ffffffffffffc
DB<6> printf("0x%x", $foo-3)
0x3ffffffffffffc
DB<7> printf("0x%x", $foo-2)
0x3ffffffffffffe

Some very strange results. Similar problems with larger numbers​:

DB<<31>> $foo = 2**56
DB<<32>> printf("0x%x",$foo-1)
0x100000000000000
DB<<33>> printf("0x%x",$foo-7)
0xfffffffffffff8
DB<<34>> printf("0x%x",$foo-2)
0x100000000000000
DB<<35>> printf("0x%x",$foo-3)
0x100000000000000
DB<<36>> printf("0x%x",$foo-4)
0x100000000000000
DB<<37>> printf("0x%x",$foo-5)
0xfffffffffffff8

-----------------------------------------------------------------
---
Flags​:
category=core
severity=high
---
Site configuration information for perl v5.8.5​:

Configured by tmnguye3 at Mon Nov 1 16​:32​:56 PST 2004.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration​:
Platform​:
osname=linux, osvers=2.6.5-7.97-smp, archname=x86_64-linux
uname='linux plxb0325 2.6.5-7.97-smp #1 smp fri jul 2 14​:21​:59 utc 2004
x86_64 x86_64 x86_64 gnulinux '
config_args='-Uusemymalloc -Dprefix=/usr/intel/pkgs/perl/5.8.5
-Dcc=/usr/intel/pkgs/gcc/3.4.2/bin/gcc -Doptimize=none
-Dloclibpth=/usr/intel/00r1/lib -Dlocincpth=/usr/intel/00r1/include
-Duse64bitall -Uinstallusrbinperl -Duselargefiles -Dperladmin=
tmnguye3@​sedona.intel.com -Uinstallusrbinperl -des'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=define uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ccflags ='-fno-strict-aliasing
-pipe -I/usr/intel/00r1/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize=' ',
cppflags='-fno-strict-aliasing -pipe -I/usr/intel/00r1/include'
ccversion='', gccversion='3.4.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='/usr/intel/pkgs/gcc/3.4.2/bin/gcc', ldflags ='
-L/usr/intel/00r1/lib'
libpth=/usr/intel/00r1/lib /lib /usr/lib /usr/local/lib
libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.3.3'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/intel/00r1/lib'

Locally applied patches​:

---
@​INC for perl v5.8.5​:
/usr/intel/pkgs/perl/5.8.5/lib/5.8.5/x86_64-linux
/usr/intel/pkgs/perl/5.8.5/lib/5.8.5
/usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5/x86_64-linux
/usr/intel/pkgs/perl/5.8.5/lib/site_perl/5.8.5
/usr/intel/pkgs/perl/5.8.5/lib/site_perl
.

---
Environment for perl v5.8.5​:
HOME=/fs19/a/bcstrong
LANG=en_US.UTF-8
LANGUAGE (unset)
LC_ALL=C
LC_COLLATE=C
LD_LIBRARY_PATH (unset)
LOGDIR (unset)

PATH=.​:/fs19/a/bcstrong/bin​:/nfs/site/proj/dpg/arch/perfhome/bin​:/nfs/site/proj/dpg/arch/bin​:/nfs/pdx/proj/dpg/arch/scripts​:/p/hsw/rtl/proj_bin​:/p/hsw/valid/bin​:/usr/intel/bin​:/usr/bin/X11​:/usr/bin​:/bin​:/usr/lib/java/jre/bin​:/usr/sbin​:/usr/X11R6/bin​:/opt/gnome/bin​:/opt/kde3/bin​:/usr/local/bin​:/p/hsw/rtl/dbin​:/p/hsw/valid/bin​:/p/hsw/valid/av/bin​:/p/hsw/rtl/models/ucode/utils/bin
PERL_BADLANG (unset)
SHELL=/usr/intel/00r1.1/bin/tcsh

I tried this on 5.10 on an x86 running Ubuntu, and it did not fail

[p5pRT]

From @ysth

On Sun, July 12, 2009 11​:09 pm, Beeman Strong wrote​:

On Sun, Jul 12, 2009 at 8​:54 PM, Karl Williamson wrote​:

I tried this on 5.10 on an x86 running Ubuntu, and it did not fail

Strange. I just tried it on another 5.10.0 build, using
cygwin-thread-multi-64int, and got the same (buggy) result.

I think Karl must have used a 32bitint perl.

A 64bitint (amd64 ubuntu) perl 5.10 does fail.

Exponentiation doesn't preserve integer representation currently (except in
select cases?); note the following​:

$ perl -we'printf "%x\n", 2**54-1'
40000000000000
$ perl -we'printf "%x\n", int(2**54)-1'
3fffffffffffff