allow linking with gmp instead of mpir

[DanGrayson]

No description provided.

[DanGrayson]

I've started a branch for this work at https://github.com/DanGrayson/M2/tree/use-native-gmp-mpfr-pari .

But it's complicated! Basically, we are forced to let libgmp use malloc instead of continuing to tell it to use GC_malloc, which we need to do so our top level objects get garbage collected.

[mikestillman]

I was wondering about this. For instance, frobby also uses gmp integers, inside of c++ data structure memory which, I think, gc is not aware of (having been called through malloc or new). For the part of frobby we use, I think it is not an issue, but if we try to use the rest, it is.

This gmp allocation issue has been thorny for a long time!

On Jun 4, 2015, at 1:22 PM, Daniel R. Grayson notifications@github.com wrote:

I've started a branch for this work at https://github.com/DanGrayson/M2/tree/use-native-gmp-mfpr-pari https://github.com/DanGrayson/M2/tree/use-native-gmp-mfpr-pari
But it's complicated! Basically, we are forced to let libgmp use malloc instead of continuing to tell it to use GC_malloc, which we need to do so our top level objects get garbage collected.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

I'm assuming that libgc cannot find thread local storage created with the __thread compiler keyword, but I could be wrong. Thread safe 3rd party libraries (such as pari) will use this. (Actually, pari comes in a non-gmp flavor, too.) So we cannot force them to use GC_malloc. Also, we can't set the gmp allocation functions on the fly, because that info is stored in global variables. (Pari sets them on the fly, which makes the gmp version of pari not thread safe, despite what pari documentation says.)

[DanGrayson]

In order to maximize the number of native or homebrew-supplied libraries we can use under Mac OS X, it seems we should compile with clang and clang++, to match what the libraries are compiled with. So I'm trying that now.

[mikestillman]

The clang provided with mac is way too old, I think. Perhaps with the latest Xcode it is OK, but this is a bit of a problem.

On Jun 4, 2015, at 4:53 PM, Daniel R. Grayson notifications@github.com wrote:

In order to maximize the number of native or homebrew-supplied libraries we can use under Mac OS X, it seems we should compile with clang and clang++, to match what the libraries are compiled with. So I'm trying that now.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

How do you compare ages of clang?

einsteinium$ ~/Downloads/clang+llvm-3.6.1-x86_64-apple-darwin/bin/clang --version
clang version 3.6.1 (tags/RELEASE_361/final)
Target: x86_64-apple-darwin14.1.0
Thread model: posix
einsteinium$ /usr/bin/clang --version
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix

[DanGrayson]

I think the only way to fix these issue is to stop setting the GMP memory allocation functions (using mp_set_memory_functions) to versions that invoke LIBGC, because we can never be sure that third party libraries will store those pointers in a place where LIBGC can find them. So now I'm trying to finalize every top level object that contains a pointer to a GMP integer, a GMP rational, or an MPFR real.

[mikestillman]

Another option, possibly a bad one, is to use mpn routines at top level (for M2) and do memory allocation ourselves. mpn routines, I think, do no memory allocation.

How many gmp functions do you/we interface to?

On Jun 8, 2015, at 7:42 PM, Daniel R. Grayson notifications@github.com wrote:

I think the only way to fix these issue is to stop setting the GMP memory allocation functions (using mp_set_memory_functions) to versions that invoke LIBGC, because we can never be sure that third party libraries will store those pointers in a place where LIBGC can find them. So now I'm trying to finalize every top level object that contains a pointer to a GMP integer, a GMP rational, or an MPFR real.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

I think that won't work, because it's not just us using GMP routines.

On Mon, Jun 8, 2015 at 7:46 PM Mike Stillman notifications@github.com
wrote:

Another option, possibly a bad one, is to use mpn routines at top level
(for M2) and do memory allocation ourselves. mpn routines, I think, do no
memory allocation.

How many gmp functions do you/we interface to?

On Jun 8, 2015, at 7:42 PM, Daniel R. Grayson notifications@github.com
wrote:

I think the only way to fix these issue is to stop setting the GMP
memory allocation functions (using mp_set_memory_functions) to versions
that invoke LIBGC, because we can never be sure that third party libraries
will store those pointers in a place where LIBGC can find them. So now I'm
trying to finalize every top level object that contains a pointer to a GMP
integer, a GMP rational, or an MPFR real.

—
Reply to this email directly or view it on GitHub <
https://github.com/Macaulay2/M2/issues/285#issuecomment-110172694>.

—
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110173282&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=vktisaeaQJrOfBNcbmyuvOiGEzGu9Ikpeuw5GngouSw&s=3yzeBBm_le0S5GcFOI44A6WHL5Np4RYXz_YRy10BuCU&e=
.

[DanGrayson]

Hmm, I take that back... let me think about it.

[mikestillman]

I was thinking that all engine objects should be finalized too. Then I can avoid (extra) garbage collection issues in the engine.

On Jun 8, 2015, at 7:49 PM, Daniel R. Grayson notifications@github.com wrote:

Hmm, I take that back... let me think about it.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

There are lots of engine objects that might contain pointers to GMP
integers.

Another option would be to copy the array of limbs in every GMP integer
returned by 3rd
party routines into memory obtained from GC and replace the limb pointer.

On Mon, Jun 8, 2015 at 8:10 PM Mike Stillman notifications@github.com
wrote:

I was thinking that all engine objects should be finalized too. Then I can
avoid (extra) garbage collection issues in the engine.

On Jun 8, 2015, at 7:49 PM, Daniel R. Grayson notifications@github.com
wrote:

Hmm, I take that back... let me think about it.

[mikestillman]

Engine objects, once finalized, can have destructors free the gmp integers (and that is done for finalized objects now)

How much would the limb switch slow things down in the front end? Maybe not so much?

On Jun 8, 2015, at 8:54 PM, Daniel R. Grayson notifications@github.com wrote:

There are lots of engine objects that might contain pointers to GMP
integers.

Another option would be to copy the array of limbs in every GMP integer
returned by 3rd
party routines into memory obtained from GMP and replace the limb pointer.

[DanGrayson]

It might be too confusing to have GMP integers floating about, some of
which come from
malloc and some of which come from GC_malloc.

On Mon, Jun 8, 2015 at 8:58 PM Mike Stillman notifications@github.com
wrote:

Engine objects, once finalized, can have destructors free the gmp integers
(and that is done for finalized objects now)

How much would the limb switch slow things down in the front end? Maybe
not so much?

On Jun 8, 2015, at 8:54 PM, Daniel R. Grayson notifications@github.com
wrote:

There are lots of engine objects that might contain pointers to GMP
integers.

Another option would be to copy the array of limbs in every GMP integer
returned by 3rd
party routines into memory obtained from GMP and replace the limb
pointer.

On Mon, Jun 8, 2015 at 8:10 PM Mike Stillman notifications@github.com
wrote:

I was thinking that all engine objects should be finalized too. Then I
can
avoid (extra) garbage collection issues in the engine.

On Jun 8, 2015, at 7:49 PM, Daniel R. Grayson <
notifications@github.com>
wrote:

Hmm, I take that back... let me think about it.

—
Reply to this email directly or view it on GitHub <
https://github.com/Macaulay2/M2/issues/285#issuecomment-110175430>.

—
Reply to this email directly or view it on GitHub
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110178682&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=9doYGICq265eQXbh-g9VFXL_B7MycCh59kuXjanG_wQ&s=O2RQ8Uq40ZGjXptKhTUlG2kJY7h7APlgm17d4KMbqNk&e=

.

—
Reply to this email directly or view it on GitHub <
https://github.com/Macaulay2/M2/issues/285#issuecomment-110184969>.

—
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110185408&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=EO-OOPViaRZ_7pW9hBoSd7cSMz007WjLDjobety8nuI&s=XHFWsPX-SmzQvaoIMb-n1q_b5bScVonu3MtfxxNacG0&e=
.

[mikestillman]

That is true, but what we do now is also very confusing, and we have spent a lot of time trying to get the logic of it right. And if we can’t modify library code, then we need to change what we do anyway, it seems.

If all of the engine integers in use come from malloc, and all the front end ones come from GC_malloc, we should be ok, as long as when you pass the engine a gmp int, the engine considers it const (which I do now anyway). And when the engine wants to hand one to the front end, there could be a routine to create one (I think the number of cases for these two is quite small).

On Jun 8, 2015, at 9:38 PM, Daniel R. Grayson notifications@github.com wrote:

It might be too confusing to have GMP integers floating about, some of
which come from
malloc and some of which come from GC_malloc.

On Mon, Jun 8, 2015 at 8:58 PM Mike Stillman notifications@github.com
wrote:

Engine objects, once finalized, can have destructors free the gmp integers
(and that is done for finalized objects now)

How much would the limb switch slow things down in the front end? Maybe
not so much?

On Jun 8, 2015, at 8:54 PM, Daniel R. Grayson notifications@github.com
wrote:

There are lots of engine objects that might contain pointers to GMP
integers.

Another option would be to copy the array of limbs in every GMP integer
returned by 3rd
party routines into memory obtained from GMP and replace the limb
pointer.

On Mon, Jun 8, 2015 at 8:10 PM Mike Stillman notifications@github.com
wrote:

I was thinking that all engine objects should be finalized too. Then I
can
avoid (extra) garbage collection issues in the engine.

On Jun 8, 2015, at 7:49 PM, Daniel R. Grayson <
notifications@github.com>
wrote:

Hmm, I take that back... let me think about it.

—
Reply to this email directly or view it on GitHub <
https://github.com/Macaulay2/M2/issues/285#issuecomment-110175430>.

—
Reply to this email directly or view it on GitHub
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110178682&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=9doYGICq265eQXbh-g9VFXL_B7MycCh59kuXjanG_wQ&s=O2RQ8Uq40ZGjXptKhTUlG2kJY7h7APlgm17d4KMbqNk&e=

.

—
Reply to this email directly or view it on GitHub <
https://github.com/Macaulay2/M2/issues/285#issuecomment-110184969>.

—
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110185408&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=EO-OOPViaRZ_7pW9hBoSd7cSMz007WjLDjobety8nuI&s=XHFWsPX-SmzQvaoIMb-n1q_b5bScVonu3MtfxxNacG0&e=
.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

Well, I think what I'm currently trying is less confusing, because remembering to convert the malloced GMP to the GCmalloced GMP will be difficult. See https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/e/x-relem.cpp#L558 , for example.

[DanGrayson]

... and https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/d/interface.dd#L1146 .

[mikestillman]

These are exactly the routines that need to be rewritten, if we change to doing gmp both with malloc and GC_malloc. And it would be easy to do so. (There are a few more like this, as we do the same for gmp int’s too).

If you want, I can can check for other changes that would be required.

On Jun 8, 2015, at 10:27 PM, Daniel R. Grayson notifications@github.com wrote:

Well, I think what I'm currently trying is less confusing, because remembering to convert the malloced GMP to the GCmalloced GMP will be difficult. See https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/e/x-relem.cpp#L558 https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/e/x-relem.cpp#L558 , for example.

—
Reply to this email directly or view it on GitHub #285 (comment).

[mikestillman]

This is not for 1.8 though.

On Jun 8, 2015, at 10:27 PM, Daniel R. Grayson notifications@github.com wrote:

Well, I think what I'm currently trying is less confusing, because remembering to convert the malloced GMP to the GCmalloced GMP will be difficult. See https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/e/x-relem.cpp#L558 https://github.com/DanGrayson/M2/blob/64e2d6129451ee4d496043411b2839d38a688213/M2/Macaulay2/e/x-relem.cpp#L558 , for example.

—
Reply to this email directly or view it on GitHub #285 (comment).

[DanGrayson]

Right, not for 1.8.

I think you don't treat GMP numbers from the engine as read only. For example, try changing mpz_ptr to mpz_srcptr in

const RingElement *IM2_RingElement_from_Integer(const Ring *R, mpz_ptr d)
{
  return RingElement::make_raw(R, R->from_int(d));
}

[DanGrayson]

Re: "These are exactly the routines that need to be rewritten"

"These"?

[mikestillman]

I just checked the "from_int" routines, and they all consider their argument read only. I am probably using the wrong type, so that changing one type notation to mpz_srcptr can't work, as I would need to change all the locations where from_int is used too. Perhaps I should do that.

So in fact, none of these functions (the others refer to the "from_rational" functions) would need to be rewritten.

[DanGrayson]

That's probably a good idea in any case. I'll try it.

On Tue, Jun 9, 2015 at 8:24 AM Mike Stillman notifications@github.com
wrote:

I just checked the "from_int" routines, and they all consider their
argument read only. I am probably using the wrong type, so that changing
one type notation to mpz_srcptr can't work, as I would need to change all
the locations where from_int is used too. Perhaps I should do that.

So in fact, none of these functions (the others refer to the
"from_rational" functions) would need to be rewritten.

—
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Macaulay2_M2_issues_285-23issuecomment-2D110338755&d=AwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=PAZ_jTPLrSb1btCfAu4RkycWe57N0sGyzbjgeOn2wN0&m=IQPNITacGR7e5dS6jF6UUAVvH5DZj3w4xbYTn-ibWbE&s=dc4G0LwohZ95sO1PNZRhVUUnysOLz89iEGoBCZNGwtI&e=
.

[DanGrayson]

I successfully changed many of your routines to accepting mpz_srcptr on my branch.

[DanGrayson]

Re: "Another option, possibly a bad one, is to use mpn routines at top level (for M2) and do memory allocation ourselves. mpn routines, I think, do no memory allocation."

The mpfr library uses also mpz routines, and doesn't provide an interface to avoid that.

[d-torrance]

Even if Debian did include mpir, I don't think using the Macaulay2 patched version would be ideal, as other applications built using mpir might want to set the memory functions. So we'd have the same problem as gmp.

[d-torrance]

I've sent a note about this issue to the pari dev mailing list. [1] Maybe that will solve it!

[1] http://pari.math.u-bordeaux.fr/archives/pari-dev-1605/msg00003.html

[dimpase]

@d-torrance : I presume the MPIR patch can be converted to a proper runtime feature, that can be accepted by upstream.

[d-torrance]

I got a nice quick reply from the pari devs [1]! Basically, they propose adding an additional flag which could be passed to pari_init_opts() which would skip the kernel initialization (and thus skip overwriting the gmp memory functions).

[1] http://pari.math.u-bordeaux.fr/archives/pari-dev-1605/msg00004.html

[d-torrance]

I got another response from the pari devs [1]. They suggest a wrapper around the call to pari_init_opts() saving the gmp memory functions.

I'll play around with this and see if it works for the Debian package. If so, I'll submit a pull request.

[1] http://pari.math.u-bordeaux.fr/archives/pari-dev-1605/msg00005.html

[dimpase]

Actually, the PARI devs bring up their #1317 which had to do with an analogous issue with Sagemath using PARI and MPIR at the same time.
There is a Sage ticket #15774 where the patch akin to the current M2's MPIR patch (well, the patch was for PARI rather than for MPIR, but it served the same purpose) has been removed.

This suggest that M2 could use an analogous wrapper around PARI initialisation to achieve the same effect, and remove MPIR patch.

[dimpase]

here is what Sage does (not a wrapper, in fact): right after calling pari_init_opts() (which does call mp_set_memory_functions()), it calls mp_set_memory_functions() itself, passing the needed MPIR the right memory handling functions to use.

In the M2 case, I guess that the function to free memory should not do anything.

[dimpase]

f6e2f61 at https://github.com/DanGrayson/M2/ is Dan's implementation of this for M2, and it actually works!

[DanGrayson]

Oops, I removed that branch from DanGrayson/M2 and put it on Macaulay2/M2, see https://github.com/Macaulay2/M2/tree/try-not-patching-mpir -- it's progressed a bit since then. There are bugs, though.

[dimpase]

By "bugs" do you mean the same sort of memory errors, i.e. more libraries do mess around with MPIR/GMP memory allocators? Or is it something else?

[DanGrayson]

We're getting many calls to "abort()" from "free()" complaining about pointers to memory not obtained with "malloc()". It happens inside ntl.

[dimpase]

Sage applies the following patch to NTL: https://github.com/sagemath/sage/blob/master/build/pkgs/ntl/patches/new_singular.patch

(replacing new (std::nothrow) with new in NTL's smart pointer class)

Without this, Sage gets a problem with (lib)Singular...

That is to say that perhaps new (std::nothrow) does not get memory via malloc()...

[DanGrayson]

That can't be it, because the calls to abort() happen only if I strip M2-binary.

[dimpase]

I will be at the workshop around 10am, hope to have a closer look then.
On 26 May 2016 8:05 am, "Daniel R. Grayson" notifications@github.com
wrote:

That can't be it, because the calls to abort() stop happening if I strip
M2-binary.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#285 (comment)

[d-torrance]

A possible update:

I finally got around to sending a backtrace to the PARI devs of where the M2 build fails when using the Debian GMP package, and they replied saying that there was a possible bug in PARI [1]. I tried building M2 again after making a small change in the PARI code, and it worked!

(Well, for a moment. Then I ran into #460...)

[1] https://pari.math.u-bordeaux.fr/archives/pari-dev-1706/msg00010.html

[mahrud]

Is this resolved now?

[dimpase]

@d-torrance should know all about it, I believe.

[d-torrance]

Yes, linking with gmp works beautifully now!