Cannot load kernel extensions built against non-install GAP into an executable built against `libgap`

[dimpase]

libgap is broken in the current master and in 4.12.1 (but still OK on 4.11.1) on macOS 13.0.1 if io is built, as can be seen by running
tst/testlibgap/basic: (which is built by make testlibgap, for instance)

% ./tst/testlibgap/basic -l .
# Initializing GAP...
 ┌───────┐   GAP 4.13dev-169-g28d5faa built on 2022-11-30 18:44:58+0000
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-apple-darwin22-default64-kv8
 Configuration:  gmp 6.2.1, GASMAN, readline
 Loading the library and packages ...
Error, LOAD_DYN: failed to load kernel module ./pkg/io//bin/x86_64-apple-darwin2\
2-default64-kv8/io.so, dlopen(./pkg/io//bin/x86_64-apple-darwin22-default\
64-kv8/io.so, 0x0005): Symbol not found: _AssGVar
  Referenced from: <691ED98B-03DB-3A1C-98A6-D5307CB4FCF6> /Users/dima/sof\
tware/gap/pkg/io/bin/x86_64-apple-darwin22-default64-kv8/io.so
  Expected in:     <6EA6D1DA-9BAB-322E-AEE4-887EFCA0E1EF> /Users/dima/sof\
tware/gap/tst/testlibgap/.libs/basic in
  LOAD_DYN( filename ) at ./lib/files.gd:595 called from 
<function "LoadDynamicModule">( <arguments> )
 called from read-eval loop at ./pkg/io//init.g:23
Error, was not in any namespace at ./lib/variable.g:269 called from
LEAVE_NAMESPACE(  ); at ./lib/package.gi:1327 called from
ReadPackage( pkgname, "init.g" ); at ./lib/package.gi:1688 called from
LoadPackage( pair[1], pair[2], false ) at ./lib/package.gi:1933 called from
func(  ); at ./lib/system.g:228 called from
<function "CallAndInstallPostRestore">( <arguments> )
 called from read-eval loop at ./lib/init.g:700
you can 'quit;' to quit to outer loop, or
you can 'return;' to continue
brk> 

On linux this works fine:

$ ./tst/testlibgap/basic -l .
# Initializing GAP...
 ┌───────┐   GAP 4.13dev-169-g28d5faa built on 2022-11-30 21:48:54+0000
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv8
 Configuration:  gmp 6.2.1, GASMAN, readline
 Loading the library and packages ...
 Packages:   AClib 1.3.2, Alnuth 3.2.1, AtlasRep 2.1.6, AutPGrp 1.11, CRISP 1.4.5, Cryst 4.1.25, CrystCat 1.1.10, CTblLib 1.3.4, FactInt 1.6.3, FGA 1.4.0, Forms 1.2.9, GAPDoc 1.6.6, genss 1.6.8, IO 4.8.0, 
             IRREDSOL 1.4.4, LAGUNA 3.9.5, orb 4.9.0, Polenta 1.3.10, Polycyclic 2.16, PrimGrp 3.4.2, RadiRoot 2.9, recog 1.4.2, ResClasses 4.7.3, SmallGrp 1.5.1, Sophus 1.27, SpinSym 1.5.2, TomLib 1.2.9, 
             TransGrp 3.6.3, utils 0.78
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> 

Originally posted by @dimpase in #5230 (comment)

[dimpase]

also, it can be triggered non-interactively. With the following change

--- a/tst/testlibgap/basic.c
+++ b/tst/testlibgap/basic.c
@@ -6,6 +6,7 @@ int main(int argc, char ** argv)
 {
     printf("# Initializing GAP...\n");
     GAP_Initialize(argc, argv, 0, 0, 1);
+    test_eval("LoadPackage(\"io\");");
     test_eval("1+2+3;");
     test_eval("g:=FreeGroup(2);");
     test_eval("a:=g.1;");

by running make testlibgap on macOS 13 one gets the same error as above.

[dimpase]

There is no issue with GAP 4.11.1.

Unless someone can tell straight away where the regression was introduced, I'll do a git bisect to find out later today.

[ChrisJefferson]

I can't think of anything, and at first glance AssGVar doesn't look like a function we treat unusually.

[dimpase]

bisecting now, at most 10 runs to go :-)

[dimpase]

according to git bisect, the regression came from @fingolfin 's commit here;

9c343a4d91fb33900f0ac24010a99e7e7c800290 is the first bad commit
commit 9c343a4d91fb33900f0ac24010a99e7e7c800290
Author: Max Horn <max@quendi.de>
Date:   Fri Mar 4 12:22:45 2022 +0100

    buildsys: change gac to not use libtool

 Makefile.rules | 47 +++++++++++++++++++++++++++++++++------
 cnf/gac.in     | 69 +++++++++++++++++++++-------------------------------------
 configure.ac   |  1 -
 3 files changed, 65 insertions(+), 52 deletions(-)

[dimpase]

reverting 9c343a4 on the master, i.e. 28d5faa, fixes the issue.

I haven't tried running a full testsuite.

[fingolfin]

Fixed in #5235.

[dimpase]

probably not related, but testing without the bad commit produces two errors in streams.tst

gap> Read( Filename( DirectoriesLibrary( "tst" ), "testinstall.g" ) );
Architecture: x86_64-apple-darwin22-default64-kv8
...
testing: /Users/dima/software/gap/tst/testinstall/streams.tst
########> Diff in /Users/dima/software/gap/tst/testinstall/streams.tst:
214
# Input is:
for i in [ 1 .. 300 ] do
   Add( streams, OutputTextFile( fname, false ) );
od;;
# Expected output:
Error, Too many open files (internal file descriptor limit reached)
# But found:
########
########> Diff in /Users/dima/software/gap/tst/testinstall/streams.tst:
218
# Input is:
Perform( streams, CloseStream );
# Expected output:
# But found:
Error, no method found! For debugging hints type ?Recovery from NoMe\
thodFound
Error, no 1st choice method found for `CloseStream' on 1 arguments
The 1st argument is 'fail' which might point to an earlier problem

########
     120 ms (115 ms GC) and 509KB allocated for streams.tst

[fingolfin]

After looking this in details, I disagree that there is a bug here: you built a kernel extension against a non-installed gap which is not using libgap. You then tried to load it into a different executable (tst/testlibgap/basic) which is using libgap.

In the long run, I don't expect this to work at all. In the short run, I was able to find a workaround, which is in PR #5235.

But the initial assessment, that "libgap is broken in the current master" is simply not accurate. It works fine, also with kernel extensions, as long as you built the kernel extension against libgap (i.e., make install gap and build against that). We verify that this works as part of our CI, too.

If there is genuine need to load packages in tst/testlibgap/basic in the future, then we could make those tests more sophisticated, by giving them their own sysinfo.gap with suitable flags to ensure a GAP package built with it is built against the non-installed libgap, and the building the relevant packages with that. But that is a lot of effort for little noticeable gain, as it tests things in an "unnatural" scenario.

[dimpase]

once again, how does gap use libgap? The only way I know about how an executable can "use" a dynamical library is to link it. But it does not. Here is the linkage of gap installed with make install to --prefix=/tmp

% otool -L gap
gap:
	/tmp/lib/libgap.8.dylib (compatibility version 9.0.0, current version 9.0.0)
	/usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 15.0.0, current version 15.1.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/local/opt/readline/lib/libreadline.8.dylib (compatibility version 8.2.0, current version 8.2.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

there is no libgap there to talk about.