Skip to content

Conversation

@WalterBright
Copy link
Member

Reboot of #6907 because git rebase simply would not work with files that were renamed.

@dlang-bot
Copy link
Contributor

dlang-bot commented Jan 15, 2018

Thanks for your pull request, @WalterBright!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#7714"

set -uexo pipefail

HOST_DMD_VER=2.072.2 # same as in dmd/src/posix.mak
HOST_DMD_VER=2.074.1 # same as in dmd/src/posix.mak
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: auto-tester needs to be bumped too, see the discussion on the previous PR and the NG discussion - especially Martin's reply.

@wilzbach
Copy link
Contributor

In case the CI failures are unclear:


auto-tester: Fail: 1, Pending: 9

auto-tester still uses 2.068.2
CC @braddr

semaphoreci: The build failed on Semaphore.

Semaphore tests bootstrapping with the latest DMD, LDC + GDC releases.
The latest GDC release is 6.3.0+2.068.2. At #6907 Travis was used for this.

To repeat @ibuclaw from #6907:

Wouldn't want to give a precise date, but gcc-8 is first this would land in. As @MartinNowak said, you are free to make -betterC a noop for the dmd-script wrapper.

@wilzbach
Copy link
Contributor

wilzbach commented Jan 15, 2018

you are free to make -betterC a noop for the dmd-script wrapper.

It could be as simple as:

#!/bin/bash
${HOST_DMD:-gdc} "${@/-betterC/}"

Of course, you could also do this directly in the Makefile:

ifeq (gdc, $(HOST_DMD_KIND))
 BACK_FLAGS:=
else
 BACK_FLAGS:=-betterC
endif

@wilzbach
Copy link
Contributor

Of course, you could also do this directly in the Makefile:

  1. That trick does works for GDC
    [WIP] Don't set -betterC for old auto-tester + GDC #7733
  2. You probably want to bump HOST_DMD_VER to 2.074.1 in posix.mak -> this will fix Jenkins
  3. You need to add $(G_OBJS) for cxx-unittest -> this will fix auto-tester
  4. It seems like 2.068.2 requires you to use C mangling on OSX ?
  5. DAutoTest still uses 2.067.1 CC @CyberShadow

@WalterBright
Copy link
Member Author

ar rcs ../generated/linux/release/32/backend.a ../generated/linux/release/32/go.o ../generated/linux/release/32/gdag.o ../generated/linux/release/32/gother.o ../generated/linux/release/32/gflow.o ../generated/linux/release/32/gloop.o ../generated/linux/release/32/gsroa.o ../generated/linux/release/32/var.o ../generated/linux/release/32/el.o ../generated/linux/release/32/glocal.o ../generated/linux/release/32/os.o ../generated/linux/release/32/nteh.o ../generated/linux/release/32/evalu8.o ../generated/linux/release/32/cgcs.o ../generated/linux/release/32/rtlsym.o ../generated/linux/release/32/cgelem.o ../generated/linux/release/32/cgen.o ../generated/linux/release/32/cgreg.o ../generated/linux/release/32/out.o ../generated/linux/release/32/blockopt.o ../generated/linux/release/32/cg.o ../generated/linux/release/32/type.o ../generated/linux/release/32/dt.o ../generated/linux/release/32/debug.o ../generated/linux/release/32/code.o ../generated/linux/release/32/ee.o ../generated/linux/release/32/symbol.o ../generated/linux/release/32/cgcod.o ../generated/linux/release/32/cod5.o ../generated/linux/release/32/outbuf.o ../generated/linux/release/32/compress.o ../generated/linux/release/32/bcomplex.o ../generated/linux/release/32/aa.o ../generated/linux/release/32/ti_achar.o ../generated/linux/release/32/ti_pvoid.o ../generated/linux/release/32/pdata.o ../generated/linux/release/32/cv8.o ../generated/linux/release/32/backconfig.o ../generated/linux/release/32/dwarf.o ../generated/linux/release/32/dwarfeh.o ../generated/linux/release/32/varstats.o ../generated/linux/release/32/ph2.o ../generated/linux/release/32/util2.o ../generated/linux/release/32/tk.o ../generated/linux/release/32/strtold.o ../generated/linux/release/32/md5.o ../generated/linux/release/32/cg87.o ../generated/linux/release/32/cgxmm.o ../generated/linux/release/32/cgsched.o ../generated/linux/release/32/cod1.o ../generated/linux/release/32/cod2.o ../generated/linux/release/32/cod3.o ../generated/linux/release/32/cod4.o ../generated/linux/release/32/ptrntab.o ../generated/linux/release/32/elfobj.o ../generated/linux/release/32/divcoeff.o
CC=c++ /home/braddr/sandbox/at-client/release-build/install/linux/bin32/dmd -of../generated/linux/release/32/cxx-unittest -m32 -vtls -J../generated/linux/release/32 -J../res -L-lstdc++ -version=MARS  -w -de -O -release -inline -version=NoMain ../generated/linux/release/32/cxxfrontend.o dmd/access.d dmd/aggregate.d dmd/aliasthis.d dmd/apply.d dmd/argtypes.d dmd/arrayop.d dmd/arraytypes.d dmd/astcodegen.d dmd/attrib.d dmd/builtin.d dmd/canthrow.d dmd/cli.d dmd/clone.d dmd/compiler.d dmd/complex.d dmd/cond.d dmd/constfold.d dmd/cppmangle.d dmd/cppmanglewin.d dmd/ctfeexpr.d dmd/dcast.d dmd/dclass.d dmd/declaration.d dmd/delegatize.d dmd/denum.d dmd/dimport.d dmd/dinifile.d dmd/dinterpret.d dmd/dmacro.d dmd/dmangle.d dmd/dmodule.d dmd/doc.d dmd/dscope.d dmd/dstruct.d dmd/dsymbol.d dmd/dsymbolsem.d dmd/dtemplate.d dmd/dversion.d dmd/escape.d dmd/expression.d dmd/expressionsem.d dmd/func.d dmd/hdrgen.d dmd/id.d dmd/impcnvtab.d dmd/imphint.d dmd/init.d dmd/initsem.d dmd/inline.d dmd/inlinecost.d dmd/intrange.d dmd/json.d dmd/lambdacomp.d dmd/lib.d dmd/libelf.d dmd/libmach.d dmd/link.d dmd/mars.d dmd/mtype.d dmd/nogc.d dmd/nspace.d dmd/objc.d dmd/opover.d dmd/optimize.d dmd/parse.d dmd/permissivevisitor.d dmd/sapply.d dmd/templateparamsem.d dmd/sideeffect.d dmd/statement.d dmd/staticassert.d dmd/target.d dmd/typesem.d dmd/traits.d dmd/transitivevisitor.d dmd/parsetimevisitor.d dmd/visitor.d dmd/typinf.d dmd/utils.d dmd/scanelf.d dmd/scanmach.d dmd/statement_rewrite_walker.d dmd/statementsem.d dmd/staticcond.d dmd/safe.d dmd/blockexit.d dmd/printast.d dmd/semantic2.d dmd/semantic3.d dmd/irstate.d dmd/toctype.d dmd/glue.d dmd/gluelayer.d dmd/todt.d dmd/tocsym.d dmd/toir.d dmd/dmsc.d dmd/tocvdebug.d dmd/s2ir.d dmd/toobj.d dmd/e2ir.d dmd/eh.d dmd/iasm.d dmd/objc_glue.d dmd/backend/bcomplex.d dmd/backend/cc.d dmd/backend/cdef.d dmd/backend/cgcv.d dmd/backend/code.d dmd/backend/cv4.d dmd/backend/dt.d dmd/backend/el.d dmd/backend/global.d dmd/backend/obj.d dmd/backend/oper.d dmd/backend/outbuf.d dmd/backend/rtlsym.d dmd/backend/code_x86.d dmd/backend/iasm.d dmd/backend/ty.d dmd/backend/type.d dmd/backend/exh.d dmd/backend/mach.d dmd/backend/md5.d dmd/backend/mscoff.d dmd/backend/dwarf.d dmd/backend/dwarf2.d dmd/backend/xmm.d dmd/tk/dlist.d dmd/root/aav.d dmd/root/array.d dmd/root/ctfloat.d dmd/root/file.d dmd/root/filename.d dmd/root/man.d dmd/root/outbuffer.d dmd/root/port.d dmd/root/response.d dmd/root/rmem.d dmd/root/rootobject.d dmd/root/speller.d dmd/root/stringtable.d dmd/root/hash.d ../generated/linux/release/32/newdelete.o ../generated/linux/release/32/lexer.a ../generated/linux/release/32/go.o ../generated/linux/release/32/gdag.o ../generated/linux/release/32/gother.o ../generated/linux/release/32/gflow.o ../generated/linux/release/32/gloop.o ../generated/linux/release/32/gsroa.o ../generated/linux/release/32/var.o ../generated/linux/release/32/el.o ../generated/linux/release/32/glocal.o ../generated/linux/release/32/os.o ../generated/linux/release/32/nteh.o ../generated/linux/release/32/evalu8.o ../generated/linux/release/32/cgcs.o ../generated/linux/release/32/rtlsym.o ../generated/linux/release/32/cgelem.o ../generated/linux/release/32/cgen.o ../generated/linux/release/32/cgreg.o ../generated/linux/release/32/out.o ../generated/linux/release/32/blockopt.o ../generated/linux/release/32/cg.o ../generated/linux/release/32/type.o ../generated/linux/release/32/dt.o ../generated/linux/release/32/debug.o ../generated/linux/release/32/code.o ../generated/linux/release/32/ee.o ../generated/linux/release/32/symbol.o ../generated/linux/release/32/cgcod.o ../generated/linux/release/32/cod5.o ../generated/linux/release/32/outbuf.o ../generated/linux/release/32/compress.o ../generated/linux/release/32/bcomplex.o ../generated/linux/release/32/aa.o ../generated/linux/release/32/ti_achar.o ../generated/linux/release/32/ti_pvoid.o ../generated/linux/release/32/pdata.o ../generated/linux/release/32/cv8.o ../generated/linux/release/32/backconfig.o ../generated/linux/release/32/dwarf.o ../generated/linux/release/32/dwarfeh.o ../generated/linux/release/32/varstats.o ../generated/linux/release/32/ph2.o ../generated/linux/release/32/util2.o ../generated/linux/release/32/tk.o ../generated/linux/release/32/strtold.o ../generated/linux/release/32/md5.o ../generated/linux/release/32/cg87.o ../generated/linux/release/32/cgxmm.o ../generated/linux/release/32/cgsched.o ../generated/linux/release/32/cod1.o ../generated/linux/release/32/cod2.o ../generated/linux/release/32/cod3.o ../generated/linux/release/32/cod4.o ../generated/linux/release/32/ptrntab.o ../generated/linux/release/32/elfobj.o
CC="c++" /home/braddr/sandbox/at-client/release-build/install/linux/bin32/dmd -of../generated/linux/release/32/dmd -m32 -vtls -J../generated/linux/release/32 -J../res -L-lstdc++ -version=MARS  -w -de -O -release -inline dmd/access.d dmd/aggregate.d dmd/aliasthis.d dmd/apply.d dmd/argtypes.d dmd/arrayop.d dmd/arraytypes.d dmd/astcodegen.d dmd/attrib.d dmd/builtin.d dmd/canthrow.d dmd/cli.d dmd/clone.d dmd/compiler.d dmd/complex.d dmd/cond.d dmd/constfold.d dmd/cppmangle.d dmd/cppmanglewin.d dmd/ctfeexpr.d dmd/dcast.d dmd/dclass.d dmd/declaration.d dmd/delegatize.d dmd/denum.d dmd/dimport.d dmd/dinifile.d dmd/dinterpret.d dmd/dmacro.d dmd/dmangle.d dmd/dmodule.d dmd/doc.d dmd/dscope.d dmd/dstruct.d dmd/dsymbol.d dmd/dsymbolsem.d dmd/dtemplate.d dmd/dversion.d dmd/escape.d dmd/expression.d dmd/expressionsem.d dmd/func.d dmd/hdrgen.d dmd/id.d dmd/impcnvtab.d dmd/imphint.d dmd/init.d dmd/initsem.d dmd/inline.d dmd/inlinecost.d dmd/intrange.d dmd/json.d dmd/lambdacomp.d dmd/lib.d dmd/libelf.d dmd/libmach.d dmd/link.d dmd/mars.d dmd/mtype.d dmd/nogc.d dmd/nspace.d dmd/objc.d dmd/opover.d dmd/optimize.d dmd/parse.d dmd/permissivevisitor.d dmd/sapply.d dmd/templateparamsem.d dmd/sideeffect.d dmd/statement.d dmd/staticassert.d dmd/target.d dmd/typesem.d dmd/traits.d dmd/transitivevisitor.d dmd/parsetimevisitor.d dmd/visitor.d dmd/typinf.d dmd/utils.d dmd/scanelf.d dmd/scanmach.d dmd/statement_rewrite_walker.d dmd/statementsem.d dmd/staticcond.d dmd/safe.d dmd/blockexit.d dmd/printast.d dmd/semantic2.d dmd/semantic3.d dmd/irstate.d dmd/toctype.d dmd/glue.d dmd/gluelayer.d dmd/todt.d dmd/tocsym.d dmd/toir.d dmd/dmsc.d dmd/tocvdebug.d dmd/s2ir.d dmd/toobj.d dmd/e2ir.d dmd/eh.d dmd/iasm.d dmd/objc_glue.d dmd/backend/bcomplex.d dmd/backend/cc.d dmd/backend/cdef.d dmd/backend/cgcv.d dmd/backend/code.d dmd/backend/cv4.d dmd/backend/dt.d dmd/backend/el.d dmd/backend/global.d dmd/backend/obj.d dmd/backend/oper.d dmd/backend/outbuf.d dmd/backend/rtlsym.d dmd/backend/code_x86.d dmd/backend/iasm.d dmd/backend/ty.d dmd/backend/type.d dmd/backend/exh.d dmd/backend/mach.d dmd/backend/md5.d dmd/backend/mscoff.d dmd/backend/dwarf.d dmd/backend/dwarf2.d dmd/backend/xmm.d dmd/tk/dlist.d dmd/root/aav.d dmd/root/man.d dmd/root/response.d dmd/root/speller.d ../generated/linux/release/32/newdelete.o ../generated/linux/release/32/backend.a ../generated/linux/release/32/lexer.a
../generated/linux/release/32/cod2.o: In function `cdmul(CodeBuilder&, elem*, unsigned int*)':
cod2.c:(.text+0x2d1b): undefined reference to `choose_multiplier'
cod2.c:(.text+0x2f77): undefined reference to `udiv_coefficients'

It's a mystery why this is failing. It links on my machine. Both names are in divcoeff.o.

@WalterBright WalterBright force-pushed the divcoeff2 branch 2 times, most recently from c979866 to e4c4861 Compare February 15, 2018 07:31
@WalterBright
Copy link
Member Author

@wilzbach can you please have a look at divcoeff.o and see what the symbols in it are?

@wilzbach
Copy link
Contributor

wilzbach commented Mar 22, 2018

Okay, I tried to add the workaround to optionally add -betterC only if the host compiler supports it (in a separate commit).

Regarding the linker error that appears until 2.074:

../generated/linux/release/64/backend.a(divcoeff.o):dmd/backend/divcoeff.d:function u128Div(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, unsigned long*): error: undefined reference to '_D8divcoeff8__assertFiZv'
../generated/linux/release/64/backend.a(divcoeff.o):dmd/backend/divcoeff.d:function choose_multiplier: error: undefined reference to '_D8divcoeff8__assertFiZv'
../generated/linux/release/64/backend.a(divcoeff.o):dmd/backend/divcoeff.d:function choose_multiplier: error: undefined reference to '_D8divcoeff8__assertFiZv'
../generated/linux/release/64/backend.a(divcoeff.o):dmd/backend/divcoeff.d:function choose_multiplier: error: undefined reference to '_D8divcoeff8__assertFiZv'

Looking at the assert mangling:

>cat divcoeff.72.s | grep -C 1 assert
	public	_D8divcoeff7__arrayZ
	public	_D8divcoeff8__assertFiZv
	public	_D8divcoeff15__unittest_failFiZv
--
	extrn	_d_unittest
	extrn	_d_assert
	extrn	__start_deh
--
		mov	EDI,037h
		call	  _D8divcoeff8__assertFiZv@PLT32
L3C:		mov	dword ptr -048h[RBP],1
--
		mov	EDI,078h
		call	  _D8divcoeff8__assertFiZv@PLT32
L38:		cmp	-018h[RBP],EDI
--
		mov	EDI,079h
		call	  _D8divcoeff8__assertFiZv@PLT32
L47:		cmp	RSI,1
--
L59:		mov	EDI,07Ah
		call	  _D8divcoeff8__assertFiZv@PLT32
L63:		mov	dword ptr 0FFFFFF68h[RBP],0
--
L293:		mov	EDI,0D1h
		call	  _D8divcoeff8__assertFiZv@PLT32
L29D:		mov	EDX,0FFFFFF78h[RBP]
--
		mov	EDI,0F6h
		call	  _D8divcoeff8__assertFiZv@PLT32
L85:		jmp short	L91
--
.text._D8divcoeff7__arrayZ	ends
.text._D8divcoeff8__assertFiZv	segment
	assume	CS:.text._D8divcoeff8__assertFiZv
_D8divcoeff8__assertFiZv:
		push	RBP
--
		mov	RDX,-8[RBP]
		call	  _d_assert@PLT32
		nop
.text._D8divcoeff8__assertFiZv	ends
.text._D8divcoeff15__unittest_failFiZv	segment
> cat divcoeff.74.s | grep assert
	public	_D8divcoeff12__ModuleInfoZ
	extrn	_d_assertp
	extrn	__start_minfo
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L43:		mov	dword ptr -048h[RBP],1
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L3F:		cmp	-018h[RBP],EDI
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L55:		cmp	RSI,1
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L78:		mov	dword ptr 0FFFFFF68h[RBP],0
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L2B9:		mov	EDX,0FFFFFF78h[RBP]
--
		lea	RDI,_TMP0@PC32[RIP]
		call	  _d_assertp@PLT32
L8C:		jmp short	L98

Here's a simple idea that avoids the linker error and works both with 2.068 and 2.074.1

diff --git a/src/dmd/backend/divcoeff.d b/src/dmd/backend/divcoeff.d
index 6d4a0a0cf..26b25d57f 100644
--- a/src/dmd/backend/divcoeff.d
+++ b/src/dmd/backend/divcoeff.d
@@ -52,7 +52,7 @@ void u128Div(ullong xh, ullong xl, ullong yh, ullong yl, ullong *pqh, ullong *pq
 
     //ullong xxh = xh, xxl = xl, yyh = yh, yyl = yl;
 
-    assert(yh || yl);           // no div-by-0 bugs
+    dassert(yh || yl);           // no div-by-0 bugs
 
     // left justify y
     uint shiftcount = 1;
@@ -102,6 +102,24 @@ void u128Div(ullong xh, ullong xl, ullong yh, ullong yl, ullong *pqh, ullong *pq
     }
 }
 
+extern(C) void _d_assert_msg(string msg, string file, uint line);
+extern(C) void _d_assert(string file, uint line);
+
+extern(D) void dassert(bool condition, string msg = "", string file = __FILE__, uint line = __LINE__)
+{
+    if (!condition)
+    {
+        if (msg.length)
+        {
+            _d_assert_msg(msg, file, line);
+        }
+        else
+        {
+            _d_assert(file, line);
+        }
+    }
+}
+
 /************************************
  * Implement Algorithm 6.2: Selection of multiplier and shift count
  * Params:
@@ -117,9 +135,9 @@ void u128Div(ullong xh, ullong xl, ullong yh, ullong yl, ullong *pqh, ullong *pq
 
 extern (C) bool choose_multiplier(int N, ullong d, int prec, ullong *pm, int *pshpost)
 {
-    assert(N == 32 || N == 64);
-    assert(prec <= N);
-    assert(d > 1 && (d & (d - 1)));
+    dassert(N == 32 || N == 64);
+    dassert(prec <= N);
+    dassert(d > 1 && (d & (d - 1)));
 
     // Compute b such that 2**(b-1) < d <= 2**b
     // which is the number of significant bits in d
@@ -206,7 +224,7 @@ extern (C) bool choose_multiplier(int N, ullong d, int prec, ullong *pm, int *ps
         mhighbit = mhighh & 1;
     }
     else
-        assert(0);
+        dassert(0);
 
     *pshpost = shpost;
     return mhighbit;
@@ -243,7 +261,7 @@ extern (C) bool udiv_coefficients(int N, ullong d, int *pshpre, ullong *pm, int
         }
         *pshpre = e;
         mhighbit = choose_multiplier(N, d, N - e, pm, pshpost);
-        assert(mhighbit == false);
+        dassert(mhighbit == false);
     }
     else
         *pshpre = 0;

@wilzbach
Copy link
Contributor

posix.mak: add dependency on the bootstrap compiler to the backend 0516631

This fixes Jenkins.

Experiment with setting -betterC only on newer build hosts be61d60

This fixes SemaphoreCI.

The remaining CIs (auto-tester + DAutoTest) fail due to mentioned assert linker errors and could be fixed by the mentioned diff / hack.

The AppVeyor build failure is probably due to divcoeff.c still being there:

1>c1xx : fatal error C1083: Cannot open source file: '..\dmd\backend\divcoeff.c': No such file or directory
...
2>LINK : fatal error LNK1181: cannot open input file 
'C:\projects\dmd\generated\Windows\Release\Win32\dmd_backend.lib'

Maybe 2fe4d85 fixes this.

@wilzbach
Copy link
Contributor

Here's a simple idea that avoids the linker error and works both with 2.068 and 2.074.1

Oh looking at auto-tester the linker errors between Linux and FreeBSD seem different. Maybe the root case is the same.
FYI: I pushed the patch as an additional commit just to see whether they have the same root cause or a different one.

@WalterBright
Copy link
Member Author

The FreeBSD linker errors are:

dmd/backend/cod2.c:(.text+0x226e): undefined reference to `choose_multiplier'
dmd/backend/cod2.c:(.text+0x2505): undefined reference to `udiv_coefficients'

Since those are C mangled, how are they different on FreeBSD?

@wilzbach wilzbach closed this Mar 24, 2018
@wilzbach wilzbach reopened this Mar 24, 2018
@wilzbach
Copy link
Contributor

The AppVeyor build failure is probably due to divcoeff.c still being there:
Maybe 2fe4d85 fixes this.

Well, at least the good news is that the FreeBSD errors are the last CI errors and all other CI seem to be happy (though I'm not sure why SemaphoreCI isn't showing up here - it used to).

Since those are C mangled, how are they different on FreeBSD?

I have no idea as well. Might it be related to this:

ld: GNU ld 2.17.50 [FreeBSD] 2007-07-03
HOST_CXX(c++): FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) 

Maybe the old C++ compiler on FreeBSD used to have a different mangling scheme?
I just pushed another commit to test whether the C++ mangling would work and test whether it's really a problem with the C mangling.

@wilzbach
Copy link
Contributor

Well, at least the good news is that the FreeBSD errors are the last CI errors

This might be a lot faster/easier for someone familiar with FreeBSD.
Maybe @dkgroot or @jmdavis know something about the linker error?

@jacob-carlborg
Copy link
Contributor

Maybe the old C++ compiler on FreeBSD used to have a different mangling scheme?
I just pushed another commit to test whether the C++ mangling would work

Now it fails on Linux and macOS as well.

and test whether it's really a problem with the C mangling.

C doesn't have a mangling.

@andralex
Copy link
Member

Quite coincidentally I was planning to post in the getopt newsgroup thread that --start-group and --end-group are good examples of command line processing where the order of flags is relevant.

@WalterBright WalterBright force-pushed the divcoeff2 branch 3 times, most recently from 5e1af26 to 6bc0c42 Compare March 30, 2018 19:33
@jmdavis
Copy link
Member

jmdavis commented Mar 30, 2018

Not sure if it is still relevant but when I worked with linux a long time ago, I've been fighting with ld about the order of libraries given on the command line as it wouldn't look up symbols referenced in a library within libraries or object files specified before on the command line. Maybe FreeBSD is still using such a linker?

As I understand it, FreeBSD is still using GNU's ld as its default linker (even in FreeBSD CURRENT). They're moving towards switching to using clang's as the default, but they're still working out the issues caused by that switch.

@WalterBright
Copy link
Member Author

This is failing on all autotester Posix platforms, not just FreeBSD. It's also not an ordering issue, because there should be only one divcoeff.o.

@wilzbach
Copy link
Contributor

Another idea I had is that it's maybe related to ENABLE_RELEASE=1 which is set here for auto-tester:

dmd/posix.mak

Lines 10 to 11 in c83e4b5

auto-tester-build: toolchain-info
$(QUIET)$(MAKE) -C src -f posix.mak auto-tester-build ENABLE_RELEASE=1

But it doesn't make a difference for me locally :/

FYI: the build script that is supposedly used by auto-tester is here: https://github.com/braddr/at-client/blob/master/src/do_build_dmd.sh
(EXTRA_ARGS is set here: https://github.com/braddr/at-client/blob/master/src/setup_env.sh) - though again nothing stands out there ...

@braddr
Copy link
Member

braddr commented Mar 30, 2018

Is no one able (I haven't tried yet) to reproduce this outside the tester? If it's failing as reliably and widely as indicated that shouldn't be difficult. Why are you working so hard to do this through the tester when it's almost certainly NOT the tester and easily reproduced outside it?

@dnadlinger
Copy link
Contributor

Reproducing it outside the tester isn't really required; the fact that GNU ld (apart from --start-group/--end-group and so on) resolves symbols in the order given on the command line is well-known (although it seems to have slipped Walter's mind), and not respecting that is asking for trouble either way.

The simplest solution is to add divcoeff.o to backend.a.

@wilzbach
Copy link
Contributor

wilzbach commented Mar 30, 2018

Is no one able (I haven't tried yet) to reproduce this outside the tester?

@braddr nope. I even tried all LTSes of Ubuntu. Here's an example with the oldest that's not EOL yet:

docker run --rm -ti ubuntu:14.04
apt-get update && apt-get install curl g++ make git unzip
git clone https://github.com/WalterBright/dmd
cd dmd
sed -i "s/2.074.1/2.068.2/" -i src/posix.mak
git checkout divcoeff2
make -f posix.mak AUTO_BOOTSTRAP=1

Compiles flawlessly

> g++ --version
g++ (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
> ldd --version
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.14) 2.19

@WalterBright
Copy link
Member Author

Reproducing it outside the tester isn't really required; the fact that GNU ld (apart from --start-group/--end-group and so on) resolves symbols in the order given on the command line is well-known (although it seems to have slipped Walter's mind),

Order has nothing to do with this problem.

The simplest solution is to add divcoeff.o to backend.a.

That's how it started out. With it in, out, explicitly before or after backend.a makes no difference to the error.

@WalterBright
Copy link
Member Author

Let's see what the state of this is:

  • fails on all autotester posix systems
  • works on other posix systems (like circleci)
  • works on my machine (even using dmd 2.068)
  • objdump on divcoeff.o shows the correct symbols are in there
  • nm on backend.a shows the correct symbols are in there and in divcoeff.o
  • linking divcoeff.o with test.c works
  • I checked the logs to see if divcoeff.c was being compiled instead, another dead end

What it smells like to me is there's another divcoeff.o on the autotester somewhere that it is picking up. Perhaps the make clean is not working?

Anyhow, this is proving impossible to debug remotely, and cannot be reproduced outside of the autotester.

@dnadlinger
Copy link
Contributor

Order has nothing to do with this problem.

Ah, apologies, I must have jumped a line when looking at the old logs, and GitHub helpfully hid the earlier discussion. From experience, getting the order wrong would cause exactly that sort of hard-to-debug problem that only appears on certain linker versions when the stars align just the wrong way, though.

@braddr
Copy link
Member

braddr commented Mar 31, 2018

Did any of you attempt building the target that the auto-tester runs:

make MODEL=64 -f posix.mak auto-tester-build

Fails for me quite easily and reproducible. And, notably, NOT specifying that target (going with whatever is the default) doesn't. A few minutes trying each of the targets in that dependency list and... it's cxx_unittest.

@WalterBright
Copy link
Member Author

Thanks for taking a look at this, @braddr ! But I'm not sure I'm understanding your results - is the cxx_unittest the problem? While you're at it, are you able to see the source of the problem?

@rainers
Copy link
Member

rainers commented Mar 31, 2018

is the cxx_unittest the problem?

Yes. The ouput from the parallel jobs makes it a bit confusing, but it's displayed in this line

gmake[1]: *** [posix.mak:605: ../generated/freebsd/release/32/cxx-unittest] Error 1

While you're at it, are you able to see the source of the problem?

The command line is just missing divcoeff.o. Either add $(G_DOBJS) as a dependency to $G/cxx-unittest or use backend.a instead of all object files explicitly.

@WalterBright
Copy link
Member Author

@rainers thanks! Trying it now...

@@ -1,258 +1,257 @@
<?xml version="1.0" encoding="utf-8"?>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect you have converted newlines in the VS projects.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean that vs projects have to have CRLF line endings? Sheesh.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's mandatory, but the next one who saves it from within VS will change the whole file again.

Copy link
Contributor

@wilzbach wilzbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Jenkins failure is unrelated (see dlang/ci#190), so feel free to pull anytime you are ready Walter ;-)

//ullong xxh = xh, xxl = xl, yyh = yh, yyl = yl;

assert(yh || yl); // no div-by-0 bugs
// assert(yh || yl); // no div-by-0 bugs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the dassert workaround for DAutoTest I proposed here?
#7714 (comment)
If the body is version(assert) it should behave like built-in assert? Alternatively it's really just 2.067 that needs special casing.

Or maybe @CyberShadow can update DAutoTest to use 2.068.2 as initial bootstrap compiler?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to redo the asserts, as that doesn't scale to the rest of the back end. What's needed is to upgrade the boot compiler to 2.074. But that's for after this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as that doesn't scale to the rest of the back end.

Well, I thought it helps porting when the assert's are still enabled, but yeah this can be done in another PR

$G/backend.a: $(G_OBJS) $(SRC_MAKE)
$(AR) rcs $@ $(G_OBJS)
$G/backend.a: $(G_OBJS) $(G_DOBJS) $(SRC_MAKE)
$(HOST_DMD_RUN) -lib -of$G/backend.a $(G_OBJS)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove this line, now?

$(AR) rcs $@ $(G_OBJS)
$G/backend.a: $(G_OBJS) $(G_DOBJS) $(SRC_MAKE)
$(HOST_DMD_RUN) -lib -of$G/backend.a $(G_OBJS)
$(HOST_DMD_RUN) -lib -of$G/backend.a $@ $(G_DOBJS)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that the problems has been identified this could be reverted to $(AR) rcs $@ $(G_OBJS)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use ar? dmd can create libraries just fine. I put it on 2 lines as the log file was cutting off the end of the line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a good reason for using $(AR) - gdmd doesn't support it.
-> #8111

@WalterBright
Copy link
Member Author

Now that it's working, how about getting this pulled?

@wilzbach wilzbach merged commit 62110cd into dlang:master Apr 2, 2018
@wilzbach
Copy link
Contributor

wilzbach commented Apr 2, 2018

So let's see what CIs break now ...

@WalterBright WalterBright deleted the divcoeff2 branch April 2, 2018 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants