Cross-compilation issues with -O2 due to alignment optimization

[ks156]

I use exqlite as a dependency in several Elixir projects that I cross-compile in a Yocto environment.

I’ve noticed crashes on the target related to sqlite3_nif.so. My investigations pointed to alignment issues.

I fixed the problem by changing, in the Makefile (line 41), the optimization flag from -O2 to -Os in order to disable alignment optimizations (see https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html).

Has anyone else run into similar issues when cross-compiling this library? Would it be possible to consider changing the default optimization on line 41 to -Os, or to add a branch in the Makefile’s ifneq ($(CROSSCOMPILE),) section that set the CFLAGS to -Os when cross-compiling for Linux systems?

---

Last lines of strace :

pselect6_time64(0, NULL, NULL, NULL, NULL, NULL <unfinished ...>) = ?
+++ killed by SIGBUS (core dumped) +++
Bus error (core dumped)

GDB debug :

#0  sqlite3WhereBegin (pParse=pParse@entry=0xb2ffe234, pTabList=pTabList@entry=0xb2a4b62c, pWhere=pWhere@entry=0x0, pOrderBy=pOrderBy@entry=0xb2a4b42c, pResultSet=0xb2a47f9c, pSelect=pSelect@entry=0xb2a4b3ac, wctrlFlags=wctrlFlags@entry=0, iAuxArg=320) at c_src/sqlite3.c:170142
#1  0xb056b50a in sqlite3Select (pParse=pParse@entry=0xb2ffe234, p=<optimized out>, pDest=pDest@entry=0xb2ffdca0) at c_src/sqlite3.c:151935
#2  0xb058b484 in yy_reduce (yypParser=yypParser@entry=0xb2ffdd34, yyruleno=84, yyLookaheadToken=..., pParse=pParse@entry=0xb2ffe234, yyLookahead=<optimized out>) at c_src/sqlite3.c:178174
#3  0xb058d036 in sqlite3Parser (yyminor=..., yymajor=<optimized out>, yyp=0xb2ffdd34) at c_src/sqlite3.c:179625
#4  sqlite3RunParser (pParse=pParse@entry=0xb2ffe234, zSql=<optimized out>, zSql@entry=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid") at c_src/sqlite3.c:180962
#5  0xb058d714 in sqlite3Prepare (db=db@entry=0xb6141184, zSql=zSql@entry=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid", nBytes=nBytes@entry=-1, prepFlags=prepFlags@entry=128, pReprepare=pReprepare@entry=0x0, ppStmt=ppStmt@entry=0xb2ffe3f8, pzTail=pzTail@entry=0xb2ffe3f4)
    at c_src/sqlite3.c:143541
#6  0xb058d97a in sqlite3LockAndPrepare (db=db@entry=0xb6141184, zSql=zSql@entry=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid", nBytes=nBytes@entry=-1, prepFlags=prepFlags@entry=128, pOld=pOld@entry=0x0, ppStmt=ppStmt@entry=0xb2ffe3f8, pzTail=pzTail@entry=0xb2ffe3f4)
    at c_src/sqlite3.c:143616
#7  0xb057fb1c in sqlite3_prepare_v2 (pzTail=0xb2ffe3f4, ppStmt=0xb2ffe3f8, nBytes=-1, zSql=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid", db=0xb6141184) at c_src/sqlite3.c:143703
#8  sqlite3_exec (db=db@entry=0xb6141184, zSql=zSql@entry=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid", xCallback=0xb0592215 <sqlite3InitCallback>, pArg=pArg@entry=0xb2ffe440, pzErrMsg=pzErrMsg@entry=0x0) at c_src/sqlite3.c:137279
#9  0xb0580192 in sqlite3InitOne (db=0xb6141184, iDb=0, pzErrMsg=pzErrMsg@entry=0xb2ffeab0, mFlags=<optimized out>) at c_src/sqlite3.c:143129
#10 0xb0580582 in sqlite3Init (db=db@entry=0xb6141184, pzErrMsg=pzErrMsg@entry=0xb2ffeab0) at c_src/sqlite3.c:143204
#11 0xb05805a0 in sqlite3ReadSchema (pParse=pParse@entry=0xb2ffeaac) at c_src/sqlite3.c:143230
#12 0xb058b478 in yy_reduce (yypParser=yypParser@entry=0xb2ffe5ac, yyruleno=84, yyLookaheadToken=..., pParse=pParse@entry=0xb2ffeaac, yyLookahead=<optimized out>) at c_src/sqlite3.c:178172
#13 0xb058d036 in sqlite3Parser (yyminor=..., yymajor=<optimized out>, yyp=0xb2ffe5ac) at c_src/sqlite3.c:179625
#14 sqlite3RunParser (pParse=pParse@entry=0xb2ffeaac, zSql=<optimized out>, zSql@entry=0xb6e000e4 "SELECT max(iteration) from store") at c_src/sqlite3.c:180962
#15 0xb058d714 in sqlite3Prepare (db=db@entry=0xb6141184, zSql=zSql@entry=0xb6e000e4 "SELECT max(iteration) from store", nBytes=nBytes@entry=33, prepFlags=prepFlags@entry=128, pReprepare=pReprepare@entry=0x0, ppStmt=ppStmt@entry=0xb1f80184, pzTail=pzTail@entry=0x0)
    at c_src/sqlite3.c:143541
#16 0xb058d97a in sqlite3LockAndPrepare (db=0xb6141184, zSql=0xb6e000e4 "SELECT max(iteration) from store", nBytes=33, prepFlags=prepFlags@entry=128, pOld=pOld@entry=0x0, ppStmt=ppStmt@entry=0xb1f80184, pzTail=pzTail@entry=0x0) at c_src/sqlite3.c:143616
#17 0xb058e0b8 in sqlite3_prepare_v3 (db=<optimized out>, zSql=<optimized out>, nBytes=<optimized out>, prepFlags=prepFlags@entry=0, ppStmt=ppStmt@entry=0xb1f80184, pzTail=pzTail@entry=0x0) at c_src/sqlite3.c:143724
#18 0xb04f34ac in exqlite_prepare (env=0xb2ffecc0, argc=2, argv=<optimized out>) at c_src/sqlite3_nif.c:513
#19 0x00576dbe in erts_call_dirty_nif ()
#20 0x00461efc in erts_dirty_process_main ()
#21 0x0043bd5a in _start ()


Thread 20 "erts_dios_8" received signal SIGBUS, Bus error.
[Switching to Thread 0xb2fff420 (LWP 29455)]
sqlite3WhereBegin (pParse=pParse@entry=0xb2ffe234, pTabList=pTabList@entry=0xb2a4b62c, pWhere=pWhere@entry=0x0, pOrderBy=pOrderBy@entry=0xb2a4b42c, pResultSet=0xb2a47f9c, pSelect=pSelect@entry=0xb2a4b3ac, wctrlFlags=wctrlFlags@entry=0, iAuxArg=320) at c_src/sqlite3.c:170142
170142	  pWInfo->aiCurOnePass[0] = pWInfo->aiCurOnePass[1] = -1;

0  sqlite3WhereBegin (pParse=pParse@entry=0xb2ffe234, pTabList=pTabList@entry=0xb2a4b62c, pWhere=pWhere@entry=0x0, pOrderBy=pOrderBy@entry=0xb2a4b42c, pResultSet=0xb2a47f9c, pSelect=pSelect@entry=0xb2a4b3ac, wctrlFlags=wctrlFlags@entry=0, iAuxArg=320) at c_src/sqlite3.c:170142
        nByteWInfo = <optimized out>
        nTabList = <optimized out>
        pWInfo = 0xb2a4763c
        v = 0xb2a488fc
        notReady = <optimized out>
        sWLB = {pWInfo = 0x0, pWC = 0x0, pNew = 0x0, pOrSet = 0x0, bldFlags1 = 0 '\000', bldFlags2 = 0 '\000', iPlanLimit = 0}
        pMaskSet = <optimized out>
        pLevel = <optimized out>
        pLoop = <optimized out>
        ii = <optimized out>
        db = 0xb6141184
        rc = <optimized out>
        bFordelete = 0 '\000'
        whereBeginError = <optimized out>
#1  0xb056b50a in sqlite3Select (pParse=pParse@entry=0xb2ffe234, p=<optimized out>, pDest=pDest@entry=0xb2ffdca0) at c_src/sqlite3.c:151935
        wctrlFlags = 0
        pWin = 0x0
        i = <optimized out>
        j = <optimized out>
        pWInfo = <optimized out>
        v = <optimized out>
        isAgg = <optimized out>
        pEList = 0xb2a47f9c
        pTabList = <optimized out>
        pWhere = 0x0
        pGroupBy = <optimized out>
        pHaving = <optimized out>
        pAggInfo = 0x0
        rc = 1
        sDistinct = {isTnct = 0 '\000', eTnctType = 0 '\000', tabTnct = 1, addrTnct = 6983360}
        sSort = {pOrderBy = 0xb2a4b42c, nOBSat = 0, iECursor = 1, regReturn = 0, labelBkOut = 0, addrSortIndex = 1, labelDone = 0, labelOBLopt = 0, sortFlags = 1 '\001', pDeferredRowLoad = 0x0}
        iEnd = -1
        db = 0xb6141184
        pMinMaxOrderBy = 0x0
        minMaxFlag = <optimized out>
#2  0xb058b484 in yy_reduce (yypParser=yypParser@entry=0xb2ffdd34, yyruleno=84, yyLookaheadToken=..., pParse=pParse@entry=0xb2ffe234, yyLookahead=<optimized out>) at c_src/sqlite3.c:178174
        dest = {eDest = 9 '\t', iSDParm = 0, iSDParm2 = 0, iSdst = 0, nSdst = 0, zAffSdst = 0x0, pOrderBy = 0x0}
        yylhsminor = <optimized out>
        yygoto = <optimized out>
        yyact = <optimized out>
        yymsp = 0xb2ffdd50
--Type <RET> for more, q to quit, c to continue without paging--
        yysize = <optimized out>
#3  0xb058d036 in sqlite3Parser (yyminor=..., yymajor=<optimized out>, yyp=0xb2ffdd34) at c_src/sqlite3.c:179625
        yyruleno = <optimized out>
        yyminorunion = {yyinit = 0, yy0 = {z = 0x0, n = 0}, yy9 = 0, yy28 = {a = 0, b = 0x0}, yy125 = 0x0, yy204 = 0x0, yy205 = {eType = 0, pExpr = 0x0}, yy319 = 0x0, yy342 = 0x0, yy361 = 0x0, yy402 = 0x0, yy403 = 0x0, yy421 = {pOn = 0x0, pUsing = 0x0}, yy444 = 0 '\000', yy481 = {
            value = 0, mask = 0}, yy483 = 0x0, yy502 = 0, yy563 = 0x0, yy590 = 0x0, yy637 = 0x0}
        yyact = <optimized out>
        yypParser = 0xb2ffdd34
        pParse = 0xb2ffe234
        yyminorunion = <optimized out>
        yyact = <optimized out>
        yypParser = <optimized out>
        pParse = <optimized out>

        yyruleno = <optimized out>
#4  sqlite3RunParser (pParse=pParse@entry=0xb2ffe234, zSql=<optimized out>, zSql@entry=0xb2a4b7ac "SELECT*FROM\"main\".sqlite_master ORDER BY rowid") at c_src/sqlite3.c:180962
        nErr = 0
        pEngine = 0xb2ffdd34
        n = <optimized out>
        tokenType = 1
        lastTokenParsed = <optimized out>
        db = <optimized out>
        mxSqlLen = 999999954
        pParentParse = 0xb2ffe234

(gdb) print *pWInfo
$1 = {pParse = 0xb2a4718c, pTabList = 0x0, pOrderBy = 0x0, pResultSet = 0x0, pSelect = 0x0, aiCurOnePass = {0, 0}, iContinue = 0, iBreak = 0, savedNQueryLoop = 0, wctrlFlags = 0, iLimit = 0, nLevel = 0 '\000', nOBSat = 0 '\000', eOnePass = 0 '\000', eDistinct = 0 '\000',
  bDeferredSeek = 0, untestedTerms = 0, bOrderedInnerLoop = 0, sorted = 0, bStarDone = 0, bStarUsed = 0, nRowOut = 0, iTop = 0, iEndWhere = 0, pLoops = 0x0, pMemToFree = 0x0, revMask = 0, sWC = {pWInfo = 0x0, pOuter = 0x0, op = 0 '\000', hasOr = 0 '\000', nTerm = 0, nSlot = 0,
    nBase = 0, a = 0x0, aStatic = {{pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {
        pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0,
        wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0,
        nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000',
        eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0,
        leftCursor = 0, u = {x = {leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {
            leftColumn = 0, iField = 0}, pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}, {pExpr = 0x0, pWC = 0x0, truthProb = 0, wtFlags = 0, eOperator = 0, nChild = 0 '\000', eMatchOp = 0 '\000', iParent = 0, leftCursor = 0, u = {x = {leftColumn = 0, iField = 0},
          pOrInfo = 0x0, pAndInfo = 0x0}, prereqRight = 0, prereqAll = 0}}}, sMaskSet = {bVarSelect = 0, n = 0, ix = {0 <repeats 64 times>}}, a = {{iLeftJoin = 0, iTabCur = 0, iIdxCur = 0, addrBrk = 0, addrNxt = 0, addrSkip = 0, addrCont = 0, addrFirst = 0, addrBody = 0, regBignull = 0,
      addrBignull = 0, regFilter = 0, pRJ = 0x0, iFrom = 0 '\000', op = 0 '\000', p3 = 0 '\000', p5 = 0 '\000', p1 = 0, p2 = 0, u = {in = {nIn = 0, aInLoop = 0x0}, pCoveringIdx = 0x0}, pWLoop = 0x0, notReady = 0}}}

[warmwaffles]

@ks156 is there any way you could use the precompiled binary and see if this bug exists there too? This may be something deeper.

[warmwaffles]

Has anyone else run into similar issues when cross-compiling this library? Would it be possible to consider changing the default optimization on line 41 to -Os, or to add a branch in the Makefile’s ifneq ($(CROSSCOMPILE),) section that set the CFLAGS to -Os when cross-compiling for Linux systems?

I'm open to a PR for this.

[ks156]

@ks156 is there any way you could use the precompiled binary and see if this bug exists there too? This may be something deeper.

I also suspect that there may be an issue deeper than mere alignment optimizations, so before submitting a PR I’ll test a precompiled binary. I’ll download and try a version compatible with the target system (ARM Cortex-A7) and keep you informed.

[warmwaffles]

Thank you.

[ks156]

I’ve tested with the precompiled library (exqlite-nif-2.17-armv7l-linux-gnueabihf-0.29.0), and it works fine.

I can open a PR to adjust the optimization when CROSSCOMPILE contains linux if you’d like. However, I’d prefer not to change the default optimization here.

Moreover, I don’t think that part is quite right either, since you’d end up with two different optimization levels when cross-compiling on Android. I’m not sure how it would behave in that case. Instead, I’d suggest filtering out the -O2 flag before appending -Os to the CFLAGS.

If you agree, here’s the patch I’m proposing:

diff --git a/Makefile b/Makefile
index 9ffc70a..1ecfac4 100644
--- a/Makefile
+++ b/Makefile
@@ -52,8 +52,13 @@ OBJ = $(SRC:c_src/%.c=$(BUILD)/%.o)

 ifneq ($(CROSSCOMPILE),)
        ifeq ($(CROSSCOMPILE), Android)
+               CFLAGS:=$(filter-out -O2,$(CFLAGS))
                CFLAGS += -fPIC -Os -z global
                LDFLAGS += -fPIC -shared -lm
+       else ifeq ($(findstring linux,$(CROSSCOMPILE)),linux)
+               CFLAGS:=$(filter-out -O2,$(CFLAGS))
+               CFLAGS += -fPIC -Os -fvisibility=hidden
+               LDFLAGS += -fPIC -shared
        else
                CFLAGS += -fPIC -fvisibility=hidden
                LDFLAGS += -fPIC -shared

[ks156]

By the way, the issue could also be related to the GCC version.

I was previously using an older Yocto release (Dunfell) with GCC 9.3.0 and never encountered this problem. Now I’m compiling the library with GCC 13.3.0 (Scarthgap), and that's how I saw the problem. The yocto recipe has not changed and I'm compiling my project the same way on dunfell and scarthgap.

Which GCC version does the CI use? Any idea?

[warmwaffles]

Here's the deps that are installed from ubuntu-latest

exqlite/.github/workflows/linux-precompile.yml

Lines 99 to 105 in 4a3c718

	- name: Install system dependecies
	run: |
	alias sudo=`which sudo`
	$sudo apt-get update
	$sudo apt-get install -y \
	build-essential automake autoconf pkg-config wget curl \
	bc m4 unzip zip gcc g++ ca-certificates libssl-dev

So it's what ever is upstream. I don't pin the GCC version.

[ks156]

Here's the deps that are installed from ubuntu-latest

exqlite/.github/workflows/linux-precompile.yml

Lines 99 to 105 in 4a3c718
- name: Install system dependecies
run: |
alias sudo=which sudo
$sudo apt-get update
$sudo apt-get install -y \
build-essential automake autoconf pkg-config wget curl \
bc m4 unzip zip gcc g++ ca-certificates libssl-dev

So it's what ever is upstream. I don't pin the GCC version.

I'm not familiar with the github pipelines, but I understand here that the container is ubuntu 20.04, so it probably use an old version of GCC (seems to be gcc 9.4.0)

[warmwaffles]

Fix is live under v0.32.1, can you verify it resolves your issue?

[ks156]

Fix is live under v0.32.1, can you verify it resolves your issue?

Thanks. I'll remove my local patches and update the version of exqlite in my projects, and let you know. I'll test it today

[ks156]

Fix is live under v0.32.1, can you verify it resolves your issue?

Thanks. I'll remove my local patches and update the version of exqlite in my projects, and let you know. I'll test it today

I removed my local patches, updated exqlite to 0.32.1, and recompiled everything in my cross-compilation environment. After deploying to my targets, everything is working fine now. Thank you.

I also tested the precompiled exqlite-nif-2.17-armv7l-linux-gnueabihf-0.32.1 library, and I can confirm that it still works.

This issue can be closed.

[warmwaffles]

Thanks for your help @ks156. If you encounter anything else, please open an issue or submit a PR with a fix 😄