Skip to content

Commit

Permalink
CI: simplify sqlite3.so fix for PyPy
Browse files Browse the repository at this point in the history
Instead of forcing the use of OS provided libsqlite3.so through
LD_PRELOAD, we now just delete PyPy's copy of the library. This is based
on a suggestion by Armin Rigo in the following StackOverflow discussion:
https://stackoverflow.com/questions/65476852/
  • Loading branch information
tomka committed Jan 1, 2021
1 parent c0897e5 commit ba82681
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 21 deletions.
3 changes: 1 addition & 2 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ jobs:
- run: pip install -r django/requirements-test.txt coveralls
- run: bash scripts/travis/setup_database.sh
- run: bash scripts/travis/configure_catmaid.sh
- run: bash scripts/travis/configure_pypy_libs.sh
- run: |
cd django/projects
python manage.py migrate --noinput
Expand All @@ -65,8 +66,6 @@ jobs:
python manage.py makemigrations catmaid
name: Check migrations are up to date
- run: |
source scripts/travis/configure_pypy_libs.sh
if_pypy_force_os_sqlite_lib
cd django/projects
coverage run manage.py test catmaid.tests
name: Run tests
Expand Down
3 changes: 1 addition & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,8 @@ before_install:
- bash scripts/travis/install_postgres.sh
- bash scripts/travis/install_python.sh
- bash scripts/travis/install_gdal.sh
- source scripts/travis/configure_pypy_libs.sh
- bash scripts/travis/configure_pypy_libs.sh
- nvm install 11
- if_pypy_force_os_sqlite_lib
install:
# Install requirements for running CATMAID and its unit tests
- ./scripts/travis/install_requirements.sh
Expand Down
37 changes: 20 additions & 17 deletions scripts/travis/configure_pypy_libs.sh
Original file line number Diff line number Diff line change
@@ -1,23 +1,26 @@
#!/usr/bin/env bash
#
# Prepare environment to run Django with PyPy.

# Unforunately needed for PyPy at the moment, because it comes with its own
# libsqlite3.so.0, which doesn't have column metadata compiled in, which in turn
# is needed by GDAL and Django. This forces the use of the libsqlite3 library
# provided by the operationg system. This fixes running CATMAID's back-end test
# suit with PyPy on both Travis CI and GH Actions, which failed before due to
# the following error:
# Prepare environment to run Django with PyPy. This effectively means removing
# any copy of the shared library libsqlite3.so that comes with the existing PyPy
# installations.
#
# This is unforunately needed for PyPy at the moment, because it comes with its
# own libsqlite3.so.0 on both Travis and GitHub Actions, which doesn't have
# column metadata compiled in, which in turn is needed by GDAL and Django. In
# order to make PyPy use the OS provided libsqlite3.so, we delete PyPy's copy.
# This fixes running CATMAID's back-end test suit with PyPy on both Travis CI
# and GH Actions, which failed before due to the following error:
#
# OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name
#
# Since the OS level library is compiled with SQLITE_ENABLE_COLUMN_METADATA=1,
# this is the library we want to use.
if_pypy_force_os_sqlite_lib() {
if python --version | grep PyPy ; then
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
echo "PyPy detected: override PyPy's own copy of libsqlite3.so with OS provided version"
else
echo "Not running PyPy, no library override needed"
fi
}
# this is the library we want to use. More information can also be found here:
#
# https://stackoverflow.com/questions/65476852/

set -ex

# Delete all files that have libsqlite3.so in their path name as well as PyPy
# (regardless of the case of the P).
echo 'Attempting to delete all PyPy copies of libsqlite3.so in order to use OS copy'
sudo time find / -path /proc -prune -o -regex '.*[Pp]y[Pp]y.*libsqlite3\.so.*' -type f -exec rm -f {} +

0 comments on commit ba82681

Please sign in to comment.