v2.8.0: DataMan 1D.Serial Test Fails (GCC, No-MPI build)

[ax3l]

Describe the bug
I am trying to deploy ADIOS 2.8.0 via conda-forge.

A DataMan test fails for Linux no-MPI builds: conda-forge/adios2-feedstock#44

To Reproduce
Please see link.

        Start 571: Engine.DataMan.DataManEngineTest.1D.Serial
571/695 Test #571: Engine.DataMan.DataManEngineTest.1D.Serial ..............................................................***Failed    0.40 sec
Note: Google Test filter = DataManEngineTest.1D
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from DataManEngineTest
[ RUN      ] DataManEngineTest.1D
$SRC_DIR/testing/adios2/engine/dataman/TestDataMan1D.cpp:243: Failure
Expected equality of these values:
  readString
    Which is: "\x10\x19MsM\x7F\0\0\t\0\0\0\0\0\0\0some text\0\0\0\0\0\0\0"
  "some text"
[  FAILED  ] DataManEngineTest.1D (385 ms)
[----------] 1 test from DataManEngineTest (385 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (385 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DataManEngineTest.1D

 1 FAILED TEST

Expected behavior
Pass all tests on a limited resource like CI (<= 2 cores).

Desktop (please complete the following information):

• OS/Platform: 20.04.4 on Azure Pipelines via conda-forge
• Build
  • GCC 10.3.0
  • CMake 3.23.0
  • No-MPI build

Additional context
Conda-Forge deployment (WIP): conda-forge/adios2-feedstock#44

CC DataMan component author @JasonRuonanWang
CC release manager @chuckatkins

[JasonRuonanWang]

I have tried everything I could think of but still cannot reproduce this on any of my machines. Any suggestions?

[ax3l]

Thanks! Did you build with -DADIOS2_USE_MPI=OFF?

[ax3l]

The ZeroMQ version on conda-forge is 4.3.4 in those failing tests, and it seems to work well when MPI is on (both with OpenMPI and MPICH).

[ax3l]

I can reproduce the crashing test on my Ubuntu 20.04 LTS with these commands:

conda create -y -n adios-2.8.0-dev -c conda-forge python numpy gnuconfig compilers make pkg-config cmake flex bison mpi4py libffi hdf5 zeromq bzip2 blosc libpng zfp
source activate adios-2.8.0-dev

Collecting package metadata (current_repodata.json): done Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.11.0
latest version: 4.12.0

Please update conda by running

$ conda update -n base conda

Package Plan

environment location: /home/axel/sw/miniconda3/envs/adios-2.8.0-dev

added / updated specs:
- bison
- blosc
- bzip2
- cmake
- compilers
- flex
- gnuconfig
- hdf5
- libffi
- libpng
- make
- mpi4py
- numpy
- pkg-config
- python
- zeromq
- zfp

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
binutils_linux-64-2.36     |       hf3e587d_9          24 KB  conda-forge
bison-3.8                  |       h9c3ff4c_0         818 KB  conda-forge
c-compiler-1.4.1           |       h166bdaf_0           5 KB  conda-forge
cmake-3.23.0               |       h5432695_1        16.3 MB  conda-forge
compilers-1.4.1            |       ha770c72_0           5 KB  conda-forge
cxx-compiler-1.4.1         |       h924138e_0           5 KB  conda-forge
expat-2.4.8                |       h27087fc_0         187 KB  conda-forge
flex-2.6.4                 |    h58526e2_1004         330 KB  conda-forge
fortran-compiler-1.4.1     |       h2a4ca65_0           5 KB  conda-forge
gcc-10.3.0                 |       he2824d0_9          24 KB  conda-forge
gcc_impl_linux-64-10.3.0   |      hf2f2afa_14        46.6 MB  conda-forge
gcc_linux-64-10.3.0        |       hc39de41_9          25 KB  conda-forge
gfortran-10.3.0            |       h18518b4_9          24 KB  conda-forge
gfortran_impl_linux-64-10.3.0|      h73f4979_14        14.7 MB  conda-forge
gfortran_linux-64-10.3.0   |       hb09a455_9          24 KB  conda-forge
gnuconfig-2020.11.07       |       hd8ed1ab_0          26 KB  conda-forge
gxx-10.3.0                 |       he2824d0_9          24 KB  conda-forge
gxx_impl_linux-64-10.3.0   |      hf2f2afa_14        11.4 MB  conda-forge
gxx_linux-64-10.3.0        |       h2593f52_9          24 KB  conda-forge
libgcc-devel_linux-64-10.3.0|      he6cfe16_14         3.3 MB  conda-forge
libsanitizer-10.3.0        |      h26c7422_14         6.0 MB  conda-forge
libstdcxx-devel_linux-64-10.3.0|      he6cfe16_14         9.6 MB  conda-forge
m4-1.4.18                  |    h516909a_1001         172 KB  conda-forge
mpi-1.0                    |            mpich           4 KB  conda-forge
mpi4py-3.1.3               |  py310h853ac07_0         3.3 MB  conda-forge
mpich-4.0.1                |     h846660c_100         6.0 MB  conda-forge
setuptools-61.3.1          |  py310hff52083_0         1.3 MB  conda-forge
------------------------------------------------------------
                                       Total:       120.2 MB

The following NEW packages will be INSTALLED:

_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
binutils conda-forge/linux-64::binutils-2.36.1-hdd6e379_2
binutils_impl_lin~ conda-forge/linux-64::binutils_impl_linux-64-2.36.1-h193b22a_2
binutils_linux-64 conda-forge/linux-64::binutils_linux-64-2.36-hf3e587d_9
bison conda-forge/linux-64::bison-3.8-h9c3ff4c_0
blosc conda-forge/linux-64::blosc-1.21.0-h9c3ff4c_0
bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
c-ares conda-forge/linux-64::c-ares-1.18.1-h7f98852_0
c-compiler conda-forge/linux-64::c-compiler-1.4.1-h166bdaf_0
ca-certificates conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
cmake conda-forge/linux-64::cmake-3.23.0-h5432695_1
compilers conda-forge/linux-64::compilers-1.4.1-ha770c72_0
cxx-compiler conda-forge/linux-64::cxx-compiler-1.4.1-h924138e_0
expat conda-forge/linux-64::expat-2.4.8-h27087fc_0
flex conda-forge/linux-64::flex-2.6.4-h58526e2_1004
fortran-compiler conda-forge/linux-64::fortran-compiler-1.4.1-h2a4ca65_0
gcc conda-forge/linux-64::gcc-10.3.0-he2824d0_9
gcc_impl_linux-64 conda-forge/linux-64::gcc_impl_linux-64-10.3.0-hf2f2afa_14
gcc_linux-64 conda-forge/linux-64::gcc_linux-64-10.3.0-hc39de41_9
gfortran conda-forge/linux-64::gfortran-10.3.0-h18518b4_9
gfortran_impl_lin~ conda-forge/linux-64::gfortran_impl_linux-64-10.3.0-h73f4979_14
gfortran_linux-64 conda-forge/linux-64::gfortran_linux-64-10.3.0-hb09a455_9
gnuconfig conda-forge/noarch::gnuconfig-2020.11.07-hd8ed1ab_0
gxx conda-forge/linux-64::gxx-10.3.0-he2824d0_9
gxx_impl_linux-64 conda-forge/linux-64::gxx_impl_linux-64-10.3.0-hf2f2afa_14
gxx_linux-64 conda-forge/linux-64::gxx_linux-64-10.3.0-h2593f52_9
hdf5 conda-forge/linux-64::hdf5-1.12.1-nompi_h4df4325_104
kernel-headers_li~ conda-forge/noarch::kernel-headers_linux-64-2.6.32-he073ed8_15
keyutils conda-forge/linux-64::keyutils-1.6.1-h166bdaf_0
krb5 conda-forge/linux-64::krb5-1.19.3-h08a2579_0
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2
libblas conda-forge/linux-64::libblas-3.9.0-13_linux64_openblas
libcblas conda-forge/linux-64::libcblas-3.9.0-13_linux64_openblas
libcurl conda-forge/linux-64::libcurl-7.82.0-h2283fc2_0
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libev conda-forge/linux-64::libev-4.33-h516909a_1
libffi conda-forge/linux-64::libffi-3.4.2-h7f98852_5
libgcc-devel_linu~ conda-forge/linux-64::libgcc-devel_linux-64-10.3.0-he6cfe16_14
libgcc-ng conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_14
libgfortran-ng conda-forge/linux-64::libgfortran-ng-11.2.0-h69a702a_14
libgfortran5 conda-forge/linux-64::libgfortran5-11.2.0-h5c6108e_14
libgomp conda-forge/linux-64::libgomp-11.2.0-h1d223b6_14
liblapack conda-forge/linux-64::liblapack-3.9.0-13_linux64_openblas
libnghttp2 conda-forge/linux-64::libnghttp2-1.47.0-he49606f_0
libnsl conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
libopenblas conda-forge/linux-64::libopenblas-0.3.18-pthreads_h8fe5266_0
libpng conda-forge/linux-64::libpng-1.6.37-h21135ba_2
libsanitizer conda-forge/linux-64::libsanitizer-10.3.0-h26c7422_14
libsodium conda-forge/linux-64::libsodium-1.0.18-h36c2ea0_1
libssh2 conda-forge/linux-64::libssh2-1.10.0-ha35d2d1_2
libstdcxx-devel_l~ conda-forge/linux-64::libstdcxx-devel_linux-64-10.3.0-he6cfe16_14
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-11.2.0-he4da1e4_14
libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
libuv conda-forge/linux-64::libuv-1.43.0-h7f98852_0
libzlib conda-forge/linux-64::libzlib-1.2.11-h166bdaf_1014
lz4-c conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_1
m4 conda-forge/linux-64::m4-1.4.18-h516909a_1001
make conda-forge/linux-64::make-4.3-hd18ef5c_1
mpi conda-forge/linux-64::mpi-1.0-mpich
mpi4py conda-forge/linux-64::mpi4py-3.1.3-py310h853ac07_0
mpich conda-forge/linux-64::mpich-4.0.1-h846660c_100
ncurses conda-forge/linux-64::ncurses-6.3-h9c3ff4c_0
numpy conda-forge/linux-64::numpy-1.22.3-py310h45f3432_0
openssl conda-forge/linux-64::openssl-3.0.2-h166bdaf_1
pip conda-forge/noarch::pip-22.0.4-pyhd8ed1ab_0
pkg-config conda-forge/linux-64::pkg-config-0.29.2-h36c2ea0_1008
python conda-forge/linux-64::python-3.10.4-h2660328_0_cpython
python_abi conda-forge/linux-64::python_abi-3.10-2_cp310
readline conda-forge/linux-64::readline-8.1-h46c0cb4_0
rhash conda-forge/linux-64::rhash-1.4.1-h7f98852_0
setuptools conda-forge/linux-64::setuptools-61.3.1-py310hff52083_0
sqlite conda-forge/linux-64::sqlite-3.37.1-h4ff8645_0
sysroot_linux-64 conda-forge/noarch::sysroot_linux-64-2.12-he073ed8_15
tk conda-forge/linux-64::tk-8.6.12-h27826a3_0
tzdata conda-forge/noarch::tzdata-2022a-h191b570_0
wheel conda-forge/noarch::wheel-0.37.1-pyhd8ed1ab_0
xz conda-forge/linux-64::xz-5.2.5-h516909a_1
zeromq conda-forge/linux-64::zeromq-4.3.4-h9c3ff4c_1
zfp conda-forge/linux-64::zfp-0.5.5-h9c3ff4c_8
zlib conda-forge/linux-64::zlib-1.2.11-h166bdaf_1014
zstd conda-forge/linux-64::zstd-1.5.2-ha95c52a_0

cd to ADIOS 2.8.0 source directory:

rm -rf build
cmake -S . -B build -DADIOS2_USE_MPI=OFF
cmake --build build -j 3

• cmake.log
• make.log

ctest --test-dir build --verbose -R Engine.DataMan.DataManEngineTest.1D.Serial

Internal ctest changing into directory: /home/axel/src/adios2/build UpdateCTestConfiguration from :/home/axel/src/adios2/build/DartConfiguration.tcl Parse Config file:/home/axel/src/adios2/build/DartConfiguration.tcl Add coverage exclude regular expressions. Add coverage exclude: .*/thirdparty/.* Add coverage exclude: .*/testing/.* Add coverage exclude: .*/examples/.* UpdateCTestConfiguration from :/home/axel/src/adios2/build/DartConfiguration.tcl Parse Config file:/home/axel/src/adios2/build/DartConfiguration.tcl Test project /home/axel/src/adios2/build Constructing a list of tests Done constructing a list of tests Updating test list for fixtures Added 0 tests to meet fixture requirements Checking test dependency graph... Checking test dependency graph end test 571 Start 571: Engine.DataMan.DataManEngineTest.1D.Serial

571: Test command: /home/axel/src/adios2/build/bin/Test.Engine.DataMan.1D.Serial "--gtest_filter=DataManEngineTest.1D"
571: Test timeout computed to be: 1500
571: Note: Google Test filter = DataManEngineTest.1D
571: [==========] Running 1 test from 1 test suite.
571: [----------] Global test environment set-up.
571: [----------] 1 test from DataManEngineTest
571: [ RUN ] DataManEngineTest.1D
571: /home/axel/src/adios2/testing/adios2/engine/dataman/TestDataMan1D.cpp:243: Failure
571: Expected equality of these values:
571: readString
571: Which is: "\xF0\xD8\x1C\xC5" "c\x7F\0\0\t\0\0\0\0\0\0\0some text\0\0\0\0\0\0\0"
571: "some text"
571: [ FAILED ] DataManEngineTest.1D (486 ms)
571: [----------] 1 test from DataManEngineTest (486 ms total)
571:
571: [----------] Global test environment tear-down
571: [==========] 1 test from 1 test suite ran. (486 ms total)
571: [ PASSED ] 0 tests.
571: [ FAILED ] 1 test, listed below:
571: [ FAILED ] DataManEngineTest.1D
571:
571: 1 FAILED TEST
1/1 Test #571: Engine.DataMan.DataManEngineTest.1D.Serial ...***Failed 0.50 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.54 sec

The following tests FAILED:
571 - Engine.DataMan.DataManEngineTest.1D.Serial (Failed)
Errors while running CTest
Output from these tests are in: /home/axel/src/adios2/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[ax3l]

@JasonRuonanWang let me know if you can reproduce it with that workflow as well :)

[JasonRuonanWang]

@ax3l #3155 should fix this problem. Although I haven't fully reproduced the problem you saw, I did see some weird behaviors after compiling using gcc 10. I believe it's caused by the same thing.

[ax3l]

Thanks a lot, I'll give it a try in conda-forge/adios2-feedstock#45 and report back! 👍

[ax3l]

First test locally on maser at 6313d31: this still shows the same error.

Are you sure that's not related to something decoding/copying in ZeroMQ?

571:     Which is: "\xF0(\xD5\xEEs\x7F\0\0\t\0\0\0\0\0\0\0some text\0\0\0\0\0\0\0"
571:   "some text"

looks like extra padding to me.

[ax3l]

Ah no, I should have used release_28 on 55dbc88.

Awesome, thanks a lot - yes, that fixes it 🎉

[JasonRuonanWang]

Ah no, I should have used release_28 on 55dbc88.

Awesome, thanks a lot - yes, that fixes it 🎉

Thanks for providing the details. Let me know if there is other issues!