ThreadSanitizer reports lock-order-inversion in evp_extra_test

[mspncp]

@bernd-edlinger pointed me to his observation in #9437 (comment) that the ThreadSanitizer reports errors for the evp_extra_test. This here is only the first one reported.

It can be reproduced on current master (d0cf719) as follows:

./config -g -fsanitize=thread
make -j 16
TSAN_OPTIONS=second_deadlock_stack=1 OPENSSL_TEST_RAND_ORDER=1563972976 util/shlib_wrap.sh /home/msp/src/openssl/test/evp_extra_test

The first issue reported by the ThreadSanitizer is a lock-order-inversion caused by

openssl_ctx_get_data(), which takes the context lock

openssl/crypto/context.c

Line 195 in d0cf719

CRYPTO_THREAD_read_lock(ctx->lock);

and ossl_provider_forall_loaded() which takes the store lock

openssl/crypto/provider_core.c

Line 587 in d0cf719

CRYPTO_THREAD_read_lock(store->lock);

Although the issue occurs during the test, I think it reveals a fundamental locking problem IMHO.

Here are the call stacks. (Forget about the #frame numbers, but the source code locations are real links!)

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=32344)
Cycle in lock order graph: M44 (0x7b10000003c0) => M7 (0x7b1000000000) => M44

Mutex M7 acquired here while holding mutex M44 in main thread:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f7059)
#3 ossl_namemap_stored crypto/core_namemap.c:92 (libcrypto.so.3+0x1f7c31)
#4 put_method_in_store crypto/evp/evp_fetch.c:126 (libcrypto.so.3+0x1d58af)
#5 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f780a)
#6 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f748c)
#7 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x2054ab)
#8 ossl_provider_forall_loaded crypto/provider_core.c:627 (libcrypto.so.3+0x2054ab)
#9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#10 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#12 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bad13)
#13 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1bb400)
#14 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1bb49a)
#15 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x287cd0)
#16 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x286e34)
#17 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1dc979)
#18 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1dc979)
#19 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x599b)
#20 run_tests test/testutil/driver.c:353 (evp_extra_test+0x8f0d)
#21 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M44 previously acquired by the same thread here:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 ossl_provider_forall_loaded crypto/provider_core.c:587 (libcrypto.so.3+0x2052e2)
#3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#6 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bad13)
#7 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1bb400)
#8 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1bb49a)
#9 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x287cd0)
#10 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x286e34)
#11 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1dc979)
#12 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1dc979)
#13 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x599b)
#14 run_tests test/testutil/driver.c:353 (evp_extra_test+0x8f0d)
#15 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M44 acquired here while holding mutex M7 in main thread:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 ossl_provider_forall_loaded crypto/provider_core.c:587 (libcrypto.so.3+0x2052e2)
#3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d4993)
#7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x242720)
#8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2445b4)
#9 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246242)
#10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246242)
#11 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x24672d)
#12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x24672d)
#13 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f6afe)
#14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1f9aeb)
#15 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f7082)
#16 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246d4a)
#17 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x246d4a)
#18 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x248b41)
#19 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x248b41)
#20 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf0ff4)
#21 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf0ff4)
#22 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf1ff2)
#23 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf1ff2)
#24 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xef871)
#25 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xef871)
#26 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xefdf4)
#27 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xefdf4)
#28 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xefe90)
#29 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252365)
#30 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252365)
#31 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24ece1)
#32 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e3fe2)
#33 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x532a)
#34 run_tests test/testutil/driver.c:386 (evp_extra_test+0x8d63)
#35 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M7 previously acquired by the same thread here:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f7059)
#3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246d4a)
#4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x246d4a)
#5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x248b41)
#6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x248b41)
#7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf0ff4)
#8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf0ff4)
#9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf1ff2)
#10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf1ff2)
#11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xef871)
#12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xef871)
#13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xefdf4)
#14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xefdf4)
#15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xefe90)
#16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252365)
#17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252365)
#18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24ece1)
#19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e3fe2)
#20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x532a)
#21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x8d63)
#22 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

[mspncp]

Side note for the curious: the handy links were produced from the original tsan output using the following query-replace-regexp command in Emacs:

(query-replace-regexp "\\([a-z0-9/_.]+\\):\\([0-9]+\\)" "[\\1:\\2](https://github.com/openssl/openssl/blob/master/\\1#L\\2)" nil nil nil nil nil)

[paulidale]

Any chance of a quick check to see if #9477 fixes this too?
This looks suspiciously similar to the RAND initialisation loop of #9455

[bernd-edlinger]

The lock order inversions still prevail, but the too deep recursive read locks
and the memory use after free seem to be fixed.

$ TSAN_OPTIONS=second_deadlock_stack=1 OPENSSL_TEST_RAND_ORDER=1563972976 make test V=1 TESTS=test_evp_extra
make depend && make _tests
make[1]: Entering directory '/home/ed/OPC/openssl'
make[1]: Leaving directory '/home/ed/OPC/openssl'
make[1]: Entering directory '/home/ed/OPC/openssl'
( cd test; \
  mkdir -p test-runs; \
  SRCTOP=../. \
  BLDTOP=../. \
  RESULT_D=test-runs \
  PERL="/usr/bin/perl" \
  EXE_EXT= \
  OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd` \
  OPENSSL_DEBUG_MEMORY=on \
    /usr/bin/perl .././test/run_tests.pl test_evp_extra )
../test/recipes/30-test_evp_extra.t .. 
1..1
    # RAND SEED 1563972976
    # Subtest: /home/ed/OPC/openssl/test/evp_extra_test
    1..13
    ok 1 - test_EVP_DigestVerifyInit
    ok 2 - test_EVP_SM2_verify
        # Subtest: test_d2i_AutoPrivateKey
        1..3
        ok 1 - iteration 2
        ok 2 - iteration 1
        ok 3 - iteration 3
    ok 3 - test_d2i_AutoPrivateKey
    ok 4 - test_HKDF
    ok 5 - test_EVP_PKCS82PKEY
        # Subtest: test_EVP_MD_fetch
        1..5
        ok 1 - iteration 2
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M248 (0x7b1000000e80) => M242 (0x7b1000000380) => M248

  Mutex M242 acquired here while holding mutex M248 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (fips.so+0x45612)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (fips.so+0x410a0)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #13 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #14 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #15 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #16 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #17 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #18 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #19 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #20 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M248 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #7 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #8 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #9 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #10 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #11 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #12 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #13 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #14 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M248 acquired here while holding mutex M242 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #13 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #15 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #17 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #18 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #19 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #20 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #21 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #22 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #23 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #24 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #25 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M242 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M245 (0x7b1000000ec0) => M242 (0x7b1000000380) => M245

  Mutex M242 acquired here while holding mutex M245 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (fips.so+0x54ebb)
    #4 ossl_method_store_add crypto/property/property.c:204 (fips.so+0x5579d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M245 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M245 acquired here while holding mutex M242 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #20 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #22 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #24 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #25 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #26 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #27 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #28 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #29 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #30 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #31 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #32 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M242 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 2 - iteration 4
        ok 3 - iteration 1
        ok 4 - iteration 3
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M589 (0x7b1000002240) => M583 (0x7b10000022c0) => M589

  Mutex M583 acquired here while holding mutex M589 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (fips.so+0x45612)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (fips.so+0x410a0)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #13 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #14 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #15 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #16 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #17 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #18 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #19 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #20 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M589 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #7 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #8 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #9 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #10 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #11 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #12 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #13 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #14 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M589 acquired here while holding mutex M583 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #13 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #15 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #17 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #18 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #19 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #20 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #21 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #22 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #23 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #24 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #25 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M583 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M586 (0x7b1000002280) => M583 (0x7b10000022c0) => M586

  Mutex M583 acquired here while holding mutex M586 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (fips.so+0x54ebb)
    #4 ossl_method_store_add crypto/property/property.c:204 (fips.so+0x5579d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M586 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M586 acquired here while holding mutex M583 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #20 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #22 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #24 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #25 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #26 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #27 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #28 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #29 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #30 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #31 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #32 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M583 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 5 - iteration 5
    ok 6 - test_EVP_MD_fetch
        # Subtest: test_EVP_PKEY_check
        1..5
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M47 (0x7b10000003c0) => M10 (0x7b1000000000) => M47

  Mutex M10 acquired here while holding mutex M47 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (libcrypto.so.3+0x1f9912)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (libcrypto.so.3+0x1d7140)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #13 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #14 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #15 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #16 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #17 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #18 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #19 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #20 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M47 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (libcrypto.so.3+0x206eb5)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #7 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #8 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #9 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #10 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #11 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #12 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #13 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #14 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #15 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M47 acquired here while holding mutex M10 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (libcrypto.so.3+0x206eb5)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d61f4)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x2434db)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2451e4)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246e1e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246e1e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x247340)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x247340)
    #13 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f884e)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1fb77b)
    #15 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f8da2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #17 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #18 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #19 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #20 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #21 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #22 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #23 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #24 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #25 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #26 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #27 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #28 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #29 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #30 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #31 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #32 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #33 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #34 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #35 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M10 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M44 (0x7b1000000180) => M10 (0x7b1000000000) => M44

  Mutex M10 acquired here while holding mutex M44 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (libcrypto.so.3+0x23d46b)
    #4 ossl_method_store_add crypto/property/property.c:204 (libcrypto.so.3+0x23dd4d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #14 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #15 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #16 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #17 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #18 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #19 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #20 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #21 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M44 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (libcrypto.so.3+0x20817a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (libcrypto.so.3+0x23d9cd)
    #3 ossl_property_write_lock crypto/property/property.c:76 (libcrypto.so.3+0x23d9cd)
    #4 ossl_method_store_add crypto/property/property.c:202 (libcrypto.so.3+0x23dc7d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #14 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #15 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #16 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #17 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #18 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #19 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #20 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #21 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M44 acquired here while holding mutex M10 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (libcrypto.so.3+0x20817a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (libcrypto.so.3+0x23d9cd)
    #3 ossl_property_write_lock crypto/property/property.c:76 (libcrypto.so.3+0x23d9cd)
    #4 ossl_method_store_add crypto/property/property.c:202 (libcrypto.so.3+0x23dc7d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d61f4)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x2434db)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2451e4)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246e1e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246e1e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x247340)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x247340)
    #20 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f884e)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1fb77b)
    #22 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f8da2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #24 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #25 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #26 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #27 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #28 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #29 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #30 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #31 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #32 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #33 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #34 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #35 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #36 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #37 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #38 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #39 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #40 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #41 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #42 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M10 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 1 - iteration 2
        ok 2 - iteration 4
        ok 3 - iteration 1
        ok 4 - iteration 3
        ok 5 - iteration 5
    ok 7 - test_EVP_PKEY_check
        # Subtest: test_invalide_ec_char2_pub_range_decode
        1..3
        ok 1 - iteration 2
        ok 2 - iteration 1
        ok 3 - iteration 3
    ok 8 - test_invalide_ec_char2_pub_range_decode
        # Subtest: test_set_get_raw_keys
        1..7
        ok 1 - iteration 3
        ok 2 - iteration 6
        ok 3 - iteration 2
        ok 4 - iteration 5
        ok 5 - iteration 1
        ok 6 - iteration 4
        ok 7 - iteration 7
    ok 9 - test_set_get_raw_keys
    ok 10 - test_EVP_DigestSignInit
    ok 11 - test_X509_PUBKEY_inplace
    ok 12 - test_EVP_Enveloped
    ok 13 - test_EVP_SM2
ThreadSanitizer: reported 6 warnings
/home/ed/OPC/openssl/util/shlib_wrap.sh /home/ed/OPC/openssl/test/evp_extra_test => 66
not ok 1 - running evp_extra_test

#   Failed test 'running evp_extra_test'
#   at ../test/recipes/30-test_evp_extra.t line 21.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 

Test Summary Report
-------------------
../test/recipes/30-test_evp_extra.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=1,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.14 cusr  0.02 csys =  0.19 CPU)
Result: FAIL
Makefile:198: recipe for target '_tests' failed
make[1]: *** [_tests] Error 1
make[1]: Leaving directory '/home/ed/OPC/openssl'
Makefile:196: recipe for target 'tests' failed
make: *** [tests] Error 2

[bernd-edlinger]

My guess is that all providers need to document their dependencies before
they are loaded, and not just on demand.
The loader should make a graph out of it and sort it top down, and make sure
all dependencies are satisfied (and already initialized) before initializing each provider.

[paulidale]

Thanks for the output. The test ran okay for me whereas previously it deadlocked hence my optimism. I'll try to have another look at this one tomorrow.

I'm not sure what to do about the dependency graph. This test case jumps all over the place with respect to algorithm usage (& hence initialisation). Having the graph and initialising in a topologically sorted order won't necessarily address the problem with the lock inversion. E.g. initialising a public key algorithm then symmetric ciphers could start one way whereas reversing these could be another. Always initialising everything would work but it would be a big change from the current lazy model and one I'm not sure how palatable it would be. A single global initialisation lock might be the solution.

[bernd-edlinger]

I have of course not a ready solution. I just have the impression that it is not at all
clear why a aes-ctr provider is not allowed to use a rand provider. Or when you add
later add a replacement aes-ctr provider that is not aware of the possible recursion,
how will we know that it is incompatible with the rand-provider, maybe also replacement,
using more ciphers than the original one.

I mean from application code, it is probably not possible to know in advance what it will need.
But I think from a rand or aes-ctr provider we should expect a manifest describing all possible
dependencies so it is clear from the beginning that we have something incompatible.

[paulidale]

So long as the algorithm is fetched rather than being used, it should work. The CTR DRBG vs AES-CTR is a good example BTW.

[bernd-edlinger]

The drbg and aes-ctr provider is good today, but not so clear what will be added in the future.
What is not okay, is recursive new_func callbacks, with mutex held.

That may work today, but not when we do not know what new_func is actually doing.
So what would happen if the provider is a third-party component?

Why is there a recursion in the new_func ?

diff --git a/crypto/context.c b/crypto/context.c
index cc1ec0e..1fb7074 100644
--- a/crypto/context.c
+++ b/crypto/context.c
@@ -148,7 +148,11 @@ static void openssl_ctx_generic_new(void *parent_ign, void *ptr_ign,
                                     long argl_ign, void *argp)
 {
     const OPENSSL_CTX_METHOD *meth = argp;
+static int recurs = 0;
+recurs++;
+OPENSSL_assert(recurs == 1);
     void *ptr = meth->new_func(crypto_ex_data_get_openssl_ctx(ad));
+recurs--;
 
     if (ptr != NULL)
         CRYPTO_set_ex_data(ad, index, ptr);

It would not need recursion, if the CTR is initialized before RAND
and RAND is initialized before RSA, right?

[paulidale]

@levitte @mattcaswell thoughts on the recursion question?

[paulidale]

Rather than initilisation, isn't it more that CTR DRBG needs an AES CTR cipher context?

When a call is made to get data from the DRBG, the caller won't know what other cryptographic algorithm will be required. It's up to the specific DRBG to determine this.

This is looking more problematic the more I dig in.

[paulidale]

I've emailed the project list for wider discussion.