[C++] Valgrind detect memory leak when using s3fs(InitializeS3/FinalizeS3)

[allinux]

Describe the bug, including details regarding any error messages, version, and platform.

• OS
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.3 LTS
  Release: 22.04
  Codename: jammy

• LIBS
  aws-sdk-cpp 1.11.65
  arrow 12.0.1

• TEST CODE

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>

#include <arrow/status.h>
#include <arrow/filesystem/s3fs.h>

int main(int argc, char** argv) {
    auto globalOption = arrow::fs::S3GlobalOptions{};
    arrow::fs::InitializeS3(globalOption).ok();
    arrow::fs::FinalizeS3().ok();
}

• REPORT

==530999== HEAP SUMMARY:
==530999==     in use at exit: 46,319 bytes in 327 blocks
==530999==   total heap usage: 47,961 allocs, 47,634 frees, 4,270,889 bytes allocated
==530999== 
==530999== 224 bytes in 1 blocks are definitely lost in loss record 153 of 234
==530999==    at 0x484880F: malloc (vg_replace_malloc.c:431)
==530999==    by 0x9AE0ED: CRYPTO_zalloc (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xAD37D7: ENGINE_new (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xE18A44: s2n_rand_init (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xE15C4B: s2n_init (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDCC7C9: aws_tls_init_static_state (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xD7555F: aws_mqtt_library_init (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xD63473: Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x8CF250: Aws::InitializeCrt() (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x8CD776: Aws::InitAPI(Aws::SDKOptions const&) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x485F01: arrow::fs::InitializeS3(arrow::fs::S3GlobalOptions const&) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x479A27: main (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999== 
==530999== 304 bytes in 1 blocks are possibly lost in loss record 154 of 234
==530999==    at 0x484D80C: calloc (vg_replace_malloc.c:1554)
==530999==    by 0x40147D9: calloc (rtld-malloc.h:44)
==530999==    by 0x40147D9: allocate_dtv (dl-tls.c:375)
==530999==    by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==530999==    by 0x4C3B834: allocate_stack (allocatestack.c:430)
==530999==    by 0x4C3B834: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==530999==    by 0xE83580: aws_thread_launch (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDC418B: s_run (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDBF517: s_event_loop_group_new (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDBF73B: aws_event_loop_group_new (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xD71E62: Aws::Crt::Io::EventLoopGroup::EventLoopGroup(unsigned short, aws_allocator*) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x495482: std::_Function_handler<std::shared_ptr<Aws::Crt::Io::ClientBootstrap> (), arrow::fs::(anonymous namespace)::AwsInstance::DoInitialize(arrow::fs::S3GlobalOptions const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x8CD8CF: Aws::InitAPI(Aws::SDKOptions const&) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x485F01: arrow::fs::InitializeS3(arrow::fs::S3GlobalOptions const&) (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x479A27: main (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999== 
==530999== 304 bytes in 1 blocks are possibly lost in loss record 155 of 234
==530999==    at 0x484D80C: calloc (vg_replace_malloc.c:1554)
==530999==    by 0x40147D9: calloc (rtld-malloc.h:44)
==530999==    by 0x40147D9: allocate_dtv (dl-tls.c:375)
==530999==    by 0x40147D9: _dl_allocate_tls (dl-tls.c:634)
==530999==    by 0x4C3B834: allocate_stack (allocatestack.c:430)
==530999==    by 0x4C3B834: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==530999==    by 0xE83401: aws_thread_launch (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDBEA79: s_aws_event_loop_group_shutdown_async (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xE83CC0: aws_ref_count_release (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xDBC31B: s_client_bootstrap_destroy_impl (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xE83CC0: aws_ref_count_release (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0xD70D9C: Aws::Crt::Io::ClientBootstrap::~ClientBootstrap() (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x8CF591: std::shared_ptr<Aws::Crt::Io::ClientBootstrap>::~shared_ptr() (in /home/allinux/workspace/projects/c/arrow_exam02/build/arrow_exam02)
==530999==    by 0x4BEB494: __run_exit_handlers (exit.c:113)
==530999==    by 0x4BEB60F: exit (exit.c:143)
==530999== 
==530999== LEAK SUMMARY:
==530999==    definitely lost: 224 bytes in 1 blocks
==530999==    indirectly lost: 0 bytes in 0 blocks
==530999==      possibly lost: 608 bytes in 2 blocks
==530999==    still reachable: 45,487 bytes in 324 blocks
==530999==         suppressed: 0 bytes in 0 blocks
==530999== Reachable blocks (those to which a pointer was found) are not shown.
==530999== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==530999== 
==530999== For lists of detected and suppressed errors, rerun with: -s
==530999== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

Component(s)

C++

[kou]

Is this reproduced by the following?

#include <aws/core/Aws.h>

int main(int argc, char** argv) {
  Aws::SDKOptions options;
  Aws::InitAPI(options);
  Aws::ShutdownAPI(options);
}

If so, it's a problem of AWS SDK for C++ not Apache Arrow C++.

[assignUser]

I recall coming across this in the past and I think kou is right with it being an aws issue...

[allinux]

#include <aws/core/Aws.h>

int main(int argc, char** argv) {
  Aws::SDKOptions options;
  Aws::InitAPI(options);
  Aws::ShutdownAPI(options);
}

sample code above has no memory leaks.

==539449== HEAP SUMMARY:
==539449==     in use at exit: 0 bytes in 0 blocks
==539449==   total heap usage: 47,946 allocs, 47,946 frees, 4,268,580 bytes allocated
==539449== 
==539449== All heap blocks were freed -- no leaks are possible
==539449== 
==539449== For lists of detected and suppressed errors, rerun with: -s
==539449== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[allinux]

This issue is reproducible in docker containers.
https://github.com/allinux/arrow-test

[kou]

Could you try 13.0.0 too?

[allinux]

Yes, I really appreciate your time.

[allinux]

There is no problem with version 13.0.0.
Thank U for reply.

==2292333== HEAP SUMMARY:
==2292333==     in use at exit: 14,398 bytes in 216 blocks
==2292333==   total heap usage: 34,340 allocs, 34,124 frees, 2,773,707 bytes allocated
==2292333== 
==2292333== LEAK SUMMARY:
==2292333==    definitely lost: 0 bytes in 0 blocks
==2292333==    indirectly lost: 0 bytes in 0 blocks
==2292333==      possibly lost: 416 bytes in 1 blocks
==2292333==    still reachable: 13,982 bytes in 215 blocks
==2292333==                       of which reachable via heuristic:
==2292333==                         length64           : 456 bytes in 2 blocks
==2292333==         suppressed: 0 bytes in 0 blocks
==2292333== Rerun with --leak-check=full to see details of leaked memory
==2292333== 
==2292333== For lists of detected and suppressed errors, rerun with: -s
==2292333== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[allinux]

Solved and closed.