Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SEGV after failing to map contiguous pages by MmapAllocator #9135

Closed
czentgr opened this issue Mar 18, 2024 · 5 comments
Closed

SEGV after failing to map contiguous pages by MmapAllocator #9135

czentgr opened this issue Mar 18, 2024 · 5 comments
Labels
bug Something isn't working triage Newly created issue that needs attention.

Comments

@czentgr
Copy link
Collaborator

czentgr commented Mar 18, 2024

Bug description

Under certain conditions memory cannot be allocated by the MemoryAllocator. This occasionally on clusters running TPCDS queries.

Prestissimo does not explicitly set the option useMmapArena and the default value of false is used by Velox. As a result, contiguous memory requests use the mmap system call.

The code in allocateContiguousImpl performing the allocation may call mmap.

    if (useMmapArena_) {.   <<<---- this is false in prestissimo
      std::lock_guard<std::mutex> l(arenaMutex_);
      data = managedArenas_->allocate(AllocationTraits::pageBytes(maxPages));
    } else {
      data = ::mmap(
          nullptr,
          AllocationTraits::pageBytes(maxPages),
          PROT_READ | PROT_WRITE,
          MAP_PRIVATE | MAP_ANONYMOUS,
          -1,
          0);
    }

If the mmap call fails then data will be set to MAP_FAILED (-1) and errno is set.
The code has a TODO comment to handle MAP_FAILED which == -1.

  // TODO: add handling of MAP_FAILED.
  if (data == nullptr) {
...

The proposed solution to prevent the SEGV and return the error (fail query with insufficient resources) would be to check for MAP_FAILED

  if (data == nullptr || data == MAP_FAILED) {
// handle failed allocation

System information

Host with 122 GiB memory available. Async data cache enabled by default.

system-memory-gb=108
query-memory-gb=102
query.max-memory-per-node=102GB
memory-arbitrator-kind=SHARED
task.max-drivers-per-task=16

Relevant logs

I20240313 23:03:44.306201    18 PeriodicServiceInventoryManager.cpp:115] Announcement succeeded: HTTP 202. State: active.
I20240313 23:03:45.901907  1803 Task.cpp:1818] Terminating task 20240313_230250_00207_d49um.7.0.1.0 with state Finished after running for 54913 ms.
I20240313 23:03:45.903502  1680 TaskManager.cpp:581] Deleting task 20240313_230250_00207_d49um.7.0.1.0
W20240313 23:03:54.756481  1839 MmapAllocator.cpp:79] [MEM] Exceeded memory allocator limit when allocating 32 pages with capacity of 28311552 pages
W20240313 23:03:55.109093  1834 MmapAllocator.cpp:373] [MEM] Could not advise away enough for 1024 pages for total allocation of 1024 pages
W20240313 23:03:55.309314  1722 MmapAllocator.cpp:467] [MEM] Exceeded memory allocator limit when allocating 512 new pages for total allocation of 67584 pages, the memory allocator capacity is 28311552 pages
I20240313 23:03:56.011111  1928 Task.cpp:1092] All drivers (16) finished for task 20240313_230250_00207_d49um.10.0.1.0 after running for 65052 ms.
W20240313 23:03:56.134788  1911 MmapAllocator.cpp:484] [MEM] Could not advise away enough for 512 pages for growing allocation of 130560 pages
W20240313 23:03:56.165632  1710 MmapAllocator.cpp:373] [MEM] Could not advise away enough for 3072 pages for total allocation of 3072 pages
I20240313 23:03:56.434197  1810 Task.cpp:1092] All drivers (48) finished for task 20240313_230250_00207_d49um.6.0.7.0 after running for 65475 ms.
I20240313 23:03:56.497978  1800 Task.cpp:1818] Terminating task 20240313_230250_00207_d49um.6.0.7.0 with state Finished after running for 65539 ms.	
I20240313 23:03:56.499540  1800 TaskManager.cpp:581] Deleting task 20240313_230250_00207_d49um.6.0.7.0
W20240313 23:03:57.153400  1816 MemoryAllocator.cpp:334] [MEM] madvise hugepage errno=Cannot allocate memory
*** Aborted at 1710371037 (unix time) try "date -d @1710371037" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0xffffffffffffffff) received by PID 8 (TID 0x7e3e52ffd700) from PID 18446744073709551615; stack trace: ***
    @     0x7f3214f8ed20 (unknown)
    @     0x7f32132fe77a __memset_evex_unaligned_erms
    @          0x3b14652 facebook::velox::exec::HashTable<>::allocateTables()
    @          0x3b2e876 facebook::velox::exec::HashTable<>::checkSize()
    @          0x3b2f9db facebook::velox::exec::HashTable<>::setHashMode()
    @          0x3b2886c facebook::velox::exec::HashTable<>::decideHashMode()
    @          0x3b2f4e3 facebook::velox::exec::HashTable<>::prepareJoinTable()
    @          0x3afa0c3 facebook::velox::exec::HashBuild::finishHashBuild()
    @          0x3afa359 facebook::velox::exec::HashBuild::noMoreInputInternal()
    @          0x398940f facebook::velox::exec::Driver::runInternal()
    @          0x398b75d facebook::velox::exec::Driver::run()
    @          0x398b91f _ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallIZN8facebook5velox4exec6Driver7enqueueESt10shared_ptrIS9_EEUlvE_EEvRNS1_4DataE
    @          0x40fad07 folly::detail::function::FunctionTraits<>::operator()()
    @     0x7f32173a9024 folly::ThreadPoolExecutor::runTask()
    @          0x40f6df3 folly::CPUThreadPoolExecutor::threadRun()
    @     0x7f32173b586b std::__invoke_impl<>()
    @     0x7f32173b4239 std::__invoke<>()
    @     0x7f32173b260b _ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x7f32173b0c40 std::_Bind<>::operator()<>()
    @     0x7f32173af210 folly::detail::function::FunctionTraits<>::callSmall<>()
    @     0x7f3213c77b23 (unknown)
    @     0x7f3214f841ca start_thread
    @     0x7f321327e8d3 __GI___clone
    @                0x0 (unknown)
    @          0x40f6df3 folly::CPUThreadPoolExecutor::threadRun()
	
	
    @     0x7f32173b586b std::__invoke_impl<>()
	
	
    @     0x7f32173b4239 std::__invoke<>()
	
	
    @     0x7f32173b260b _ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
	
	
    @     0x7f32173b0c40 std::_Bind<>::operator()<>()
	
	
    @     0x7f32173af210 folly::detail::function::FunctionTraits<>::callSmall<>()
	
	
    @     0x7f3213c77b23 (unknown)
	
	
    @     0x7f3214f841ca start_thread
	
	
    @     0x7f321327e8d3 __GI___clone
	
	
    @                0x0 (unknown)
@czentgr czentgr added bug Something isn't working triage Newly created issue that needs attention. labels Mar 18, 2024
@czentgr
Copy link
Collaborator Author

czentgr commented Mar 19, 2024

FYI @majetideepak

@ethanyzhang
Copy link

A different stack trace I see:

*** Aborted at 1710860656 (unix time) try "date -d @1710860656" if you are using GNU date *** |  
-- | --
  | PC: @                0x0 (unknown) |  
  | *** SIGSEGV (@0x18) received by PID 8 (TID 0x7d9a037fe700) from PID 24; stack trace: *** |  
  | @     0x7f883794dd20 (unknown) |  
  | @     0x7f8835cbcae7 __memmove_evex_unaligned_erms |  
  | @          0x39f6a5c facebook::velox::exec::RowContainer::initializeRow() |  
  | @          0x3b39219 _ZZN8facebook5velox4exec9HashBuild8addInputESt10shared_ptrINS0_9RowVectorEEENKUlT_E_clIiEEDaS6_ |  
  | @          0x3b42791 facebook::velox::exec::HashBuild::addInput() |  
  | @          0x39cbd44 facebook::velox::exec::Driver::runInternal() |  
  | @          0x39ce8cd facebook::velox::exec::Driver::run() |  
  | @          0x39cea8f _ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallIZN8facebook5velox4exec6Driver7enqueueESt10shared_ptrIS9_EEUlvE_EEvRNS1_4DataE |  
  | @          0x4141b67 folly::detail::function::FunctionTraits<>::operator()() |  
  | @     0x7f8839d68024 folly::ThreadPoolExecutor::runTask() |  
  | @          0x413dc53 folly::CPUThreadPoolExecutor::threadRun() |  
  | @     0x7f8839d7486b std::__invoke_impl<>() |  
  | @     0x7f8839d73239 std::__invoke<>() |  
  | @     0x7f8839d7160b _ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE |  
  | W20240319 15:04:16.895465  4159 MmapAllocator.cpp:79] [MEM] Exceeded memory allocator limit when allocating 32 pages with capacity of 57409536 pages |  
  | @     0x7f8839d6fc40 std::_Bind<>::operator()<>() |  
  | @     0x7f8839d6e210 folly::detail::function::FunctionTraits<>::callSmall<>() |  
  | @     0x7f8836636b23 (unknown) |  
  | @     0x7f88379431ca start_thread |  
  | @     0x7f8835c3d8d3 __GI___clone |  
  | @                0x0 (unknown)

@czentgr
Copy link
Collaborator Author

czentgr commented Mar 19, 2024

Yes, I think ultimately it comes from the same source.

The stack to lead up to it from RowContainer::newRow which calls the initializeRow function that crashes.

MmapAllocator::allocateContiguousImpl
MmapAllocator::allocateContiguousWithoutRetry
MemoryAllocator::allocateContiguous
MemoryPoolImpl::allocateContiguous
AllocationPool::newRunImpl
AllocationPool::allocateFixed
RowContainer::newRow

The MmapAllocator::allocateContiguousImpl returns an Allocation that points to -1. Later the pointer is used first in initializeRow when the memset is called.

@wills-feng
Copy link
Contributor

After we handle the MAP_FAILED case and return false in that function, the SEGV issue is gone. Meanwhile we output the memory stats when data is MAP_FAILED.

The benchmark run was tpcds10k and on 16nodes cluster with 219 system-memory-gb setting for each node.

W20240323 04:37:37.417330  3268 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 8 pages with capacity of 57409536 pages
W20240323 04:37:43.431779  3337 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 16 pages with capacity of 57409536 pages
I20240323 04:37:44.118325    19 PeriodicServiceInventoryManager.cpp:115] Announcement succeeded: HTTP 202. State: active.
W20240323 04:37:44.954460  3326 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 128 pages with capacity of 57409536 pages

W20240323 04:37:47.917161  3377 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 256 pages with capacity of 57409536 pages
W20240323 04:37:55.762253  3528 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 16 pages with capacity of 57409536 pages
W20240323 04:38:01.240581  3359 MmapAllocator.cpp:80] [MEM] Exceeded memory allocator limit when allocating 128 pages with capacity of 57409536 pages
I20240323 04:38:08.545166    39 PeriodicTaskManager.cpp:636] Spill memory usage: current[0B] peak[0B]
I20240323 04:38:08.580164    40 PeriodicTaskManager.cpp:593] Updated memory arbitrator stats: STATS[numRequests 6176 numSucceeded 6176 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 numReserves 73 numReleases 72 queueTime 3m 5s arbitrationTime 3m 28s reclaimTime 0us shrunkMemory 0B reclaimedMemory 0B maxCapacity 208.00GB freeCapacity 196.61GB]
I20240323 04:38:08.580229    40 PeriodicTaskManager.cpp:595] Memory arbitrator stats change: STATS[numRequests 45 numSucceeded 45 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 numReserves 0 numReleases 0 queueTime 52.56ms arbitrationTime 80.72ms reclaimTime 0us shrunkMemory 0B reclaimedMemory 0B maxCapacity 208.00GB freeCapacity 196.61GB]
I20240323 04:38:08.796352    35 PeriodicTaskManager.cpp:413] Cache stats:
Cache size: 207.92GB tinySize: 748.92KB large size: 207.92GB
Cache entries: 51106 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2409
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75951 eviction checks: 445371 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
E20240323 04:38:08.921046  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 125.41MB allocated pages 57377430 mapped pages 57409432 external mapped pages 2765334
[size 1: 20876(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18779(293MB) allocated 18827 mapped]
[size 8: 21199(662MB) allocated 21281 mapped]
[size 16: 7984(499MB) allocated 8311 mapped]
[size 32: 18627(2328MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23836(11918MB) allocated 23869 mapped]
[size 256: 190706(190706MB) allocated 190715 mapped]
]
E20240323 04:38:08.925364  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.92GB tinySize: 748.92KB large size: 207.92GB
Cache entries: 51106 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2409
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75951 eviction checks: 445371 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57378541 cached pages: 54504065
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 121.07MB allocated pages 57378541 mapped pages 57409432 external mapped pages 2765334
[size 1: 20875(81MB) allocated 21540 mapped]
[size 2: 16872(131MB) allocated 22397 mapped]
[size 4: 18783(293MB) allocated 18827 mapped]
[size 8: 21201(662MB) allocated 21281 mapped]
[size 16: 7961(497MB) allocated 8311 mapped]
[size 32: 18633(2329MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23836(11918MB) allocated 23869 mapped]
[size 256: 190706(190706MB) allocated 190715 mapped]
]
E20240323 04:38:08.931684  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.931730  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 123.25MB allocated pages 57377984 mapped pages 57409432 external mapped pages 2765334
[size 1: 20874(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18780(293MB) allocated 18827 mapped]
[size 8: 21204(662MB) allocated 21281 mapped]
[size 16: 7960(497MB) allocated 8311 mapped]
[size 32: 18633(2329MB) allocated 18690 mapped]
[size 64: 26855(6713MB) allocated 26920 mapped]
[size 128: 23836(11918MB) allocated 23869 mapped]
[size 256: 190703(190703MB) allocated 190715 mapped]
]
E20240323 04:38:08.934777  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.91GB tinySize: 748.89KB large size: 207.91GB
Cache entries: 51104 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2411
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75953 eviction checks: 445373 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57377958 cached pages: 54503150
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 123.35MB allocated pages 57377958 mapped pages 57409432 external mapped pages 2765334

[size 1: 20874(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18781(293MB) allocated 18827 mapped]
[size 8: 21204(662MB) allocated 21281 mapped]
[size 16: 7958(497MB) allocated 8311 mapped]
[size 32: 18633(2329MB) allocated 18690 mapped]
[size 64: 26855(6713MB) allocated 26920 mapped]
[size 128: 23836(11918MB) allocated 23869 mapped]
[size 256: 190703(190703MB) allocated 190715 mapped]
]
E20240323 04:38:08.940786  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.940829  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 135.11MB allocated pages 57374947 mapped pages 57409432 external mapped pages 2765334
[size 1: 20873(81MB) allocated 21540 mapped]
[size 2: 16872(131MB) allocated 22397 mapped]
[size 4: 18781(293MB) allocated 18827 mapped]
[size 8: 21204(662MB) allocated 21281 mapped]
[size 16: 7963(497MB) allocated 8311 mapped]
[size 32: 18632(2329MB) allocated 18690 mapped]
[size 64: 26854(6713MB) allocated 26920 mapped]
[size 128: 23833(11916MB) allocated 23869 mapped]
[size 256: 190693(190693MB) allocated 190715 mapped]
]
E20240323 04:38:08.943756  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.90GB tinySize: 748.83KB large size: 207.90GB
Cache entries: 51100 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2415
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75957 eviction checks: 445377 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57375750 cached pages: 54500107
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 131.98MB allocated pages 57375750 mapped pages 57409432 external mapped pages 2765334
[size 1: 20874(81MB) allocated 21540 mapped]
[size 2: 16876(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21211(662MB) allocated 21281 mapped]
[size 16: 7975(498MB) allocated 8311 mapped]
[size 32: 18636(2329MB) allocated 18690 mapped]
[size 64: 26860(6715MB) allocated 26920 mapped]
[size 128: 23833(11916MB) allocated 23869 mapped]
[size 256: 190693(190693MB) allocated 190715 mapped]
]
E20240323 04:38:08.949721  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.949761  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 148.25MB allocated pages 57371583 mapped pages 57409432 external mapped pages 2765334
[size 1: 20873(81MB) allocated 21540 mapped]
[size 2: 16876(131MB) allocated 22397 mapped]
[size 4: 18783(293MB) allocated 18827 mapped]
[size 8: 21211(662MB) allocated 21281 mapped]
[size 16: 7963(497MB) allocated 8311 mapped]
[size 32: 18636(2329MB) allocated 18690 mapped]
[size 64: 26858(6714MB) allocated 26920 mapped]
[size 128: 23831(11915MB) allocated 23869 mapped]
[size 256: 190679(190679MB) allocated 190715 mapped]
]
E20240323 04:38:08.952646  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.89GB tinySize: 748.79KB large size: 207.89GB
Cache entries: 51097 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2418
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75960 eviction checks: 445385 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57371703 cached pages: 54496130
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 147.79MB allocated pages 57371703 mapped pages 57409432 external mapped pages 2765334
[size 1: 20873(81MB) allocated 21540 mapped]
[size 2: 16876(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21211(662MB) allocated 21281 mapped]
[size 16: 7971(498MB) allocated 8311 mapped]
[size 32: 18636(2329MB) allocated 18690 mapped]
[size 64: 26858(6714MB) allocated 26920 mapped]
[size 128: 23831(11915MB) allocated 23869 mapped]
[size 256: 190679(190679MB) allocated 190715 mapped]
]
E20240323 04:38:08.958513  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.958555  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 160.50MB allocated pages 57368449 mapped pages 57409432 external mapped pages 2765334
[size 1: 20873(81MB) allocated 21540 mapped]
[size 2: 16875(131MB) allocated 22397 mapped]
[size 4: 18781(293MB) allocated 18827 mapped]
[size 8: 21209(662MB) allocated 21281 mapped]
[size 16: 7976(498MB) allocated 8311 mapped]
[size 32: 18635(2329MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23830(11915MB) allocated 23869 mapped]
[size 256: 190667(190667MB) allocated 190715 mapped]
]
E20240323 04:38:08.961386  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.87GB tinySize: 748.76KB large size: 207.87GB
Cache entries: 51095 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2420
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75962 eviction checks: 445390 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57368641 cached pages: 54492812
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 159.75MB allocated pages 57368641 mapped pages 57409432 external mapped pages 2765334
[size 1: 20873(81MB) allocated 21540 mapped]
[size 2: 16875(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 7989(499MB) allocated 8311 mapped]
[size 32: 18643(2330MB) allocated 18690 mapped]
[size 64: 26858(6714MB) allocated 26920 mapped]
[size 128: 23830(11915MB) allocated 23869 mapped]
[size 256: 190667(190667MB) allocated 190715 mapped]
]
E20240323 04:38:08.967202  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.967252  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 169.36MB allocated pages 57366179 mapped pages 57409432 external mapped pages 2765334
[size 1: 20871(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21212(662MB) allocated 21281 mapped]
[size 16: 7982(498MB) allocated 8311 mapped]
[size 32: 18640(2330MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23830(11915MB) allocated 23869 mapped]
[size 256: 190657(190657MB) allocated 190715 mapped]
]
E20240323 04:38:08.970023  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.86GB tinySize: 748.64KB large size: 207.86GB
Cache entries: 51087 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2428
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75970 eviction checks: 445398 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57366311 cached pages: 54490057
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 168.85MB allocated pages 57366311 mapped pages 57409432 external mapped pages 2765334
[size 1: 20871(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21212(662MB) allocated 21281 mapped]
[size 16: 7991(499MB) allocated 8311 mapped]
[size 32: 18640(2330MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23830(11915MB) allocated 23869 mapped]
[size 256: 190657(190657MB) allocated 190715 mapped]
]
E20240323 04:38:08.975678  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.975719  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 179.29MB allocated pages 57363639 mapped pages 57409432 external mapped pages 2765334
[size 1: 20871(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18782(293MB) allocated 18827 mapped]
[size 8: 21212(662MB) allocated 21281 mapped]
[size 16: 7997(499MB) allocated 8311 mapped]
[size 32: 18640(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190647(190647MB) allocated 190715 mapped]
]
E20240323 04:38:08.978440  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.85GB tinySize: 748.60KB large size: 207.85GB
Cache entries: 51084 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2431
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75973 eviction checks: 445401 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57363607 cached pages: 54487289
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 179.41MB allocated pages 57363607 mapped pages 57409432 external mapped pages 2765334
[size 1: 20871(81MB) allocated 21540 mapped]
[size 2: 16873(131MB) allocated 22397 mapped]
[size 4: 18784(293MB) allocated 18827 mapped]
[size 8: 21214(662MB) allocated 21281 mapped]
[size 16: 8013(500MB) allocated 8311 mapped]
[size 32: 18643(2330MB) allocated 18690 mapped]
[size 64: 26864(6716MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190647(190647MB) allocated 190715 mapped]
]
E20240323 04:38:08.984169  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.984211  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 187.32MB allocated pages 57361581 mapped pages 57409432 external mapped pages 2765334
[size 1: 20871(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21214(662MB) allocated 21281 mapped]
[size 16: 8018(501MB) allocated 8311 mapped]
[size 32: 18647(2330MB) allocated 18690 mapped]
[size 64: 26860(6715MB) allocated 26920 mapped]
[size 128: 23831(11915MB) allocated 23869 mapped]
[size 256: 190638(190638MB) allocated 190715 mapped]
]
E20240323 04:38:08.986992  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.84GB tinySize: 748.54KB large size: 207.84GB
Cache entries: 51080 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2435
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75977 eviction checks: 445405 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57361885 cached pages: 54484431
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 186.14MB allocated pages 57361885 mapped pages 57409432 external mapped pages 2765334
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18787(293MB) allocated 18827 mapped]
[size 8: 21214(662MB) allocated 21281 mapped]
[size 16: 7984(499MB) allocated 8311 mapped]
[size 32: 18647(2330MB) allocated 18690 mapped]
[size 64: 26860(6715MB) allocated 26920 mapped]
[size 128: 23831(11915MB) allocated 23869 mapped]
[size 256: 190638(190638MB) allocated 190715 mapped]
]
E20240323 04:38:08.992779  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.992820  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 202.25MB allocated pages 57357761 mapped pages 57409432 external mapped pages 2765334
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 7986(499MB) allocated 8311 mapped]
[size 32: 18646(2330MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23828(11914MB) allocated 23869 mapped]
[size 256: 190624(190624MB) allocated 190713 mapped]
]
E20240323 04:38:08.993736  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 200.12MB allocated pages 57358305 mapped pages 57409432 external mapped pages 2765846
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 7989(499MB) allocated 8311 mapped]
[size 32: 18646(2330MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23828(11914MB) allocated 23869 mapped]
[size 256: 190624(190624MB) allocated 190713 mapped]
]
E20240323 04:38:08.995599  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.83GB tinySize: 748.48KB large size: 207.83GB
Cache entries: 51076 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2439
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75981 eviction checks: 445409 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57358449 cached pages: 54480195
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 199.56MB allocated pages 57358449 mapped pages 57409432 external mapped pages 2765846
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 7998(499MB) allocated 8311 mapped]
[size 32: 18646(2330MB) allocated 18690 mapped]
[size 64: 26857(6714MB) allocated 26920 mapped]
[size 128: 23828(11914MB) allocated 23869 mapped]
[size 256: 190624(190624MB) allocated 190713 mapped]
]
E20240323 04:38:09.001332  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:08.996912  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.83GB tinySize: 748.48KB large size: 207.83GB
Cache entries: 51076 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2439
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75981 eviction checks: 445409 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096273
Allocated pages: 57358497 cached pages: 54480195
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 199.37MB allocated pages 57358497 mapped pages 57409432 external mapped pages 2765846
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 8000(500MB) allocated 8311 mapped]
[size 32: 18644(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23828(11914MB) allocated 23869 mapped]
[size 256: 190614(190614MB) allocated 190713 mapped]
]
E20240323 04:38:09.002688  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.002051  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 209.70MB allocated pages 57355854 mapped pages 57409432 external mapped pages 2765846
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16869(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 8004(500MB) allocated 8311 mapped]
[size 32: 18644(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190607(190607MB) allocated 190713 mapped]
]
E20240323 04:38:09.002728  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 217.07MB allocated pages 57353966 mapped pages 57409432 external mapped pages 2765846
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16869(131MB) allocated 22397 mapped]
[size 4: 18786(293MB) allocated 18827 mapped]
[size 8: 21213(662MB) allocated 21281 mapped]
[size 16: 8007(500MB) allocated 8311 mapped]
[size 32: 18644(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190607(190607MB) allocated 190713 mapped]
]
E20240323 04:38:09.004604  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.81GB tinySize: 748.34KB large size: 207.81GB
Cache entries: 51066 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2449
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75991 eviction checks: 445419 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57354141 cached pages: 54475568
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 216.39MB allocated pages 57354141 mapped pages 57409432 external mapped pages 2765846
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18787(293MB) allocated 18827 mapped]
[size 8: 21214(662MB) allocated 21281 mapped]
[size 16: 8019(501MB) allocated 8311 mapped]
[size 32: 18644(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190607(190607MB) allocated 190713 mapped]
]
E20240323 04:38:09.010218  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
I20240323 04:38:09.010246  3751 AsyncDataCache.cpp:799] [CACHE] Backoff in allocation contention for 23.67ms
E20240323 04:38:09.005873  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.81GB tinySize: 748.34KB large size: 207.81GB
Cache entries: 51066 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2449
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75991 eviction checks: 445419 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57354205 cached pages: 54475568
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 216.14MB allocated pages 57354205 mapped pages 57409432 external mapped pages 2765846
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16870(131MB) allocated 22397 mapped]
[size 4: 18791(293MB) allocated 18827 mapped]
[size 8: 21217(663MB) allocated 21281 mapped]
[size 16: 8028(501MB) allocated 8311 mapped]
[size 32: 18645(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190607(190607MB) allocated 190713 mapped]
]
E20240323 04:38:09.011552  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.011590  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 225.28MB allocated pages 57351864 mapped pages 57408920 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16872(131MB) allocated 22397 mapped]
[size 4: 18790(293MB) allocated 18827 mapped]
[size 8: 21216(663MB) allocated 21281 mapped]
[size 16: 8030(501MB) allocated 8311 mapped]
[size 32: 18645(2330MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190599(190599MB) allocated 190713 mapped]
]
E20240323 04:38:09.014307  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.80GB tinySize: 748.32KB large size: 207.80GB
Cache entries: 51065 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2450
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75992 eviction checks: 445423 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57352203 cached pages: 54473520
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 223.96MB allocated pages 57352203 mapped pages 57408920 external mapped pages 2765334
[size 1: 20879(81MB) allocated 21540 mapped]
[size 2: 16875(131MB) allocated 22397 mapped]
[size 4: 18793(293MB) allocated 18827 mapped]
[size 8: 21222(663MB) allocated 21281 mapped]
[size 16: 8058(503MB) allocated 8311 mapped]
[size 32: 18653(2331MB) allocated 18690 mapped]
[size 64: 26856(6714MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190599(190599MB) allocated 190713 mapped]
]
E20240323 04:38:09.020015  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.020058  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 234.78MB allocated pages 57349432 mapped pages 57408920 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16874(131MB) allocated 22397 mapped]
[size 4: 18798(293MB) allocated 18827 mapped]
[size 8: 21224(663MB) allocated 21281 mapped]
[size 16: 8070(504MB) allocated 8311 mapped]
[size 32: 18654(2331MB) allocated 18690 mapped]
[size 64: 26855(6713MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190588(190588MB) allocated 190713 mapped]
]
E20240323 04:38:09.022784  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.79GB tinySize: 748.29KB large size: 207.79GB
Cache entries: 51063 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2452
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 75994 eviction checks: 445432 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57351018 cached pages: 54470377
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 228.59MB allocated pages 57351018 mapped pages 57408920 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16877(131MB) allocated 22397 mapped]
[size 4: 18804(293MB) allocated 18827 mapped]
[size 8: 21231(663MB) allocated 21281 mapped]
[size 16: 8089(505MB) allocated 8311 mapped]
[size 32: 18665(2333MB) allocated 18690 mapped]
[size 64: 26862(6715MB) allocated 26920 mapped]
[size 128: 23831(11915MB) allocated 23869 mapped]
[size 256: 190588(190588MB) allocated 190713 mapped]
]
E20240323 04:38:09.028748  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.028807  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 235.23MB allocated pages 57349317 mapped pages 57408920 external mapped pages 2765334
[size 1: 20885(81MB) allocated 21540 mapped]
[size 2: 16878(131MB) allocated 22397 mapped]
[size 4: 18808(293MB) allocated 18827 mapped]
[size 8: 21233(663MB) allocated 21281 mapped]
[size 16: 8081(505MB) allocated 8311 mapped]
[size 32: 18660(2332MB) allocated 18690 mapped]
[size 64: 26864(6716MB) allocated 26920 mapped]
[size 128: 23827(11913MB) allocated 23869 mapped]
[size 256: 190585(190585MB) allocated 190713 mapped]
]
E20240323 04:38:09.034021  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 242.97MB allocated pages 57347336 mapped pages 57409432 external mapped pages 2765846
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16886(131MB) allocated 22397 mapped]
[size 4: 18812(293MB) allocated 18827 mapped]
[size 8: 21237(663MB) allocated 21281 mapped]
[size 16: 8107(506MB) allocated 8311 mapped]
[size 32: 18663(2332MB) allocated 18690 mapped]
[size 64: 26867(6716MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190570(190570MB) allocated 190713 mapped]
]
E20240323 04:38:09.031642  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.76GB tinySize: 748.16KB large size: 207.76GB
Cache entries: 51054 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2461
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76003 eviction checks: 445441 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57347432 cached pages: 54463910
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 242.59MB allocated pages 57347432 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16885(131MB) allocated 22397 mapped]
[size 4: 18811(293MB) allocated 18827 mapped]
[size 8: 21238(663MB) allocated 21281 mapped]
[size 16: 8036(502MB) allocated 8311 mapped]
[size 32: 18663(2332MB) allocated 18690 mapped]
[size 64: 26867(6716MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190570(190570MB) allocated 190713 mapped]
]
E20240323 04:38:09.038813  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.039031  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 261.77MB allocated pages 57342523 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16888(131MB) allocated 22397 mapped]
[size 4: 18812(293MB) allocated 18827 mapped]
[size 8: 21241(663MB) allocated 21281 mapped]
[size 16: 8042(502MB) allocated 8311 mapped]
[size 32: 18667(2333MB) allocated 18690 mapped]
[size 64: 26873(6718MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190554(190554MB) allocated 190713 mapped]
]
E20240323 04:38:09.037946  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.76GB tinySize: 748.16KB large size: 207.76GB
Cache entries: 51054 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2461
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76003 eviction checks: 445441 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57342991 cached pages: 54459814
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 259.94MB allocated pages 57342991 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16885(131MB) allocated 22397 mapped]
[size 4: 18817(294MB) allocated 18827 mapped]
[size 8: 21241(663MB) allocated 21281 mapped]
[size 16: 8036(502MB) allocated 8311 mapped]
[size 32: 18670(2333MB) allocated 18690 mapped]
[size 64: 26874(6718MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190554(190554MB) allocated 190713 mapped]
]
E20240323 04:38:09.044456  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.041893  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.75GB tinySize: 748.13KB large size: 207.75GB
Cache entries: 51052 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2463
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76005 eviction checks: 445443 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57342335 cached pages: 54459814
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 262.50MB allocated pages 57342335 mapped pages 57408920 external mapped pages 2765334
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16886(131MB) allocated 22397 mapped]
[size 4: 18818(294MB) allocated 18827 mapped]
[size 8: 21240(663MB) allocated 21281 mapped]
[size 16: 8031(501MB) allocated 8311 mapped]
[size 32: 18670(2333MB) allocated 18690 mapped]
[size 64: 26874(6718MB) allocated 26920 mapped]
[size 128: 23825(11912MB) allocated 23869 mapped]
[size 256: 190554(190554MB) allocated 190713 mapped]
]
E20240323 04:38:09.047709  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.047753  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 272.65MB allocated pages 57339738 mapped pages 57408920 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16889(131MB) allocated 22397 mapped]
[size 4: 18818(294MB) allocated 18827 mapped]
[size 8: 21247(663MB) allocated 21281 mapped]
[size 16: 8042(502MB) allocated 8311 mapped]
[size 32: 18677(2334MB) allocated 18690 mapped]
[size 64: 26870(6717MB) allocated 26920 mapped]
[size 128: 23822(11911MB) allocated 23869 mapped]
[size 256: 190546(190546MB) allocated 190713 mapped]
]
E20240323 04:38:09.051606  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 279.14MB allocated pages 57338075 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16888(131MB) allocated 22397 mapped]
[size 4: 18822(294MB) allocated 18827 mapped]
[size 8: 21250(664MB) allocated 21281 mapped]
[size 16: 8027(501MB) allocated 8311 mapped]
[size 32: 18681(2335MB) allocated 18690 mapped]
[size 64: 26871(6717MB) allocated 26920 mapped]
[size 128: 23822(11911MB) allocated 23869 mapped]
[size 256: 190536(190536MB) allocated 190713 mapped]
]
E20240323 04:38:09.050536  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.73GB tinySize: 747.99KB large size: 207.73GB
Cache entries: 51042 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2473
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76015 eviction checks: 445453 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57338139 cached pages: 54454282
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 278.89MB allocated pages 57338139 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16888(131MB) allocated 22397 mapped]
[size 4: 18822(294MB) allocated 18827 mapped]
[size 8: 21250(664MB) allocated 21281 mapped]
[size 16: 8029(501MB) allocated 8311 mapped]
[size 32: 18681(2335MB) allocated 18690 mapped]
[size 64: 26871(6717MB) allocated 26920 mapped]
[size 128: 23822(11911MB) allocated 23869 mapped]
[size 256: 190536(190536MB) allocated 190713 mapped]
]
E20240323 04:38:09.056095  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.056138  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 294.46MB allocated pages 57334154 mapped pages 57409432 external mapped pages 2765846
[size 1: 20876(81MB) allocated 21540 mapped]
[size 2: 16886(131MB) allocated 22397 mapped]
[size 4: 18821(294MB) allocated 18827 mapped]
[size 8: 21249(664MB) allocated 21281 mapped]
[size 16: 8035(502MB) allocated 8311 mapped]
[size 32: 18678(2334MB) allocated 18690 mapped]
[size 64: 26869(6717MB) allocated 26920 mapped]
[size 128: 23820(11910MB) allocated 23869 mapped]
[size 256: 190522(190522MB) allocated 190713 mapped]
]
E20240323 04:38:09.055198  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.71GB tinySize: 747.93KB large size: 207.71GB
Cache entries: 51038 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2477
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76019 eviction checks: 445457 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57334313 cached pages: 54450201
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 293.84MB allocated pages 57334313 mapped pages 57409432 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16888(131MB) allocated 22397 mapped]
[size 4: 18826(294MB) allocated 18827 mapped]
[size 8: 21253(664MB) allocated 21281 mapped]
[size 16: 8052(503MB) allocated 8311 mapped]
[size 32: 18679(2334MB) allocated 18690 mapped]
[size 64: 26869(6717MB) allocated 26920 mapped]
[size 128: 23820(11910MB) allocated 23869 mapped]
[size 256: 190522(190522MB) allocated 190713 mapped]
]
E20240323 04:38:09.063012  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.058771  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.71GB tinySize: 747.93KB large size: 207.71GB
Cache entries: 51038 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2477
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76019 eviction checks: 445457 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57334562 cached pages: 54450201
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 292.87MB allocated pages 57334562 mapped pages 57409432 external mapped pages 2765846
[size 1: 20876(81MB) allocated 21540 mapped]
[size 2: 16887(131MB) allocated 22397 mapped]
[size 4: 18825(294MB) allocated 18827 mapped]
[size 8: 21252(664MB) allocated 21281 mapped]
[size 16: 8050(503MB) allocated 8311 mapped]
[size 32: 18679(2334MB) allocated 18690 mapped]
[size 64: 26869(6717MB) allocated 26920 mapped]
[size 128: 23820(11910MB) allocated 23869 mapped]
[size 256: 190522(190522MB) allocated 190713 mapped]
]
E20240323 04:38:09.064047  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.064111  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 310.82MB allocated pages 57329965 mapped pages 57408920 external mapped pages 2765334
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16889(131MB) allocated 22397 mapped]
[size 4: 18830(294MB) allocated 18830 mapped]
[size 8: 21255(664MB) allocated 21281 mapped]
[size 16: 8064(504MB) allocated 8311 mapped]
[size 32: 18680(2335MB) allocated 18690 mapped]
[size 64: 26869(6717MB) allocated 26920 mapped]
[size 128: 23820(11910MB) allocated 23869 mapped]
[size 256: 190506(190506MB) allocated 190713 mapped]
]
E20240323 04:38:09.066720  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.70GB tinySize: 747.88KB large size: 207.70GB
Cache entries: 51035 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2480
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76022 eviction checks: 445460 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57330379 cached pages: 54446095
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 309.21MB allocated pages 57330379 mapped pages 57408932 external mapped pages 2765334
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16892(131MB) allocated 22397 mapped]
[size 4: 18830(294MB) allocated 18830 mapped]
[size 8: 21255(664MB) allocated 21281 mapped]
[size 16: 8085(505MB) allocated 8311 mapped]
[size 32: 18680(2335MB) allocated 18690 mapped]
[size 64: 26869(6717MB) allocated 26920 mapped]
[size 128: 23820(11910MB) allocated 23869 mapped]
[size 256: 190506(190506MB) allocated 190713 mapped]
]
E20240323 04:38:09.072160  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.072201  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 323.14MB allocated pages 57326811 mapped pages 57408932 external mapped pages 2765334
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16896(132MB) allocated 22397 mapped]
[size 4: 18835(294MB) allocated 18838 mapped]
[size 8: 21269(664MB) allocated 21281 mapped]
[size 16: 8097(506MB) allocated 8311 mapped]
[size 32: 18689(2336MB) allocated 18697 mapped]
[size 64: 26877(6719MB) allocated 26920 mapped]
[size 128: 23815(11907MB) allocated 23869 mapped]
[size 256: 190495(190495MB) allocated 190713 mapped]
]
E20240323 04:38:09.075006  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.68GB tinySize: 747.81KB large size: 207.68GB
Cache entries: 51030 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2485
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76027 eviction checks: 445465 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57328392 cached pages: 54442319
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 316.97MB allocated pages 57328392 mapped pages 57409196 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16902(132MB) allocated 22397 mapped]
[size 4: 18840(294MB) allocated 18840 mapped]
[size 8: 21280(665MB) allocated 21281 mapped]
[size 16: 8069(504MB) allocated 8311 mapped]
[size 32: 18695(2336MB) allocated 18697 mapped]
[size 64: 26872(6718MB) allocated 26920 mapped]
[size 128: 23821(11910MB) allocated 23869 mapped]
[size 256: 190495(190495MB) allocated 190713 mapped]
]
E20240323 04:38:09.080940  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.096529  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 328.47MB allocated pages 57325448 mapped pages 57409412 external mapped pages 2765334
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16901(132MB) allocated 22397 mapped]
[size 4: 18830(294MB) allocated 18842 mapped]
[size 8: 21251(664MB) allocated 21287 mapped]
[size 16: 8067(504MB) allocated 8311 mapped]
[size 32: 18664(2333MB) allocated 18702 mapped]
[size 64: 26853(6713MB) allocated 26920 mapped]
[size 128: 23818(11909MB) allocated 23869 mapped]
[size 256: 190477(190477MB) allocated 190711 mapped]
]
E20240323 04:38:09.097841  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 344.23MB allocated pages 57321412 mapped pages 57409412 external mapped pages 2765846
[size 1: 20877(81MB) allocated 21540 mapped]
[size 2: 16900(132MB) allocated 22397 mapped]
[size 4: 18829(294MB) allocated 18842 mapped]
[size 8: 21250(664MB) allocated 21287 mapped]
[size 16: 8069(504MB) allocated 8311 mapped]
[size 32: 18664(2333MB) allocated 18702 mapped]
[size 64: 26853(6713MB) allocated 26920 mapped]
[size 128: 23818(11909MB) allocated 23869 mapped]
[size 256: 190477(190477MB) allocated 190711 mapped]
]
E20240323 04:38:09.100278  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.66GB tinySize: 747.66KB large size: 207.66GB
Cache entries: 51020 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2495
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76037 eviction checks: 445475 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57320400 cached pages: 54437033
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 348.19MB allocated pages 57320400 mapped pages 57409412 external mapped pages 2765846
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16888(131MB) allocated 22397 mapped]
[size 4: 18827(294MB) allocated 18842 mapped]
[size 8: 21220(663MB) allocated 21287 mapped]
[size 16: 8065(504MB) allocated 8311 mapped]
[size 32: 18646(2330MB) allocated 18702 mapped]
[size 64: 26853(6713MB) allocated 26920 mapped]
[size 128: 23818(11909MB) allocated 23869 mapped]
[size 256: 190477(190477MB) allocated 190711 mapped]
]
E20240323 04:38:09.107064  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.101714  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.66GB tinySize: 747.66KB large size: 207.66GB
Cache entries: 51020 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2495
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76037 eviction checks: 445475 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57320537 cached pages: 54437033
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 347.65MB allocated pages 57320537 mapped pages 57409412 external mapped pages 2765846
[size 1: 20878(81MB) allocated 21540 mapped]
[size 2: 16887(131MB) allocated 22397 mapped]
[size 4: 18827(294MB) allocated 18842 mapped]
[size 8: 21220(663MB) allocated 21287 mapped]
[size 16: 8073(504MB) allocated 8311 mapped]
[size 32: 18646(2330MB) allocated 18702 mapped]
[size 64: 26853(6713MB) allocated 26920 mapped]
[size 128: 23818(11909MB) allocated 23869 mapped]
[size 256: 190477(190477MB) allocated 190711 mapped]
]
E20240323 04:38:09.108418  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.131866  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 373.39MB allocated pages 57313948 mapped pages 57408900 external mapped pages 2765334
[size 1: 20886(81MB) allocated 21540 mapped]
[size 2: 16894(131MB) allocated 22397 mapped]
[size 4: 18820(294MB) allocated 18842 mapped]
[size 8: 21208(662MB) allocated 21287 mapped]
[size 16: 8027(501MB) allocated 8311 mapped]
[size 32: 18621(2327MB) allocated 18702 mapped]
[size 64: 26826(6706MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190450(190450MB) allocated 190711 mapped]
]
E20240323 04:38:09.132555  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 384.19MB allocated pages 57311184 mapped pages 57409412 external mapped pages 2765846
[size 1: 20886(81MB) allocated 21540 mapped]
[size 2: 16894(131MB) allocated 22397 mapped]
[size 4: 18820(294MB) allocated 18842 mapped]
[size 8: 21208(662MB) allocated 21287 mapped]
[size 16: 8034(502MB) allocated 8311 mapped]
[size 32: 18621(2327MB) allocated 18702 mapped]
[size 64: 26826(6706MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190450(190450MB) allocated 190711 mapped]
]
E20240323 04:38:09.135048  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.63GB tinySize: 747.60KB large size: 207.63GB
Cache entries: 51016 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2499
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76041 eviction checks: 445479 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57311207 cached pages: 54429821
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 384.10MB allocated pages 57311207 mapped pages 57409412 external mapped pages 2765846
[size 1: 20887(81MB) allocated 21540 mapped]
[size 2: 16895(131MB) allocated 22397 mapped]
[size 4: 18821(294MB) allocated 18842 mapped]
[size 8: 21204(662MB) allocated 21287 mapped]
[size 16: 8042(502MB) allocated 8311 mapped]
[size 32: 18615(2326MB) allocated 18702 mapped]
[size 64: 26825(6706MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190450(190450MB) allocated 190711 mapped]
]
E20240323 04:38:09.141470  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.136657  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.63GB tinySize: 747.60KB large size: 207.63GB
Cache entries: 51016 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2499
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76041 eviction checks: 445479 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57311053 cached pages: 54429821
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 384.70MB allocated pages 57311053 mapped pages 57409412 external mapped pages 2765846
[size 1: 20885(81MB) allocated 21540 mapped]
[size 2: 16895(131MB) allocated 22397 mapped]
[size 4: 18819(294MB) allocated 18842 mapped]
[size 8: 21202(662MB) allocated 21287 mapped]
[size 16: 8013(500MB) allocated 8311 mapped]
[size 32: 18615(2326MB) allocated 18702 mapped]
[size 64: 26825(6706MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190450(190450MB) allocated 190711 mapped]
]
E20240323 04:38:09.143445  3750 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.149595  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 398.95MB allocated pages 57307406 mapped pages 57408900 external mapped pages 2765334
[size 1: 20884(81MB) allocated 21540 mapped]
[size 2: 16892(131MB) allocated 22397 mapped]
[size 4: 18821(294MB) allocated 18842 mapped]
[size 8: 21183(661MB) allocated 21287 mapped]
[size 16: 8015(500MB) allocated 8311 mapped]
[size 32: 18598(2324MB) allocated 18702 mapped]
[size 64: 26821(6705MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190440(190440MB) allocated 190711 mapped]
]
E20240323 04:38:09.152616  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.62GB tinySize: 747.47KB large size: 207.62GB
Cache entries: 51007 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2508
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76050 eviction checks: 445488 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57306799 cached pages: 54427015
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 401.32MB allocated pages 57306799 mapped pages 57408900 external mapped pages 2765334
[size 1: 20885(81MB) allocated 21540 mapped]
[size 2: 16879(131MB) allocated 22397 mapped]
[size 4: 18821(294MB) allocated 18842 mapped]
[size 8: 21167(661MB) allocated 21287 mapped]
[size 16: 8026(501MB) allocated 8311 mapped]
[size 32: 18588(2323MB) allocated 18702 mapped]
[size 64: 26821(6705MB) allocated 26920 mapped]
[size 128: 23823(11911MB) allocated 23869 mapped]
[size 256: 190440(190440MB) allocated 190711 mapped]
]
E20240323 04:38:09.158784  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.161846  3751 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 417.39MB allocated pages 57302683 mapped pages 57408900 external mapped pages 2765334
[size 1: 20883(81MB) allocated 21540 mapped]
[size 2: 16879(131MB) allocated 22397 mapped]
[size 4: 18821(294MB) allocated 18842 mapped]
[size 8: 21164(661MB) allocated 21287 mapped]
[size 16: 8046(502MB) allocated 8311 mapped]
[size 32: 18586(2323MB) allocated 18702 mapped]
[size 64: 26820(6705MB) allocated 26920 mapped]
[size 128: 23822(11911MB) allocated 23869 mapped]
[size 256: 190426(190426MB) allocated 190711 mapped]
]
E20240323 04:38:09.164783  3751 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.61GB tinySize: 747.41KB large size: 207.61GB
Cache entries: 51003 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2512
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76054 eviction checks: 445492 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57303159 cached pages: 54423096
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 415.54MB allocated pages 57303159 mapped pages 57408900 external mapped pages 2765334
[size 1: 20887(81MB) allocated 21540 mapped]
[size 2: 16883(131MB) allocated 22397 mapped]
[size 4: 18825(294MB) allocated 18842 mapped]
[size 8: 21168(661MB) allocated 21287 mapped]
[size 16: 8080(505MB) allocated 8311 mapped]
[size 32: 18586(2323MB) allocated 18702 mapped]
[size 64: 26820(6705MB) allocated 26920 mapped]
[size 128: 23822(11911MB) allocated 23869 mapped]
[size 256: 190426(190426MB) allocated 190711 mapped]
]
E20240323 04:38:09.170701  3751 MmapAllocator.cpp:415] [MEM] Mmap failed with 512 pages use MmapArena false
E20240323 04:38:09.180087  3750 MmapAllocator.cpp:404] [MEM] Data Mmap failedMemory Allocator[MMAP total capacity 219.00GB free capacity 437.51MB allocated pages 57297533 mapped pages 57408900 external mapped pages 2765334
[size 1: 20889(81MB) allocated 21540 mapped]
[size 2: 16887(131MB) allocated 22397 mapped]
[size 4: 18826(294MB) allocated 18842 mapped]
[size 8: 21171(661MB) allocated 21287 mapped]
[size 16: 8187(511MB) allocated 8311 mapped]
[size 32: 18585(2323MB) allocated 18702 mapped]
[size 64: 26816(6704MB) allocated 26920 mapped]
[size 128: 23817(11908MB) allocated 23869 mapped]
[size 256: 190401(190401MB) allocated 190711 mapped]
]
E20240323 04:38:09.183209  3751 Exceptions.h:69] Line: /app/presto-native-execution/velox/velox/common/memory/MemoryPool.cpp:1191, Function:handleAllocationFailure, Expression:  allocateContiguous failed with 512 pages from Memory Pool[op.10.0.0.Aggregation LEAF root[20240323_043257_00076_tjfjm_72] parent[node.10] MMAP track-usage thread-safe]<max capacity 208.00GB unlimited capacity used 5.19GB available 7.50MB reservation [used 5.19GB, reserved 5.20GB, min 0B] counters [allocs 41, frees 15, reserves 0, releases 0, collisions 0])> Mmap failed with 512 pages use MmapArena false Failed to evict from cache state: AsyncDataCache:
Cache size: 207.60GB tinySize: 747.37KB large size: 207.59GB
Cache entries: 51000 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2515
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76057 eviction checks: 445495 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57301058 cached pages: 54419726
, Source: RUNTIME, ErrorCode: MEM_ALLOC_ERROR
E20240323 04:38:09.183156  3750 MmapAllocator.cpp:406] [MEM] AsyncDataCache:
Cache size: 207.58GB tinySize: 747.31KB large size: 207.58GB
Cache entries: 50996 read pins: 160 write pins: 0 pinned shared: 586.47MB pinned exclusive: 0B
 num write wait: 7416 empty entries: 2519
Cache access miss: 127057 hit: 513945 hit bytes: 1.63TB eviction: 76061 eviction checks: 445499 aged out: 0
Prefetch entries: 22 bytes: 115.99MB
Alloc Megaclocks 1096275
Allocated pages: 57298267 cached pages: 54415698
Backing: Memory Allocator[MMAP total capacity 219.00GB free capacity 434.64MB allocated pages 57298267 mapped pages 57408900 external mapped pages 2765334
[size 1: 20891(81MB) allocated 21540 mapped]
[size 2: 16889(131MB) allocated 22397 mapped]
[size 4: 18828(294MB) allocated 18842 mapped]
[size 8: 21173(661MB) allocated 21287 mapped]
[size 16: 8223(513MB) allocated 8311 mapped]
[size 32: 18585(2323MB) allocated 18702 mapped]
[size 64: 26816(6704MB) allocated 26920 mapped]
[size 128: 23817(11908MB) allocated 23869 mapped]
[size 256: 190401(190401MB) allocated 190711 mapped]
]

@wills-feng wills-feng mentioned this issue Mar 27, 2024
Closed
facebook-github-bot pushed a commit that referenced this issue Apr 6, 2024
@facebook-github-bot
Handling MAP_FAILED in MmapAllocator (#9228)
84ae6bf 
Summary:
This PR is addressing this issue #9135

Pull Request resolved: #9228

Reviewed By: Yuhta

Differential Revision: D55609322

Pulled By: xiaoxmeng

fbshipit-source-id: c1921fd5551c553384fde4ffddf80ad659471094
@czentgr
Copy link
Collaborator Author

czentgr commented Apr 11, 2024

PR resolved the issue.

@czentgr czentgr closed this as completed Apr 11, 2024
Joe-Abraham pushed a commit to Joe-Abraham/velox that referenced this issue Jun 7, 2024
@Joe-Abraham
Handling MAP_FAILED in MmapAllocator (facebookincubator#9228)
fd3c4c6 
Summary:
This PR is addressing this issue facebookincubator#9135

Pull Request resolved: facebookincubator#9228

Reviewed By: Yuhta

Differential Revision: D55609322

Pulled By: xiaoxmeng

fbshipit-source-id: c1921fd5551c553384fde4ffddf80ad659471094
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage Newly created issue that needs attention.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wills-feng @ethanyzhang @czentgr