LeoFS's gateway do not response or return HTTP error code 5xx

[minhchuduc]

Environments:
Leofs version 1.2.6, installed using .deb file
Erlang R16B03-1 (erts-5.10.4) [source] [64-bit halfword] [smp:16:16] [async-threads:10] [kernel-poll:false]
Ubuntu 14.04 on Baremetal
Kernel 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 GNU/Linux

On a terminal i run stress test:

siege -c 500 -r 2 http://hehe1.s3.mydomain.com/300MB_file

(i checked to sure that 300MB_file is cached on disk @leofs_gateway node
and i monitored network @leofs_gateway node to sure that it not get data from storage nodes)

On another terminal i download some small file to test its responsiveness:

wget http://hehe1.s3.mydomain.com/IMAGES/0478e05a-d5e5-47ac-bab9-26278a2de1ec.jpg

And i receive 3 types of response randomly:

1: OK

 (it delay a bit, sometimes quite long, then...)
Saving to: `0478e05a-d5e5-47ac-bab9-26278a2de1ec.jpg.13'

100%[======================================================================================================>] 229,449     --.-K/s   in 0.002s

2015-03-04 13:44:10 (115 MB/s) - `0478e05a-d5e5-47ac-bab9-26278a2de1ec.jpg.13' saved [229449/229449]

Download time is "0.002s"
It's mean Time to first byte is so long (http://en.wikipedia.org/wiki/Time_To_First_Byte)

--

2: ERROR 503: Service Unavailable

...
HTTP request sent, awaiting response... 503 Service Unavailable
2015-03-04 13:44:11 ERROR 503: Service Unavailable.

3: ERROR 500: Internal Server Error

...
HTTP request sent, awaiting response... 500 Internal Server Error
2015-03-04 13:45:20 ERROR 500: Internal Server Error.

Output of my "leofs-adm status gateway_0@172.x.y.z":

-------------------------------+------------------
             Item              |       Value
-------------------------------+------------------
 Config-1: basic
--------------------------------------------------
 [basic]
-------------------------------+------------------
                       version | 1.2.6
                using protocol | s3
                       log dir | ./log/erlang
-------------------------------+------------------
 [http server related for rest/s3 api]
-------------------------------+------------------
                listening port | 80
            listening ssl port | 443
           number of_acceptors | 256
-------------------------------+------------------
 [cache-related]
-------------------------------+------------------
       http cache [true|false] | false
       number of cache_workers | 16
                  cache expire | 600
         cache max content len | 3123456789
            ram cache capacity | 6123456789
        disk cache capacity    | 30123456789
        disk cache threshold   | 20123456
        disk cache data dir    | ./cache/data
        disk cache journal dir | ./cache/journal
-------------------------------+------------------
 [large object related]
-------------------------------+------------------
      max number of chunk objs | 1000
           chunk object length | 5242880
             max object length | 5242880000
     reading  chunk obj length | 5242880
     threshold of chunk length | 5767168
-------------------------------+------------------
 Config-2: watchdog
-------------------------------+------------------
 [rex(rpc-proc)]               |
             watch interval(s) | 5
        threshold mem capacity | 335544320
-------------------------------+------------------
 [cpu]                         |
              watchdog eanbled | disabled
             watch interval(s) | 5
        threshold cpu load avg | 5.0
         threshold cpu util(%) | 100
-------------------------------+------------------
 [io]                          |
              watchdog enabled | enabled
             watch interval(s) | 1
        threshold input size/s | 134217728
       threshold output size/s | 134217728
-------------------------------+------------------
 Status-1: RING hash
-------------------------------+------------------
             current ring hash | 9d2edfe6
            previous ring hash | 9d2edfe6
-------------------------------+------------------
 Status-2: Erlang VM
-------------------------------+------------------
                    vm version | 5.10.4
               total mem usage | 75322280
              system mem usage | 54630464
               procs mem usage | 20690424
                 ets mem usage | 5585976
                         procs | 728/1048576
                   kernel_poll | true
              thread_pool_size | 32
-------------------------------+------------------

[yosukehara]

Here is the original - leo-project/leo_gateway#15

[yosukehara]

Thank you for your report. We'd like to know your environments and system information to check this issue. I've put partial of LeoFS's issue template as follows. It would be nice if you answer them.

In this situation, it seems the storage nodes become very high-load because of retrieving chunk objects of a large-object and also, network traffic may be very high between the gateway and the storage node(s).

We'd like to know the watchdog configuration of the gateway-node and the storage-node(s).

• Reference:
  • Watchdog

Environments

• What version of Erlang are you using?
• What kind of virtualization(VMWare/Docker/Xen...) are you using or using Baremetal?

System information

• Attach the result of issueing a status command(leofs-adm status).
• Attach the result of issueing a status command to a specific node(leofs-adm status $nodename) as necessary.
• Attach configuration files(leo_(gateway, storage).conf)
• Attach error log files if that exists.

[minhchuduc]

Hi, it added some info in this issue:
"(i checked to sure that 300MB_file is cached on disk @leofs_gateway node
and i monitored network @leofs_gateway node to sure that it not get data from storage nodes)"

Another problem i saw is: %util of disk @leofs_gateway start very low and increase very high

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.07    0.00    0.59   10.01    0.00   89.34
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00 19228.00   34.00 1231.00  1620.00 27008.00    45.26   122.67   92.75   22.71   94.68   0.76  96.00
sdb               0.00 19227.00   22.00 1330.00   640.00 33140.00    49.97    68.32   49.74   14.18   50.33   0.58  78.80

I checked and found that because of swap increase so much on this stress test (i attached htop screenshot of leof_gateway)

When i stop stress test, memory usage is return to normal.
Maybe it (full memory) is the cause of the problem i reported.

[yosukehara]

A large object is divided to chunk objects in order to equalize disk usage into the storage cluster.
If you request retrieving many large object to LeoFS (similar to this case), the storage node's disk util become high.

We recommend you turn on watchdog of every LeoFS storage node.

• LeoFS Storage's watchdog configuration:
  • https://github.com/leo-project/leo_storage/blob/develop/priv/leo_storage.conf#L79-L95
  • https://github.com/leo-project/leo_storage/blob/develop/priv/leo_storage.conf#L114-L148

Reference:

• LeoFS Storage

[yosukehara]

Regarding LeoFS Gateway of this chart (htop), it seems OS page cache does not exists, which may be the root cause.

[mocchira]

@minhchuduc Thank you for detailed report.
IMHO, The point of this issue is whether 22G memory requested by leo_gateway with 500 concurrent connections is vaild or NOT.
So we will look into this issue on our environment AND report the result on this thread.

Thanks.

[mocchira]

@minhchuduc
Regarding except memory usage,
The behavior you receive 3 types of response randomly is expected behavior.

Since number of_acceptors set to 256 limits concurrent processes on leo_gateway,
There is a possibility that some processes tcp:accepted have to wait for other processes finish downloading.

• 1: OK means this connection could get started donwloading within a certain timeout( probably be determined at siege )
• 2: ERROR 503: Service Unavailable means this connection could get started downloading but limited operations due to I/O watchdog mechanizm detect high I/O utilization.
• 3: ERROR 500: Internal Server Error means this connection got timeout due to not start downloading within a certain timeout

[mocchira]

@minhchuduc if you spare time,
Could you try the same stress test with modifying below parameter in leo_(gateway|storage).conf.

### erlang.distribution_buffer_size = 32768 
erlang.distribution_buffer_size = 1024

https://github.com/leo-project/leo_gateway/blob/develop/priv/leo_gateway.conf#L338

Since the default setting consume 32MB memory for send buffer per one tcp connection at most,
This might cause high memory usage.

• Related links
  • http://erlang.org/doc/man/erlang.html#system_info_dist_buf_busy_limit
  • http://erlang.org/doc/man/erl.html#+zdbbl

[minhchuduc]

i changed to

erlang.distribution_buffer_size = 1024

and stress test again with 200 concurrent connections
The memory slowly increase to 2.3GB, 2.4GB, 2.5,.... and suddenly jump to 6GB, 8GB, 11GB,... then jump back and stable at 8GB memory usage.

I open another terminal and download some small files:

wget http://hehe1.s3.mydomain.com/IMAGES/0478e05a-d5e5-47ac-bab9-26278a2de1ec.jpg

90% request is resonse very fast
10% fail fast with response

HTTP request sent, awaiting response... 503 Service Unavailable
2015-03-04 16:05:45 ERROR 503: Service Unavailable.

Quite good!

Therefore, with this configuration, when i want to increase concurrent connections, i can increase number of_acceptors and install more ram for server. That's right?

[minhchuduc]

But if i stress test with 300 concurrent connections, the problem happen again (out of memory, then using swap)

Maybe this will happen if concurrent connections > number of_acceptors

[mocchira]

@minhchuduc Thanks for quick try!

Therefore, with this configuration, when i want to increase concurrent connections, i can increase ?>> number of_acceptors and install more ram for server. That's right?

Right.

But if i stress test with 300 concurrent connections, the problem happen again (out of memory, >> then using swap)
Maybe this will happen if concurrent connections > number of_acceptors

This behavior should NOT happen even if concurrent connections > number of_acceptors.
Instead should be responding to 503 to clients.
So we will continue to survey this issue.

[mocchira]

@minhchuduc good morning.

I conducted stress tests that is similar with your cases as much as possible.

The different points are described below.

Differences of leo_gateway spec

• OS: CentOS6.5
• CPU Cores: 24
• Memory: 96GB
• Network bandwidth: 10Gbps
• Erlang Runtime: 17B04

The result are very stable in our environment even if concurrent connections > number of_acceptors.

siege commands

siege -c 200 -r 2 http://leofs13.rit.rakuten.com:8080/bbb/300m.bin
siege -c 300 -r 2 http://leofs13.rit.rakuten.com:8080/bbb/300m.bin
siege -c 400 -r 2 http://leofs13.rit.rakuten.com:8080/bbb/300m.bin

siege.log

Date & Time	Trans	Elap Time	Data Trans	Resp Time	Trans Rate	Throughput	Concurrent	OKAY	Failed
2015-03-05 10:29:13	400	145.85	120000	65.34	2.74	822.76	179.21	400	0
2015-03-05 10:35:29	600	321.29	180000	146.14	1.87	560.24	272.91	600	0
2015-03-05 10:46:11	800	473.50	240000	217.65	1.69	506.86	367.74	800	0

htop on leo_gateway

concurrent clients: 200

concurrent clients: 300

concurrent clients: 400

Since the memory usage of leo_gateway is stable in our environment,
I'm curious what consumes your memory.
So could you issue the below commands on leo_gateway remote shell?

### login leo_gateway remote shell
${LEOFS_INSTALL_PATH}/leo_gateway/bin/leo_gateway remote_console

erlang:memory().
recon_alloc:memory(used).
recon_alloc:memory(allocated).
recon_alloc:memory(allocated_types).
recon_alloc:fragmentation(current).
recon_alloc:fragmentation(max).

also I will try with R16B03-1 from now and share the result later.

[minhchuduc]

Hi, this is the output of your commands:
1. when normal operation:

(gateway_0@172.30.1.22)1> erlang:memory().
[{total,73859504},
 {processes,21225936},
 {processes_used,21224720},
 {system,52633568},
 {atom,1098921},
 {atom_used,1090104},
 {binary,5710328},
 {code,32171158},
 {ets,5624560},
 {low,31949248}]
(gateway_0@172.30.1.22)2> recon_alloc:memory(used).
42108232
(gateway_0@172.30.1.22)3> recon_alloc:memory(allocated).
102506496
(gateway_0@172.30.1.22)4> recon_alloc:memory(allocated_types).
[{binary_alloc,11538432},
 {driver_alloc,4980736},
 {eheap_alloc,11014144},
 {ets_alloc,6029312},
 {fix_alloc,4456448},
 {ll_alloc,37748736},
 {sl_alloc,4456448},
 {std_alloc,4456448},
 {temp_alloc,17825792}]
(gateway_0@172.30.1.22)5> recon_alloc:fragmentation(current).
[{{ll_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7752342224121094},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,29264112},
   {mbcs_carriers_size,37748736}]},
 {{eheap_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.2561391194661458},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,402872},
   {mbcs_carriers_size,1572864}]},
 {{temp_alloc,16},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,15},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,14},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,13},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,12},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,11},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,9},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,8},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,7},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,6},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,5},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,4},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,3},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{binary_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.25419921875},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,333184},
   {mbcs_carriers_size,...}]},
 {{eheap_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.314495849609375},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,...},
   {...}]},
 {{ets_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.4506683349609375},
   {sbcs_block_size,0},
   {sbcs_carriers_size,...},
   {...}|...]},
 {{eheap_alloc,1},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.50721435546875},
   {sbcs_block_size,...},
   {...}|...]},
 {{eheap_alloc,9},
  [{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
 {{eheap_alloc,5},[{sbcs_usage,...},{...}|...]},
 {{binary_alloc,3},[{...}|...]},
 {{binary_alloc,...},[...]},
 {{...},...},
 {...}|...]
(gateway_0@172.30.1.22)6> recon_alloc:fragmentation(max).
[{{ll_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7760336134168837},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,29294288},
   {mbcs_carriers_size,37748736}]},
 {{binary_alloc,9},
  [{sbcs_usage,0.999228181282759},
   {mbcs_usage,0.8046424333439317},
   {sbcs_block_size,146801936},
   {sbcs_carriers_size,146915328},
   {mbcs_block_size,18140168},
   {mbcs_carriers_size,22544384}]},
 {{binary_alloc,2},
  [{sbcs_usage,0.9994894157744948},
   {mbcs_usage,0.7169175581498579},
   {sbcs_block_size,5241824256},
   {sbcs_carriers_size,5244502016},
   {mbcs_block_size,4134584},
   {mbcs_carriers_size,5767168}]},
 {{binary_alloc,11},
  [{sbcs_usage,0.9992311762051131},
   {mbcs_usage,0.5996848658511513},
   {sbcs_block_size,83887072},
   {sbcs_carriers_size,83951616},
   {mbcs_block_size,5973744},
   {mbcs_carriers_size,9961472}]},
 {{binary_alloc,1},
  [{sbcs_usage,0.999448423889045},
   {mbcs_usage,0.8096575055803571},
   {sbcs_block_size,5098958784},
   {sbcs_carriers_size,5101772800},
   {mbcs_block_size,2971456},
   {mbcs_carriers_size,3670016}]},
 {{binary_alloc,3},
  [{sbcs_usage,0.9996136359688093},
   {mbcs_usage,0.6782316320082721},
   {sbcs_block_size,3446897616},
   {sbcs_carriers_size,3448229888},
   {mbcs_block_size,3022504},
   {mbcs_carriers_size,4456448}]},
 {{eheap_alloc,9},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.40552475873161764},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,1807200},
   {mbcs_carriers_size,4456448}]},
 {{eheap_alloc,4},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.4598783605238971},
   {sbcs_block_size,2359824},
   {sbcs_carriers_size,2371584},
   {mbcs_block_size,2049424},
   {mbcs_carriers_size,4456448}]},
 {{binary_alloc,0},
  [{sbcs_usage,0.9999871064911264},
   {mbcs_usage,0.43275669642857145},
   {sbcs_block_size,15414069160},
   {sbcs_carriers_size,15414267904},
   {mbcs_block_size,1588224},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,6},
  [{sbcs_usage,0.9993929476351351},
   {mbcs_usage,0.416595458984375},
   {sbcs_block_size,3332120},
   {sbcs_carriers_size,3334144},
   {mbcs_block_size,1528912},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,10},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.46281869070870535},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,1698552},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,2},
  [{sbcs_usage,0.9781622385700389},
   {mbcs_usage,0.5684670840992647},
   {sbcs_block_size,2059368},
   {sbcs_carriers_size,2105344},
   {mbcs_block_size,2533344},
   {mbcs_carriers_size,4456448}]},
 {{binary_alloc,10},
  [{sbcs_usage,0.9992294679901408},
   {mbcs_usage,0.6789689497514204},
   {sbcs_block_size,141559192},
   {sbcs_carriers_size,141668352},
   {mbcs_block_size,3915728},
   {mbcs_carriers_size,5767168}]},
 {{binary_alloc,8},
  [{sbcs_usage,0.9992285080015613},
   {mbcs_usage,0.6198963838465074},
   {sbcs_block_size,115344416},
   {sbcs_carriers_size,115433472},
   {mbcs_block_size,2762536},
   {mbcs_carriers_size,4456448}]},
 {{binary_alloc,4},
  [{sbcs_usage,0.9992275243320491},
   {mbcs_usage,0.807218017578125},
   {sbcs_block_size,325061216},
   {sbcs_carriers_size,325312512},
   {mbcs_block_size,5290184},
   {mbcs_carriers_size,6553600}]},
 {{eheap_alloc,3},
  [{sbcs_usage,0.9985569761668321},
   {mbcs_usage,0.6567927769252232},
   {sbcs_block_size,4118720},
   {sbcs_carriers_size,4124672},
   {mbcs_block_size,2410440},
   {mbcs_carriers_size,3670016}]},
 {{binary_alloc,6},
  [{sbcs_usage,0.9992278545639567},
   {mbcs_usage,0.8376312255859375},
   {sbcs_block_size,403705192},
   {sbcs_carriers_size,404017152},
   {mbcs_block_size,4830760},
   {mbcs_carriers_size,5767168}]},
 {{eheap_alloc,12},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.3268839518229167},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,514144},
   {mbcs_carriers_size,1572864}]},
 {{temp_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.00948333740234375},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,9944},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,16},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.01744842529296875},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,18296},
   {mbcs_carriers_size,...}]},
 {{temp_alloc,15},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.020294189453125},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,...},
   {...}]},
 {{temp_alloc,14},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.03044891357421875},
   {sbcs_block_size,0},
   {sbcs_carriers_size,...},
   {...}|...]},
 {{temp_alloc,12},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.06250762939453125},
   {sbcs_block_size,...},
   {...}|...]},
 {{temp_alloc,11},
  [{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
 {{temp_alloc,10},[{sbcs_usage,...},{...}|...]},
 {{temp_alloc,9},[{...}|...]},
 {{temp_alloc,...},[...]},
 {{...},...},
 {...}|...]

2. when running-out-of memory:

(gateway_0@172.30.1.22)7> erlang:memory().
[{total,16140425584},
 {processes,27558176},
 {processes_used,27555136},
 {system,16112867408},
 {atom,1098921},
 {atom_used,1090104},
 {binary,16060039768},
 {code,32171158},
 {ets,5635168},
 {low,32177464}]
(gateway_0@172.30.1.22)8> recon_alloc:memory(used).
16128435208
(gateway_0@172.30.1.22)9> recon_alloc:memory(allocated).
16191262720
(gateway_0@172.30.1.22)10> recon_alloc:memory(allocated_types).                                                                       [305/9667]
[{binary_alloc,16092696576},
 {driver_alloc,4718592},
 {eheap_alloc,18874368},
 {ets_alloc,6029312},
 {fix_alloc,4456448},
 {ll_alloc,37748736},
 {sl_alloc,4456448},
 {std_alloc,4456448},
 {temp_alloc,17825792}]
(gateway_0@172.30.1.22)11> recon_alloc:fragmentation(current).
[{{ll_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7752342224121094},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,29264112},
   {mbcs_carriers_size,37748736}]},
 {{eheap_alloc,13},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.38880484444754465},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,1426920},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,8},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.2522176106770833},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,396704},
   {mbcs_carriers_size,1572864}]},
 {{binary_alloc,13},
  [{sbcs_usage,0.9992269833138173},
   {mbcs_usage,0.227850341796875},
   {sbcs_block_size,99615480},
   {sbcs_carriers_size,99692544},
   {mbcs_block_size,298648},
   {mbcs_carriers_size,1310720}]},
 {{temp_alloc,16},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,15},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,14},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,13},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,12},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,11},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,9},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,8},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,7},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,6},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,5},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,4},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,3},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,1048576}]},
 {{temp_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,0},
   {mbcs_carriers_size,...}]},
 {{temp_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.0},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,...},
   {...}]},
 {{eheap_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.4365946451822917},
   {sbcs_block_size,0},
   {sbcs_carriers_size,...},
   {...}|...]},
 {{ets_alloc,10},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.450714111328125},
   {sbcs_block_size,...},
   {...}|...]},
 {{eheap_alloc,6},
  [{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
 {{eheap_alloc,9},[{sbcs_usage,...},{...}|...]},
 {{eheap_alloc,1},[{...}|...]},
 {{eheap_alloc,...},[...]},
 {{...},...},
 {...}|...]
(gateway_0@172.30.1.22)12> recon_alloc:fragmentation(max).
[{{ll_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7760336134168837},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,29294288},
   {mbcs_carriers_size,37748736}]},
 {{binary_alloc,9},
  [{sbcs_usage,0.999228181282759},
   {mbcs_usage,0.8046424333439317},
   {sbcs_block_size,146801936},
   {sbcs_carriers_size,146915328},
   {mbcs_block_size,18140168},
   {mbcs_carriers_size,22544384}]},
 {{binary_alloc,2},
  [{sbcs_usage,0.9994894157744948},
   {mbcs_usage,0.7169175581498579},
   {sbcs_block_size,5241824256},
   {sbcs_carriers_size,5244502016},
   {mbcs_block_size,4134584},
   {mbcs_carriers_size,5767168}]},
 {{binary_alloc,11},
  [{sbcs_usage,0.9992311762051131},
   {mbcs_usage,0.5996848658511513},
   {sbcs_block_size,83887072},
   {sbcs_carriers_size,83951616},
   {mbcs_block_size,5973744},
   {mbcs_carriers_size,9961472}]},
 {{binary_alloc,1},
  [{sbcs_usage,0.999448423889045},
   {mbcs_usage,0.8096575055803571},
   {sbcs_block_size,5098958784},
   {sbcs_carriers_size,5101772800},
   {mbcs_block_size,2971456},
   {mbcs_carriers_size,3670016}]},
 {{binary_alloc,3},
  [{sbcs_usage,0.9996136359688093},
   {mbcs_usage,0.6782316320082721},
   {sbcs_block_size,3446897616},
   {sbcs_carriers_size,3448229888},
   {mbcs_block_size,3022504},
   {mbcs_carriers_size,4456448}]},
 {{eheap_alloc,4},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.4598783605238971},
   {sbcs_block_size,2359824},
   {sbcs_carriers_size,2371584},
   {mbcs_block_size,2049424},
   {mbcs_carriers_size,4456448}]},
 {{binary_alloc,0},
  [{sbcs_usage,0.9999871064911264},
   {mbcs_usage,0.43275669642857145},
   {sbcs_block_size,15414069160},
   {sbcs_carriers_size,15414267904},
   {mbcs_block_size,1588224},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,9},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.49423038258272056},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,2202512},
   {mbcs_carriers_size,4456448}]},
 {{eheap_alloc,6},
  [{sbcs_usage,0.9993929476351351},
   {mbcs_usage,0.416595458984375},
   {sbcs_block_size,3332120},
   {sbcs_carriers_size,3334144},
   {mbcs_block_size,1528912},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,10},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.46281869070870535},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,1698552},
   {mbcs_carriers_size,3670016}]},
 {{eheap_alloc,2},
  [{sbcs_usage,0.9781622385700389},
   {mbcs_usage,0.5684670840992647},
   {sbcs_block_size,2059368},
   {sbcs_carriers_size,2105344},
   {mbcs_block_size,2533344},
   {mbcs_carriers_size,4456448}]},
 {{binary_alloc,10},
  [{sbcs_usage,0.9992294679901408},
   {mbcs_usage,0.6789689497514204},
   {sbcs_block_size,141559192},
   {sbcs_carriers_size,141668352},
   {mbcs_block_size,3915728},
   {mbcs_carriers_size,5767168}]},
 {{binary_alloc,8},
  [{sbcs_usage,0.9992285080015613},
   {mbcs_usage,0.6198963838465074},
   {sbcs_block_size,115344416},
   {sbcs_carriers_size,115433472},
   {mbcs_block_size,2762536},
   {mbcs_carriers_size,4456448}]},
 {{eheap_alloc,13},
  [{sbcs_usage,0.9991396201768489},
   {mbcs_usage,0.5494341169084821},
   {sbcs_block_size,1272760},
   {sbcs_carriers_size,1273856},
   {mbcs_block_size,2016432},
   {mbcs_carriers_size,3670016}]},
 {{binary_alloc,4},
  [{sbcs_usage,0.9992275243320491},
   {mbcs_usage,0.807218017578125},
   {sbcs_block_size,325061216},
   {sbcs_carriers_size,325312512},
   {mbcs_block_size,5290184},
   {mbcs_carriers_size,6553600}]},
 {{eheap_alloc,3},
  [{sbcs_usage,0.9985569761668321},
   {mbcs_usage,0.6567927769252232},
   {sbcs_block_size,4118720},
   {sbcs_carriers_size,4124672},
   {mbcs_block_size,2410440},
   {mbcs_carriers_size,3670016}]},
 {{binary_alloc,6},
  [{sbcs_usage,0.9992278545639567},
   {mbcs_usage,0.8376312255859375},
   {sbcs_block_size,403705192},
   {sbcs_carriers_size,404017152},
   {mbcs_block_size,4830760},
   {mbcs_carriers_size,5767168}]},
 {{binary_alloc,13},
  [{sbcs_usage,0.9992308168144307},
   {mbcs_usage,0.3266550699869792},
   {sbcs_block_size,183502904},
   {sbcs_carriers_size,183644160},
   {mbcs_block_size,513784},
   {mbcs_carriers_size,1572864}]},
 {{temp_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.00948333740234375},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,9944},
   {mbcs_carriers_size,...}]},
 {{eheap_alloc,11},
  [{sbcs_usage,0.9950412888601037},
   {mbcs_usage,0.44184657505580355},
   {sbcs_block_size,1573216},
   {sbcs_carriers_size,1581056},
   {mbcs_block_size,...},
   {...}]},
 {{temp_alloc,15},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.020294189453125},
   {sbcs_block_size,0},
   {sbcs_carriers_size,...},
   {...}|...]},
 {{temp_alloc,14},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.03044891357421875},
   {sbcs_block_size,...},
   {...}|...]},
 {{temp_alloc,16},
  [{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
 {{temp_alloc,12},[{sbcs_usage,...},{...}|...]},
 {{temp_alloc,11},[{...}|...]},
 {{temp_alloc,...},[...]},
 {{...},...},
 {...}|...]

[mocchira]

@minhchuduc Thank you.

Found the guilty one from your output

{{binary_alloc,0},
  [{sbcs_usage,0.9999871064911264},
   {mbcs_usage,0.43275669642857145},
   {sbcs_block_size,15414069160},
   {sbcs_carriers_size,15414267904},
   {mbcs_block_size,1588224},
   {mbcs_carriers_size,3670016}]}

This binary allocator held around 16GB.

and, in our env with R16B03-1,
We've succeeded in reproducing this issue!
So this might be Erlang VM problems(we will look for R16B03-1 issues which could cause this problem, and try to find temporary solution for R16B03-1 if possible).

If we could specify this issue caused by R16B03-1,
we will close this issue.

[yosukehara]

@mocchira We need to provide packages of LeoFS w/Erlang 17.4. from the next time and the next version.