High memory usage while doing wallet rescan

[zebpayamarthakur]

Whenever we do wallet rescan bcoin process takes around 15.5 GB of RAM & then crashes.
After scanning 25000-30000 blocks bcoin process reaches around 15.5 GB RAM.

Sometimes bcoin process didn't crashes immediately, in that case we are unable to SSH server, as bcoin using majority of RAM.

Please look into this on urgent basis.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                        
18103 ubuntu    20   0 10.269g 9.023g  75012 R 109.6 57.6  21:41.92 bcoin

Below is my bcoin config -

# bcoin config file (~/.bcoin/bcoin.conf)

#
# Options
#

network: main

#
# Node
#

prefix: /blockchain/bcoindb
db: leveldb
# max-files: 64
# cache-size: 100

#
# Workers
#

workers: true
workers-size: 4
# workers-timeout: 5000

#
# Logger
#

log-level: debug
log-console: true
log-file: true

#
# Chain
#

prune: false
checkpoints: true
coin-cache: 100
# entry-cache: 5000
index-tx: true
index-address: true

#
# Mempool
#

# mempool-size: 100
# limit-free: true
# limit-free-relay: 15
# reject-absurd-fees: true
# replace-by-fee: false
# persistent-mempool: false

#
# Pool
#

# selfish: false
# compact: true
# bip37: false
# listen: true
# max-outbound: 8
# max-inbound: 30

# Proxy Server (browser=websockets, node=socks)
# proxy: foo:bar@127.0.0.1:9050
# onion: true
# upnp: true

# Custom list of DNS seeds
# seeds: seed.bitcoin.sipa.be

# Local Host & Port (to listen on)
host: ::
# port: 8333

# Public Host & Port (to advertise to peers)
# public-host: 1.2.3.4
# public-port: 8444

# Always try to connect to these nodes.
# nodes: 127.0.0.1,127.0.0.2

# Only try to connect to these nodes.
# only: 127.0.0.1,127.0.0.2

#
# Miner
#

# coinbase-flags: mined by bcoin
# coinbase-address: 1111111111111111111114oLvT2,1111111111111111111114oLvT2
# preverify: false
# max-block-weight: 4000000
# reserved-block-weight: 4000
# reserved-block-sigops: 400

#
# HTTP
#

http-host: ::
# http-port: 8332
# ssl: true
# ssl-cert: @/ssl/cert.crt
# ssl-key: @/ssl/priv.key
api-key: test-api-password
# no-auth: false
# cors: false

[pinheadmz]

Thanks for posting @zebpayamarthakur -- how big is your walletDB? Do you have multiple wallets? Multiple accounts and addresses?

[braydonf]

Possibly related issue #551

[zebpayamarthakur]

Thanks for posting @zebpayamarthakur -- how big is your walletDB? Do you have multiple wallets? Multiple accounts and addresses?

Wallet DB size is around 385 MB
I have only one watch only wallet currently
I have around 50 receiving address & 10 change addresses in wallet
Using default account only

[pinheadmz]

What version of node? node --version

[zebpayamarthakur]

What version of node? node --version

9.11.2

[pinheadmz]

Ok interesting, there have been other issues about memory usage in node v9 that don't occur in v8 or v10: #754

[zebpayamarthakur]

Ok interesting, there have been other issues about memory usage in node v9 that don't occur in v8 or v10: #754

Ok. I'll try updating node version to 10.15.3 LTS.
Thanks

[zebpayamarthakur]

Ok interesting, there have been other issues about memory usage in node v9 that don't occur in v8 or v10: #754

Updated node to 10.15.3 & started wallet rescan again.
Still having same issue.

[braydonf]

I think others have experienced a similar issue during rescan.

[braydonf]

This is possibly related to #864?