fix: lnd backend mode switching bug #724

reliveyy · 2020-09-16T14:06:04Z

No description provided.

raladev · 2020-09-17T12:40:10Z

switching to native mode does not work for testnet, it seems conf file is missed:
Screenshot from 2020-09-17 15-12-52
also when i switched to light from native, i got locked lndbtc, but i cant unlock it using unlock command (because xud is not locked). So i used lndbtc-lncli unlock, but status of lnd in getinfo call was Disconnected and there was no BTC currency in getbalance call. I got good state for lndbtc only after xud restart. So, we need to do something with that.
Screenshot from 2020-09-17 15-37-02
Screenshot from 2020-09-17 15-36-58
Screenshot from 2020-09-17 15-36-51
just a note, switching will not work for connext for now, because of persistence of URI in connext db

raladev

Above

kilrau · 2020-09-17T16:09:20Z

* just a note, switching will not work for connext for now, because of persistence of URI in connext db

As discussed, we can ignore this for now. I opened #727 as follow-up.

reliveyy · 2020-09-22T12:03:49Z

@raladev

switching to native mode does not work for testnet, it seems conf file is missed

Sorry I've tried many times and I cannot reproduce .conf file missing case for bitcoind, litecoind & geth

also when i switched to light from native, i got locked lndbtc, but i cant unlock it using unlock command (because xud is not locked). So i used lndbtc-lncli unlock, but status of lnd in getinfo call was Disconnected and there was no BTC currency in getbalance call. I got good state for lndbtc only after xud restart. So, we need to do something with that.

It's confirmed. When L1 nodes switching modes we should restart xud otherwise unlock won't work. I think it's because the xud cannot tell underlying nodes configuration changes only from the RPC responses. But it's ticky to implement this feature in xud-docker because we do not presist the previous configurations.

A simple workaround would be comparing lnd.conf in utils and detect mode changes of L1 nodes. But a proper fix should be dumping all configuration keys as a snapshot and persisted in network_dir. I will create another PR for the proper fix.

reliveyy · 2020-09-22T14:39:14Z

We argee not to apply such kind of xud-docker workaround fix for such unlock case during the call. So @kilrau could you confirm on the first issue @raladev reported.

Change bitcoind mode to native
Check bitcoind work or not? For me it's syncing normally. (I think the log line ".conf file missing" is normal and bitcoind should work anyway)

sangaman · 2020-09-22T15:45:24Z

I'll respond in the related xud issue, I would expect auto unlock to work as long as the password doesn't change between the two lnd backends.

kilrau · 2020-09-22T16:25:46Z

Thanks, I am testing lndbtc light -> native -> light now on a fresh testnet setup.

kilrau · 2020-09-22T17:35:17Z

synced a fresh testnet env with xud -b fix-lnd-mode-switching

testnet > status
┌───────────┬────────────────────────────────────────────────┐
│ SERVICE   │ STATUS                                         │
├───────────┼────────────────────────────────────────────────┤
│ bitcoind  │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ litecoind │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ geth      │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ lndbtc    │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ lndltc    │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ connext   │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ boltz     │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ xud       │ Ready                                          │
└───────────┴────────────────────────────────────────────────┘

exit env (env still running), configure testnet.conf with bitcoind mode = "native"
xud -b fix-lnd-mode-switching, changes are detected and sync starts:

kilrau@K-Yoga:~/.xud-docker/testnet$ xud
1) Simnet
2) Testnet
3) Mainnet
Please choose the network: 2
Pulling image exchangeunion/utils:latest
🚀 Launching testnet environment
🌍 Checking for updates...
- Image exchangeunion/bitcoind:latest: missing
- Image exchangeunion/lndbtc:latest: missing
- Image exchangeunion/lndltc:latest: missing
- Container testnet_bitcoind_1: missing
- Container testnet_lndbtc_1: outdated (image)
- Container testnet_lndltc_1: outdated (image, environment)
A new version is available. Would you like to upgrade (Warning: this may restart your environment and cancel all open orders)? [Y/n] 
Pulling exchangeunion/bitcoind:latest...
cbdbe7a5bc2a: Already exists
9b54e87c68ea: Pull complete
6cdec4ccef1d: Pull complete
Pulling exchangeunion/lndbtc:latest...
cbdbe7a5bc2a: Already exists
7c770b46e6c5: Pull complete
fda3510e7a89: Pull complete
77980096db42: Pull complete
8510cdbf3734: Pull complete
b875395838d7: Pull complete
b86e61887933: Pull complete
167c9778b496: Pull complete
f5faddd47d15: Pull complete
Pulling exchangeunion/lndltc:latest...
cbdbe7a5bc2a: Already exists
d7949b6a87b2: Pull complete
73e60e3bd8a2: Pull complete
b2ed116f0675: Pull complete
386dac52c528: Pull complete
9a88e336db70: Pull complete
94fd5cc86218: Pull complete
3edc969906ef: Pull complete
bbc199154fde: Pull complete
Creating testnet_bitcoind_1...
Recreating testnet_lndbtc_1...
Recreating testnet_lndltc_1...

🏃 Warming up...


                           .___           __  .__   
          ___  _____ __  __| _/     _____/  |_|  |  
          \  \/  /  |  \/ __ |    _/ ___\   __\  |  
           >    <|  |  / /_/ |    \  \___|  | |  |__
          /__/\_ \____/\____ |     \___  >__| |____/
                \/          \/         \/           
--------------------------------------------------------------

testnet > status
┌───────────┬────────────────────────────────────────────────┐
│ SERVICE   │ STATUS                                         │
├───────────┼────────────────────────────────────────────────┤
│ bitcoind  │ Syncing 0.00% (0/110000)                       │
├───────────┼────────────────────────────────────────────────┤
│ litecoind │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ geth      │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ lndbtc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ lndltc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ connext   │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ boltz     │ btc down; ltc down                             │
├───────────┼────────────────────────────────────────────────┤
│ xud       │ Ready                                          │
└───────────┴────────────────────────────────────────────────┘

lndbtc gets marked as disconnected from here : https://paste.ubuntu.com/p/DgHt2q3pvr/

Syncing

testnet > status
┌───────────┬────────────────────────────────────────────────┐
│ SERVICE   │ STATUS                                         │
├───────────┼────────────────────────────────────────────────┤
│ bitcoind  │ Syncing 10.04% (184333/1835564)                │
├───────────┼────────────────────────────────────────────────┤
│ litecoind │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ geth      │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ lndbtc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ lndltc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ connext   │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ boltz     │ btc down; ltc down                             │
├───────────┼────────────────────────────────────────────────┤
│ xud       │ Ready                                          │
└───────────┴────────────────────────────────────────────────┘

Syncing done, lndbtc stays disconnected & locked

testnet > status
┌───────────┬────────────────────────────────────────────────┐
│ SERVICE   │ STATUS                                         │
├───────────┼────────────────────────────────────────────────┤
│ bitcoind  │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ litecoind │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ geth      │ Ready (light mode)                             │
├───────────┼────────────────────────────────────────────────┤
│ lndbtc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ lndltc    │ Wallet locked. Unlock with xucli unlock.       │
├───────────┼────────────────────────────────────────────────┤
│ connext   │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ boltz     │ btc down; ltc down                             │
├───────────┼────────────────────────────────────────────────┤
│ xud       │ Ready                                          │
└───────────┴────────────────────────────────────────────────┘
testnet > lndbtc-lncli getinfo
[lncli] Wallet is encrypted. Please unlock using 'lncli unlock', or set password using 'lncli create' if this is the first time starting lnd.

To be fixed:

lndltc was restarted and couldn't get unlocked in the same wy even though I didn't change a thing in the config for lndltc. @reliveyy

kilrau

Above

kilrau · 2020-09-24T13:46:52Z

Update:

the lndltc problem is gone, not sure what was wrong there

Todo:

now lndbtc fails to start with the following error after switching from light -> native on this branch. Reproduced twice:

2020-09-24 13:44:35,912 INFO spawned: 'lnd' with pid 82
[entrypoint] Enabling native mode
Waiting for lnd-bitcoin onion address...
Onion address for lnd-bitcoin is cg5a4nxib5icty2zhtleqnx2zaskmvwrm2x5rc7p42rlzjydumivbcyd.onion
Attempting automatic RPC configuration to bitcoind
unable to load RPC credentials for bitcoind: unable to extract RPC credentials: open /root/.bitcoin/bitcoin.conf: no such file or directory, cannot start w/o RPC connection
2020-09-24 13:44:35,996 CRIT uncaptured python exception, closing channel <POutputDispatcher at 140316755739696 for <Subprocess at 140316756005936 with name lnd in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Invalid seek [/usr/lib/python3.8/site-packages/supervisor/supervisord.py|runforever|218] [/usr/lib/python3.8/site-packages/supervisor/dispatchers.py|handle_read_event|270] [/usr/lib/python3.8/site-packages/supervisor/dispatchers.py|record_output|204] [/usr/lib/python3.8/site-packages/supervisor/dispatchers.py|_log|173] [/usr/lib/python3.8/site-packages/supervisor/loggers.py|info|327] [/usr/lib/python3.8/site-packages/supervisor/loggers.py|log|345] [/usr/lib/python3.8/site-packages/supervisor/loggers.py|emit|227] [/usr/lib/python3.8/site-packages/supervisor/loggers.py|doRollover|264])
2020-09-24 13:44:35,998 INFO exited: lnd (exit status 1; not expected)
2020-09-24 13:44:36,999 INFO gave up: lnd entered FATAL state, too many start retries too quickly

reliveyy requested a review from raladev September 16, 2020 14:06

reliveyy self-assigned this Sep 16, 2020

raladev requested changes Sep 17, 2020

View reviewed changes

kilrau added the P1 top priority label Sep 17, 2020

reliveyy force-pushed the fix-lnd-mode-switching branch from 3029fcd to 178b477 Compare September 22, 2020 09:06

kilrau self-assigned this Sep 22, 2020

kilrau mentioned this pull request Sep 22, 2020

LND autounlock doesn't work if lnd backend changed ExchangeUnion/xud#1913

Open

kilrau self-requested a review September 22, 2020 17:38

kilrau suggested changes Sep 22, 2020

View reviewed changes

reliveyy force-pushed the fix-lnd-mode-switching branch from 178b477 to 3bd6c06 Compare September 29, 2020 12:22

reliveyy added 5 commits September 29, 2020 23:18

fix: lnd backend mode switching bug

fd43b93

fix sed

811b032

fix script

eabcd73

remove neutrino options when in other modes

87084bb

fix bitcoind hostname in lndltc

feb9643

reliveyy force-pushed the fix-lnd-mode-switching branch from 3bd6c06 to feb9643 Compare September 29, 2020 15:18

kilrau added P2 mid priority and removed P1 top priority labels Nov 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: lnd backend mode switching bug #724

fix: lnd backend mode switching bug #724

reliveyy commented Sep 16, 2020

raladev commented Sep 17, 2020 •

edited

Loading

raladev left a comment

kilrau commented Sep 17, 2020

reliveyy commented Sep 22, 2020

reliveyy commented Sep 22, 2020 •

edited

Loading

sangaman commented Sep 22, 2020

kilrau commented Sep 22, 2020

kilrau commented Sep 22, 2020 •

edited

Loading

kilrau left a comment

kilrau commented Sep 24, 2020

fix: lnd backend mode switching bug #724

Are you sure you want to change the base?

fix: lnd backend mode switching bug #724

Conversation

reliveyy commented Sep 16, 2020

raladev commented Sep 17, 2020 • edited Loading

raladev left a comment

Choose a reason for hiding this comment

kilrau commented Sep 17, 2020

reliveyy commented Sep 22, 2020

reliveyy commented Sep 22, 2020 • edited Loading

sangaman commented Sep 22, 2020

kilrau commented Sep 22, 2020

kilrau commented Sep 22, 2020 • edited Loading

kilrau left a comment

Choose a reason for hiding this comment

kilrau commented Sep 24, 2020

raladev commented Sep 17, 2020 •

edited

Loading

reliveyy commented Sep 22, 2020 •

edited

Loading

kilrau commented Sep 22, 2020 •

edited

Loading