feat(backup): write xud db every 3 minutes max

[sangaman]

This changes the behavior of the xud backup from backing up the xud db every time it changes (which is very often) to every 3 minutes, provided that it has changed in the preceding 3 minutes. Note that this approach still needs refinement to prevent corrupted backups as described in issue #1652.

Closes #1368.

[kilrau]

You can run this with bash xud.sh -b backup/xud-db-timer once https://travis-ci.org/github/ExchangeUnion/xud-docker/builds/697757895 passed.

[kilrau]

Tested this today on the same setup from yesterday:

• maker on fast-disk machine
• taker on a slow disk machine

Swaps are still comparably slow but faster compared to yesterday (ltc/btc swaps in about 10s total time to complete). Nowhere near the 30s timeout. eth/btc still hits the timeout in most cases.

As per our recommended setup, backups are written to an external USB pen drive on both machines, so backup writes should not influence disk I/O on the main disk where the main xud db resides. Reads/locks of the xud main db will.

[kilrau]

I ran this branch on two environments, swapping on testnet. Watched the xud backup file change and (on taker side) there are still changes being made to the backup file during the swap and are not delayed to the next 3 minutes interval (xud backup file changed at 10:15 and 10:16, where I did a swap):

kilrau@beast:/media/USB/xud-backup/02-Testnet$ ls -la
total 84
drwxr-xr-x 2 root root  4096 Jun  8 16:10 .
drwxr-xr-x 5 root root  4096 May  1 11:34 ..
-rw-r--r-- 1 root root   461 Jun 12 16:04 lnd-BTC
-rw-r--r-- 1 root root   461 Jun 12 16:04 lnd-LTC
-rw-r--r-- 1 root root 69632 Jun 13 10:15 xud
kilrau@beast:/media/USB/xud-backup/02-Testnet$ ls -la
total 84
drwxr-xr-x 2 root root  4096 Jun  8 16:10 .
drwxr-xr-x 5 root root  4096 May  1 11:34 ..
-rw-r--r-- 1 root root   461 Jun 12 16:04 lnd-BTC
-rw-r--r-- 1 root root   461 Jun 12 16:04 lnd-LTC
-rw-r--r-- 1 root root 69632 Jun 13 10:16 xud
kilrau@beast:/media/USB/xud-backup/02-Testnet$ 

On maker side the db changes in larger intervals and not during the swap. But I didn't test this thoroughly enough to verify that it is different on maker/taker side, so I might be wrong here.

[kilrau]

Backup tests need to be adjusted:

  ● Backup › should write XUD database backups on new event

    Failed: undefined

      149 |   });

      150 | 

    > 151 |   test('should write XUD database backups on new event', async () => {

          |   ^

      152 |     fs.writeFileSync(

      153 |       xudDatabasePath,

      154 |       backups.xud.event,

      at Env.it (../../node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:89:24)

      at Suite.<anonymous> (Backup.spec.ts:151:3)

      at Object.<anonymous> (Backup.spec.ts:60:1)

And can you comment out that raiden db test for now?

FAIL  test/jest/Backup.spec.ts (6.527 s)

  ● Backup › should write Raiden backups on startup

    ENOENT: no such file or directory, open 'backup-test/raiden'

      109 |   test('should write Raiden backups on startup', () => {

      110 |     expect(

    > 111 |       fs.readFileSync(

          |          ^

      112 |         path.join(backupdir, 'raiden'),

      113 |         'utf8',

      114 |       ),

      at Object.<anonymous> (Backup.spec.ts:111:10)

  ● Backup › should write Raiden backups on new event

    Failed: undefined

      116 |   });

      117 | 

    > 118 |   test('should write Raiden backups on new event', async () => {

          |   ^

      119 |     fs.writeFileSync(

      120 |       raidenDatabasePath,

      121 |       backups.raiden.event,

      at Env.it (../../node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:89:24)

      at Suite.<anonymous> (Backup.spec.ts:118:3)

      at Object.<anonymous> (Backup.spec.ts:60:1)

  ● Backup › should write XUD database backups on new event

    Failed: undefined

      149 |   });

      150 | 

    > 151 |   test('should write XUD database backups on new event', async () => {

          |   ^

      152 |     fs.writeFileSync(

      153 |       xudDatabasePath,

      154 |       backups.xud.event,

      at Env.it (../../node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:89:24)

      at Suite.<anonymous> (Backup.spec.ts:151:3)

      at Object.<anonymous> (Backup.spec.ts:60:1)

[michael1011]

Backup tests need to be adjusted:

These tests should be either removed completely or need to be rewritten since the functionality of the backup daemon changed.

[sangaman]

I updated the tests to conform to the new backup approach.

[kilrau]

#1655 (review) still needs to be addressed though. Can do some more testing today if required.

[sangaman]

@kilrau In my tests I confirmed this is only updating the backup at most every 3 minutes, can you please double check on your end if you're still seeing that behavior? And confirm xud is on commit cfdf8b6c6bb9ed1a2c7f7b1a512504a387b5074b.

fixed