Feat/subscribealerts (#1997)

* feat: removeorder output changed to a more meaningful message (#1526)

* fix(p2p): don't reconnect peers when pool closed (#1965)

This ensures that we don't attempt to reconnect to peers that have
disconnected from us after we have started closing the p2p pool. This
may help prevent scenarios where we unintentionally attempt to
reconnect to peers after shutting down xud.

> Should be tested against [#1668 (comment)](#1668 (comment)) @raladev

re-connection after shutdown is disappeared, but my xud still can not be gracefully  terminated, it waits something:

```
28/10/2020 05:17:43.164 [CONNEXT] trace: sending request to /balance/0x69C3d485623bA3f382Fc0FB6756c4574d43C1618
^C28/10/2020 05:17:44.087 [GLOBAL] info: XUD is shutting down
28/10/2020 05:17:44.088 [LND-BTC] info: new status: Disconnected
28/10/2020 05:17:44.089 [LND-LTC] info: new status: Disconnected
28/10/2020 05:17:44.090 [CONNEXT] info: new status: Disconnected
28/10/2020 05:17:44.093 [P2P] debug: Peer 03ece33a30db1dbce4b62fa96a5e9541138a24997ef5672eebed2d332270e39542 (OzoneYellow): closing socket. reason: Shutdown
28/10/2020 05:17:44.095 [HTTP] info: http server has closed
28/10/2020 05:17:44.096 [RPC] info: gRPC server completed shutdown
28/10/2020 05:17:44.097 [P2P] trace: Sent Disconnecting packet to 03ece33a30db1dbce4b62fa96a5e9541138a24997ef5672eebed2d332270e39542 (OzoneYellow): "{\"body\":{\"reason\":9},\"header\":{\"id\":\"95133be0-1917-11eb-b75b-73d0f0278756\"}}"
28/10/2020 05:17:44.109 [ORDERBOOK] debug: removed all orders for peer 03ece33a30db1dbce4b62fa96a5e9541138a24997ef5672eebed2d332270e39542 (OzoneYellow)
28/10/2020 05:17:44.118 [GLOBAL] info: XUD shutdown gracefully
```

* feat(lnd): change gRPC client options

* fix(connext): not enough balance for closechannel (#1963)

This introduces better error handling for Connext when using
`closeChannel` to remove funds from a Connext channel and specifying an
amount to remove that is greater than the available balance.

* feat: reserved capacity checks on PlaceOrder (#1949)

This rejects orders that would put our total reserved balance over our
total capacity for either the outbound or inbound currency. The sum of
the inbound & outbound amounts for a newly placed order are added to
the amounts reserved by open orders, and if either of these amounts
exceed the corresponding capacity then the request to place the order is
rejected.

An exception to this are inbound limits for Connext currencies, since we
have the ability to dynamically request additional inbound collateral
via our "lazy collateral" approach.

It is still possible for market orders to cause our open orders to
exceed our capacity. This is a difficult problem to avoid entirely, as
the price that market orders will execute at is unknown until the
execution is complete. Even if we simulate the matching routine, we
won't know which matches will succeed until we attempt a swap.

Instead, we generously assume that market orders will execute at the
best quoted price for purposes of these capacity checks. For users that
simultaneously place limit orders and market orders for the same
currencies, it should be made clear that market orders may use up their
available balance needed for their limit orders to succeed.

Closes #1947.

* fix(cli): openchannel assertion error for string amount (#1950)

Fixes #1643.

* feat(swapclient): auto init wallets on xud unlock (#1973)

This adds a new feature to xud to automatically attempt to create a
wallet for any new swap client configured after an xud node has been
created. Effectively this only changes the behavior for lnd clients, as
this is already the existing behavior for Connext. The process for
initializing has now been standardized instead of the ad hoc approach
used previously.

If xud tries to unlock an lnd node and gets an error message indicating
that the wallet has not been created, then it will generate a client &
currency specific seed mnemonic using seedutil and call `InitWallet`
with that seed and the existing xud password, such that the wallet
funds and node identity for the new lnd client can be unlocked and
restored along with the rest of lnd.

Closes #1929.

* feat(rpc): runtime addcurrency for lnd & connext (#1746)

Co-authored-by: Le Premier Homme <interjoint1@gmail.com>

* refactor(rpc): rename reserved TradingLimits fields

This renames the `reservedOutbound` and `reservedInbound` fields on the
`TradingLimits` call to `reservedSell` and `reservedBuy` respectively.

* fix(rpc): no success if no channels to close (#1689) (#1942)

Co-authored-by: rsercano <ozdemirsercan27@gmail.com>
Co-authored-by: Daniel McNally <mcnallydp@gmail.com>

* fix: tls certificate check on startup (#1510)

* fix: alias missing in streamorders (#1725) (#1962)

* fix(lnd): handling hold invoice check errors (#1969)

This adds better error handling for when the test calls to verify lnd
hold invoices are available fail due to connectivity reasons. Previously
any error that occurred at this step would cause us to set lnd's status
to `NoHoldInvoiceSupport` including connection issues. There was also a
bug that caused us to try to set the status to connected even when a
hold invoice status check failed.

This could result in the unusual behavior of status going to
`Disconnected` upon a call failing due to the grpc `UNAVAILABLE` error
status, then being set to `NoHoldInvoiceSupport` and then to
`ConnectionVerified`. Now we only set `NoHoldInvoiceSupport` when the
test calls fail for a reason other than `UNAVAILABLE`, and we only set
the status to `ConnectionVerified` when the hold invoice calls succeed.

Closes #1968.

* feat(lnd): SendPaymentV2

This migrates the call we use to send payments with lnd from the
deprecated `SendPaymentSync` to `SendPaymentV2` which allows for multi
path payments, among other improvements. As part of this change, the
lnd proto files have been updated to their v0.11.x versions and the
version of lnd used in simulation tests has been updated to v0.11.1 as
well.

Closes #1590.

* Revert "feat: removeorder output changed to a more meaningful message (#1526)"

* fix: use regtest instead of regnet arg

This fixes a bug where the xud flag to set the network was incorrectly
configured as `regnet` when it should be `regtest` to match what xud
expects.

* fix(lnd): don't calculate negative capacities

This fixes a bug in the logic for calculating the inbound & outbound
amount/capacity totals. We subtract the channel reserve amounts from the
balances when determining how large a payment the channel could support,
however we should not end up with a negative number.

* feat: new grpc call for subscribring alerts such as low balance (#864)

* fix: changes removeorder output to a more meaningful message (#1526) (#1986)

Co-authored-by: rsercano <ozdemirsercan27@gmail.com>
Co-authored-by: Daniel McNally <mcnallydp@gmail.com>
Co-authored-by: Karl Ranna <karl@karlranna.com>
Co-authored-by: Le Premier Homme <interjoint1@gmail.com>

bin/xud

@@ -42,8 +42,8 @@ const { argv } = require('yargs')
       type: 'boolean',
       default: undefined,
     },
-    regnet: {
-      describe: 'Whether to run XUD on regnet',
+    regtest: {
+      describe: 'Whether to run XUD on regtest',
       type: 'boolean',
       default: undefined,
     },

lib/Xud.ts

@@ -130,6 +130,12 @@ class Xud extends EventEmitter {
           nodeKey = await NodeKey.generate();
           await nodeKey.toFile(nodeKeyPath);
         }
+
+        // we need to initialize connext every time xud starts, even in noencrypt mode
+        // the call below is in lieu of the UnlockNode/CreateNode call flow
+        await this.swapClientManager.initConnext(
+          nodeKey.childSeed(SwapClientType.Connext),
+        );
       } else if (this.rpcServer) {
         this.rpcServer.grpcService.locked = true;
         const initService = new InitService(this.swapClientManager, nodeKeyPath, nodeKeyExists, this.config.dbpath);
@@ -194,12 +200,6 @@ class Xud extends EventEmitter {
       // wait for components to initialize in parallel
       await Promise.all(initPromises);
 
-      // We initialize Connext separately because it
-      // requires a NodeKey.
-      await this.swapClientManager.initConnext(
-        nodeKey.childSeed(SwapClientType.Connext),
-      );
-
       // initialize pool and start listening/connecting only once other components are initialized
       await this.pool.init();
 

lib/cli/command.ts

@@ -38,6 +38,18 @@ const loadXudConfig = async (argv: Arguments<any>) => {
   }
 };
 
+const getTlsCert = (certPath: string) => {
+  try {
+    return fs.readFileSync(certPath);
+  } catch (err) {
+    if (err.code === 'ENOENT') {
+      throw `tls cert could not be found at ${certPath}, it may take several seconds to be created on xud's first run`;
+    }
+
+    throw err;
+  }
+};
+
 /**
  * A generic function to instantiate an XU client.
  * @param argv the command line arguments
@@ -46,7 +58,7 @@ export const loadXudClient = async (argv: Arguments<any>) => {
   await loadXudConfig(argv);
 
   const certPath = argv.tlscertpath || path.join(argv.xudir, 'tls.cert');
-  const cert = fs.readFileSync(certPath);
+  const cert = getTlsCert(certPath);
   const credentials = grpc.credentials.createSsl(cert);
 
   return new XudClient(`${argv.rpchost}:${argv.rpcport}`, credentials);
@@ -56,7 +68,7 @@ export const loadXudInitClient = async (argv: Arguments<any>) => {
   await loadXudConfig(argv);
 
   const certPath = argv.tlscertpath || path.join(argv.xudir, 'tls.cert');
-  const cert = fs.readFileSync(certPath);
+  const cert = getTlsCert(certPath);
   const credentials = grpc.credentials.createSsl(cert);
 
   return new XudInitClient(`${argv.rpchost}:${argv.rpcport}`, credentials);

lib/cli/commands/openchannel.ts

@@ -37,6 +37,10 @@ export const builder = (argv: Argv) => argv
   .example('$0 openchannel ETH 0.5', 'deposit 0.5 into an ETH Connext channel without specifying a remote node');
 
 export const handler = async (argv: Arguments<any>) => {
+  if (isNaN(argv.amount)) {
+    throw 'amount must be a number';
+  }
+
   const request = new OpenChannelRequest();
   if (argv.node_identifier) {
     request.setNodeIdentifier(argv.node_identifier);

lib/cli/commands/removeorder.ts

@@ -1,7 +1,7 @@
 import { Arguments, Argv } from 'yargs';
-import { RemoveOrderRequest } from '../../proto/xudrpc_pb';
+import { RemoveOrderRequest, RemoveOrderResponse } from '../../proto/xudrpc_pb';
 import { callback, loadXudClient } from '../command';
-import { coinsToSats } from '../utils';
+import { coinsToSats, satsToCoinsStr } from '../utils';
 
 export const command = 'removeorder <order_id> [quantity]';
 
@@ -18,11 +18,23 @@ export const builder = (argv: Argv) => argv
   .example('$0 removeorder 79d2cd30-8a26-11ea-90cf-439fb244cf44', 'remove an order by id')
   .example('$0 removeorder 79d2cd30-8a26-11ea-90cf-439fb244cf44 0.1', 'remove 0.1 quantity from an order by id');
 
+const displayOutput = (orderId: string, removeOrderResponse: RemoveOrderResponse.AsObject) => {
+  const removedCurrency = removeOrderResponse.pairId.split('/')[0];
+  if (removeOrderResponse.quantityOnHold === 0 && removeOrderResponse.remainingQuantity === 0) {
+    console.log(`Order ${orderId} successfully removed.`);
+  } else {
+    console.log(`
+Order ${orderId} partially removed, remaining quantity: \
+${satsToCoinsStr(removeOrderResponse.remainingQuantity)} ${removedCurrency}, \
+on hold: ${satsToCoinsStr(removeOrderResponse.quantityOnHold)} ${removedCurrency}`);
+  }
+};
+
 export const handler = async (argv: Arguments<any>) => {
   const request = new RemoveOrderRequest();
   request.setOrderId(argv.order_id);
   if (argv.quantity) {
     request.setQuantity(coinsToSats(argv.quantity));
   }
-  (await loadXudClient(argv)).removeOrder(request, callback(argv));
+  (await loadXudClient(argv)).removeOrder(request, callback(argv, displayOutput.bind(undefined, argv.order_id)));
 };

lib/cli/commands/streamorders.ts

@@ -1,9 +1,8 @@
-import { ServiceError, status } from 'grpc';
 import { Arguments, Argv } from 'yargs';
 import { XudClient } from '../../proto/xudrpc_grpc_pb';
 import * as xudrpc from '../../proto/xudrpc_pb';
-import { setTimeoutPromise } from '../../utils/utils';
 import { loadXudClient } from '../command';
+import { onStreamError, waitForClient } from '../utils';
 
 export const command = 'streamorders [existing]';
 
@@ -26,20 +25,8 @@ const ensureConnection = async (argv: Arguments, printError?: boolean) => {
   if (!client) {
     client = await loadXudClient(argv);
   }
-  client.waitForReady(Date.now() + 3000, (error: Error | null) => {
-    if (error) {
-      if (error.message === 'Failed to connect before the deadline') {
-        console.error(`could not connect to xud at ${argv.rpchost}:${argv.rpcport}, is xud running?`);
-        process.exit(1);
-      }
 
-      if (printError) console.error(`${error.name}: ${error.message}`);
-      setTimeout(ensureConnection.bind(undefined, argv, printError), 3000);
-    } else {
-      console.log('Successfully connected, subscribing for orders');
-      streamOrders(argv);
-    }
-  });
+  waitForClient(client, argv, ensureConnection, streamOrders, printError);
 };
 
 const streamOrders = (argv: Arguments<any>) => {
@@ -57,15 +44,7 @@ const streamOrders = (argv: Arguments<any>) => {
   // adding end, close, error events only once,
   // since they'll be thrown for three of subscriptions in the corresponding cases, catching once is enough.
   ordersSubscription.on('end', reconnect.bind(undefined, argv));
-  ordersSubscription.on('error', async (err: ServiceError) => {
-    if (err.code === status.UNIMPLEMENTED) {
-      console.error("xud is locked, run 'xucli unlock', 'xucli create', or 'xucli restore' then try again");
-      process.exit(1);
-    }
-    console.warn(`Unexpected error occured: ${err.message}, reconnecting in 1 second`);
-    await setTimeoutPromise(1000);
-    await ensureConnection(argv);
-  });
+  ordersSubscription.on('error', onStreamError.bind(undefined, ensureConnection.bind(undefined, argv)));
 
   const swapsRequest = new xudrpc.SubscribeSwapsRequest();
   swapsRequest.setIncludeTaker(true);

lib/cli/commands/subscribealerts.ts

@@ -0,0 +1,51 @@
+import { Arguments, Argv } from 'yargs';
+import { XudClient } from '../../proto/xudrpc_grpc_pb';
+import * as xudrpc from '../../proto/xudrpc_pb';
+import { loadXudClient } from '../command';
+import { AlertType } from '../../constants/enums';
+import { onStreamError, waitForClient } from '../utils';
+
+export const command = 'subscribealerts';
+
+export const describe = 'subscribe alerts such as low balance';
+
+export const builder = (argv: Argv) => argv
+    .option('pretty', {
+      type: 'boolean',
+    })
+    .example('$0 subscribealerts -j', 'prints alert payload in a JSON structure')
+    .example('$0 subscribealerts', 'prints alert message only');
+
+export const handler = async (argv: Arguments) => {
+  await ensureConnection(argv, true);
+};
+
+let client: XudClient;
+
+const ensureConnection = async (argv: Arguments, printError?: boolean) => {
+  if (!client) {
+    client = await loadXudClient(argv);
+  }
+
+  waitForClient(client, argv, ensureConnection, subscribeAlerts, printError);
+};
+
+const subscribeAlerts = (argv: Arguments<any>) => {
+  const request = new xudrpc.SubscribeAlertsRequest();
+  const alertsSubscription = client.subscribeAlerts(request);
+
+  alertsSubscription.on('data', (alert: xudrpc.Alert) => {
+    if (argv.json) {
+      console.log(JSON.stringify(alert, undefined, 2));
+    } else {
+      console.log(`${AlertType[alert.getType()]}: ${alert.getMessage()}`);
+    }
+  });
+  alertsSubscription.on('end', reconnect.bind(undefined, argv));
+  alertsSubscription.on('error', onStreamError.bind(undefined, ensureConnection.bind(undefined, argv)));
+};
+
+const reconnect = async (argv: Arguments) => {
+  console.log('Stream has closed, trying to reconnect');
+  await ensureConnection(argv, false);
+};