release-notes-0.14.0.md

Release Notes

Networking & Tor

Connectivity mode

A new flag has been added to enable a hybrid tor connectivity mode, where tor is only used for onion address connections, and clearnet for everything else. This new behavior can be added using the tor.skip-proxy-for-clearnet-targets flag.

Onion service

The Onion service created upon lnd startup is now deleted during lnd shutdown using DEL_ONION.

Tor connection

A new health check, tor connection, is added to lnd's liveness monitor upon startup. This check will ensure the liveness of the connection between the Tor daemon and lnd's tor controller. To enable it, please use the following flags,

healthcheck.torconnection.attempts=xxx
healthcheck.torconnection.timeout=xxx
healthcheck.torconnection.backoff=xxx
healthcheck.torconnection.internal=xxx

Read more about the usage of the flags in the sample-lnd.conf.

LN Peer-to-Peer Network

Bitcoin Blockheaders in Ping Messages

In this release, we implement a long discussed mechanism to use the Lightning Network as a redundant block header source. By sending our latest block header with each ping message, we give peers another source (outside of the Bitcoin P2P network) they can use to spot check their chain state. Peers can also use this information to detect if they've been eclipsed from the traditional Bitcoin P2P network itself.

As is, we only send this data in Ping messages (which are periodically sent), in the future we could also move to send them as the partial payload for our pong messages, and also randomize the payload size requested as well.

The ListPeers RPC call will now also include a hex encoded version of the last ping message the peer has sent to us.

Backend Enhancements & Optimizations

Full remote database support

lnd now stores all its data in the same remote/external database such as etcd instead of only the channel state and wallet data. This makes lnd fully stateless and therefore makes switching over to a new leader instance almost instantaneous. Read the guide on leader election for more information.

Postgres database support

This release adds support for Postgres as a database backend to lnd. Postgres has several advantages over the default bbolt backend:

• Better handling of large data sets.
• On-the-fly database compaction (auto vacuum).
• Database replication.
• Inspect data while lnd is running (bbolt opens the database exclusively).
• Usage of industry-standard tools to manage the stored data, get performance metrics, etc.

Furthermore, the SQL platform opens up possibilities to improve lnd's performance in the future. Bbolt's single-writer model is a severe performance bottleneck, whereas Postgres offers a variety of locking models. Additionally, structured tables reduce the need for custom serialization/deserialization code in lnd, saving developer time and limiting the potential for bugs.

Instructions for enabling Postgres can be found in docs/postgres.md.

In-memory path finding

Finding a path through the channel graph for sending a payment doesn't involve any database queries anymore. The channel graph is now kept fully in-memory for up a massive performance boost when calling QueryRoutes or any of the SendPayment variants. Keeping the full graph in memory naturally comes with increased RAM usage. Users running lnd on low-memory systems are advised to run with the routing.strictgraphpruning=true configuration option that more aggressively removes zombie channels from the graph, reducing the number of channels that need to be kept in memory.

There is a fallback option db.no-graph-cache=true that can be used when running a Bolt (bbolt) based database backend. Using the database for path finding is considerably slower than using the in-memory graph cache but uses less RAM. The fallback option is not available for etcd or Postgres database backends because of the way they handle long-running database transactions that are required for the path finding operations.

Protocol Extensions

Explicit Channel Negotiation

A new protocol extension has been added known as explicit channel negotiation. This allows a channel initiator to signal their desired channel type to use with the remote peer. If the remote peer supports said channel type and agrees, the previous implicit negotiation based on the shared set of feature bits is bypassed, and the proposed channel type is used. Feature bits 44/45 are used to signal this new feature.

Script Enforced Channel Leases

A new channel commitment type that builds upon the recently introduced anchors commitment format has been introduced to back channel leases resulting from Lightning Pool. This new channel commitment type features an additional spend requirement on any commitment and HTLC outputs that pay directly to the channel initiator. The channel initiator must now wait for the channel lease maturity, expressed as an absolute height of the chain, to be met before being able to sweep their funds, on top of the usual CSV delay requirement. See the linked pull request for more details.

Re-Usable Static AMP Invoices

AMP invoices are now fully re-usable, meaning it's possible for an lnd node to use a static AMP invoice multiple times. An AMP invoice can be created by adding the --amp flag to lncli addinvoice. From there repeated payments can be made to the invoice using lncli payinvoice. On the receiver side, notifications will still come in as normal, but notifying only the new "sub-invoice" paid each time.

A new field has been added to the main Invoice proto that allows callers to easily scan to see the current state of all repeated payments to a given payment_addr.

A new LookupInvoiceV2 RPC has been added to the invoicerpcserver which allows callers to look up an AMP invoice by set_id, opting to only return relevant HTLCs, or to look up an AMP invoice by its payment_addr, but omit all HTLC information. The first option is useful when a caller wants to get information specific to a repeated payment, omitting the thousands of possible other payment attempts. The second option is useful when a caller wants to obtain the base invoice information, and then use the first option to extract the custom records (or other details) of the prior payment attempts.

RPC Server

• Return payment address and add index from addholdinvoice call.

• The versions of several gRPC related libraries were bumped and the main rpc.proto was renamed to lightning.proto to fix a warning related to protobuf file name collisions.

• Stub code for interacting with lnrpc from a WASM context through JSON messages was added.

• The updatechanpolicy call now detects invalid and pending channels, and returns a policy update failure report.

• LND now reports to systemd that RPC is ready (port bound, certificate generated, macaroons created, in case of wallet-unlock-password-file wallet unlocked). This can be used to avoid misleading error messages from dependent services if they use After systemd option.

• Delete a specific payment, or its failed HTLCs.

• A new state, WalletState_SERVER_ACTIVE, is added to the state server. This state indicates whether the lnd server and all its subservers have been fully started or not.

• Adds an option to the BakeMacaroon rpc "allow-external-permissions," which makes it possible to bake a macaroon with external permissions. That way, the baked macaroons can be used for services beyond LND. Also adds a new CheckMacaroonPermissions rpc that checks that the macaroon permissions and other restrictions are being followed. It can also check permissions not native to LND.

• A new RPC middleware interceptor was added that allows external tools to hook into lnd's RPC server and intercept any requests made with custom macaroons (and also the responses to those requests).

• Adds NOT_FOUND status code for LookupInvoice

• The FundingPsbtFinalize step is a safety measure that assures the final signed funding transaction has the same TXID as was registered during the funding flow and was used for the commitment transactions. This step is cumbersome to use if the whole funding process is completed external to lnd. We allow the finalize step to be skipped for such cases. The API user/script will need to make sure things are verified (and possibly cleaned up) properly. An example script was added to the PSBT documentation to show the simplified process.

Batched channel funding

Multiple channels can now be opened in a single transaction in a safer and more straightforward way by using the BatchOpenChannel RPC or the command line version of that RPC called lncli batchopenchannel. More information can be found in the PSBT documentation.

Wallet

• It is now possible to fund a psbt without specifying any outputs. This option is useful for CPFP bumping of unconfirmed outputs or general utxo consolidation.

• The internal wallet can now also be created or restored by using an extended master root key (xprv) instead of an aezeed only. This allows wallet integrators to use existing seed mechanism that might already be in place. It is still not supported to use the same seed/root key on multiple lnd instances simultaneously though.

• Publish transaction is now reachable through lncli.

• Prior to this release, when running on simnet or regtest, lnd would skip the check on wallet synchronization during its startup. In doing so, the integration test can bypass the rule set by bitcoind, which considers the node is out of sync when the last block is older than 2 hours(more discussion). This synchronization check is put back now as we want to make the integration test more robust in catching real world situations. This also means it might take longer to start an lnd node when running in simnet or regtest, something developers need to watch out from this release.

Remote signing

It is now possible to delegate any operation that needs access to private keys to a remote signer that serves signing requests over RPC. More information can be found in the new remote signing document.

Security

• The release signature verification script was overhauled to fix some possible attack vectors and user errors. The public keys used to verify the signatures against are no longer downloaded form Keybase but instead are kept in the lnd git repository. This allows for a more transparent way of keeping track of changes to the signing keys.

Admin macaroon permissions

The default file permissions of admin.macaroon were changed from 0600 to 0640. This makes it easier to allow other users to manage LND. This is safe on common Unix systems because they always create a new group for each user.

If you use a strange system or changed group membership of the group running LND you may want to check your system to see if it introduces additional risk for you.

Custom peer messages

Lightning nodes have a connection to each of their peers for exchanging messages. In regular operation, these messages coordinate processes such as channel opening and payment forwarding.

The lightning spec however also defines a custom range (>= 32768) for experimental and application-specific peer messages.

With this release, custom peer message exchange is added to open up a range of new possibilities. Custom peer messages allow the lightning protocol with its transport mechanisms (including tor) and public key authentication to be leveraged for application-level communication. Note that peers exchange these messages directly. There is no routing/path finding involved.

Safety

• Locally force closed channels are now kept in the channel.backup file until their time lock has fully matured.

• Cooperative closes optimistically shutdown the associated link before closing the channel.

Build System

• The illumos operating system has been dropped from the set of release binaries. We now also build with postgres and etcd in the main release binaries.

• A new pre-submit check has been added to ensure that all PRs (aside from merge commits) add an entry in the release notes folder that at least links to PR being added.

• A new build target itest-race to help uncover undetected data races with our itests.

• The itest error whitelist check was removed to reduce the number of failed Travis builds.

• A flake in the Neutrino integration tests with anchor sweeps was addressed.

• The lnwire fuzz tests have been fixed and now run without crashing.

• A flake in the race unit tests was addressed that lead to failed tests sometimes when the CPU of the GitHub CI runner was strained too much.

• Reduce the number of parallel itest runs to 2 on ARM.

• Fix Travis itest parallelism

• All CI, containers, and automated release artifact building now all use Go 1.17.1. All build tags have been updated accordingly to comply with the new Go 1.17.1 requirements.

• All integration tests (except the ARM itests) were moved from Travis CI to GitHub Actions.

• The LndMobile iOS build has been updated to work with newer gomobile versions that output in the xcframework packaging format. Applications that use the iOS build will have to be updated to include an xcframework instead of a framework.

• Upgrade the sub packages to 1.16

• CI has been upgraded to build against bitcoind 22.0

• Update to the latest neutrino version

Documentation

• Outdated warning about unsupported pruning was replaced with clarification that LND does support pruning.

• Clarified 'ErrReservedValueInvalidated' error string to explain that the error is triggered by a transaction that would deplete funds already reserved for potential future anchor channel closings (via CPFP) and that more information (e.g., specific sat amounts) can be found in the debug logs.

• Updated C# grpc docs to use Grpc.Net.Client. The Grpc.Core NuGet package is in maintenance mode. Grpc.Net.Client is now the recommended implementation.

Misc

• The direct use of certain syscalls in packages such as bbolt or lnd's own healthcheck package made it impossible to import lnd code as a library into projects that are compiled to WASM binaries. That problem was fixed by guarding those syscalls with build tags.

• The only way to retrieve hophints for a given node was to create an invoice with the addInvoice rpc interface. However, now the function has been exposed in the go package invoicesrpc.

• The DeleteAllPayments and DeletePayment RPC methods can now be called from the command line with the new lncli deletepayments command.

• Add more verbose error printed to console when lnd fails loading the user specified config.

• Make it possible to add more than one RPC Listener when calling lnd.Main. And add MacChan field for passing back lnd's admin macaroon back to the program calling lnd, when needed.

• The --amp-reuse CLI flag has been removed as the latest flavor of AMP now natively supports static invoices

• Using go get to install go executables is now deprecated. Migrate to go install our lnrpc proto dockerfile Migrate go get to go install

• The premature update map has been revamped using an LRU cache

Code Health

Code cleanup, refactor, typo fixes

• Fix logging typo to log channel point on cooperative closes

• Refactor the interaction between the htlcswitch and peer packages for cleaner separation.

• Moved the original breach handling and timelock UTXO handling into the contract court package

• Unused error check removed.

• Shorten Pull Request check list by referring to the CI checks that are in place.

• Added minor fixes to contribution guidelines.

• Fixed typo in dest_custom_records description comment.

• Fixed payment test error message.

• Bumped version of github.com/miekg/dns library to fix a Dependabot alert.

• Fixed timeout flakes in async payment benchmark tests.

• State, subscribechannelevents, subscribepeerevents, subscribeinvoices, subscribetransactions, subscribechannelgraph and subscribechannelbackups no longer logs certain errors.

• Flake fix in async bidirectional payment test.

• Fixed context timeout when closing channels in tests.

• Fixed transaction not found in mempool flake in commitment deadline itest.

• Fixed a missing import and git tag in the healthcheck package.

• Fixed a data race in payment unit test.

• Missing dots in cmd interface.

• Link channel point logging.

• Canceling the chain notifier no longer logs certain errors.

• Fixed context leak in integration tests, and properly handled context timeout.

• Removed nested db tx.

• Fixed wallet recovery itests on Travis ARM.

• Integration tests save embedded etcd logs to help debugging flakes.

• Replace reference to JWT library with CVE

• Replace reference to XZ library with CVE

• Replace reference to mongo library with CVE

• Fixed restore backup file test flake with bitcoind.

• Timing fix in AMP itest.

• Upgraded miekg/dns to improve the security posture.

• server.go: dedupe pubkey output in debug/log msgs.

• Fixed flakes caused by graph topology subcription.

• Order of the start/stop on subsystems are changed to promote better safety.

• Fixed flake that occurred when testing the new optimistic link shutdown.

• Respect minimum relay fee during commitment fee updates.

• Replace reference to protobuf library with OSV

• Only upload itest logs on failure, fix more flakes.

• The interfaces for signing messages and the code for initializing a wallet was refactored as a preparation for supporting remote signing.

• Include htlc amount in bandwidth hints.

• Fix REST/WebSocket API itest that lead to overall test timeout.

• Fix missing label on sweep transactions.

• Fixed flake that occurred with the switch dust forwarding test under the data race unit build.

• Run channeldb tests on postgres

• Fixed two flakes in itests that were caused by things progressing too fast.

• Fixes two issues around configuration parsing and error logging.

Database

• Ensure single writer for legacy code when using etcd backend.

• When starting/restarting, lnd will clean forwarding packages, payment circuits and keystones for closed channels, which will potentially free up disk space for long running nodes that have lots of closed channels.

• Optimized payment sequence generation to make LNDs payment throughput (and latency) with better when using etcd.

• More robust commit queue design to make it less likely that we retry etcd transactions and make the commit queue more scalable.

• Flatten the payment-htlcs-bucket in order to make it possible to prefetch all htlc attempts of a payment in one DB operation. Migration may fail for extremely large DBs with many payments (10+ million). Be careful and backup your channel.db if you have that many payments. Deleting all failed payments beforehand makes migration safer and faster too.

• Prefetch payments on hot paths to reduce roundtrips to the remote DB backend.

Performance improvements

• Update MC store in blocks to make payment throughput better when using etcd.

• The lnwire package now uses a write buffer pool when encoding/decoding messages. Such that most of the heap escapes are fixed, resulting in less memory being used when running lnd.

• lnd will now no longer (in a steady state) need to open a new database transaction each time a private key needs to be derived for signing or ECDH operations. This results in a massive performance improvement across several routine operations at the

• When decrypting incoming encrypted brontide messages on the wire, we'll now properly re-use the buffer that was allocated for the ciphertext to store the plaintext. When combined with the buffer pool, this ensures that we no longer need to allocate a new buffer each time we decrypt an incoming message, as we recycle these buffers in the peer.

• The DescribeGraph and GetNetworkInfo calls have been optimized by caching the response periodically, or using the new channel graph cache directly. This should significantly cut down on the garbage these two calls generate.

Log system

• Save compressed log files from logrorate during itest.

Mission control

• Interpretation of channel disabled errors was changed to only penalize t he destination node to consider mobile wallets with hub nodes.

Bug Fixes

• A bug has been fixed that would cause lnd to try to bootstrap using the current DNS seeds when in SigNet mode.

• A validation check for sane CltvLimit and FinalCltvDelta has been added for REST-initiated payments.

• A bug has been fixed with Neutrino's RegisterConfirmationsNtfn and RegisterSpendNtfn calls that would cause notifications to be missed.

• A bug has been fixed when registering for spend notifications in the txnotifier. A re-org notification would previously not be dispatched in certain scenarios.

• Catches up on blocks in the router in order to fix an "out of order" error that crops up.

• Fix healthcheck might be running after the max number of attempts are reached.

• Fix crash with empty AMP or MPP record in invoice.

• Config setting sync-freelist was ignored in certain cases.

• The underlying gRPC connection of a WebSocket is now properly closed when the WebSocket end of a connection is closed. A bug with the write deadline that caused connections to suddenly break was also fixed in the same PR.

• A bug has been fixed in Neutrino that would result in transactions being rebroadcast even after they had been confirmed. Lnd is updated to use the version of Neutrino containing this fix.

• A bug has been fixed that would result in nodes not reconnecting to their persistent outbound peers if the peer's IP address changed.

• Use the change output index when validating the reserved wallet balance for SendCoins/SendMany calls

• A bug has been fixed where certain channels couldn't be passed to lncli getchaninfo due to their 8-byte compact ID being too large for an int64.

• Dedup stored peer addresses before creating connection requests to prevent redundant connection requests

• A concurrent map writes crash was fixed in the btcwallet dependency.

• A bug has been fixed that would at times cause intercepted HTLCs to be re-notified, which could lead to higher-level HTLC mismanagement issues.

• Do not error log when an invoice that has been canceled and GC'd is expired

• Don't print bucket names with special characters when compacting

• Fix pathfinding crash when inbound policy is unknown

• Stagger connection attempts to multi-address peers to ensure that the peer doesn't close the first successful connection in favour of the next if the first one was successful

• Fixed an issue with external listeners and the --noseedbackup development flag.

• Fixed a bug that caused the RPC middleware request ID not to be the same for intercept messages belonging to the same intercepted gRPC request/response pair.

• Fix deadlock when using the graph cache

• Fixes a bug that would cause pruned nodes to stall out

• Add Postgres connection limit

Documentation

The code contribution guidelines have been updated to mention the new requirements surrounding updating the release notes for each new change.

Contributors (Alphabetical Order)

• Abubakar Nur Khalil
• Adrian-Stefan Mares
• Alex Bosworth
• Alyssa Hertig
• András Bánki-Horváth
• Bjarne Magnussen
• Carla Kirk-Cohen
• Carsten Otto
• Conner Fromknecht
• Elle Mouton
• ErikEk
• Eugene Siegel
• Hampus Sjöberg
• Harsha Goli
• Jesse de Wit
• Johan T. Halseth
• Johnny Holton
• Joost Jager
• Jordi Montes
• Juan Pablo Civile
• Kishin Kato
• Leonhard Weese
• Martin Habovštiak
• Michael Rhee
• Naveen Srinivasan
• Olaoluwa Osuntokun
• Oliver Gugger
• Priyansh Rastogi
• Roei Erez
• Simon Males
• Stevie Zollo
• Torkel Rogstad
• Wilmer Paulino
• Yong Yu
• Zero-1729
• benthecarman
• de6df1re
• github2k20
• mateuszmp
• nathanael
• offerm
• positiveblue
• xanoni