Add introduction

0.introduction/introduction.md

@@ -0,0 +1,193 @@
+# The Bitcoin Optech Scaling Cookbook
+
+The Bitcoin Optech scaling cookbook is a guide to effective practices and
+techniques that engineers interacting with the Bitcoin blockchain can adopt
+today. These practices are good for your business (they make your operations
+run more efficiently and save you money) and good for the ecosystem (they minimise
+your usage of common network resources and ensure that the Bitcoin network
+remains available for as many users and use cases as possible).
+
+What is it that we mean by _scaling_ when talking about the Bitcoin blockchain?
+In general, we can call a system 'scalable' if the resources consumed by the
+system scales no worse than linearly with the number of users or units of
+activity in the system. The Bitcoin base layer as a payment network does not
+appear to achieve this definition of scalability. Since the Bitcoin network
+makes use of a peer-to-peer gossip protocol in which every node validates every
+transaction, the global amount of computation, bandwidth and storage required
+scales (in at least some senses) quadratically with the number of users. n
+users validating transactions from n users results in the order of n^2
+validations. Even if we radically reduce the security by requiring new users to
+be 'light clients' and not validate the full state of the system, adding new
+users by increasing the transaction throughput is still at best linear in total
+global resource cost.
+
+And there's another problem. In many systems, the costs of the additional
+resources consumed is borne by the party benefiting from the additional
+activity. If a company needs to acquire additional computing resource to serve
+new customers, then that company would pay for the additional servers or cloud
+computing resources and benefit from the increased revenue generated by the
+customers. That isn't the case with on-chain Bitcoin transactions. Each full
+node operator bears the resource costs of validating transactions, in the form
+of bandwidth, compute, memory and storage. Naively increasing the transaction
+throughput of the Bitcoin network would therefore benefit parties creating additional
+transactions at the expense of everyone running a full node. Many users
+recognise that a vital component of maintaining a decentralized network is keeping
+the cost of running a full node low and allowing as many users as possible to
+run a full node. An attempt in 2017 to double the block size (and hence increase
+the resource requirements of running a full node), failed to gain consensus and
+was [withdrawn by its proponents][segwit2x].
+
+[segwit2x]: https://lists.linuxfoundation.org/pipermail/bitcoin-segwit2x/2017-November/000685.html
+
+So if scaling up the underlying resource requirements is not currently an
+option (and in any case could not deliver the global scale that Bitcoin
+proponents would like to see), scaling Bitcoin becomes a question of
+how to maximize the utility that can be delivered _within the constraints_ of
+the system, or in other words, how we can achieve _more with less_? 
+
+The utility that the Bitcoin blockchain delivers is _payments_, since that’s
+what users want to use the Bitcoin system for. To confuse the logical concept
+of a Bitcoin _transaction_ with the social concept of a payment is a
+conspicuous (and sadly very common) error. A _payment_ is a transfer of value
+from one party to another. In Bitcoin, a _transaction_is a transformation on
+the global set of unspent transaction outputs, consuming some UTXOs and
+producing some new UTXOs.  _Transaction_ can also refer to a data structure
+that is passed over the P2P network that both describes the transformation on
+the UTXO set and carries a witness that the transformation is valid. In neither
+of these definitions does a transaction correspond to a payment - each
+transaction can realize one, many or even no payments.
+
+To try to measure scalability with a metric like _transactions per second_
+(_tps_) is therefore flawed. Firstly, _tps_ is not measuring what we’re
+interested in, and secondly, not all payments are equally useful.
+
+To give a concrete Example of the first point, if high-frequency Bitcoin users
+switched from sending many single-recipient transactions per day to sending a
+single batched transaction per day, the average blockchain weight per _payment_
+would decrease, but the average blockchain weight per _transaction_ would
+increase.  We'd very likely see a Bitcoin network with fewer transactions per
+day, but a greater number of payments. Utility is up, but _tps_ is down!
+
+The second reason is that not all transactions are equally useful. Since each
+transaction is simply a transformation on the set of UTXOs, the Bitcoin
+consensus rules do not recognise any difference in importance between different
+transactions. The transaction produced by an institutional investor moving
+hundreds of Bitcoin into a cold storage multisig output as part of a long-term
+investment strategy could look identical to the transaction for a customer
+paying for a cup of coffee. Each counts as one 'transaction' but the first user
+probably gets much more utility from his transaction than the second user. The
+former may be prepared to pay a high fee to have his transaction recorded in a
+decentralized, trustless, immutable ledger, whereas the latter may be equally
+satisfied to have his payment made on a second-layer like lightning, a
+side-chain or even a trusted off-chain payment network.
+
+The Bitcoin network has a mechanism for users to express the urgency and
+utility of their transactions through transaction fees, but at the time of
+writing, the fee market is underdeveloped, inelastic and immature. As the network
+develops, it is expected that the fee market will become more sophisticated.
+
+When considered in this light, many practices which might not at first be
+considered as 'scaling technologies' can be considered as such. For example, a
+functioning fee market, and above all a reliable way to 'bump' fees on a
+transaction that has not yet included in a block, allows users to better
+express their time preference for a transaction to be confirmed. That means
+that the scarce block space can be allocated more efficiently to those users
+who value it more, and therefore the total utility delivered by the network is
+increased. This book takes this wider view of 'scalability'. Any technique that
+can increase the usefulness of the network in aggregate can be thought of as
+contributing to scaling.
+
+Take this book then as a guide to extracting the most possible value out of
+this scarce and precious resource called the Bitcoin blockchain. The protocol
+wizards continue to innovate on the base and second layers, and exciting
+developments like Schnorr signatures, taproot, signature aggregation, eltoo,
+atomic multi path payments and channel splicing are all in the works. But for
+now, we can all do our bit to minimize our impact on the base layer and allow
+as many users as possible to transact permissionlessly and trustlessly.
+
+# About Bitcoin Optech
+
+Bitcoin Optech was formed in early 2018. Our aim is to help Bitcoin companies
+adopt the best scaling technologies and practices available to make efficient
+use of the blockchain, and thereby help Bitcoin to scale to more users and use
+cases.
+
+During late 2017 and early 2018, we saw demand for Bitcoin usage explode and
+many users and companies were unready for the sudden spike in transaction
+volume. Block space suddenly became a competitive a rivalrous good, and
+transaction fees jumped as demand exceeded supply.
+
+At the same time, many well-understood scaling technologies were being
+underutilized. Segwit usage remained below 40%, some high-frequency users
+of the chain were sending payments as individual transactions rather than
+batching, fee estimates were failing to react to fee market conditions and
+poor coin selection led to wallets accumulating low value UTXOs and paying
+very high fees.
+
+The end result was a poor and costly experience for users, some use cases
+being priced out entirely, and Bitcoin failing to scale to meet demand. At
+Bitcoin Optech, we believe that seawalls are best built during low tide, and
+hope that Bitcoin companies will take advantage of conditions of low network
+usage and transaction fees to implement these scaling techniques. Those
+that do will be best positioned to continue offering their customers the
+best reliability and value when Bitcoin usage surges again.
+
+# Intended Audience
+
+This book is intended for engineers who work in developer or operational roles
+at companies that interact with the Bitcoin blockchain. It is meant as a
+practical guide for people who are already familiar with standard Bitcoin
+concepts like ScriptPubKeys, ScriptSigs and witnesses; the SCRIPT language and
+standard output types; public and private keys; and transaction serialization.
+See [Mastering Bitcoin][antonopoulos] or [Bitcoin and Cryptocurrency
+Technologies][narayanan] for a beginners guide to those concepts or
+[btcinformation.org][] for a comprehensive reference.
+
+[antonopoulos]: https://bitcoinbook.info/
+[narayanan]: http://bitcoinbook.cs.princeton.edu/
+[btcinformation.org]: https://btcinformation.org/en/developer-documentation
+
+Likewise, this book is not overly concerned with the cryptographic
+underpinnings of Bitcoin, except where those are directly required for
+improving the efficiency and scalability of Bitcoin operations. Cryptography is
+a fascinating subject and [Dan Boneh's Applied Cryptography course][boneh] is
+an excellent introduction for those wishing to begin their exploration.
+
+[boneh]: https://crypto.stanford.edu/~dabo/courses/OnlineCrypto/
+
+# Conventions
+
+Most terms are used as readers would expect. The script in a transaction output
+which encumers the output is the scriptPubKey. The script in the transaction
+input which unlocks the funds is the scriptSig. The *scriptCode* is the
+actually executed script (which is the redeemscript in non-segwit P2SH, and the
+witnesscript in P2WSH/P2WPKH).
+
+The smallest unit of account is referred to as a satoshi by convention, and
+10^8 satoshis is named a bitcoin. No other units of account are used.
+
+The party making a payment is called the _spender_ (_sender_ could be
+ambiguously refer to a party sending Bitcoin or a party sending data). There is
+no equivalent word in English to disambiguate the receiver of a payment and
+the receiver of data, so we use the word _recipient_ to refer to one receiving
+a payment, and live with the ambiguity.
+
+## Transaction weight and virtual bytes
+
+The consensus rule for block size is that the combined _weight_ of
+transactions must be less than 4,000,000. The weight of a single
+transaction is calculated as follows:
+
+- The _base size_ of a transaction is the number of bytes to serialize the
+  block without witness
+- The _size_ of a transaction is the number of bytes to serialize the block
+  with witness
+- The _weight_ is 3 x base size + size
+
+This formula means that the bytes in the witness are 'discounted' by a factor
+of 4. They count less towards the weight of the transaction than bytes in
+the base transaction.
+
+The _virtual size_ of a transaction is weight / 4 and is measured in _virtual bytes_
+or _vBytes_. For transactions without any segwit inputs, the base size, size and
+virtual size are all equal.

1.fee_bumping/fee_bumping.md

@@ -211,17 +211,15 @@ increase the combined feerate across the parent and child transactions.
 
 ## How does CPFP work?
 
-When constructing a new block, miners are incentivized to fill the 1vMB(see
-footnote on virtual bytes) with the set of transactions that maximize the
-transaction fees. If all unconfirmed transactions were independant, this would
-be a very straightforward operation - the miner would select the transaction
-with the highest feerate and add it to the candidate block. She'd then take the
-transaction with the next highest feerate and add it to the block. She'd
-continue to do this until the block was full. This trivially maximizes her
-profit from the block (with a little complication around the final few bytes of
-the block to ensure that she'd maximally filled the block).
-
-[//]: # (TODO: remove reference to footnote)
+When constructing a new block, miners are incentivized to fill the 1vMB with
+the set of transactions that maximize the transaction fees. If all unconfirmed
+transactions were independant, this would be a very straightforward operation -
+the miner would select the transaction with the highest feerate and add it to
+the candidate block. She'd then take the transaction with the next highest
+feerate and add it to the block. She'd continue to do this until the block was
+full. This trivially maximizes her profit from the block (with a little
+complication around the final few bytes of the block to ensure that she'd
+maximally filled the block).
 
 However, unconfirmed transactions _aren't_ independant. It is possible to have
 chains of unconfirmed transactions by spending the output from a transaction
@@ -390,26 +388,3 @@ miner incentive-compatible - a miner who is trying to maximize his revenue will
 accept both RBF’ed transactions and CPFP packages. Individual nodes’ mempools
 (which should be a node’s best guess for what will be included in the next
 blocks) should therefore also accept RBF’ed transactions and CPFP packages.
-
-[//]: # (TODO: Move this explanation to a more appropriate place in the book)
-[//]: # (make sure to remote reference to this footnote when that happens)
-
-## Transaction weight and virtual bytes
-
-The consensus rule for block size is that the combined _weight_ of
-transactions must be less than 4,000,000. The weight of a single
-transaction is calculated as follows:
-
-- The _base size_ of a transaction is the number of bytes to serialize the
-  block without witness
-- The _size_ of a transaction is the number of bytes to serialize the block
-  with witness
-- The _weight_ is 3 x base size + size
-
-This formula means that the bytes in the witness are 'discounted' by a factor
-of 4. They count less towards the weight of the transaction than bytes in
-the base transaction.
-
-The _virtual size_ of a transaction is weight / 4 and is measured in _virtual bytes_
-or _vBytes_. For transactions without any segwit inputs, the base size, size and
-virtual size are all equal.