Background
Some users performed a public stress test of the BSV network, beginning on May 12, 2020 and apparently continuing for a week. During the first day of this public stress test, approximately 2 million low fee transactions were broadcast on the BSV network. The vast majority of these transactions had a fee rate just over 0.25 sats/byte which is above the default relay fee, meaning the transactions were fully propagated around the network, but below the default transaction acceptance (or mining) fee of 0.5 sats/byte, meaning that most miners would not attempt to mine them. This resulted in a large mempool building up consuming as much as 1.7GB of memory. Note that the default mempool limit in BitcoinSV Node software is 1GB; however, most mining pools/groups have set their mempool limit significantly higher.
This “mempool buildup” event occurred just one day after the BTC block reward halving (on May 11), which caused a large influx of BTC miners to BSV, diluting the effective hash rate of the pre-existing dedicated BSV miners. This was apparent because the BSV network saw many 1MB blocks being mined in the days after May 11, suggesting that these new miners on BSV had imported configurations directly from their BTC setups. As such, the one mining pool we previously knew was mining with a transaction acceptance fee rate of 0.25 sats/byte (Mempool) was reduced to finding less than 1% of blocks on or around May 12, and was unable to process the backlog on their own. The mempool buildup was eventually cleared by TAAL, when it mined a world record 309MB block later on May 12.
Network impact
The BSV Node team runs quite a few bitcoind instances on mainnet to monitor the state of the network. What we saw on our bitcoin daemons, as well as what we learned from various miners, was very encouraging. The BSV Node software performed well within operating limits throughout this mempool buildup event. Some instances with the default 1GB mempool limit did not keep all of the transactions but they also didn’t slow down when they reached the 1GB limit. Neither did those instances need to keep all of the transactions in memory. The load from this event was a fraction of what BSV is subjected to daily on the BSV Scaling Test Network so this is unsurprising to us.
So what happened? – Relay fee vs transaction acceptance (mining) fee
It is important to note that there are two different fee settings in Bitcoin node software:
- the relay fee, which determines whether a miner will accept a transaction into its mempool and broadcast it to other miners. In BSV Node software, this is set by default at 0.25 sats/byte.
- the transaction acceptance (mining) fee, which is what is needed for a miner to accept the transaction and add it into a block found by the miner. This is set by default at 0.5 sats/byte.
Thus, the default relay fee (0.25 sats/byte) is lower than the default transaction acceptance (mining) fee (0.5 sats/byte). Until late last year, these different fees were both set equally at 1 sat/byte so the difference in their function was not very apparent; furthermore, it was a given that if your transaction wasn’t rejected when you sent it, then it would also be mined rapidly. This was a substantial improvement over the BTC experience but it still had a significant downside: it left no room for miners to compete on fees.
Let’s hypothetically say that both relay and transaction acceptance fee rates are set by most miners to 1 sat/byte, and one miner wants to offer a lower rate (on both fees) of 0.8 sats/byte. A miner is of course free to offer the lower rate, but to take advantage of it as a user, you have a couple of important caveats:
- You will lose all double spend protection because other miners will ignore the lower fee transaction, will not be able to detect a later transaction is a double spend.
- You will not get the transaction relayed by other miners (because your relay fee of 0.8 sats/byte is lower than the 1 sat/byte required by the other miners), so to even get the transaction to the discounting miner, you have to find that miner’s node and submit it to them directly, via the Bitcoin peer-to-peer protocol.
The second problem can be solved with tools like Miner ID (to find the miner API endpoints) and Merchant API (to discover fees and to submit transactions easily to the discounting miners). The first problem is more concerning because if the starting point makes competitive price cutting uncompetitive due to immediate loss of utility, it is a very strong barrier to competition happening in the first place. We have seen the effect of this on BCH where after more than two years fee rates have remained stagnant.
By separating the two fees and leaving a gap in between them, we have a starting point where competitive pricing becomes possible. Any miner can reduce transaction acceptance fee as low as the relay fee, and transactions will still be fully propagated. Any miner could reduce the relay fee at any time, but in doing so, they would fully aware that they’d be on their own accepting those transactions for mining, for which they may have a market, but they would also be aware they have to have a plan to clear them out of their own mempool. Presumably by mining them.
But what happens if no one is mining these low fee transactions? Won’t they just build up until the nodes run out of memory?
In the short term, NO because the BSV Node software handles this scenario gracefully: once the mempool reaches the configured limit of memory usage, it will start evicting lowest fee transactions to make room for new transactions. This costs the miners nothing so they can maintain this state indefinitely whilst still taking all the transactions that meet their fee requirement. This is scenario is very different to the one you might encounter on BTC when the mempool fills up because of one critical factor: It does not push the cost of a transaction fee UP, it just puts a miner configured floor under it. In the real Bitcoin, competition is the force that drives prices down, not full mempools. And users will always have certainty that they can get a transaction accepted by using known fee rate. It CAN potentially create a fee market incentivising lower fee transactions to increase their fees though CPFP but the critical difference between this market and the BTC market is that it has a known ceiling.
This mempool memory limit exists simply because it is preferable to having no fall back option. But it is not intended to ever be reached. And for several reasons, this is likely to be a very rare event:
- Miners generally have very large mempools, most are configured with at least 8GB and they can raise this limit relatively easily in real time if there is an operational need to do so.
- If you are a transaction sender, it is in your interests to discover what fee is required to get your transaction mined using services like Merchant API. Otherwise, users risk having their transactions stuck in a mempool for days or weeks, and unable to access those funds during that time without using a CPFP payment to nudge it over the line. Once the behaviour of “fee discovery” is widely adopted and probably also automated, it is unlikely we will see this type of mempool buildup scenario again. The transactions with low fees won’t be there by mistake, they’ll be limited to users that intentionally want to test the market or who don’t care how long it takes to have their transactions mined.
- The mempool has a timeout: by default it’s two weeks, so it won’t keep filling up forever.
- There is always likely to be at least one miner that will intervene if the mempool gets out of hand to sweep the mempool as we saw TAAL did on May 12 – mining a world record sized 309MB block.
- Miners always have the hardline option of raising their relay fee and dropping all transactions below their transaction acceptance fee. I doubt they want to do this, but if they believe someone is trying to force them to drop fees by attempting to exhaust their mempool, they might just react by doing the opposite and demonstrating that miners are entirely free to choose what they include in their block.
It’s important to note that it is not much of a burden for miners to hold these transactions in memory. They don’t cause any significant CPU overhead; the transactions just sit there waiting. So as long as miners have enough memory (which most of them do), they can afford to hold transactions for as long as needed. Nevertheless, the BSV Node team is planning further improvements to offer more options to miners in these scenarios. The first will be the option to evict transactions to disk rather than holding them in memory. This will allow transactions to be kept for longer if that’s what the miner wants to do. The second option is more flexible eviction policies where if a transaction doesn’t meet their transaction acceptance (mining) fee, the miner can accept it and hold in mempool for a short period (perhaps one day) before evicting it.
So why did miners intervene?
Given that the BSV Node software was handling things just fine, you may be wondering why didn’t the miners just leave the mempool buildup as is? Eventually, Mempool would have mined those transactions because it was accepting 0.25 sat/byte transactions. One reason a miner (TAAL) decided to clear the mempool was due to some BSV application services reporting problems. I can speculate that because we had the attention of many BTC miners at the time due to the BTC block reward halving, it was probably a good time to demonstrate that a miner COULD.
Also I think there is a recognition among miners that the process of separating the relay fee from the transaction acceptance (mining) fee, and adjusting user behaviour is one that inevitably comes with some growing pains. And until users of Bitcoin (especially producers of high transaction volumes) adjust to new ways of interacting with the network that are more in line with Satoshi’s vision, a few hiccups will happen and miners don’t want services on the network to suffer needless outages. As the ecosystem professionalizes further, I’d expect miners to take a harder stance along the lines of “if you’re maintaining a mempool, you need to make sure you can handle it because we can and it isn’t our job to look after you”.
