Difference between revisions of "Compacting Channel DB"

From PlebNet Wiki
Jump to navigation Jump to search
 
(14 intermediate revisions by the same user not shown)
Line 1: Line 1:
The BOLT db, or channel.db, stores payments sent by your LND node, including failed payments, failed htlcs from payments and payment probes. The database is located under <code>~/umbrel/lnd/data/graph/mainnet/channel.db></code> on Umbrel.
The BOLT db, or channel.db, stores the entire history of payments processed by your LND node, including failed payments, failed htlcs from payments, and payment probes. The database is located at <code>~/umbrel/lnd/data/graph/mainnet/channel.db</code> on Umbrel.


The file continuously grows and may eventually exceed the available disk space and/or slow things down. 1GB or 2GB size is reasonable, but your node may begin to slow down when the file reaches 6 - 8Gb.
The file continuously grows and may eventually exceed the available disk space and/or slow things down. 1GB or 2GB size is reasonable, but your node may begin to slow down when the file reaches 6 - 8Gb.


Compact & prune channel.db periodically. Pruning refers to marking payment records for deletion. Compaction refers to removing the marked records from the database and thus reducing its size.
Monitor channel.db, compact & prune it when needed. Pruning refers to marking payment records for deletion. Compaction refers to removing the marked records from the database and thus reducing its size.


== Compaction ==
== Monitor channel.db size ==


See configuration settings to enable compaction: [[Special:MyLanguage/LND_Configuration_Settings#Database_optimizations|enable compaction of channel.db]].
Ways to monitor channel.db:


Compaction makes a copy of channel.db during lnd restart, removes the marked records, and restores the file. Make sure to have enough disk space to support compaction, or lnd may fail to start.
* periodically check the channel.db size by <code>ls -l <channel.db path></code>.
* install [https://github.com/itsneski/lightning-jet#telegram-bot Lightning Jet telegram bot]; the bot will notify when channel.db grows over the threshold.
* run [https://github.com/itsneski/lightning-jet Lightning Jet] <code>jet monitor</code> or <code>jet channeldb</code> tools; the tools will show a warning when channel db grows over the threshold.


== Pruning ==
== Create a backup ==


Pruning can be automated and manual. Automated pruning can be [[Special:MyLanguage/LND_Configuration_Settings#Database_optimizations|configured via the lnd.conf]]. Compaction will then delete channel.db pruned records.
Backup channel.db payments prior to pruning and compaction: <code>lncli listpayments > listpayments.bak</code>. Place <code>listpayments.bak</code> in a backup folder under your home directory, e.g. <code>~/listpayments_backup</code>. Indicate current date in the file name, e.g. <code>listpayments.bak.2021.12.29</code>.


Manual pruning is needed in case automated pruning could not sufficiently reduce the size of channel.db. Node operators can do manual pruning in a few different ways: identify and reset channels with the biggest disk footprint; manually remove payments from the channel.db.
== Enable pruning ==


Node operators can identify channels with the biggest footprint through [https://api.lightning.community/#listchannels lncli listchannels], the <code>num_updates</code> filed will indicate the number of channel state updates a particular channel did. Node operators can also use <code>bos <alias/pubkey></code> as part of [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] tool. [https://github.com/itsneski/lightning-jet Lightning Jet] provides way to monitor channel.db and receive warnings via <code>jet monitor</code> (including [https://github.com/itsneski/lightning-jet#telegram-bot telegram bot] notications).
[[Special:MyLanguage/LND_Configuration_Settings#BOLT_Database_optimizations_.28channel.db.29|Update lnd.conf]] to enable automated pruning.


Once a channel with the biggest footprint is identified, the node operator can re-establish it by closing the channel and creating a new one. Please note that there is a cost associated with closing and creating channels, so node operators should use this option carefully.  
Note: in lnd14 [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] automatically prunes the obsolete channel.db records generated as part of payment probes.


In addition, node operators can prune config.db manually by calling the [https://api.lightning.community/#deleteallpayments LND API DeleteAllPayments], with <code>failed_payments_only</code> and <code>failed_htlcs_only</code> set to true.  lncli does not currently support the call so you will need to write a script (in say Python or JavaScript) to call this method. [https://github.com/itsneski/lightning-jet/blob/main/tools/prune-payments Example of a JavaScript code]. There is also a hidden bos command <code>bos delete-payments-history</code> which will delete all payments, including successful ones. You can find the details [https://github.com/niteshbalusu11/BOS-Commands-Document#delete-payments-history here].
== Prune manually ==


Manual pruning may take time to complete depending on the channel.db size. Make sure to wait until it completes and don't interrupt the process. Once pruning completes verify that the number of payments in the channel.db is zero by running <code>lncli listpayments | jq '.payments' | jq length</code>.  Delete <code>~/umbrel/lnd/data/graph/mainnet/channel.db.last-compacted</code> to trigger compaction on restart. Next restart you node: <code>restart lnd cd ~/umbrel ; docker-compose restart lnd; docker-compose ps</code>
Prune manually when automated pruning is not able to sufficiently reduce the size of channel.db.


== Upcoming lnd release ==
=== Clean failed payments ===


[https://github.com/lightningnetwork/lnd/releases The next major lnd14 release] will introduce new ways to manage the channel.db size. It'll allow more granular deletion of records from the channel.db. [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] will automatically prune obsolete channel.db records generated as part of payment probes. In addition, <code>bos clean-failed-payments</code> will take over as a visible command (currently in dry run mode).
Ways to clean failed payments based on lnd version, <code>lncli -v</code>. Note: use this with caution and only when there is no other choice. Make sure to identify your version of lnd prior to calling any of the methods. Do not call a method multiple times in case of a failure. Message on Plebnet Noderunnes or Plebnet Advanced telegram groups in case of a failure.
 
* lnd13: [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] <code>bos delete-payments-history</code>. This command will delete all payments, including successful ones, use it with caution. Details [https://github.com/niteshbalusu11/BOS-Commands-Document#delete-payments-history here].
* lnd14: [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] <code>bos clean-failed-payments</code>. The command is safe to use as it only cleans failed payments.
* lnd13+: call [https://github.com/itsneski/lightning-jet Lightning Jet] <code>tools/prune-payments</code> tool; the tool will only remove failed payments & htlcs (lnd13 and up).
* lnd13+: write a script to call [https://api.lightning.community/#deleteallpayments LND API DeleteAllPayments] ([https://github.com/itsneski/lightning-jet/blob/main/tools/prune-payments example of a JavaScript code]), with <code>failed_payments_only</code> and <code>failed_htlcs_only</code> set to true. Please note that the flags are supported in lnd13 and up, and are not supported in lnd12. Take extra caution before calling this method, as if called in lnd12 it'll remove all payments, including good ones.
 
=== Re-establish channels with the biggest footprint ===
 
Ways to identify channels with the biggest footprint:
 
* call [https://api.lightning.community/#listchannels lncli listchannels], the <code>num_updates</code> field indicates the number of channel state updates for a channel.
* [https://github.com/alexbosworth/balanceofsatoshis BalanceOfSatoshis] <code>bos <alias/pubkey></code>.
* [https://github.com/itsneski/lightning-jet Lightning Jet] <code>jet list-channels</code> returns top 10 channels based on footprint.
 
Re-establish a channel(s) with the biggest footprint by closing the channel and creating a new one. Please note that there is a cost associated with closing and creating channels, so node operators should use this option carefully.
 
== Compact ==
 
Steps to enable and trigger compaction:
 
# [[Special:MyLanguage/LND_Configuration_Settings#BOLT_Database_optimizations_.28channel.db.29|Update lnd.conf]] to enable compaction.
# Make sure to have enough disk space to support compaction, or lnd may fail to start. Compaction makes a copy of channel.db during lnd restart.
# Delete <code>~/umbrel/lnd/data/graph/mainnet/channel.db.last-compacted</code> to trigger compaction on restart.
# Restart lnd: <code>cd ~/umbrel ; docker-compose restart lnd; docker-compose ps</code>.

Latest revision as of 18:39, 30 December 2021

The BOLT db, or channel.db, stores the entire history of payments processed by your LND node, including failed payments, failed htlcs from payments, and payment probes. The database is located at ~/umbrel/lnd/data/graph/mainnet/channel.db on Umbrel.

The file continuously grows and may eventually exceed the available disk space and/or slow things down. 1GB or 2GB size is reasonable, but your node may begin to slow down when the file reaches 6 - 8Gb.

Monitor channel.db, compact & prune it when needed. Pruning refers to marking payment records for deletion. Compaction refers to removing the marked records from the database and thus reducing its size.

Monitor channel.db size

Ways to monitor channel.db:

  • periodically check the channel.db size by ls -l <channel.db path>.
  • install Lightning Jet telegram bot; the bot will notify when channel.db grows over the threshold.
  • run Lightning Jet jet monitor or jet channeldb tools; the tools will show a warning when channel db grows over the threshold.

Create a backup

Backup channel.db payments prior to pruning and compaction: lncli listpayments > listpayments.bak. Place listpayments.bak in a backup folder under your home directory, e.g. ~/listpayments_backup. Indicate current date in the file name, e.g. listpayments.bak.2021.12.29.

Enable pruning

Update lnd.conf to enable automated pruning.

Note: in lnd14 BalanceOfSatoshis automatically prunes the obsolete channel.db records generated as part of payment probes.

Prune manually

Prune manually when automated pruning is not able to sufficiently reduce the size of channel.db.

Clean failed payments

Ways to clean failed payments based on lnd version, lncli -v. Note: use this with caution and only when there is no other choice. Make sure to identify your version of lnd prior to calling any of the methods. Do not call a method multiple times in case of a failure. Message on Plebnet Noderunnes or Plebnet Advanced telegram groups in case of a failure.

  • lnd13: BalanceOfSatoshis bos delete-payments-history. This command will delete all payments, including successful ones, use it with caution. Details here.
  • lnd14: BalanceOfSatoshis bos clean-failed-payments. The command is safe to use as it only cleans failed payments.
  • lnd13+: call Lightning Jet tools/prune-payments tool; the tool will only remove failed payments & htlcs (lnd13 and up).
  • lnd13+: write a script to call LND API DeleteAllPayments (example of a JavaScript code), with failed_payments_only and failed_htlcs_only set to true. Please note that the flags are supported in lnd13 and up, and are not supported in lnd12. Take extra caution before calling this method, as if called in lnd12 it'll remove all payments, including good ones.

Re-establish channels with the biggest footprint

Ways to identify channels with the biggest footprint:

Re-establish a channel(s) with the biggest footprint by closing the channel and creating a new one. Please note that there is a cost associated with closing and creating channels, so node operators should use this option carefully.

Compact

Steps to enable and trigger compaction:

  1. Update lnd.conf to enable compaction.
  2. Make sure to have enough disk space to support compaction, or lnd may fail to start. Compaction makes a copy of channel.db during lnd restart.
  3. Delete ~/umbrel/lnd/data/graph/mainnet/channel.db.last-compacted to trigger compaction on restart.
  4. Restart lnd: cd ~/umbrel ; docker-compose restart lnd; docker-compose ps.