graph/db: expand TestPopulateViaMigration for easy testing

[ellemouton]

Testing the Graph SQL migration

Steps:

1. Check out this branch
2. Find the TestPopulateViaMigration test in graph/db/benchmark_test.go.
3. Follow the steps in that test closely
   3.1 Steps 1 - 6 is all about setting up the source DB (this can even be a bbolt channel.db file).
   3.2 Steps 7 - 10 is about setting up the source database (sqlite or postgres)
4. Run the following (you can remove the kvdb_postgres tag if neither your destination or source is postgres)

make unit pkg=graph/db tags="test_db_sqlite kvdb_sqlite kvdb_postgres" case=TestPopulateViaMigration verbose=1

Example successful output:

=== RUN   TestPopulateViaMigration
2025-08-13 16:20:28.243 [INF]: Starting migration of the graph store from KV to SQL
2025-08-13 16:20:28.250 [DBG]: Migrated 1 nodes (157.35 nodes/sec)
2025-08-13 16:20:37.773 [INF]: Migrated 16216 nodes from KV to SQL in 9.529524833s (skipped 0 nodes due to invalid TLV streams)
2025-08-13 16:20:37.773 [DBG]: Migrating source node from KV to SQL
2025-08-13 16:20:37.781 [INF]: Migrated source node with pubkey 0256812a1cb2539a5500f3c1c20db5ed7626e3878a552e3356c032c7bec2b3060e to SQL
2025-08-13 16:20:38.095 [DBG]: Migrated 1 channels (3.18 channels/sec)
2025-08-13 16:20:48.096 [DBG]: Migrated 23177 channels (2317.49 channels/sec)
2025-08-13 16:20:58.097 [DBG]: Migrated 37544 channels (1436.62 channels/sec)
2025-08-13 16:21:07.765 [INF]: Migrated 51239 channels and 82029 policies from KV to SQL in 29.983330209s(skipped 0 channels and 0 policies due to invalid TLV streams)
2025-08-13 16:21:07.768 [DBG]: Migrated 1 prune log entries (377.88 entries/sec)
2025-08-13 16:21:17.768 [DBG]: Migrated 164509 prune log entries (16450.79 entries/sec)
2025-08-13 16:21:27.768 [DBG]: Migrated 328502 prune log entries (16399.22 entries/sec)
2025-08-13 16:21:29.944 [INF]: Migrated 361511 prune log entries from KV to SQL in 22.178933583s. The prune tip is: height 892530, hash: 0000000000000000000157abc0fc27f519eff6dc0ab6b6ab9a44505b737b2802
2025-08-13 16:21:29.947 [DBG]: Migrated 1 closed scids (347.34 entries/sec)
2025-08-13 16:21:30.966 [INF]: Migrated 17931 closed SCIDs from KV to SQL in 1.021946292s
2025-08-13 16:21:30.969 [DBG]: Migrated 1 zombie index entries (367.15 entries/sec)
2025-08-13 16:21:40.969 [DBG]: Migrated 92206 zombie index entries (9220.41 entries/sec)
2025-08-13 16:21:50.969 [DBG]: Migrated 182574 zombie index entries (9036.77 entries/sec)
2025-08-13 16:21:57.720 [INF]: Migrated 238183 zombie channels from KV to SQL in 26.75310375s
2025-08-13 16:21:57.720 [INF]: Finished migration of the graph store from KV to SQL in 1m29.475824792s
--- PASS: TestPopulateViaMigration (89.79s)
PASS
ok      github.com/lightningnetwork/lnd/graph/db        90.132s

Debugging:

• if you're using a postgres destination and you see an error like this:

  could not migrate nodes: could not migrate nodes: could not persist node(..): upserting node(..): sql: no rows in result set
  

  it likely means you're running the migration test a second time and forgot to first drop & recreate the previous postgres database.
  See step 9 in TestPopulateViaMigration

[gemini-code-assist]

Summary of Changes

Hello @ellemouton, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the TestPopulateViaMigration test function within the graph/db package. The primary goal is to make it much easier for developers to manually test the graph database migration process from various KV (key-value) store backends (BBolt, SQLite, Postgres) to SQL backends (SQLite, Postgres). The changes introduce clear, step-by-step instructions and configurable parameters directly within the test, streamlining the setup and execution of migration tests for different database configurations.

Highlights

• Improved Test Configurability: The TestPopulateViaMigration function has been expanded with detailed, commented steps, allowing users to easily configure source and destination database types (BBolt, SQLite, Postgres) and their respective connection parameters.
• Refactored BBolt Connection: The logic for connecting to a BBolt database for testing has been extracted into a new helper function, kvdbBBolt, improving code modularity and reusability.
• Dynamic ChainHash: The ChainHash parameter used in the MigrateGraphToSQL function within the test is now dynamically configurable, allowing for more flexible testing scenarios.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors the database connection logic for bbolt and significantly expands TestPopulateViaMigration to make it a flexible helper for testing database migrations. The changes are well-structured and improve testability. I've added a couple of minor comments regarding a missing function comment and a typo, in line with the project's style guide.

[bhandras]

😂

[guggero]

Great stuff, LGTM 🎉

[ellemouton]

(cc @guggero for override pls 🙏 i added the tag too late it seems & re-running that check doesnt re-check if the tag is present. dont wanna re-run the whole ci for this test that is skipped anyways.

[bitromortac]

I tested the migration from bbolt to sqlite and it took about a minute for mainnet data, ending up with a 90 MB sqlite graph db file. I also didn't see excessive memory usage. Great work 🎉🎉!!