Coins not discovered within initial sync might never be found

[thomaseizinger]

Describe the bug

BDK does not seem to always find all coins on-chain if the wallet has been in use for a while and we don't have a local database.

To Reproduce

1. Use in-memory database.
2. Have a wallet whose external descriptor has been used for more than CACHE_ADDR_BATCH_SIZE addresses.

I've made a private repository here (contains the testnet coin descriptor, please don't steal them 🙃 ) and invited a few select maintainers of this repository: https://github.com/thomaseizinger/bdk-missing-coins-bug

Expected behavior

BDK to always find all coins as long as they are within a specified stop_gap.

Build environment

• BDK tag/commit: 0.15.0
• OS+version: Linux thomas-thinkpad 5.10.89-1-MANJARO
• Rust/Cargo version: 1.57.0
• Rust/Cargo target: x86_64-unknown-linux-gnu

Additional context

I've tested this also with the esplora-ureq backend but without luck.

[thomaseizinger]

I think the issue might be because we always go from 0 to max_address_param:

bdk/src/wallet/mod.rs

Line 1506 in fdb272e

self.cache_addresses(KeychainKind::External, 0, max_address)?;

Instead, what I think we should be doing is going from last_index_with_activity to last_index_with_activity + max_address_param! That way, consecutive syncs of the wallet will scan forward in batches until there are at least max_address_param addresses without activity.

[LLFourn]

Yes. I've noticed this as well. IMO what we should be doing is caching on-demand rather than batching some arbitrary number stuff up front. I think #461 made this easier.