History and balance mismatch

[yhaenggi]

I've tried the history mode of wallet-tool and it came up with this (values replaced with meaningful text)

BUG ERROR: wallet balance (BIGGERNUMBER) does not match balance from history (SMALLERNUMBER)
BUG ERROR: wallet utxo count (HIGHERCOUNT) does not match utxo count from history (LOWERCOUNT)

[chris-belcher]

I had this while testing. It happened to me because two joinmarket transactions actually belonged to the "" account instead of "joinmarket-wallet-whatever".

When I made it search "" as well as the other account, all transactions appears.

https://github.com/JoinMarket-Org/joinmarket/blob/master/wallet-tool.py#L293

You could try debugging by adding a likely wallet name to the list of accounts to search. Maybe you used two joinmarket wallets in the lifetime of that wallet.dat. Ideally there's be a way to obtain all wallet transactions regardless of which account they're in.

[yhaenggi]

I havent used multiple joinmarket wallets. there is only "" and "joinmarket-wallet-random" as accounts.

[yhaenggi]

i just checked and saw i deleted the yigen logfile some time ago, i'll have to recover it for proper results. this would cause less utxo in history and also much less balance (which both are the case).
im closing this therefore, sry for the trouble.

[chris-belcher]

wallet-tool history doesnt read from the yigen log file at all. It only reads from the blockchain. You can delete your log file and wallet-tool history will work in exactly the same way

Reopening

[Onefox]

i also have a case where this happend, i moved some coins via the sendpayment.py and the transaction is missing in the history and now the balance differ

[chris-belcher]

It's confirmed I assume(?)
If you open Bitcoin-Qt, does the transaction appear in the transactions tab?

[Onefox]

Yes it is, i am on console only, but with the normal joinmarket command where i can see all the addresses in the different mix depths and balances. There is the right balance.
Only with the history command i get the wrong one and this error.

Its console only i don't think bitcoin-qt would work there or?

[meeDamian]

I'd say I see it ~60% of the time when running python wallet-tool.py wallet.json history.

Also, which might be related(?), when I was trying to add notifications somewhat like here. I've noticed that my confirm_callback() fn is never called, even though I do have coinjoins happening.

Basically, while potentially earned is logged many times:

$ grep "potentially earned" logs/*.log | wc -l
A_NONZERO_NUMBER

then just earned, that should be logged by this happens never…

$ grep "earned" logs/*.log | grep -v "potentially" | wc -l
0

[chris-belcher]

That's very strange mmDamian, obviously you're getting unconfirmed notifies but not confirmed notifies, so -walletnotify isn't the problem. I wonder if there's a crash or bug somewhere in confirm_fun(), maybe your log file can tell us what happened?

[meeDamian]

@chris-belcher Here are some scrubbed logs that do contain potentially earned, but not earned.

Also, something I should probably also mention, is that I'm using pruned Bitcoin Core.

[chris-belcher]

@meeDamian Double check your -walletnotify config. I'm 80% sure that's the problem. Since the 'potentially earned' print can happen even if your -walletnotify config is wrong.

I use a pruned node too, that shouldn't make a difference.

[meeDamian]

This is my ~/.bitcoin/bitcoin.conf (minus user and password):

$ grep -v rpc ~/.bitcoin/bitcoin.conf
server=1
prune=10240
dbcache=1111
daemon=1

# for JoinMarket
walletnotify=curl -sI --connect-timeout 1 http://localhost:62602/walletnotify?%s
alertnotify=curl -sI --connect-timeout 1 http://localhost:62602/alertnotify?%s

I've restarted all: bitcoin node, joinmarket and machine couple of times since applying that config.

[chris-belcher]

I don't know but for some reason the notify messages are not reaching joinmarket.

[meeDamian]

Ok, my machine was allergic to localhost apparently. Swapping it with 127.0.0.1 in walletnotify and alertnotify and adding notify_host = 127.0.0.1 to joinmarket.cfg did the trick.

@chris-belcher But getting back to the original problem, do you thinks this could've been the cause behind balances mismatch?

[chris-belcher]

I doubt it, since wallet-tool history doesn't depend on walletnotify working.

If the balance mismatch happens every time, try changing this line https://github.com/JoinMarket-Org/joinmarket/blob/master/wallet-tool.py#L310

EDIT: it moves over time, here's the link with the SHA

joinmarket/wallet-tool.py

Line 399 in 8c2b6d8

for wn in [wallet_name, ""]:

Make it for wn in [wallet_name, "", "*"]:

and see if it fixes it

[meeDamian]

I only have one wallet there.

After doing some digging through, it seems that it only shows if there's a tx in-between being broadcast and confirmed. My best guess would be that, while the wallet, based on its memory pool(?), has already increased its total balance, the relevant transaction is not yet returned by listtransactions, hence not yet summed up in balance, making total_wallet_balance > balance. Same would apply to utxos.

In which case only the BUG ERROR: wording would be misleading :)

[ian-kelling]

I just hit this. I had a previous wallet, but I immediately deleted it
without making any transactions at all. Adding * to the wallets does not help.

It began after the very first coinjoin. It happens every time I run history.
I don't know if it will persist after further transactions.

The relevant log from yield-generator-basic.py on commit 05b758c, various data replaced with dashes.

 -----,--- [MCThread    ] [INFO ]  obtained tx
{'ins': [{'outpoint': {'hash':
-----
}
 -----,--- [MCThread    ] [INFO ]  potentially earned = ---
 -----,--- [MCThread    ] [INFO ]  goodtx
 -----,--- [CoreNotifyTh] [INFO ]  started bitcoin core notify listening thread, host=localhost port=62602
 ---,--- [CoreNotifyTh] [INFO ]  saw tx on network, removed_utxos=
{---': {'address': u'---',
                                                                        'value': ---}}
 ---,--- [CoreNotifyTh] [INFO ]  modifying orders. to_cancel=[]
to_announce=[{'ordertype': 'reloffer', 'oid': 0, 'minsize': ---, 'txfee': ---, 'maxsize': ---, 'cjfee': '---'}]

[ian-kelling]

Update: I ran history again after at least one more coinjoin, and did not hit this error.

[chris-belcher]

We should at least make the error message less scary.

[mecampbellsoup]

@chris-belcher not to be a nag, but could you please use SHAs when linking to GH since master moves over time? Per: #534 (comment)

Thanks, you rock!

[chris-belcher]

That line is this:

joinmarket/wallet-tool.py

Line 399 in 8c2b6d8

for wn in [wallet_name, ""]:

I also edited that comment above.