Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wallet TX count and time indexing #605

Closed
wants to merge 70 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
680e20a
test: add utility for test fs cleanup
braydonf Sep 10, 2018
5746ee5
test: add utility to sleep for desired ms
braydonf Sep 10, 2018
248c3d4
wallet: add test cases for wallet tx count and time indexing
braydonf Sep 11, 2018
41a371f
wallet: add db layout for tx count and monotonic time
braydonf Sep 11, 2018
a53f966
wallet: implement monotonic time
braydonf Sep 11, 2018
fe9f3f3
wallet: add tx count index for confirmed txs
braydonf Sep 11, 2018
b5a9074
wallet: add rpc methods for tx count and monotonic time
braydonf Sep 11, 2018
4f96017
test: restore count to generateBlocks regtest util
braydonf Sep 12, 2018
c931791
docs: after is exclusive
braydonf Sep 12, 2018
c289780
wallet: monotonic time indexing
braydonf Sep 13, 2018
56ef021
test: add reorg to wallet time test
braydonf Sep 21, 2018
bbaa222
test: monotonic time reorg, incomplete
braydonf Oct 17, 2018
9b2c585
wallet: rollback monotonic time
braydonf Oct 23, 2018
0f99886
wallet: add spv monotonic time reorg
braydonf Oct 23, 2018
54ff89a
wallet: reorg tx monotonic time and count indexes
braydonf Oct 25, 2018
99a396e
wallet: refactor monotonic index method
braydonf Oct 27, 2018
2c5ddb0
wallet: options for list history methods
braydonf Oct 27, 2018
1def771
wallet: add indexing for unconfirmed count and time
braydonf Nov 6, 2018
f9cc58d
wallet: add methods to query for unconfirmed txs
braydonf Nov 8, 2018
ea7c27c
test: confirm and unconfirm txs
braydonf Nov 12, 2018
9adf54c
wallet: combine `getPending` and `listUnconfirmed`
braydonf Nov 12, 2018
69e2f1b
wallet: http endpoints for history and unconfirmed txs
braydonf Nov 20, 2018
d4afc1d
wallet: test spv node http endpoints and w/ fixes
braydonf Nov 21, 2018
dafad8d
test: remove `.skip` function call, it's not implemented in bmocha
braydonf Dec 5, 2018
9dc1673
test: wallet rescan test
braydonf Dec 18, 2018
e8c6410
wallet: use height + index for count index
braydonf Dec 19, 2018
30d79d4
test: remove spv from wallet rescan test
braydonf Dec 19, 2018
d8ff085
test: assertions for wallet rescan test
braydonf Dec 19, 2018
a0e5729
wallet: use median time past for time indexing
braydonf Dec 20, 2018
d9f4601
wallet: linting
braydonf Dec 20, 2018
857d07c
wallet: use brhash for http after param
braydonf Dec 20, 2018
7b41f51
test: fix spv test when cpu is loaded
braydonf Dec 20, 2018
6d1748c
wallet: minor cleanup
braydonf Dec 20, 2018
bd29e9f
wallet: cleanup indexes
braydonf Dec 20, 2018
00c652a
wallet: make configurable max transactions limits
braydonf Dec 20, 2018
4108b58
wallet: minor cleanup
braydonf Dec 20, 2018
43199ec
wallet: fix configurable txs limits
braydonf Dec 20, 2018
f66c89d
test: bump wallet http before hook time
braydonf Dec 20, 2018
be59b32
wallet: fix http time tx queries
braydonf Dec 21, 2018
ce7ca8b
wallet: minor linting
braydonf Dec 21, 2018
a803c98
wallet: check rpc min args length
braydonf Dec 21, 2018
49cc51b
test: minor linting
braydonf Dec 21, 2018
5b51439
wallet: fix sorting of time indexing
braydonf Dec 21, 2018
abe2353
wallet: use brhash instead of manually reversing
braydonf Dec 21, 2018
3f41dae
docs: minor linting
braydonf Dec 21, 2018
2fe41f2
wallet: fix sorting of confirmed time index
braydonf Dec 21, 2018
f386a39
wallet: refactor date handling for api
braydonf Dec 21, 2018
0974f72
test: refactor utils
braydonf Dec 21, 2018
8de45e5
wallet: minor cleanup and adjustments
braydonf Dec 21, 2018
cda883a
wallet: minor cleanup and update
braydonf Dec 21, 2018
561b1ae
wallet: update rpc methods with new history functions
braydonf Dec 22, 2018
3fd5216
wallet: use unix epoch time in seconds for rpc history cmds
braydonf Dec 23, 2018
fb46596
test: increase lookahead for test wallet
braydonf Dec 23, 2018
794b417
test: refactor initial blocks
braydonf Dec 25, 2018
1dda1a8
wallet client: add getMedianTime
pinheadmz Jan 8, 2019
cfb8359
wallet: list confirmed and unconfirmed history
braydonf Jan 30, 2019
f0cbe7f
test: update for segwit activation on regtest
braydonf Feb 12, 2019
10ee54f
changelog: update for transaction history api changes
braydonf Feb 12, 2019
7d3c403
wallet: check if options available
braydonf Mar 1, 2019
02b3141
test: listtransactions is deprecated
braydonf Mar 21, 2019
4c559f1
changelog: add note about max-txs
braydonf Mar 26, 2019
b14237d
wallet: minor, refactor and comments
braydonf Mar 26, 2019
62b26fd
test: use `testdir` for prefixes
braydonf May 15, 2019
6833408
test: updates wallet tests to use `forValue`
braydonf May 16, 2019
6024507
test: updates for nyc and ci
braydonf Jun 5, 2019
f339e8c
node: restore error logger argument
braydonf Jun 5, 2019
f75a157
wallet: remove http history date argument
braydonf Jun 5, 2019
1d8c94b
wallet: remove deprecated methods
braydonf Jun 5, 2019
069e3d6
test: minor, cleanup todos
braydonf Jun 5, 2019
7d8ddf8
test: use bsert instead of assert util
braydonf Jun 27, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
wallet: fix http time tx queries
  • Loading branch information
braydonf committed Jun 27, 2019
commit be59b32c75faa3f889e0c42c1d84fe9cb23dae5b
20 changes: 17 additions & 3 deletions lib/wallet/http.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ const KeyRing = require('../primitives/keyring');
const Mnemonic = require('../hd/mnemonic');
const HDPrivateKey = require('../hd/private');
const HDPublicKey = require('../hd/public');
const util = require('../utils/util');
const common = require('./common');

/**
Expand Down Expand Up @@ -769,9 +770,15 @@ class HTTP extends Server {
const acct = valid.str('account');
const reverse = valid.bool('reverse', false);
const limit = valid.u32('limit', 100);

const after = valid.brhash('after');
const time = valid.u32('time');
let time = valid.u32('time');
let date = valid.str('date');

if (date && !time) {
date = new Date(date);
enforce(date.toString() !== 'Invalid Date', 'Invalid date.');
time = util.time(date);
}

enforce(limit <= this.options.maxTxs,
braydonf marked this conversation as resolved.
Show resolved Hide resolved
`Limit above max of ${this.options.maxTxs}.`);
Expand Down Expand Up @@ -814,7 +821,14 @@ class HTTP extends Server {
const reverse = valid.bool('reverse');
const limit = valid.u32('limit');
const after = valid.brhash('after');
const time = valid.u32('time');
let time = valid.u32('time');
let date = valid.str('date');

if (date && !time) {
date = new Date(date);
enforce(date.toString() !== 'Invalid Date', 'Invalid date.');
time = util.time(date);
}

enforce(limit <= this.options.maxTxs,
`Limit above max of ${this.options.maxTxs}.`);
Expand Down
6 changes: 2 additions & 4 deletions lib/wallet/rpc.js
Original file line number Diff line number Diff line change
Expand Up @@ -1280,8 +1280,7 @@ class RPC extends RPCBase {
`Limit above max of ${this.maxTxs}.`);

const date = new Date(timestamp);
/* eslint eqeqeq: "off" */
if (date == 'Invalid Date')
if (date.toString() === 'Invalid Date')
throw new RPCError(errs.INVALID_PARAMETER, 'Invalid timestamp.');

const time = util.time(date);
Expand Down Expand Up @@ -1370,8 +1369,7 @@ class RPC extends RPCBase {
throw new RPCError(errs.INVALID_PARAMETER, 'Limit above max of 100.');

const date = new Date(timestamp);
/* eslint eqeqeq: "off" */
if (date == 'Invalid Date')
if (date.toString() === 'Invalid Date')
throw new RPCError(errs.INVALID_PARAMETER, 'Invalid timestamp.');

const time = util.time(date);
Expand Down
68 changes: 50 additions & 18 deletions lib/wallet/txdb.js
Original file line number Diff line number Diff line change
Expand Up @@ -1846,24 +1846,40 @@ class TXDB {

let max = null;
let min = null;
let parse = null;

if (options.reverse) {
min = layout.G.min();
max = layout.G.max(acct, options.time);
if (acct !== -1) {
if (options.reverse) {
min = layout.G.min();
max = layout.G.max(acct, options.time);
} else {
min = layout.G.min(acct, options.time);
max = layout.G.max();
}
parse = (key) => {
const [,,hash] = layout.G.decode(key);
return hash;
}
} else {
min = layout.G.min(acct, options.time);
max = layout.G.max();
if (options.reverse) {
min = layout.g.min();
max = layout.g.max(options.time);
} else {
min = layout.g.min(options.time);
max = layout.g.max();
}
parse = (key) => {
const [,hash] = layout.g.decode(key);
return hash;
}
}

const keys = await this.bucket.keys({
gte: min,
lte: max,
limit: 1,
reverse: options.reverse,
parse: (key) => {
const [,,hash] = layout.G.decode(key);
return hash;
}
parse: parse
});

const txid = keys.length > 0 ? keys[0] : null;
Expand Down Expand Up @@ -2049,24 +2065,40 @@ class TXDB {

let max = null;
let min = null;
let parse = null;

if (options.reverse) {
min = layout.W.min();
max = layout.W.max(acct, options.time);
if (acct !== -1) {
if (options.reverse) {
min = layout.W.min();
max = layout.W.max(acct, options.time);
} else {
min = layout.W.min(acct, options.time);
max = layout.W.max();
}
parse = (key) => {
const [,,hash] = layout.W.decode(key);
return hash;
}
} else {
min = layout.W.min(acct, options.time);
max = layout.W.max();
if (options.reverse) {
min = layout.w.min();
max = layout.w.max(options.time);
} else {
min = layout.w.min(options.time);
max = layout.w.max();
}
parse = (key) => {
const [,hash] = layout.w.decode(key);
return hash;
}
}

const keys = await this.bucket.keys({
gte: min,
lte: max,
limit: 1,
reverse: options.reverse,
parse: (key) => {
const [,,hash] = layout.W.decode(key);
return hash;
}
parse: parse
});

const txid = keys.length > 0 ? keys[0] : null;
Expand Down
92 changes: 90 additions & 2 deletions test/wallet-http-test.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ const {
const testPrefix = '/tmp/bcoin-fullnode';
const spvTestPrefix = '/tmp/bcoin-spvnode';
const genesisTime = 1534965859;
const genesisDate = new Date(genesisTime * 1000);

const ports = {
full: {
Expand All @@ -40,6 +41,8 @@ describe('Wallet TX HTTP Pagination', function() {
let node, spvnode, wallet = null;
let nclient, wclient, spvwclient = null;
let coinbase = null;
let unconfirmedTime = null;
let unconfirmedDate = null;

before(async () => {
await rimraf(testPrefix);
Expand Down Expand Up @@ -77,11 +80,14 @@ describe('Wallet TX HTTP Pagination', function() {
blocks: 125
});

// Generate unconfirmed transactions for the
// fullnode and spv wallet
unconfirmedTime = Math.floor(Date.now() / 1000);
unconfirmedDate = new Date(unconfirmedTime * 1000);

// TODO remove this
await sleep(5000);

// Generate unconfirmed transactions for the
// fullnode and spv wallet
await generateTxs({
wclient,
spvwclient,
Expand Down Expand Up @@ -174,6 +180,26 @@ describe('Wallet TX HTTP Pagination', function() {
assert.strictEqual(two[49].confirmations, 125);
assert.notStrictEqual(two[0].hash, one[99].hash);
});

it('with datetime (MTP in ISO 8601)', async () => {
const history = await wclient.get('/wallet/test/tx/history', {
limit: 100,
date: new Date(),
reverse: true
});
assert.strictEqual(history.length, 100);
assert(history[0].confirmations < history[99].confirmations);
});

it('with datetime (MTP in epoch seconds)', async () => {
const history = await wclient.get('/wallet/test/tx/history', {
limit: 100,
time: Math.ceil(Date.now() / 1000),
reverse: true
});
assert.strictEqual(history.length, 100);
assert(history[0].confirmations < history[99].confirmations);
});
});

describe('confirmed txs (asc)', function() {
Expand Down Expand Up @@ -211,6 +237,26 @@ describe('Wallet TX HTTP Pagination', function() {
assert.strictEqual(two[11].confirmations, 102);
assert.notStrictEqual(two[0].hash, one[11].hash);
});

it('with datetime (MTP in ISO 8601)', async () => {
const history = await wclient.get('/wallet/test/tx/history', {
limit: 100,
date: genesisDate.toString(),
reverse: false
});
assert.strictEqual(history.length, 100);
assert(history[0].confirmations > history[99].confirmations);
});

it('with datetime (MTP in epoch seconds)', async () => {
const history = await wclient.get('/wallet/test/tx/history', {
limit: 100,
time: genesisTime,
reverse: false
});
assert.strictEqual(history.length, 100);
assert(history[0].confirmations > history[99].confirmations);
});
});

describe('unconfirmed txs (dsc)', function() {
Expand Down Expand Up @@ -254,6 +300,26 @@ describe('Wallet TX HTTP Pagination', function() {

assert.notStrictEqual(two[0].hash, one[3].hash);
});

it('with datetime (MTP in ISO 8601)', async () => {
const history = await wclient.get('/wallet/test/tx/unconfirmed', {
limit: 100,
date: new Date(Date.now() + 2000),
reverse: true
});
assert.strictEqual(history.length, 19);
assert(history[0].mtime > history[18].mtime);
});

it('with datetime (MTP in epoch seconds)', async () => {
const history = await wclient.get('/wallet/test/tx/unconfirmed', {
limit: 100,
time: Math.ceil((Date.now() + 2000) / 1000),
reverse: true
});
assert.strictEqual(history.length, 19);
assert(history[0].mtime > history[18].mtime);
});
});

describe('unconfirmed txs (asc)', function() {
Expand Down Expand Up @@ -307,6 +373,28 @@ describe('Wallet TX HTTP Pagination', function() {

assert.notStrictEqual(two[0].hash, one[3].hash);
});

it('with datetime (MTP in ISO 8601)', async () => {
const history = await wclient.get('/wallet/test/tx/unconfirmed', {
limit: 100,
date: unconfirmedDate.toString(),
reverse: false
});
// TODO
//assert.strictEqual(history.length, 19);
assert(history[0].mtime < history[10].mtime);
});

it('with datetime (MTP in epoch seconds)', async () => {
const history = await wclient.get('/wallet/test/tx/unconfirmed', {
limit: 100,
time: unconfirmedTime,
reverse: false
});
// TODO
// assert.strictEqual(history.length, 19);
assert(history[0].mtime < history[10].mtime);
});
});
});

Expand Down