Stake tables: tickets, votes, misses by chappjc · Pull Request #300 · decred/dcrdata

chappjc · 2017-12-15T23:58:11Z

IMPORTANT: This requires removing and rebuilding all DBs, including dcrdata.sqlt.db, the PostgreSQL tables, and the stakedb (the ffldb_stake folder).

Adds stake-related tables, including: tickets, votes, misses, winners.
Note that this PR modifies apitypes.TicketPoolInfo to include height at which the result applies, and winning tickets drawn at that height. Existing fields are unchanged.

TODO:

winners stored only in sqlite, need to store in the full DB (e.g. postgresql) too
SELECT statements
show missed votes (the ticket that could have been spent as a vote) on the block pages
~~- [ ] proof-of-concept page to exercise the table queries~~ This was ill-defined

Bonuses:

changes the syntax for rebuilddb2 command line tool so that "-u" is removed, but the behaviour is now the default. Use "-a/--addrspends-no-batch" to do the address table updates online (slow, impractical for initial build).
duplicate entries in vins and vouts are removed prior to their indexing so that the index may be UNIQUE.
Switch a bunch of "dup check"-enabled insert statements to use the "upsert" instead of "on conflict ... do nothing" behaviour.

chappjc · 2018-01-05T18:54:38Z

@raedah @RogueElement The only visible change is showing missed votes on the block page, mainly as a test. The added tables (tickets, votes, misses) should support a lot of new functionality though.

The README also needs an update since all DBs need recreation (SQLite, PostgreSQL, and stakedb/ffl_stakedb).

I also need to review my TODOs as I think there are some things still to do... updating ticket spend status for revokes comes to mind.

chappjc

some comments for reviewers by way of a self-review

chappjc · 2018-01-05T19:09:08Z

+	insertTxRow = insertTxRow0 + `RETURNING id;`
+	//insertTxRowChecked = insertTxRow0 + `ON CONFLICT (tx_hash, block_hash) DO NOTHING RETURNING id;`
+	upsertTxRow = insertTxRow0 + `ON CONFLICT (tx_hash, block_hash) DO UPDATE 
+		SET tx_hash = $8, block_hash = $1 RETURNING id;`


not right. Need to update the block_index, block_hash, etc.

chappjc · 2018-01-05T19:10:10Z

+	insertMissRow = insertMissRow0 + `RETURNING id;`
+	// insertVoteRowChecked = insertMissRow0 + `ON CONFLICT (ticket_hash, block_hash) DO NOTHING RETURNING id;`
+	upsertMissRow = insertMissRow0 + `ON CONFLICT (ticket_hash, block_hash) DO UPDATE 
+		SET ticket_hash = $4, block_hash = $2 RETURNING id;`


should really be a do nothing, but the ...DO NOTHING RETURNING Id; bit from the commented line wasn't working properly

chappjc · 2018-01-05T19:12:39Z

+	insertVoteRow = insertVoteRow0 + `RETURNING id;`
+	// insertVoteRowChecked = insertVoteRow0 + `ON CONFLICT (tx_hash, block_hash) DO NOTHING RETURNING id;`
+	upsertVoteRow = insertVoteRow0 + `ON CONFLICT (tx_hash, block_hash) DO UPDATE 
+		SET tx_hash = $3, block_hash = $4 RETURNING id;`


should also be a do nothing, but not for the reasons mentioned in the other comment.

chappjc · 2018-01-05T19:12:48Z

+		$8, $9, $10, $11) `
+	insertTicketRow = insertTicketRow0 + `RETURNING id;`
+	// insertTicketRowChecked = insertTicketRow0 + `ON CONFLICT (tx_hash, block_hash) DO NOTHING RETURNING id;`
+	upsertTicketRow = insertTicketRow0 + `ON CONFLICT (tx_hash, block_hash) DO UPDATE 


should also be a do nothing, but not for the reasons mentioned in the other comment.

chappjc · 2018-01-05T19:13:44Z

-	insertBlockRow         = insertBlockRow0 + `RETURNING id;`
-	insertBlockRowChecked  = insertBlockRow0 + `ON CONFLICT (hash) DO NOTHING RETURNING id;`
+	insertBlockRow = insertBlockRow0 + `RETURNING id;`
+	// insertBlockRowChecked  = insertBlockRow0 + `ON CONFLICT (hash) DO NOTHING RETURNING id;`


the DO NOTHING RETURNING id; wasn't working, had to put the upsert

chappjc · 2018-01-05T19:17:01Z

 		}
 	}

+	stakeDB, err := stakedb.NewStakeDatabase(client, activeChain, "pg_rebuild_stakedb")


ChainDB requires a stake database now to get the winning tickets

chappjc · 2018-01-05T19:17:55Z

+	bestBlock          int64
+	lastBlock          map[chainhash.Hash]uint64
+	addressCounts      *addressCounter
+	stakeDB            *stakedb.StakeDatabase


stakeDB added for winning vote access

chappjc · 2018-01-05T19:20:44Z

 	}
 }

+type TicketTxnIDGetter struct {


@chappjc comment this to indicate it's a thread-safe cache for transaction id to DB row IDs, required by ChainDB for efficient random access to a transactions DB row IDs without having to query the table (by hash or whatever). A cache miss will automatically call RetrieveTicketIDByHash

chappjc · 2018-01-05T19:25:47Z

 		return 0, err
 	}

+	updatesPerDBTx := 500


should go back to 2000 or so. I didn't have a good reason for this change.

chappjc · 2018-01-05T19:28:57Z


+		var winners []string
+		//prevBlockHash := block.MsgBlock().Header.PrevBlock
+		for {


A little wait loop to facilitate the simultaneous sync of sqlite (which manages the shared stakedb) and postgresql.

chappjc · 2018-01-11T23:14:01Z

Last commit (43ecdc2) adds ticket pool and ticket spend info to a ticket's transaction page.

For example:

Statuses:

chappjc · 2018-01-11T23:16:04Z

Now there are two visible changes:

showing missed votes table on each block page
showing ticket pool and spend status on each ticket's transaction page

This is a good enough proof of concept for these DB extensions, so I'd like to review and merge this now. Aside from perhaps some docs, this PR is done... and I am done with SQL for a while. 🤢

chappjc · 2018-01-12T07:34:04Z

Running on https://beta.dcrdata.org/
Example links above updated.

raedah · 2018-01-28T02:41:18Z

Here are some suggestions for the labels. Capitalized to match the rest of the top items. Using commas instead of slashes to show multiple steps. Using labels that correspond with expected ticket states in decrediton.

immature / unspent -> Immature
live / unspent -> In Ticket Pool
voted -> Voted
expired / unspent -> Expired, Unrevoked
expired / revoked -> Expired, Revoked
missed / unspent -> Missed, Unrevoked
missed / revoked -> Missed, Revoked

chappjc · 2018-01-28T03:05:15Z

Thanks for the comments. Good ideas, will incorporate!

Add stakeDB to ChainDB, so it can get winning tickets. Modify apitypes.TicketPoolInfo to include height at which the result applies, and winning tickets drawn at that height. dcrpg.SyncChainDB() requests ticket pool info for a specific block from stake DB, and if it is not yet stored in the pool info cache, it briefly sleeps and asks again. The idea is that during startup, the dcrsqlite sync that actually keeps the stake DB updated with ConnectBlock needs to catch up to this height first. Update SQLite stakeinfo table to include winners. Store winners. ChargePoolInfoCache - function for sqlitedb to read N past blocks' winners into stakedb's pool info cache

Disable dup check when batch updating address spend info. Delete duplicate tickets, votes, and misses too. Add check for index UNIQUE (indisunique), to allow fast exit from delete duplicates functions.

SetSpendingForTickets sets the spending columns (spend type, vote/revoke DB ID, and spend height). The meaning of the DB ID depends on the spend type. See TODO before sync. RetieveTicketIDHeightByHash is used to get the inputs to SetSpendingForTickets for one or more ticket hashes following a call to InsertVotes, which only returns the ticket hashes. Add spend_type column in tickets table, and go enum to indicate expired, live, voted, or revoked (-1, 0, 1, 2). TODO: As with the address table updates, the ticket table updates added here need a switch to be deferred to make a full sync tolerable. Create ChainDB.DeleteDuplicates to call all the tables' delete duplicates functions. Batch tickets spending info via UpdateSpendingInfoInAllTickets. Also add SetSpendingForTickets.

…ote. For each vote, both spent ticket height, hash, and DB ID are stored, making the use of SetSpeningForTickets simple. Insert tickets returns new ticket DB IDs and new ticket dbtypes.Tx, which contains the hash. This information will be stored in the cache if online (non-batch) ticket spending info updates for the vote table are enabled. Move SetupTables() into NewChainDB().

The default is to continuously update the ticket spend info during rebuild, unlike the default for address spend info, since it is very quick with the unspent ticket cache. The batch update is actually quite slow. Add the rebuilddb2 option -r/--recoverfromdups, which removes duplicate entries in all tables. Normally only duplicates in vins and vouts table are removed prior to indexing as these are expected when votes invalidate a block. Duplicates in the other tables are not expected.

REBUILD YOUR DBs for this. Add ChainDB.CollectTicketSpendDBInfo() to sort through block transactions and pick out votes and revokes (ticket spends) and return the relevant ticket and spending DB IDs and ticket hashes. These go to SetSpendingForTickets() in storeTxns. Add new function to StakeDatabase, expires(), to get the expiring tickets in the current best node of the stake DB, and whether the expired tickets were revoked. This information is now two new fields in apitypes.TicketPoolInfo. TicketTxnIDGetter.TxnDbID() now has a bool input to indicate if the cache hit should expire after reading it. Add sqlite columns to block summary table: expires, expires_revoked. The latter is a a boolena indicating if the expires were revoked too. Update all store and retrieve queries. Add transactions table query to retrieve all revokes and the first vin_db_id. Add RetrieveAllRevokesDbIDHashHeight, using the added query. Add vins table query to get funding tx by vin DB ID. Update ChainDB.UpdateSpendingInfoInAllTickets() to update ticket rows with revocation spend info in addtion to the existing vote spend info. TODO: bench these queries as it is a bit yucky here. stakesubmission txin index in vote/revoke in variable Bound poolInfo cache size (hack on a hack)

Revert edits with Expires/ExpiresRevoked because of misunderstanding about stake.Node.

Return misses (revoked or unrevoked) for a block from InsertVotes, and use in ChainDB.storeTxns to set ticket pool statuses of unspent (unrevoked) misses. Set poolStatuses for revokes (missed or expired). TODO: expired and unrevoked Unrevoked missed and expired. SetPoolStatusForTicketsByHash, since BestNode.MissedTickets (for expired but not revoked) give hashes and we don't want to waste a SELECT just to do a UPDATE ... SET. Record unrevoked expires. -> t := MissedByBlock() -> ExistsExpiredTicket(t[i]) -> !justRevoked (already recorded) Make -R deindex the tickets table too.

Move TicketSpendType and TicketPoolStatus to dbtypes. Add ChainDB.PoolStatusForTicket, RetrieveTicketStatusByHash, and SelectTicketStatusByHash Add TxInfo.TicketInfo (and type TicketInfo).

Add: (db *StakeDatabase) SetPoolCacheCapacity (c *PoolInfoCache) SetCapacity Use SetPoolCacheCapacity right before sqliteDB.ChargePoolInfoCache, otherwise it may load data only to throw it away as later blocks added exceed cache capacity.

New type TableVersion with the semantics: // The tables are versioned as follows. The major version is the same for all // the tables. A bump of this version is used to signal that all tables should // be dropped and rebuilt. The minor versions may be different, and they are // used to indicate a change requiring a table upgrade, which would be handled // by dcrdata or rebuilddb2. The patch versions may also be different. They // indicate a change of a table's index or constraint, which may require // re-indexing and a duplicate scan/purge. No upgrades are implemented now, only rebuild or reindex. no table version report from setupTables fixup dep lock and rebase oopsie Signed-off-by: Jon Chappelow <chappjc@gmail.com>

… a ticket

dcrpg: ExistsIndex should return (false, nil) rather than an error when no rows returned Add ExistsIndex call to DeleteDuplicateVouts/Txns functions

…onnectBlock docs too

You ***MUST*** drop any PostgreSQL tables, and delete the db files after this. * initial votes and tickets tables sql * Fix transactions table update statements. * InsertTickets and InsertVotes * Track winning tickets. Add stakeDB to ChainDB, so it can get winning tickets. Modify apitypes.TicketPoolInfo to include height at which the result applies, and winning tickets drawn at that height. dcrpg.SyncChainDB() requests ticket pool info for a specific block from stake DB, and if it is not yet stored in the pool info cache, it briefly sleeps and asks again. The idea is that during startup, the dcrsqlite sync that actually keeps the stake DB updated with ConnectBlock needs to catch up to this height first. Update SQLite stakeinfo table to include winners. Store winners. ChargePoolInfoCache - function for sqlitedb to read N past blocks' winners into stakedb's pool info cache * return early if sqlite sync fails * Upsert statements, remove broken ONCONFLICT ... DO NOTHING statements. * Update rebuilddb2, using temp ffl stakedb for winners. Give NewStakeDatabase an optional base file name arg. Fix sqlite queries using winners. * Add votes and tickets index/deindex to pgblockchain batch. * Change rebuilddb2 "-u" option to effectively be default. Remove "-u" and replace with "-a/--addrspends-no-batch", which means the opposite. That is, if you do NOT want the address table spend info updated at the end in its entirety, and instead want it updated DURING sync (much slower inserts!), then use "-a". * Make misses table, and add isSplit, price, and fee to tickets table. Manage both winners and last block winners (validators for last block). Validators (last winners) are used in dcrpg.InsertVotes to identify missed votes. In rebuilddb2, rewind stakedb one extra block to ensure the validators for current block are cached. NewStakeDatabase caches zero-value TPI for genesis block, which specifies no winners, but is a valid value. Fix StoreBlock to avoid referencing non-existent block prior to genesis. * Remove duplicate vins and vouts before unique index, in both rebuilddb2 and dcrpg.SyncChainDB. Use addresses upsert when dupcheck=true. Remove unneeded vouts index on just tx_hash when another existed on both tx_hash, tx_index, and tx_tree. * fix UNIQUE addresses index * slack link * Docs for stake tx processing and db insertion * some select statements for tickets table * in SyncChainDB bail from loop waiting on stakedb if it is not advancing * Add delete dup transactions function, and add index existence checkts. Disable dup check when batch updating address spend info. Delete duplicate tickets, votes, and misses too. Add check for index UNIQUE (indisunique), to allow fast exit from delete duplicates functions. * Create SetSpendingForTickets and RetrieveTicketIDHeightByHash. SetSpendingForTickets sets the spending columns (spend type, vote/revoke DB ID, and spend height). The meaning of the DB ID depends on the spend type. See TODO before sync. RetieveTicketIDHeightByHash is used to get the inputs to SetSpendingForTickets for one or more ticket hashes following a call to InsertVotes, which only returns the ticket hashes. Add spend_type column in tickets table, and go enum to indicate expired, live, voted, or revoked (-1, 0, 1, 2). TODO: As with the address table updates, the ticket table updates added here need a switch to be deferred to make a full sync tolerable. Create ChainDB.DeleteDuplicates to call all the tables' delete duplicates functions. Batch tickets spending info via UpdateSpendingInfoInAllTickets. Also add SetSpendingForTickets. * TicketTxnIDGetter to cache unspent ticket DB IDs by hash, for InsertVote. For each vote, both spent ticket height, hash, and DB ID are stored, making the use of SetSpeningForTickets simple. Insert tickets returns new ticket DB IDs and new ticket dbtypes.Tx, which contains the hash. This information will be stored in the cache if online (non-batch) ticket spending info updates for the vote table are enabled. Move SetupTables() into NewChainDB(). * Option for rebuilddb2 is now -T/--ticketspends-batch. The default is to continuously update the ticket spend info during rebuild, unlike the default for address spend info, since it is very quick with the unspent ticket cache. The batch update is actually quite slow. Add the rebuilddb2 option -r/--recoverfromdups, which removes duplicate entries in all tables. Normally only duplicates in vins and vouts table are removed prior to indexing as these are expected when votes invalidate a block. Duplicates in the other tables are not expected. * Show missed votes on block page (PoC) REBUILD YOUR DBs for this. Add ChainDB.CollectTicketSpendDBInfo() to sort through block transactions and pick out votes and revokes (ticket spends) and return the relevant ticket and spending DB IDs and ticket hashes. These go to SetSpendingForTickets() in storeTxns. Add new function to StakeDatabase, expires(), to get the expiring tickets in the current best node of the stake DB, and whether the expired tickets were revoked. This information is now two new fields in apitypes.TicketPoolInfo. TicketTxnIDGetter.TxnDbID() now has a bool input to indicate if the cache hit should expire after reading it. Add sqlite columns to block summary table: expires, expires_revoked. The latter is a a boolena indicating if the expires were revoked too. Update all store and retrieve queries. Add transactions table query to retrieve all revokes and the first vin_db_id. Add RetrieveAllRevokesDbIDHashHeight, using the added query. Add vins table query to get funding tx by vin DB ID. Update ChainDB.UpdateSpendingInfoInAllTickets() to update ticket rows with revocation spend info in addtion to the existing vote spend info. TODO: bench these queries as it is a bit yucky here. stakesubmission txin index in vote/revoke in variable Bound poolInfo cache size (hack on a hack) * Add profiling options to rebuilddb2 * fix ill-logic in pg sync halt Revert edits with Expires/ExpiresRevoked because of misunderstanding about stake.Node. * Add pool_status column to tickets table, and TicketPoolStatus type. Return misses (revoked or unrevoked) for a block from InsertVotes, and use in ChainDB.storeTxns to set ticket pool statuses of unspent (unrevoked) misses. Set poolStatuses for revokes (missed or expired). TODO: expired and unrevoked Unrevoked missed and expired. SetPoolStatusForTicketsByHash, since BestNode.MissedTickets (for expired but not revoked) give hashes and we don't want to waste a SELECT just to do a UPDATE ... SET. Record unrevoked expires. -> t := MissedByBlock() -> ExistsExpiredTicket(t[i]) -> !justRevoked (already recorded) Make -R deindex the tickets table too. * Fix sqlite block summary INSERT. * slightly less pointless txn upsert * Tx page with ticket spend / pool status. Move TicketSpendType and TicketPoolStatus to dbtypes. Add ChainDB.PoolStatusForTicket, RetrieveTicketStatusByHash, and SelectTicketStatusByHash Add TxInfo.TicketInfo (and type TicketInfo). * lane dep * stakedb: increase capacity as needed Add: (db *StakeDatabase) SetPoolCacheCapacity (c *PoolInfoCache) SetCapacity Use SetPoolCacheCapacity right before sqliteDB.ChargePoolInfoCache, otherwise it may load data only to throw it away as later blocks added exceed cache capacity. * Remove vins and vouts columns from transactoins table (they were null). * Start table version checks, now at 2.0.0 New type TableVersion with the semantics: // The tables are versioned as follows. The major version is the same for all // the tables. A bump of this version is used to signal that all tables should // be dropped and rebuilt. The minor versions may be different, and they are // used to indicate a change requiring a table upgrade, which would be handled // by dcrdata or rebuilddb2. The patch versions may also be different. They // indicate a change of a table's index or constraint, which may require // re-indexing and a duplicate scan/purge. No upgrades are implemented now, only rebuild or reindex. no table version report from setupTables fixup dep lock and rebase oopsie Signed-off-by: Jon Chappelow <chappjc@gmail.com> * explorer: add TicketStatusText to get display text for POOL STATUS of a ticket * rebuilddb2: allow clean exit when advancing stake db * dcrpg: fix DeleteDuplicateVins so it works when index does not exist dcrpg: ExistsIndex should return (false, nil) rather than an error when no rows returned Add ExistsIndex call to DeleteDuplicateVouts/Txns functions * rebuilddb2: stakedb.UseLogger * factor out calcPoolInfo so nodeMtx can remain locked through all of connectBlock docs too

chappjc added this to the 2.0 milestone Dec 15, 2017

chappjc added the work in progress label Dec 18, 2017

chappjc force-pushed the tickets-table branch from 30d3646 to c75db67 Compare December 19, 2017 18:29

chappjc force-pushed the tickets-table branch from a520c18 to 96034f8 Compare January 5, 2018 05:17

chappjc changed the title ~~[WIP] Stake tables: tickets, votes, misses~~ Stake tables: tickets, votes, misses Jan 5, 2018

chappjc added needs review needs testing and removed work in progress labels Jan 5, 2018

chappjc commented Jan 5, 2018

View reviewed changes

chappjc force-pushed the tickets-table branch 2 times, most recently from 37481d4 to 9630d0f Compare January 9, 2018 23:35

chappjc force-pushed the tickets-table branch from 43ecdc2 to 6d91336 Compare January 21, 2018 20:47

chappjc added ready to go and removed needs review needs testing labels Jan 26, 2018

chappjc force-pushed the tickets-table branch from f110374 to a9c7aea Compare January 27, 2018 15:24

chappjc force-pushed the tickets-table branch from 46639ef to d0ef4d9 Compare January 29, 2018 21:56

chappjc mentioned this pull request Jan 30, 2018

front end: user interface changes to help new users #328

Open

chappjc force-pushed the tickets-table branch 2 times, most recently from 227033f to b610f13 Compare February 5, 2018 02:45

chappjc added 4 commits February 9, 2018 11:36

Fix transactions table update statements.

b926864

initial votes and tickets tables sql

8b63a0e

InsertTickets and InsertVotes

6cb9c7c

chappjc added 22 commits February 9, 2018 11:39

Docs for stake tx processing and db insertion

83be4a9

some select statements for tickets table

838e8fe

in SyncChainDB bail from loop waiting on stakedb if it is not advancing

3d685e4

Add delete dup transactions function, and add index existence checkts.

08f2fa7

Disable dup check when batch updating address spend info. Delete duplicate tickets, votes, and misses too. Add check for index UNIQUE (indisunique), to allow fast exit from delete duplicates functions.

Add profiling options to rebuilddb2

d6c9299

fix ill-logic in pg sync halt

c1dbff5

Revert edits with Expires/ExpiresRevoked because of misunderstanding about stake.Node.

Fix sqlite block summary INSERT.

1b0ff6e

slightly less pointless txn upsert

28b1e5c

Tx page with ticket spend / pool status.

aec48ea

Move TicketSpendType and TicketPoolStatus to dbtypes. Add ChainDB.PoolStatusForTicket, RetrieveTicketStatusByHash, and SelectTicketStatusByHash Add TxInfo.TicketInfo (and type TicketInfo).

lane dep

f9c390c

stakedb: increase capacity as needed

1c3846f

Add: (db *StakeDatabase) SetPoolCacheCapacity (c *PoolInfoCache) SetCapacity Use SetPoolCacheCapacity right before sqliteDB.ChargePoolInfoCache, otherwise it may load data only to throw it away as later blocks added exceed cache capacity.

Remove vins and vouts columns from transactoins table (they were null).

6749af0

explorer: add TicketStatusText to get display text for POOL STATUS of…

a16a366

… a ticket

rebuilddb2: allow clean exit when advancing stake db

7ac12d4

dcrpg: fix DeleteDuplicateVins so it works when index does not exist

dcee4ca

dcrpg: ExistsIndex should return (false, nil) rather than an error when no rows returned Add ExistsIndex call to DeleteDuplicateVouts/Txns functions

rebuilddb2: stakedb.UseLogger

28a23a7

chappjc force-pushed the tickets-table branch from b610f13 to 28a23a7 Compare February 9, 2018 22:08

factor out calcPoolInfo so nodeMtx can remain locked through all of c…

ae55132

…onnectBlock docs too

chappjc merged commit 600b724 into master Feb 11, 2018

chappjc mentioned this pull request Feb 11, 2018

Track winning tickets #27

Closed

chappjc deleted the tickets-table branch February 27, 2018 05:12

Conversation

chappjc commented Dec 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chappjc commented Jan 5, 2018

Uh oh!

chappjc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chappjc commented Jan 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chappjc commented Jan 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chappjc commented Jan 12, 2018

Uh oh!

raedah commented Jan 28, 2018

Uh oh!

chappjc commented Jan 28, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chappjc commented Dec 15, 2017 •

edited

Loading

chappjc commented Jan 11, 2018 •

edited

Loading

chappjc commented Jan 11, 2018 •

edited

Loading