Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Boost release/v2 (filecoin-project#1520)
* feat: add local index directory Internally this is still refered to as the piece directory Co-authored-by: dirkmc <dirkmdev@gmail.com> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * refactor: merge from main into lid branch (filecoin-project#1339) * merge: main to lid (filecoin-project#1370) * add free check (filecoin-project#1315) * chore: bump version to 1.6.1 (filecoin-project#1317) * fix legacy deal verified status (filecoin-project#1324) * fix: update go-unixfsnode enough to make sure unixfs-preload is available (filecoin-project#1323) * release v1.6.2-rc1 (filecoin-project#1328) * use full path (filecoin-project#1330) * fix bug (filecoin-project#1332) * use forks of graphsync, go-data-transfer and go-fil-markets (filecoin-project#1333) * refactor: use forks of graphsync, go-data-transfer and go-fil-markets * refactor: convert from data transfer v1 to v2 voucher type * fix: index provider validation voucher type * fix: pass index provider engine link system through to graphsync's transport configurer * feat: use tagged version of boost-gfm * fix: retrieval client imports * feat: tagged version of lotus * feat: require go 1.19 * lint: fix lint errors * fix: itests * fix: cbor-gen, docsgen * fix: update CI lint version * fix: lint * fix: docgen * fix: go mod tidy * fix: protocol proxy TestOutboundForwarding * fix: docsgen * fix: update filecoin-ffi submodule * fix: prometheus duplicate register panic * fix: cleanup imports * fix: legs voucher processing * chore: release v1.6.2-rc2 (filecoin-project#1340) * release v1.6.2-rc2 * fix test * fix: flaky TestLibp2pCarServerNewTransferCancelsPreviousTransfer (filecoin-project#1350) * fix: flaky TestDealCompletionOnProcessResumption (filecoin-project#1351) * fix: occasional panic on shutdown (filecoin-project#1353) * feat: query UI (filecoin-project#1352) * log insert * fix display error * refactor code * shorten status strings * remove comment * apply suggestion * feat: add download block link to inspect page (filecoin-project#1312) * fix(devnet): update golang and lotus default versions (filecoin-project#1354) * fix(devnet): bump golang to 1.19 * chore(devnet): bump lotus default version * chore(devnet): remove unused stable env * booster-http: implement IPFS HTTP gateway (filecoin-project#1225) * feat: implement http api gateway * feat: use go-libipfs lib (instead of copying to extern) * feat: bump booster-bitswap info minor version * feat: http gateway metrics * fix: TestHttpInfo * feat: by default only serve blocks and CARs, with option to serve original files (jpg, mov etc) * fix: correct link for download root block (filecoin-project#1355) * feat: option to cleanup data for offline deals after add piece (filecoin-project#1341) * chore: add support for multiple node.js versions in makefile (filecoin-project#1356) * chore: release v.1.7.0-rc1 (filecoin-project#1357) * release v.1.7.0-rc1 * fix version * fix: dagstore initialize-all parameter (filecoin-project#1363) * fix: show verifying commp state for offline deals (filecoin-project#1364) * fix: boost run missing staging-area dir (filecoin-project#1368) * merge(wip): main to lid TODO: remoteblockstore needs to handle nil metrics * fix: flaky TestNewHttpServer (filecoin-project#1372) * feat: group agent version by binary name (filecoin-project#1369) * fix: wrap stats in nil checks for now we should probably revisit how stats are handled now that we have all 3 transports being tracked * test(fix): incorrect test urls --------- Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: dirkmc <dirkmdev@gmail.com> * fix: make devnet work for lid (filecoin-project#1375) * feat: support full addr config in boostd-data * chore: fix linting for boostd-data * feat: use addr instead of port for lid chore: update devnet to work with lid setup * chore: resolve feedback on lint changes * feat: fail deal if start epoch passed (filecoin-project#1319) * fail deal if start epoch passed * add suggestion * test: add deal expiry on startup test --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * fix: makefile * fix: db migration ordering * fix: correct rootcid formatting * fix: prevent accidental removal of valid sector index announcements fix: add cache tests and dont announce cache state fix: add unique index to sector state db fix: sealed and unsealed sector state conflict fix: ensure index provider wrapper starts after db migration has completed * chore: go mod tidy * fix: download block (filecoin-project#1440) * LID yugabyte db impl (filecoin-project#1391) * feat: yugabyte db impl * feat: run yugabyte tests against a dockerized yugabyte * fix: use out own yugabyte docker image * fix: use yugabyte 2.17.2.0 docker image * feat: piece doctor yugabyte impl * fix: go mod tidy * refactor: remove SetCarSize as its not longer being used * refactor: remove functionality to mark index as errored (not being used) * feat: implement delete commands * refactor: consolidate test params * feat: add lid yugabyte config * fix: port map yugabyte postgres to standard port * Fix yugabyte CI (filecoin-project#1433) * fix: yugabyte tests in CI * docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost * add tag * test lid * make gen * fixup * move couchbase settings under build tag --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * script to migrate from couchbase to yugabyte (filecoin-project#1445) * feat: script to migrate from couchbase to yugabyte * fix: reduce batch size for yugabyte inserts * Change service GetIndex / AddIndex to return channel instead of array (filecoin-project#1444) * feat: yugabyte db impl * feat: run yugabyte tests against a dockerized yugabyte * fix: use out own yugabyte docker image * fix: use yugabyte 2.17.2.0 docker image * feat: piece doctor yugabyte impl * fix: go mod tidy * refactor: remove SetCarSize as its not longer being used * refactor: remove functionality to mark index as errored (not being used) * feat: implement delete commands * refactor: consolidate test params * feat: add lid yugabyte config * fix: port map yugabyte postgres to standard port * Fix yugabyte CI (filecoin-project#1433) * fix: yugabyte tests in CI * docker-compose.yml ; Dockerfile.test ; connect to `yugabyte` and not localhost * add tag * test lid * make gen * fixup * move couchbase settings under build tag --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * wip: service GetIndex returns channel of records instead of array * feat: return channel from AddIndex and GetIndex --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * local index directory: recover tool (filecoin-project#1410) * initial disaster recovery tool for LID * wip * do not block on individual error * instantiate lid * report * catch signal * fixup * comment out sector already in progress * fixup * start containers with init: true * record that we dont have an unsealed copy * match deals with boost sqlite db and piece store * fixup * fixup * use logger * fixup * disable stacktrace * fixup * extract piece store away from disaster recovery struct * add more sanity checks * compare IsUnsealed vs storage find * improve safeIsUnseal * fixup * better logs * expand repodir * calc properly next offset * fixup * add sector id to logs * incr offset * break after finding expired deal * more logs * fewer logs * better logs * better error * refactor * refactor minerApi * better logs * add time around add index * pd.Start * LID benchmarking tool (filecoin-project#1276) * feat: LID benchmarking tool * fix: bench thread safety * refactor: structured logging * refactor: postgres bulk insert * lid bench: Add foundationdb impl * lid fdb: Fix Tx sizing, parallel chunk puts * lid fdb: More efficient sample generation * feat: array of piece count / blocks per piece (filecoin-project#1314) * lid bench: print add rate * lid bench: Add retry to postgres put (filecoin-project#1316) * lid bench: Make cassandra put much more robust (filecoin-project#1318) * instrumentation for bench tool (filecoin-project#1337) * instrument postgres * more instrumentation * check for err getoffsetsize * emit metrics every 10sec * ignore errors * add postgres-drop * use directly tables * fix: go mod tidy * use INSERT INTO instead of tmp tables * try to catch sig * remove transaction commit * fixup * add postgres-init * fixuop * split create and init * fixup * remove if not exist --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * feat: batch insert queries for postgres * feat: add flag to insert into postgres using tmp table * refactor: merge changes from nonsense/lid-bench * refactor: just use one database (dont create bench database) * refactor: remove unused params * refactor: command structure * fix: cassandra - dont use batch insert for PayloadToPieces * fix: create tables CQL * fix: increase payload to pieces insert parallelism * fix: use simple replication strategy * feat: use yugabyte cassandra driver * fix: remove bench binary * update metrics endpoint * fix random generated piece cid * fixup * fix: cassandra bitswap benchmark * remove foundationdb --------- Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * fix: failing tests due to bad merge * fix: flaky TestMultipleDealsConcurrent * more logs * piece doctor and sector state manager refactor (filecoin-project#1463) * fix timer.Reset and improve logs * revert randomization * piece doc: handle errors * adjust piece check * refactor unsealsectormanager * refactor piece doctor * add random ports * ignore tests * add version to boostd-data * fix ctx in Start * fix: add reader mock to fix tests * fix: pass new piece directory to provider on test restart * fix synchronisation * note that panics are not propagated in tests * carv1 panics piece directory * print panics * fix: use reader that supports Seek in piece reader mock * fix: reset mock car reader on each invocation * fix: TestOfflineDealDataCleanup * add check for nil cancel func * bump min check period for LevelDB to 5 minutes * check if sector state mgr is initialised * debug line for unflagging * commenting out TestMultipleDealsConcurrent -- flaky test -- works locally * add SectorStateUpdates pubsub * add close for pubsub * add mock sectorstatemgr * add wrapper tests * fixup * cleanup * cleanup * better names * t.Skip for test * remove TODO above println for panic * add unit tests for refreshState * rename tests * more cases * more tests * update description * better comment * better names and comments --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * Merge from main to lid branch (filecoin-project#1483) * fix statx output string (filecoin-project#1451) * fix: flaky TestMultipleDealsConcurrent (filecoin-project#1458) * Add option to serve index provider ads over http (filecoin-project#1452) * feat: option to serve index provider ads over http * fix: config naming, hostname parsing * fix: update docsgen * fix: log announce address * feat: add config for indexer direct announce urls * refactor: always announce over pubsub * fix: docsgen * test: add test case for empty announce address hostname * Add `boostd index announce-latest` command (filecoin-project#1456) * feat: boostd index announce-latest * feat: add announce-latest-http command * fix: default direct announce url * feat: update to index-provider v0.11.2 * Signal to index provider to skip announcements (filecoin-project#1457) * fix: signal to index provider to skip announcements * fix: ensure multihash lister skip error is of type ipld.ErrNotExists --------- Co-authored-by: LexLuthr <lexluthr@protocol.ai> * release v1.7.3-rc2 (filecoin-project#1460) * fix: improve stalled retrieval cancellation (filecoin-project#1449) * refactor stalled retrieval cancel * add ctx with timeout * implement suggestions * update err wrapping * fix: set short cancel timeout for unpaid retrievals only --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * feat: enable listen address for booster-http (filecoin-project#1461) * enable listen address * modify tests * fix nil ptr (filecoin-project#1470) * fix: incorrect check when import offline deal data using proposal CID (filecoin-project#1473) * fix incorrect early check * update error msg * fix(server): properly cancel graphsync requests (filecoin-project#1475) * set UI default listen address to localhost (filecoin-project#1476) * feat: display msg params in the mpool UI (filecoin-project#1471) * show msg params * fix: mpool nil pointer * fix width --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * Reset read deadline after reading deal proposal message (filecoin-project#1479) * fix: reset read deadline after reading deal proposal message * fix: increase client request deadline * feat: Show elapsed epoch and PSD wait epochs in UI (filecoin-project#1480) * show epochs * fix devnet UI, use BlockdDelaySecs * fix lint err * Update gql/resolver.go Co-authored-by: dirkmc <dirkmdev@gmail.com> --------- Co-authored-by: dirkmc <dirkmdev@gmail.com> * release v1.7.3-rc3 (filecoin-project#1481) --------- Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: LexLuthr <lexluthr@protocol.ai> Co-authored-by: Hannah Howard <hannah@hannahhoward.net> * update local index directory ui (filecoin-project#1477) * feat: update local index directory ui * comment out wrench as docker doesnt build * rearrange menu * refactor: remove sectors list --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * feat: surface indexing errors (filecoin-project#1490) * feat: log panic (instead of just printing to stdout) (filecoin-project#1491) * split flagged pieces into unsealed/sealed tables (filecoin-project#1493) * refactor: remove couchbase tests (filecoin-project#1496) * refactor: remove piece directory couchbase tests (filecoin-project#1497) * GraphQL resolvers for LID (filecoin-project#1494) * wip * rename * sectorUnsealedCopies and SectorProvingState * fix: piece directory tests (filecoin-project#1498) * log line for only sealed sectors * more logs * feat: flagged pieces (filecoin-project#1501) * check that sector has deals for unsealed sectors (filecoin-project#1502) * check that sector has deals for unsealed sectors * simplify * rename heading * piece doctor to ignore expired/slashed deals (filecoin-project#1503) * ignore expired/slashed deals * fix mocks * add timer for checkPiece * move ChainHead away from checkPiece * add nil check for fullnodeApi * add debug line * fix pagination * LID landing page: add stats around Flagged and non-Flagged pieces (filecoin-project#1508) * wip * fixup * add debug line * fixup * feat: split flagged pieces page into flagged / flagged because unsealed (filecoin-project#1509) * fix: display of no flagged pieces (filecoin-project#1511) * disable dummy panels - block stats; deal data (filecoin-project#1510) * fix unsealed field in flagged piece (filecoin-project#1515) * update ffi * fix main merge issue * fix go mod * Add info boxes on LID UI page (filecoin-project#1516) * feat: add info boxes on LID UI page * Update react/src/LID.js Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * Update react/src/LID.js Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> --------- Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * feat: replace migrate couchbase command with migrate yugavbyte (filecoin-project#1518) * remove redundant makefile (filecoin-project#1519) * remove redundant makefile * add migrate-lid to Makefile * update gitignore * move booster-bitswap and booster-http to make and make install * fix: inspect page - dont try to fetch root cid (filecoin-project#1525) * feat: add send epoch, time, elapsed epoch and elapsed time for each message in mpool to UI (filecoin-project#1523) * add message epoch/time details * implement suggestion * use moment lib * fix alerting bug * update polling interval * add logs * fix devnet: use ws instead of http to connect to boostd-data * feat: make legacy deals optional (filecoin-project#1524) * make legacy deals optional * fix gen * modify itests, create new * handle legacy stream explicitly * separate out the protocols * fix lint error * enable itest in CI * fix ci * apply suggestions * fix error after conflict resolution * refactor: simplify legacy deal response code --------- Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> * refactor: remove couchbase implementation (filecoin-project#1535) * Update lotus and boxo versions (filecoin-project#1466) (filecoin-project#1537) * Update to use packages in go-libipni * feat: update lotus version * update boxo (filecoin-project#1492) * feat: update boxo * refactor: depend on repo:Jorropo/lotus branch:boxo2 * chore: temporarily update go-fil-markets with replace directive * feat: switch itests framework ExtractFileFromCAR to use non-global IPLD registry * feat: switch booster-bitswap client fetch to use the go-ipld-prime globals via go-ipld-legacy * go fmt * chore: update dependencies and migrate to boxo * fix: update boost-gfm * fix: stop itests framework from prematurely setting listenaddrs via go-libp2p defaults that conflict with lotus * fix: docs gen * chore(deps): update deps for boxo v0.10.0 * chore(deps): update boost-gfm * fix(booster-http): update for boxo v0.10.0 * chore(deps): update to remove kubo dependency * fix(gen): update docs gen * feat: update boost-gfm to v1.26.6 * chore(deps): update lotus to master --------- --------- Co-authored-by: gammazero <gammazero@users.noreply.github.com> Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com> Co-authored-by: hannahhoward <hannah@hannahhoward.net> * feat: update boost-gfm to v1.26.7 (filecoin-project#1538) * fix: piece doctor tests (filecoin-project#1540) * refactor: build indexes for legacy deals (filecoin-project#1539) * feat: http index announcements (filecoin-project#1418) * feat(indexprovider): announce http transport refactor: isolate extended provider logic feat: announce http indexes refactor(indexprovider): use metadata.Default fix(wrapper): fix compile error * fix http ep signing bug * update comment --------- Co-authored-by: LexLuthr <lexluthr@protocol.ai> * feat: check unseal status of piece through both apis (filecoin-project#1548) * fix: metrics and Grafana (filecoin-project#1546) * fix grafna, metrics * remove dagstore from name * fix: add missing PieceDeal (PieceCid) index (filecoin-project#1551) * fix: iterate all deals to index piece (filecoin-project#1549) * fix: iterate all deals to index piece * add test, use multierror * add and update comments * refactor: separate yugabyte / leveldb tests for easier local testing (filecoin-project#1553) * feat: refactor mpool page in UI (filecoin-project#1530) * modify GQL * fix count type * fix locks * fix js * migrate config to v5 (filecoin-project#1560) * migrate config to v5 * change default version * chore: release v2.0.0-rc1 (filecoin-project#1561) * Upgrade to index-provider v0.13.4 (filecoin-project#1559) Upgrade to the latest index-provider library. * feat: add IPNI itest (filecoin-project#1563) * ipni itest * refactor test * add to circleCI * add indexer topic * Print protocol IDs exposed by f.Boost * generate topic name dynamically --------- Co-authored-by: Masih H. Derkani <m@derkani.org> * IPNI UX (filecoin-project#1562) * feat: IPNI UX * Update react/src/Ipni.js Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> * feat: server side config --------- Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> * add peerID to multiaddresses returned by boost provider retrieval-transports query (filecoin-project#1568) * add peerID to multiaddress * fix error typos * undo http-libp2p on vanilla http * fix: rename indexer config to index provider config (filecoin-project#1573) * fix retrieve output path (filecoin-project#1574) * update README.md (filecoin-project#1576) * update description (filecoin-project#1580) * tempo: increase max trace size (filecoin-project#1579) * use go-block-format --------- Co-authored-by: Jacob Heun <jacob.heun@gmail.com> Co-authored-by: dirkmc <dirkmdev@gmail.com> Co-authored-by: Jacob Heun <jacobheun@gmail.com> Co-authored-by: LexLuthr <88259624+LexLuthr@users.noreply.github.com> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: LexLuthr <lexluthr@protocol.ai> Co-authored-by: Hannah Howard <hannah@hannahhoward.net> Co-authored-by: gammazero <gammazero@users.noreply.github.com> Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com> Co-authored-by: Masih H. Derkani <m@derkani.org>
- Loading branch information
13 people
authored
Jul 20, 2023
1 parent
caea26a
commit 08ec616