Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: rename market to droplet #6005

Merged
merged 4 commits into from
Jun 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/common_build_upload.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,10 @@ jobs:
run: |
go clean --modcache && make
mkdir ./release
if [[ "${{steps.vars.outputs.repo_name}}" == "venus-market" ]]; then
mv ./market-client ./venus-market ./release
if [[ "${{steps.vars.outputs.repo_name}}" == "droplet" ]]; then
mv ./droplet-client ./droplet ./release
fi
if [[ "${{steps.vars.outputs.repo_name}}" != "venus-market" ]]; then
if [[ "${{steps.vars.outputs.repo_name}}" != "droplet" ]]; then
mv ./${{inputs.bin_name}} ./release
fi

Expand Down Expand Up @@ -262,10 +262,10 @@ jobs:
run: |
make
mkdir ./release
if [[ "${{steps.vars.outputs.repo_name}}" == "venus-market" ]]; then
mv ./market-client ./venus-market ./release
if [[ "${{steps.vars.outputs.repo_name}}" == "droplet" ]]; then
mv ./droplet-client ./droplet ./release
fi
if [[ "${{steps.vars.outputs.repo_name}}" != "venus-market" ]]; then
if [[ "${{steps.vars.outputs.repo_name}}" != "droplet" ]]; then
mv ./${{inputs.bin_name}} ./release
fi

Expand Down
4 changes: 2 additions & 2 deletions app/node/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ import (
"fmt"
"time"

"github.com/filecoin-project/venus-auth/core"
"github.com/filecoin-project/venus-auth/jwtclient"
"github.com/filecoin-project/venus/app/submodule/dagservice"
"github.com/filecoin-project/venus/app/submodule/eth"
"github.com/filecoin-project/venus/app/submodule/network"
"github.com/ipfs-force-community/sophon-auth/core"
"github.com/ipfs-force-community/sophon-auth/jwtclient"

logging "github.com/ipfs/go-log"
"github.com/libp2p/go-libp2p"
Expand Down
2 changes: 1 addition & 1 deletion app/node/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ import (
"github.com/awnumar/memguard"
"github.com/etherlabsio/healthcheck/v2"
"github.com/filecoin-project/go-jsonrpc"
"github.com/filecoin-project/venus-auth/jwtclient"
"github.com/filecoin-project/venus/app/submodule/blockstore"
chain2 "github.com/filecoin-project/venus/app/submodule/chain"
"github.com/filecoin-project/venus/app/submodule/common"
Expand All @@ -35,6 +34,7 @@ import (
_ "github.com/filecoin-project/venus/pkg/crypto/secp" // enable secp signatures
"github.com/filecoin-project/venus/pkg/metrics"
"github.com/filecoin-project/venus/pkg/repo"
"github.com/ipfs-force-community/sophon-auth/jwtclient"
cmds "github.com/ipfs/go-ipfs-cmds"
cmdhttp "github.com/ipfs/go-ipfs-cmds/http"
logging "github.com/ipfs/go-log/v2"
Expand Down
48 changes: 25 additions & 23 deletions documentation/en/venus-market-design-roadmap.md
Original file line number Diff line number Diff line change
@@ -1,59 +1,59 @@
## venus-market module design & roadmap
## droplet module design & roadmap

by Venus team

Sep 2021

## Background

As the rebranding of filecoin terminology spearheaded by [FIP0018](https://github.com/filecoin-project/FIPs/blob/master/_fips/fip-0018.md) settled, consensus has been reached across communities (developers, providers, ecosystem partners and etc) to push for taking on more storage deals to improve the public perception on the fact that most of the network storages are still commited capacities (CCs). Given the above sentiment, design and implemenation of venus-market module has been put into the spot light. A clear long-term roadmap is due for Venus community to dissus and iterate on, also as a means for better communications with filecoin eocsystem in general.
As the rebranding of filecoin terminology spearheaded by [FIP0018](https://github.com/filecoin-project/FIPs/blob/master/_fips/fip-0018.md) settled, consensus has been reached across communities (developers, providers, ecosystem partners and etc) to push for taking on more storage deals to improve the public perception on the fact that most of the network storages are still commited capacities (CCs). Given the above sentiment, design and implementation of droplet module has been put into the spot light. A clear long-term roadmap is due for Venus community to dissus and iterate on, also as a means for better communications with filecoin eocsystem in general.

While Lotus is leading the way of implementing a full-fledged market module according to the [spec](https://spec.filecoin.io/#section-systems.filecoin_markets), Venus has been making much efforts to catch up and closing the gap in regard to [markets](https://github.com/filecoin-project/venus/discussions/4532). Right now, Lotus supports a 1-to-1 (client-to-provider) storage and retrieval model where burdens of discoveries are mostly on storage clients and match making services like Estuary. Negotiation is a fairly mannual process and does not support much flexibility. As venus-team is picking up the reminiscences of the [Filecoin Component Architecture](https://docs.google.com/document/d/1ukPD8j6plLEbbzUjxfo7eCauIrOeC_tqxqYK_ls9xbc/edit#), emergent ways of how market could facilitate the dynamics between storage providers and storage clients are constatntly being intergrated into the long-term vision of Venus filecoin.

## Goals

Current roadmap for venus-market are loosely broken into the following phases.
Current roadmap for droplet are loosely broken into the following phases.

### Phase 1: peer-to-peer model (short-term)

For phase 1, venus-market will deliver a complete deal making experience as what lotus offers. This includes compatibility with lotus client where one can make deal with venus-market using lotus client, retrieve deal/data in the same way as lotus retrieves its data, setup storage ask and etc.
For phase 1, droplet will deliver a complete deal making experience as what lotus offers. This includes compatibility with lotus client where one can make deal with droplet using lotus client, retrieve deal/data in the same way as lotus retrieves its data, setup storage ask and etc.

![image-20210910170740850](https://i.loli.net/2021/09/10/seIgEWBiko6AKc2.png)

- Implementation of the one-to-one model of lotus market like module and fully interoperable with lotus implementation, which means compatibility with lotus client and more
- venus-market deployed as independent module, like venus-sealer and venus-wallet
- droplet deployed as independent module, like sophon-cluster and venus-wallet
- Implementation of a reliable market module that runs a seperate process from the main storage process
- A clear module boundary that allows interoperability and user customizations
- Flexibilities of market module to interact with existing venus infrastructures using RPC APIs
- Supports for mainnet, calibration and Nerpa
- Lightweight client: compatibility with Lotus and support for venus-market unique features including client running seperately as a process and remove dependencies for node; great for bootstraping tests on deal making process
- Lightweight client: compatibility with Lotus and support for droplet unique features including client running seperately as a process and remove dependencies for node; great for bootstraping tests on deal making process

### Phase 2: platform model

For phase 2, venus-market is taking the following approach.
For phase 2, droplet is taking the following approach.

**platform-to-peer**: venus-market as deal making backend for middle-man services like Estuary connecting client and provider. As deal market matures, instead of ineffectively advertising one's storage system in #fil-deal-market, storage middleman services like Estuary and Filswan are taking up the roles for distributing datacap more effectively to storage providers looking for deals. Given venus' unique architecture where multiple providers are sharing same infrastructure (chain services), venus-market is in a good position to provide before mentioned deal making backend for a storage middle-man service.
**platform-to-peer**: droplet as deal making backend for middle-man services like Estuary connecting client and provider. As deal market matures, instead of ineffectively advertising one's storage system in #fil-deal-market, storage middleman services like Estuary and Filswan are taking up the roles for distributing datacap more effectively to storage providers looking for deals. Given venus' unique architecture where multiple providers are sharing same infrastructure (chain services), droplet is in a good position to provide before mentioned deal making backend for a storage middle-man service.

**platform-to-platform**: venus-market as storage backend for a storage integrator (a storage provider who offers different kinds of storage products to its end user, for example, filecoin, S3, tape and etc).
**platform-to-platform**: droplet as storage backend for a storage integrator (a storage provider who offers different kinds of storage products to its end user, for example, filecoin, S3, tape and etc).

![image-20210910160837732](https://i.loli.net/2021/09/10/sRY5u6Bw9aj713H.png)

- Taking advantages of Venus' distributed architectural nature, a gateway service backend built on top of current infrastructure
- Compact API: seperation of node and venus-market data enabling local storage of some of the deal related meta data
- Compact API: seperation of node and droplet data enabling local storage of some of the deal related meta data
- Data transfer support for different protocols in addition to `Graphsync` [*](https://docs.google.com/document/d/1XWcTp2MEOVtKLpcpiFeeDvc_gTwQ0Bc6yABCTzDmeP0/edit#heading=h.1oxn84bcd1n1)
- Meta data stored locally in HA database like mySQL by venus-market
- venus-market as deal gateway for storage providers using venus chain services (venus shared modules)
- Meta data stored locally in HA database like mySQL by droplet
- droplet as deal gateway for storage providers using venus chain services (venus shared modules)
- Deal match making: multiple copies for store and faster retrieval

### Phase 3: Decentrialized market (Dp2p) model (long-term vision)

For phase 3, venus-market will look into ways to automate deal flow between client and provider using a peer-to-peer approach, giving up its role as a gateway in phase 2. Additionally, venus pool can be positioned as a retrieval node which is fully aware of deal meta that chain services helped to record.
For phase 3, droplet will look into ways to automate deal flow between client and provider using a peer-to-peer approach, giving up its role as a gateway in phase 2. Additionally, venus pool can be positioned as a retrieval node which is fully aware of deal meta that chain services helped to record.

![image-20210910171104881](https://i.loli.net/2021/09/10/VE6BLpaARrMck9x.png)

- Goals for phase 3 is not as clear cut; require more iterations as filecoin develops smart contracts and others
- auto-match deal market: a service to provide algorithmically (as opposed to manually verifying data using current fil-plus framework) verified data storing/retrieval from peer to peer
- venus-market as gateway for IPFS: options for paid IPFS node
- droplet as gateway for IPFS: options for paid IPFS node
- In time, a platform model as an easy and quick way for matchmaking might fall out of favour and a faster layer 2 protocol could be built on top of Venus to make true p2p data storage with standardized storage services governed by blockchain
- New econ market on layer 2

Expand All @@ -70,7 +70,7 @@ Design draws inspirations from the original [filecoin component document](https:

> In a multi-process architecture, the storage component would form the miner operator’s entry point for all mining and market operations (but not basic node operations) pertaining to a single storage miner actor. It depends on a node to mediate blockchain interactions. **The storage component drives these interactions**. If viewed as a system of services, the storage component is the consumer of a service provided by a node. Thus, the **storage component will depend on an RPC API be provided by a node**. This API is likely to include streaming operations in order to provide continually changing blockchain state to the component. The [mimblewimble/grin project](https://github.com/mimblewimble) is another example of this multi-process node/miner architecture.

Similar to what is described for storage component above, venus-market will be dependent on a RPC API provided by a node ie. chain services, venus shared modules. Blockchain interactions will be handled by venus chain services which can also be extended to handle authentications among others.
Similar to what is described for storage component above, droplet will be dependent on a RPC API provided by a node ie. chain services, venus shared modules. Blockchain interactions will be handled by venus chain services which can also be extended to handle authentications among others.

### Deal flow

Expand All @@ -79,31 +79,33 @@ In phase 1, louts market deal flow will be mirrored in Venus. Maintainace of the
In phase 2, proposing and accepting a deal will work as following.

- [Provider] Add storage ask bidding policy along with other deal acceptance policy
- [Client] Query asks from venus-market with filters like geo locations, redundancy, deal lifespan and etc
- [venus-market] Aggregate requirements from both providers and clients, matchmaking on demand
- [venus-market] Provider(s) and client go through rounds of real-time biding to match-make
- [Client] Query asks from droplet with filters like geo locations, redundancy, deal lifespan and etc
- [droplet] Aggregate requirements from both providers and clients, matchmaking on demand
- [droplet] Provider(s) and client go through rounds of real-time biding to match-make
- Once matched, provider proceeds to store data as in the one-to-one model

### Meta data

Platform model implementation of venus-market may store metadata on the deals it distributes to providers under its wings. Like Airbnb, it may include metrics that a repututation system of both client and provider can be built upon. Metrics like storage success rate, retrieval success rate, fast retrieval enabled? and etc.
Platform model implementation of droplet may store metadata on the deals it distributes to providers under its wings. Like Airbnb, it may include metrics that a repututation system of both client and provider can be built upon. Metrics like storage success rate, retrieval success rate, fast retrieval enabled? and etc.

### Dependencies

1. `venus` module to provide node services
2. `venus-messager` module to provide data services
3. `venus-gateway` to provide signature services
4. `venus-sealer` to provide sealing and data lookup services
2. `sophon-messager` module to provide data services
3. `sophon-gateway` to provide signature services
4. `sophon-cluster` to provide sealing and data lookup services
5. `go-fil-market` compatible with lotus one-to-one model (For compatbility with lotus only)
6. piece data from external deals
7. datastore using HA databases for deal meta data

*Note that `go-fil-market` included as dependencies is sololy for the use of compatibility with lotus. venus-market will be bundling other unique features along with compatibility with lotus.*
*Note that `go-fil-market` included as dependencies is sololy for the use of compatibility with lotus. droplet will be bundling other unique features along with compatibility with lotus.*

![模块图](https://i.loli.net/2021/09/08/7UxfVujcNPmszyR.jpg)

### Interactions

> `venus-market` renamed to `droplet`

<img src="https://i.loli.net/2021/09/08/cK2ZHE4DWmYuvLS.png" width="400" >

## Risks
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ require (
github.com/filecoin-project/specs-actors/v8 v8.0.1
github.com/filecoin-project/specs-storage v0.4.1
github.com/filecoin-project/test-vectors/schema v0.0.5
github.com/filecoin-project/venus-auth v1.11.1-0.20230511013901-7829b3effbcd
github.com/fxamacker/cbor/v2 v2.4.0
github.com/go-errors/errors v1.0.1
github.com/go-kit/kit v0.12.0
Expand All @@ -58,6 +57,7 @@ require (
github.com/hashicorp/golang-lru/v2 v2.0.2
github.com/howeyc/gopass v0.0.0-20190910152052-7cb4b85ec19c
github.com/ipfs-force-community/metrics v1.0.1-0.20211022060227-11142a08b729
github.com/ipfs-force-community/sophon-auth v1.11.1-0.20230607031351-bc57c2b78f44
github.com/ipfs/go-blockservice v0.4.0
github.com/ipfs/go-cid v0.3.2
github.com/ipfs/go-datastore v0.6.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -387,8 +387,6 @@ github.com/filecoin-project/specs-storage v0.4.1/go.mod h1:Z2eK6uMwAOSLjek6+sy0j
github.com/filecoin-project/storetheindex v0.4.30-0.20221114113647-683091f8e893 h1:6GCuzxLVHBzlz7y+FkbHh6n0UyoEGWqDwJKQPJoz7bE=
github.com/filecoin-project/test-vectors/schema v0.0.5 h1:w3zHQhzM4pYxJDl21avXjOKBLF8egrvwUwjpT8TquDg=
github.com/filecoin-project/test-vectors/schema v0.0.5/go.mod h1:iQ9QXLpYWL3m7warwvK1JC/pTri8mnfEmKygNDqqY6E=
github.com/filecoin-project/venus-auth v1.11.1-0.20230511013901-7829b3effbcd h1:l02UJuEbSUIBi3NC/+17K2gBbAzsUNQg42rNCXskOBc=
github.com/filecoin-project/venus-auth v1.11.1-0.20230511013901-7829b3effbcd/go.mod h1:PoTmfEn5lljjAQThBzX0+friJYGgi7Z3VLLujkOkCT4=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/flynn/noise v1.0.0 h1:DlTHqmzmvcEiKj+4RYo/imoswx/4r6iBlCMfVtrMXpQ=
github.com/flynn/noise v1.0.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag=
Expand Down Expand Up @@ -686,6 +684,8 @@ github.com/ipfs-force-community/go-jsonrpc v0.1.7 h1:e0ZTapGFhDY54j0QpRYN54Q3FHa
github.com/ipfs-force-community/go-jsonrpc v0.1.7/go.mod h1:jBSvPTl8V1N7gSTuCR4bis8wnQnIjHbRPpROol6iQKM=
github.com/ipfs-force-community/metrics v1.0.1-0.20211022060227-11142a08b729 h1:elS3KmzAMVrcZpmP2RMEjs9Zlwh6LfhJTfYQdj4TREs=
github.com/ipfs-force-community/metrics v1.0.1-0.20211022060227-11142a08b729/go.mod h1:mn40SioMuKtjmRumHFy/fJ26Pn028XuDjUJE9dorjyw=
github.com/ipfs-force-community/sophon-auth v1.11.1-0.20230607031351-bc57c2b78f44 h1:mezDnUywtX74k8onT/8M8FtKw1rEPFHH24DCkK78VeM=
github.com/ipfs-force-community/sophon-auth v1.11.1-0.20230607031351-bc57c2b78f44/go.mod h1:cGYCfjA/BDB/Km9P7Za6t1hmwpFlPKf8ho/CHmLG81Y=
github.com/ipfs/bbloom v0.0.1/go.mod h1:oqo8CVWsJFMOZqTglBG4wydCE4IQA/G2/SEofB0rjUI=
github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs=
github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0=
Expand Down
4 changes: 2 additions & 2 deletions venus-shared/TODO.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
- [ ] tests in actors
- [ ] venus-market
- [ ] droplet
- [ ] 类型考虑再细分到不同的命名空间下
- [ ] 对于 `venus-messager`、`venus-gateway` 的部分接口转发需要考虑更优雅的方式
- [ ] 对于 `sophon-messager`、`sophon-gateway` 的部分接口转发需要考虑更优雅的方式

- [ ] venus-devtool 中的 `lotus` 需要保持更新
2 changes: 1 addition & 1 deletion venus-shared/api/permission/permission.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (

"github.com/filecoin-project/venus/venus-shared/api"

"github.com/filecoin-project/venus-auth/core"
"github.com/ipfs-force-community/sophon-auth/core"
)

// PermissionProxy the scheduler between API and internal business
Expand Down