feat:config:force existing users to opt into new defaults #10488

ZenGround0 · 2023-03-15T21:35:32Z

Related Issues

This is the change to 1) force users to explicitly set splitstore config by uncommenting the default, 2) for new nodes do the uncommenting automatically so they will opt into the default automatically

Proposed Changes

Add a notion of uncommenting filters to config update. Then apply a filter on EnableSplitstore in the default full node config
Add a notion of loadability checks to the lotus config. Then apply a check ensuring #EnableSplitstore = is not in the config to ensure that users with old configs will reset their config explicity.

Additional Info

The patterns introduced here can hopefully be used in other sensative default change situations
We can probably revert the splitstore specific filters and loadability checks in a few months when the community is adjusted to the new defaults

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title is in the form of of <PR type>: <area>: <change being made>
- example: fix: mempool: Introduce a cache for valid signatures
- PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
- area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

node/config/load.go

Stebalien · 2023-03-16T00:22:11Z

node/config/load.go

+		poisonStr := "#EnableSplitstore =" // splitstore must be set explicitly
+		if strings.Contains(cfgRaw, poisonStr) {
+			return xerrors.Errorf("Config contains disallowed string %s, refusing to load. Please explicitly set EnableSplitstore. Set it to false to keep chain management unchanged from March 2023 set it to true if you want to discard old state by default", poisonStr)


What if I just want the default? What if I comment out one EnableSplitstore line but keep another?

Also, this won't this cause every single node to fail to start on upgrade?

What if I just want the default?
won't this cause every single node to fail to start on upgrade?

The point of this change (idea originating here is:

If you want the default and you have an existing, unset config: too bad. You have to set it explicitly or your node won't start. This way users won't accidentally overwrite data.

If you already set your existing config explicitly then your config values won't be changing so that's ok and your node starts.

If you start a new node then the default will be set explicitly for you as part of this PR's change to the default config and your node will start.

What if I comment out one EnableSplitstore line but keep another?

This is a good point I hadn't thought of. What I would like to be doing instead is parsing the TOML and making sure that EnableSplitstore field is set there. I'll go back and take a look.

The point of this change (idea originating here is:

Can we just warn loudly and let the user fix their config if they want? If they don't, isn't that their problem? It's not like non-splitstore configs aren't supported...

🤷 this honestly isn't something I thought of or particularly care about as a node operator myself. Other node operators appear to a lot and data loss is scary so it makes some sense. @jennijuju as the progenitor of this change could you weigh in?

I guess going back to the original message it looks like @jennijuju is listening to the advice of glif on this one

Im pretty sure what actually happens is that the universal store will just stay static with all the old state ( the same way the ~130 Gigs of snapshot stay around forever) but the new state will begin to be trimmed. That's still bad but its a much easier problem to recover from. I think I'll test this and if communicate it which will hopefully lead to some more chill.

Keeping this thread alive but the original concern is addressed in the latest commit. We now use regex to ensure that an uncommented EnableSplitstore = preceded only by whitespace exists in the config which should be the correct check.

@ribasushi I've confirmed that datastore/chain will stay untouched with default discard splitstore so all of the old historical data will stick around even if we don't get config dance right

Can we just warn loudly and let the user fix their config if they want? If they don't, isn't that their problem? It's not like non-splitstore configs aren't supported...

We are trying to be more proactive here to avoid full archival node operators to lose their state by mistake. (both zondax and glif have raised valid concerns imho)

I'm suggesting the other way around. Warn loudly if the user hasn't configured anything, but leave the default the way it is for existing nodes. But I guess we'll need some form of hack either way.

node/config/load.go

…tore false for full archive

node/config/load.go

cmd/lotus/config.go

node/config/load.go

magik6k

Definitely needs some testing

I don't love that we're forcing users to take action on this, tho on the other hand it seems like without splitstore most non-fullnodes require regular maintenance anyways, and splitstore nodes will have this explicitly set already.

There will be some disruption to user flow, but I don't think it will be too bad - and it can likely be mitigated by clear comms in changelog / announcements.

magik6k · 2023-03-16T07:56:36Z

node/config/load_test.go

Can we add some unit tests around EnableSplitstore handling?

node/config/load.go

magik6k

Looks good, feel free to merge whenever.

Ideally we'd give this a try on a real node too.

node/config/load_test.go

ZenGround0 · 2023-03-20T14:28:59Z

Thanks

Ideally we'd give this a try on a real node too.

My mainnet node just crashed so this is a good opportunity.

* Fix the overzealous fix (filecoin-project#10366) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * chore: deps: update to actors v10.0.0 * build: v1.20.0: release prep (filecoin-project#10368) * add upgrade epoch and bump version * Update changelog --------- Co-authored-by: Aayush <arajasek94@gmail.com> * feat: stmgr: cache migrated stateroots * feat: vm: switch to the new exec trace format (filecoin-project#10372) This is now "FVM" native. Changes include: 1. Don't treat "trace" messages like off-chain messages. E.g., don't include CIDs, versions, etc. 2. Include IPLD codecs where applicable. 3. Remove fields that aren't filled by the FVM (timing, some errors, code locations, etc.). * fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct * feat: update renew-sectors with FIP-0045 logic * Hide `lotus-worker set` command Hide the `lotus-worker set` command and print that this command will be deprecated. * chain prune hot -- hotstore online gc * remove accidental add * fix: EthAPI: Correctly get parent hash * make debugging windowPoSt-failures human readable * make gen * indent output * post worker sched: Filter out disabled workers correctly * itests: Test PoSt worker RPC error handling * post worker sched: Retry on alternative worker on RPC errors * make gen * itests: Wait for both workers in TestWindowPostWorkerDisconnected * Unbump api version as its not necessary * deps: Update go-jsonrpc to v0.2.2 * Add command to trigger moving GC of hotstore manually * Timing info and updated docs * fix: EthAPI: Make newEthBlockFromFilecoinTipSet faster and correct * fix: EthAPI: Correctly get parent hash * v1.20.1 prep * fix: state: short-circuit genesis state computation * fix: state: short-circuit genesis state computation * update changelog * Begin account for size during walks * Track size of dags relevant to compaction * Plan out moving GC limiting * GC respects target for max hotstore space * lint * rpcenc: More reliably failing TestReaderRedirectDrop * rpcenc: Propagate closeOnce in beginPost * fix: eth API: correct gateway restrictions, drop unimplemented methods * refactor: EthAPI: Drop unnecessary param from newEthTxReceipt * feat(deps): update to go-data-transfer v2, still wip * chore(deps): update go-libipfs * chore(deps): update to get to compile * update minimum Go version to 1.19.7 * use go-libipfs/files instead of ipfs/go-ipfs-files * fix drand test * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * un-un-update filecoin-ffi * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * fix logging * deprecate MarketListRetrievalDeals, make gen * Fix up approximation and logging * run make gen * small doc patch on how to make gen after api changes * fix TestPrintGroupInfo * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * register rcmgr metrics * replace make all with make deps according to review * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * cid key size * computing szPurge leads to deadlock and unneeded, remove * Review Response * Lint * lint * docsgen-cli * only register metrics once * lint * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * fmt. * lint * feat:splitstore:Splitstore enabled by default (filecoin-project#10429) Discard mode --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Review Response * fix lint. * Configur buffer and threshold * lint * make gen * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * review comments * simplify test. * feat: chain: make fetching tipset by height 1000x faster * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * run make gen * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * fmt. * fix lint. * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * simplify test. * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * make gen * change log * fiix bakcport error filecoin-project#10427 * retract v1.20.2 * go mod tidy * make gen * feat: cli: Add an EVM command to fetch a contract's bytecode * feat: eth API: reject masked ID addresses embedded in f410f payloads We'll never get an actor/account deployed to one of these addresses (although we might get a placeholder). However, converting such an address to an f4 address is definitely wrong. * chore: update the FFI for release (filecoin-project#10435) Specifically, this updates the proofs libraries and enables CUDA by default. * feat: stmgr: skip tipset execution when possible * feat: eth: optimize receipt reading This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state. * feat: api: optimize ChainGetParentReceipts Read the receipts all at once instead of fetching them one-by-one. * test: eth: reduce chances of chain-reorgs affecting the test Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways. * gateway: fix: drop overzealous guard on MsigGetVested * eth_feeHistory: migrate to using TipSetState. * Eth API: drop support for 'pending' block parameter. After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too. * Eth API: make net_version return the chain ID. * fix imports. * remove superfluous filter. * drop irrelevant test. * Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported. * fix: itest: avoid failing the test when we race the miner I've been seeing quite a few tests failing here, so I'm hoping this will deflake them a bit. * feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460) Instead: 1. Use the receipt from the message search. 2. Re-compute the gas fees that would have been charged. fixes filecoin-project#10418 Co-authored-by: raulk <raul.kripalani@gmail.com> * refactor: update cache to the new generic version (filecoin-project#10463) - Adds type safety. - Reduces allocations. - Fixes the drand cache (was storing by value, but retrieving by pointer) * fix: gateway: correctly apply the fee history lookback max This fix ensures that the full fee-history range falls into the allowable lookback rang. * Plumb through a proper Flush() method on all blockstores * feat: consensus: log ApplyBlock timing/gas stats * fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472) * fix: stmgr: don't attempt to lookup genesis state * stmgr: More correct fix for genesis state compute * Revert "Eth API: drop support for 'pending' block parameter." This reverts commit 9412753. * fix: build: drop drand incentinet servers * feat: gateway: export StateVerifierStatus * chore: update go-libp2p-pubsub to v0.9.2 * chore: update ffi to increase execution parallelism (filecoin-project#10480) This changes the default validation parallelism from 1 to 4, and makes it configurable with a LOTUS_FVM_CONCURRENCY environment variable. * fix connection gater integration test * chore: update go-libp2p to v0.26.2 * make mod-tidy ci happy * update go-libp2p-pubsub to v0.9.3 * update go-jsonrpc to v0.2.3 * ethrpc: Buffer sub messages if subscriber is slow * make lint happy * feat: apply gateway lookback limit to eth API lookback This change: 1. Introduces new "limited" API endpoints for EthGetTransactionByHash and EthGetTransactionReceipt that accept lookback-limits. 2. Implements the gateway version of these API endpoints by calling the limited variants with the default message search lookback limit. fixes filecoin-project#10412 * fix: actually apply limit * feat: expose limited apis on the gateway * fix: eth: handle a potential divide by zero in receipt handling This isn't really possible to hit on-chain at this piont (message won't be accepted) but we might as well be extra careful. fixes filecoin-project#10471 * Don't enforce walking receipt tree during compaction * Demote now common logs (filecoin-project#10516) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * feat:config:force existing users to opt into new defaults (filecoin-project#10488) * Config default does not comment out EnableSplitstore * Loadability check * Remove test used for debugging * regexp for properly safe check that config is set * regexp for safely matching the EnableSpitstore field in the config * Add instructions for undeleting config and remind users to set splitstore false for full archive * UpdateConfig small docs and functional opts * make gen * Lint * Fix * nil pointer check on validate * Unit testing of EnableSplitstore cases * Address Review --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * release v1.21.0-rc1 prep * chore: build: Update Changelog for v1.21.0 chore: build: Update Changelog for v1.21.0 * Add RPC API highlight Add RPC API highlight * Add callout to exchanges Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section. * fix: miner: correctly count sector extensions * fix: miner: call ExtendSectorExpiration2 * chore: all: bump go-libipfs (cherry picked from commit f48c626) * Initialize with same length as partition Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop. * build: docker: Update GO-version build: docker: Update GO-version * build: release: v1.21.0-rc2 build: release: v1.21.0-rc2 * Update go.mod Update go.mod * Update changelog for v1.21.0-rc2 Update changelog for v1.21.0-rc2 * perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546) * have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization * set applyTsMessages = true in CallWithGas call in shed * update test with new callwithgas api optimization for eth call * Update chain/stmgr/call.go Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation --------- Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> * feat: supply: drop genesis market locked funds * feat: supply: only grab genesis msig locks for writes * Update Changelog with backports Update Changelog with backports * Bump version in changelog Bump version in changelog * Add longer validation time cause to changelog Add longer validation time cause to changelog * refactor: stop using deprecated io/ioutil * Fix error when injecting StateManager * new actor bundle, fix ci, upgrade fx * fix fx version * upgrade go version circle ci * make linter happy :) * more go version fixing * replace deprecated mac build * removing support for macos build * Fix ctrl-c for GOOD --------- Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Jennifer Wang <jiayingw703@gmail.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Travis Person <travisperson@users.noreply.github.com> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Phi <orjan.roren@gmail.com> Co-authored-by: beck <1504068285@qq.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: hannahhoward <hannah@hannahhoward.net> Co-authored-by: Marten Seemann <martenseemann@gmail.com> Co-authored-by: Arsenii Petrovich <arsenii@protofire.io> Co-authored-by: Arsenii Petrovich <23522179+ArseniiPetrovich@users.noreply.github.com> Co-authored-by: Raúl Kripalani <raul@protocol.ai> Co-authored-by: ychiao <ychiaoli18@users.noreply.github.com> Co-authored-by: Travis Person <165274+travisperson@users.noreply.github.com> Co-authored-by: raulk <raul.kripalani@gmail.com> Co-authored-by: Peter Rabbitson <ribasushi@leporine.io> Co-authored-by: vyzo <vyzo@hackzen.org> Co-authored-by: Jorropo <jorropo.pgm@gmail.com> Co-authored-by: Mikers <snissn@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> Co-authored-by: Henrique Moniz <1785239+hmoniz@users.noreply.github.com>

…ed (filecoin-project#144) * use go-libipfs/files instead of ipfs/go-ipfs-files * fix drand test * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * un-un-update filecoin-ffi * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * fix logging * deprecate MarketListRetrievalDeals, make gen * Fix up approximation and logging * run make gen * small doc patch on how to make gen after api changes * fix TestPrintGroupInfo * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * register rcmgr metrics * replace make all with make deps according to review * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * cid key size * computing szPurge leads to deadlock and unneeded, remove * Review Response * Lint * lint * docsgen-cli * only register metrics once * lint * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * fmt. * lint * feat:splitstore:Splitstore enabled by default (filecoin-project#10429) Discard mode --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * Review Response * fix lint. * Configur buffer and threshold * lint * make gen * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * review comments * simplify test. * feat: chain: make fetching tipset by height 1000x faster * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * run make gen * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * fmt. * fix lint. * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * simplify test. * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * make gen * change log * fiix bakcport error filecoin-project#10427 * retract v1.20.2 * go mod tidy * make gen * feat: cli: Add an EVM command to fetch a contract's bytecode * feat: eth API: reject masked ID addresses embedded in f410f payloads We'll never get an actor/account deployed to one of these addresses (although we might get a placeholder). However, converting such an address to an f4 address is definitely wrong. * chore: update the FFI for release (filecoin-project#10435) Specifically, this updates the proofs libraries and enables CUDA by default. * feat: stmgr: skip tipset execution when possible * feat: eth: optimize receipt reading This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state. * feat: api: optimize ChainGetParentReceipts Read the receipts all at once instead of fetching them one-by-one. * test: eth: reduce chances of chain-reorgs affecting the test Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways. * gateway: fix: drop overzealous guard on MsigGetVested * eth_feeHistory: migrate to using TipSetState. * Eth API: drop support for 'pending' block parameter. After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too. * Eth API: make net_version return the chain ID. * fix imports. * remove superfluous filter. * drop irrelevant test. * Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported. * fix: itest: avoid failing the test when we race the miner I've been seeing quite a few tests failing here, so I'm hoping this will deflake them a bit. * feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460) Instead: 1. Use the receipt from the message search. 2. Re-compute the gas fees that would have been charged. fixes filecoin-project#10418 Co-authored-by: raulk <raul.kripalani@gmail.com> * refactor: update cache to the new generic version (filecoin-project#10463) - Adds type safety. - Reduces allocations. - Fixes the drand cache (was storing by value, but retrieving by pointer) * fix: gateway: correctly apply the fee history lookback max This fix ensures that the full fee-history range falls into the allowable lookback rang. * Plumb through a proper Flush() method on all blockstores * feat: consensus: log ApplyBlock timing/gas stats * fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472) * fix: stmgr: don't attempt to lookup genesis state * stmgr: More correct fix for genesis state compute * Revert "Eth API: drop support for 'pending' block parameter." This reverts commit 9412753. * fix: build: drop drand incentinet servers * feat: gateway: export StateVerifierStatus * chore: update go-libp2p-pubsub to v0.9.2 * chore: update ffi to increase execution parallelism (filecoin-project#10480) This changes the default validation parallelism from 1 to 4, and makes it configurable with a LOTUS_FVM_CONCURRENCY environment variable. * fix connection gater integration test * chore: update go-libp2p to v0.26.2 * make mod-tidy ci happy * update go-libp2p-pubsub to v0.9.3 * update go-jsonrpc to v0.2.3 * ethrpc: Buffer sub messages if subscriber is slow * make lint happy * feat: apply gateway lookback limit to eth API lookback This change: 1. Introduces new "limited" API endpoints for EthGetTransactionByHash and EthGetTransactionReceipt that accept lookback-limits. 2. Implements the gateway version of these API endpoints by calling the limited variants with the default message search lookback limit. fixes filecoin-project#10412 * fix: actually apply limit * feat: expose limited apis on the gateway * fix: eth: handle a potential divide by zero in receipt handling This isn't really possible to hit on-chain at this piont (message won't be accepted) but we might as well be extra careful. fixes filecoin-project#10471 * Don't enforce walking receipt tree during compaction * Demote now common logs (filecoin-project#10516) Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * feat:config:force existing users to opt into new defaults (filecoin-project#10488) * Config default does not comment out EnableSplitstore * Loadability check * Remove test used for debugging * regexp for properly safe check that config is set * regexp for safely matching the EnableSpitstore field in the config * Add instructions for undeleting config and remind users to set splitstore false for full archive * UpdateConfig small docs and functional opts * make gen * Lint * Fix * nil pointer check on validate * Unit testing of EnableSplitstore cases * Address Review --------- Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> * release v1.21.0-rc1 prep * chore: build: Update Changelog for v1.21.0 chore: build: Update Changelog for v1.21.0 * Add RPC API highlight Add RPC API highlight * Add callout to exchanges Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section. * fix: miner: correctly count sector extensions * fix: miner: call ExtendSectorExpiration2 * chore: all: bump go-libipfs (cherry picked from commit f48c626) * Initialize with same length as partition Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop. * build: docker: Update GO-version build: docker: Update GO-version * build: release: v1.21.0-rc2 build: release: v1.21.0-rc2 * Update go.mod Update go.mod * Update changelog for v1.21.0-rc2 Update changelog for v1.21.0-rc2 * perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546) * have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization * set applyTsMessages = true in CallWithGas call in shed * update test with new callwithgas api optimization for eth call * Update chain/stmgr/call.go Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation --------- Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> * feat: supply: drop genesis market locked funds * feat: supply: only grab genesis msig locks for writes * Update Changelog with backports Update Changelog with backports * Bump version in changelog Bump version in changelog * Add longer validation time cause to changelog Add longer validation time cause to changelog * refactor: stop using deprecated io/ioutil * Fix error when injecting StateManager * new actor bundle, fix ci, upgrade fx * fix fx version * upgrade go version circle ci * make linter happy :) * more go version fixing * replace deprecated mac build * removing support for macos build * Fix ctrl-c for GOOD * bypass mpool rm for set membership --------- Co-authored-by: Marten Seemann <martenseemann@gmail.com> Co-authored-by: Jiaying Wang <42981373+jennijuju@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@users.noreply.github.com> Co-authored-by: Łukasz Magiera <magik6k@gmail.com> Co-authored-by: Arsenii Petrovich <arsenii@protofire.io> Co-authored-by: zenground0 <ZenGround0@users.noreply.github.com> Co-authored-by: Arsenii Petrovich <23522179+ArseniiPetrovich@users.noreply.github.com> Co-authored-by: Raúl Kripalani <raul@protocol.ai> Co-authored-by: ychiao <ychiaoli18@users.noreply.github.com> Co-authored-by: Aayush Rajasekaran <arajasek94@gmail.com> Co-authored-by: ZenGround0 <5515260+ZenGround0@users.noreply.github.com> Co-authored-by: Travis Person <travisperson@users.noreply.github.com> Co-authored-by: Travis Person <165274+travisperson@users.noreply.github.com> Co-authored-by: Steven Allen <steven@stebalien.com> Co-authored-by: Jennifer Wang <jiayingw703@gmail.com> Co-authored-by: raulk <raul.kripalani@gmail.com> Co-authored-by: Peter Rabbitson <ribasushi@leporine.io> Co-authored-by: vyzo <vyzo@hackzen.org> Co-authored-by: Phi <orjan.roren@gmail.com> Co-authored-by: Jorropo <jorropo.pgm@gmail.com> Co-authored-by: Mikers <snissn@gmail.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-4-29.us-east-2.compute.internal> Co-authored-by: Henrique Moniz <1785239+hmoniz@users.noreply.github.com> Co-authored-by: Denis Kolegov <dnkolegov@gmail.com>

ZenGround0 added 2 commits March 15, 2023 14:47

Config default does not comment out EnableSplitstore

a8f6b19

Loadability check

6f2e058

ZenGround0 requested a review from a team as a code owner March 15, 2023 21:35

Remove test used for debugging

97be74f

jennijuju added this to the v1.21.0 milestone Mar 16, 2023

Stebalien requested changes Mar 16, 2023

View reviewed changes

regexp for properly safe check that config is set

f1ec8d6

ZenGround0 force-pushed the feat/safe-splitstore-default branch from 034c9cb to f1ec8d6 Compare March 16, 2023 03:13

ZenGround0 added 3 commits March 15, 2023 21:31

regexp for safely matching the EnableSpitstore field in the config

313a25f

Add instructions for undeleting config and remind users to set splits…

0cf9bb4

…tore false for full archive

UpdateConfig small docs and functional opts

34bb395

ZenGround0 requested a review from Stebalien March 16, 2023 05:06

ZenGround0 added 2 commits March 15, 2023 23:13

make gen

cfc6dc6

Lint

196ef19

Stebalien requested changes Mar 16, 2023

View reviewed changes

node/config/load.go Outdated Show resolved Hide resolved

node/config/load.go Outdated Show resolved Hide resolved

node/config/load.go Show resolved Hide resolved

node/config/load.go Outdated Show resolved Hide resolved

Stebalien reviewed Mar 16, 2023

View reviewed changes

cmd/lotus/config.go Outdated Show resolved Hide resolved

node/config/load.go Outdated Show resolved Hide resolved

node/config/load.go Outdated Show resolved Hide resolved

node/config/load.go Outdated Show resolved Hide resolved

Fix

dd08d17

ZenGround0 requested review from Stebalien and magik6k March 16, 2023 07:02

nil pointer check on validate

3d9ddea

magik6k reviewed Mar 16, 2023

View reviewed changes

Unit testing of EnableSplitstore cases

3817701

ZenGround0 requested review from jennijuju and magik6k March 18, 2023 03:22

magik6k approved these changes Mar 20, 2023

View reviewed changes

node/config/load_test.go Outdated Show resolved Hide resolved

Address Review

738c9c1

ZenGround0 merged commit 43da108 into master Mar 20, 2023

ZenGround0 deleted the feat/safe-splitstore-default branch March 20, 2023 16:19

rjan90 mentioned this pull request May 12, 2023

Force users to set the lotus-miner legacy markets config to false #10862

Closed

rjan90 mentioned this pull request Jul 18, 2023

Update lotus-fullnode config filecoin-project/lotus-infra#1480

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat:config:force existing users to opt into new defaults #10488

feat:config:force existing users to opt into new defaults #10488

ZenGround0 commented Mar 15, 2023

Stebalien Mar 16, 2023

ZenGround0 Mar 16, 2023

Stebalien Mar 16, 2023

ZenGround0 Mar 16, 2023

ZenGround0 Mar 16, 2023

ZenGround0 Mar 16, 2023

ZenGround0 Mar 16, 2023

ZenGround0 Mar 16, 2023

jennijuju Mar 16, 2023

Stebalien Mar 16, 2023

magik6k left a comment

magik6k Mar 16, 2023

magik6k left a comment

ZenGround0 commented Mar 20, 2023

feat:config:force existing users to opt into new defaults #10488

feat:config:force existing users to opt into new defaults #10488

Conversation

ZenGround0 commented Mar 15, 2023

Related Issues

Proposed Changes

Additional Info

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

ZenGround0 commented Mar 20, 2023