core: cache transaction indexing tail in memory #28908
Conversation
|
Hi Gary, sorry this may be a bit off-topic here. As mentioned in #28907, the I think we may need to cut the link between the rpc module and the txIndexer message flow, maybe in another pr. |
Can you please try it out and to see if any high latency occurs? My hunch is it should be performant enough as But yeah, if we do find there is notable latency introduced by this progress query, I can definitely optimize it a bit further. |
|
Ok sure. I haven't seen high latency since I applied this PR yesterday |
There was a problem hiding this comment.
I think the change is "ok", but I'm a bit curious why you still read it from db. I mean, the txindexer is the one that writes it to the db, so if you make indexTransactions return lastNum, same on IndexTransactions, then the txIndexer.run could update a field on the indexer with it. And you would not need to read it, except at the very startup.
I'm not saying it must be changed though, if you have some good reason that's fine
|
It's cheap enough to read a database entry per each indexing run. Otherwise, we need to calculate the tail manually and easier to mess it up tbh. |
|
Hi, @rjl493456442 . May I ask a question about what is the purpose of moving Are you plan to store the Just a little confused. nothing about this PR> |
|
@fynnss just for refactoring. |
* params: begin v.1.13.12 release cycle * internal/flags: fix typo (ethereum#28876) * core/types: fix and test handling of faulty nil-returning signer (ethereum#28879) This adds an error if the signer returns a nil value for one of the signature value fields. * README.md: fix travis badge (ethereum#28889) The hyperlink in the README file that directs to the Travis CI build was broken. This commit updates the link to point to the corrent build page. * eth/catalyst: allow payload attributes v1 in fcu v2 (ethereum#28882) At some point, `ForkchoiceUpdatedV2` stopped working for `PayloadAttributesV1` while `paris` was active. This was causing a few failures in hive. This PR fixes that, and also adds a gate in `ForkchoiceUpdatedV1` to disallow `PayloadAttributesV3`. * docs/postmortems: fix outdated link (ethereum#28893) * core: reset tx lookup cache if necessary (ethereum#28865) This pull request resets the txlookup cache if chain reorg happens, preventing them from remaining reachable. It addresses failures in the hive tests. * build: fix problem with windows line-endings in CI download (ethereum#28900) fixes ethereum#28890 * eth/downloader: fix skeleton cleanup (ethereum#28581) * eth/downloader: fix skeleton cleanup * eth/downloader: short circuit if nothing to delete * eth/downloader: polish the logic in cleanup * eth/downloader: address comments * deps: update memsize (ethereum#28916) * core/txpool/blobpool: post-crash cleanup and addition/removal metrics (ethereum#28914) * core/txpool/blobpool: clean up resurrected junk after a crash * core/txpool/blobpool: track transaction insertions and rejections * core/txpool/blobpool: linnnnnnnt * core/txpool: don't inject lazy resolved transactions into the container (ethereum#28917) * core/txpool: don't inject lazy resolved transactions into the container * core/txpool: minor typo fixes * core/types: fix typo (ethereum#28922) * p2p: fix accidental termination of portMappingLoop (ethereum#28911) * internal/flags: fix --miner.gasprice default listing (ethereum#28932) * all: fix typos in comments (ethereum#28881) * Makefile: add help target to display available targets (ethereum#28845) Co-authored-by: Martin HS <martin@swende.se> Co-authored-by: Felix Lange <fjl@twurst.com> * core: cache transaction indexing tail in memory (ethereum#28908) * eth, miner: fix enforcing the minimum miner tip (ethereum#28933) * eth, miner: fix enforcing the minimum miner tip * ethclient/simulated: fix failing test due the min tip change * accounts/abi/bind: fix simulater gas tip issue * core/state, core/vm: minor uint256 related perf improvements (ethereum#28944) * cmd,internal/era: implement `export-history` subcommand (ethereum#26621) * all: implement era format, add history importer/export * internal/era/e2store: refactor e2store to provide ReadAt interface * internal/era/e2store: export HeaderSize * internal/era: refactor era to use ReadAt interface * internal/era: elevate anonymous func to named * cmd/utils: don't store entire era file in-memory during import / export * internal/era: better abstraction between era and e2store * cmd/era: properly close era files * cmd/era: don't let defers stack * cmd/geth: add description for import-history * cmd/utils: better bytes buffer * internal/era: error if accumulator has more records than max allowed * internal/era: better doc comment * internal/era/e2store: rm superfluous reader, rm superfluous testcases, add fuzzer * internal/era: avoid some repetition * internal/era: simplify clauses * internal/era: unexport things * internal/era,cmd/utils,cmd/era: change to iterator interface for reading era entries * cmd/utils: better defer handling in history test * internal/era,cmd: add number method to era iterator to get the current block number * internal/era/e2store: avoid double allocation during write * internal/era,cmd/utils: fix lint issues * internal/era: add ReaderAt func so entry value can be read lazily Co-authored-by: lightclient <lightclient@protonmail.com> Co-authored-by: Martin Holst Swende <martin@swende.se> * internal/era: improve iterator interface * internal/era: fix rlp decode of header and correctly read total difficulty * cmd/era: fix rebase errors * cmd/era: clearer comments * cmd,internal: fix comment typos --------- Co-authored-by: Martin Holst Swende <martin@swende.se> * core,params: add holesky to default genesis function (ethereum#28903) * node, rpc: add configurable HTTP request limit (ethereum#28948) Adds a configurable HTTP request limit, and bumps the engine default * all: fix docstring names (ethereum#28923) * fix wrong comment * reviewers input * Update log/handler_glog.go --------- Co-authored-by: Martin HS <martin@swende.se> * ethclient/simulated: fix typo (ethereum#28952) (ethclient/simulated):fix typo * eth/gasprice: fix percentile validation in eth_feeHistory (ethereum#28954) * cmd/devp2p, eth: drop support for eth/67 (ethereum#28956) * params, core/forkid: add mainnet timestamp for Cancun (ethereum#28958) * params: add cancun timestamp for mainnet * core/forkid: add test for mainnet cancun forkid * core/forkid: update todo tests for cancun * internal/ethapi: add support for blobs in eth_fillTransaction (ethereum#28839) This change adds support for blob-transaction in certain API-endpoints, e.g. eth_fillTransaction. A follow-up PR will add support for signing such transactions. * internal/era: update block index format to be based on record offset (ethereum#28959) As mentioned in ethereum#26621, the block index format for era1 is not in line with the regular era block index. This change modifies the index so all relative offsets are based against the beginning of the block index record. * params: go-ethereum v1.13.12 stable --------- Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: alex <152680487+bodhi-crypo@users.noreply.github.com> Co-authored-by: protolambda <proto@protolambda.com> Co-authored-by: KeienWang <42377006+keienWang@users.noreply.github.com> Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: zoereco <158379334+zoereco@users.noreply.github.com> Co-authored-by: Chris Ziogas <ziogaschr@gmail.com> Co-authored-by: Dimitris Apostolou <dimitris.apostolou@icloud.com> Co-authored-by: Halimao <1065621723@qq.com> Co-authored-by: Felix Lange <fjl@twurst.com> Co-authored-by: lmittmann <3458786+lmittmann@users.noreply.github.com> Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com> Co-authored-by: Austin Roberts <austin.roberts@rivet.cloud>
This pull request tries to address issue #28907
From the ticket we can see the
txindexer.reportdominates the CPU usage. Inside of this function, only a single database read is involved. Therefore the first attempt to mitigate the situation is caching thetaildata in memory instead of retrieving it from db.Note, with this change, the indexing progress reported will have a big delay as tail is only updated at the end. It might have bad UX for reporting tx indexing progress, but should be fine.