feat: query acct history

[fubuloubu]

What I did

1. Added BaseAddress.history, which fetches the relevant AccountHistory object
2. Re-route AccountHistory.outgoing through the query manager
3. Add a new method (raises NotImplementedError by default) to ProviderAPI to search through chain history and fetch transactions by account and range of nonces, and implement in Web3Provider (brute force method, very expensive)
4. Add support to the default query engine to source AccountHistory queries (by nonce) through either ExplorerAPI (if available) or via new brute-force method
   NOTE: Change this functionality in ape-etherscan to QueryAPI, and remove from ExplorerAPI (non-breaking change)
5. Add AccountHistory.query for working with account history using a dataframe approach
   6. (WIP) Add support to the cache query engine for storing and fetching AccountTransactionQuery results

fixes: #1237
fixes: APE-572

How I did it

I did this to take better advantage of the Ape query engine, and also provide more correct results in situations where an explorer wasn't available.

How to verify it

check the new feature acct.history on some local and public networks, with and without etherscan and the cache query engine

Checklist

• All changes are completed
  • Add .query
    - [ ] Add to cache provider
• New test cases have been added
• Documentation has been updated

[fubuloubu]

NOTE: Can't implement cache querying until #994 is resolved

[fubuloubu]

Note for reviewers:
bee0692 is the big commit, adding a recursive algorithm for filtering blocks and finding transactions by nonce. After that, a bit of piping, and a refactor to move the ExplorerAPI method to the default query provider (take note of TODOs there). All actions within AccountHistory now source from the query provider (also some TODOs about sessional history). Lastly, added support for .query, __getitem__ (both index and slice), and finally a route from BaseAddress to AddressHistory to close the gaps for users

Some extra reasoning: this recursive algorithm is necessary in scenarios where there needs to be a backup in place for people to access these features, and push them towards using a better means to access history (explorer plugin, other query providers). From working on this PR, it became very apparently that there needs to be a refactor of both sessional history and the ExplorerAPI to take advantage of the query provider system more, but will leave those to future PRs to settle.

[fubuloubu]

Just one of @ApeWorX/engineering needs to approve to merge

[antazoey]

Testing issue: I connected to mainnet using Alchemy. I loaded my account with txns. And then I did this:

next(account.history.outgoing)

I got this error:

ChainError: 'stop=4' cannot be greater than account's current nonce (3).

(other feedback is small and nitpicky - mostly focused on testing)

[fubuloubu]

Testing issue: I connected to mainnet using Alchemy. I loaded my account with txns. And then I did this:

next(account.history.outgoing)

I got this error:

ChainError: 'stop=4' cannot be greater than account's current nonce (3).

(other feedback is small and nitpicky - mostly focused on testing)

believe I solved this with 723ef84

[fubuloubu]

For some reason, acct.history[0] doesn't work for an account that has one

---

EDIT: This is resolved

[antazoey]

I ran some queries. Neat! What remains? This works a lot better than before 🙇🏻‍♀️ .

I do wonder how we can utilize a sessional cache to cache receipts from the slower HTTP-based caches (I saw your TODO). It could be opt-in.

[fubuloubu]

I ran some queries. Neat! What remains? This works a lot better than before 🙇🏻‍♀️ .

Just this: #1277 (comment)

I think this is just affecting the detection of nonce = 0, the rest works

I do wonder how we can utilize a sessional cache to cache receipts from the slower HTTP-based caches (I saw your TODO). It could be opt-in.

I mean we have ape-cache. For permanently existing networks, the idea is that it pushes stuff to an SQLite DB, which is good for multi-session caching. Could implement a secondary cache provider (either extending the one under ape_cache or under ape/managers/query.py alongside DefaultQueryEngine) that's just an in-memory cache (with some cache expiration to limit RAM usage), but the thing holding ape-cache back is the layout of ReceiptAPI/BlockAPI/ContractLog data structures so that it's easier to serialize them to SQL

Also the query system needs a more general refactor of how data passes through it so that it only converts if/when necessary e.g. refactor all .query(...) methods to return Iterator[dict] (instead of a pandas DataFrame) and convert from Iterator[dict] to ReceiptAPI/BlockAPI/ContractLog if that's the intended output (usually we use columns = "*" and then cast to that type), or do obj.dict() if you want Iterator[dict] output. As long as the columns field is used appropiately this should have an issue (but obiously this might have some mypy issues with the approach)

Lastly, a lot of the stuff relating to caching contracts is something I wanted to merge into the data system, as time goes on (fetching Manifests, maybe publishing them, fetch contract types via address, proxy info, etc.)

[antazoey]

Ok I tested the heck out of this.
I think most of my feedback is related to the actual query system though.

Please let me know if there any other gaps you would like me to test.
Note, it is hard to ensure anything won't break in future releases without more tests.

[antazoey]

I wish the resulting dataframe was indexed by nonce always.
When I do:

account.history.query("transaction", start_once=3, stop_nonce=6)

it shows the result like numeric indices 0 - 3, which threw me off at first.

[fubuloubu]

Yeah I agree with this. Probably a breaking change though, issue to track for 0.7.0?

[antazoey]

i mean, you can't do anything with the data now.. it's just broken
Edit: nevermind, I thought this was something else.

ya this could be 0.7 or a later opportunity.

[fubuloubu]

Actually wait, we added this method here, so I think we could actually do it for account history (but not for block history or event history)

[fubuloubu]

nvm, it requires siginificant restructuring to how we handle the returns from the query manager in order to create an index column, so not handling this here

[ghost]

Looks good, just a few semantic comments/questions

[ghost]

[fubuloubu]

Note: works great with latest ape-etherscan, however wildcard rules are currently fetching .call_trace, .return_value and other computed properties