Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: query acct history #1277

Merged
merged 26 commits into from
Mar 22, 2023
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8d1b9ce
fix: add missing type to Iterator type hint
fubuloubu Feb 2, 2023
40e4e94
feat: add `get_transactions_by_account_nonce` to `Web3Provider`
fubuloubu Feb 2, 2023
bbeb769
feat: add method to `ProviderAPI` to fetch account history
fubuloubu Feb 2, 2023
3de79af
feat: add support for `AccountTransactionQuery` to default query eng
fubuloubu Feb 2, 2023
b6cbfd5
refactor: source `.outgoing` account history from the query manager
fubuloubu Feb 2, 2023
645cc6e
feat: add `.query` to `AccountHistory`
fubuloubu Feb 2, 2023
41393d7
feat: add support for `__getitem__` to `AccountHistory`
fubuloubu Feb 16, 2023
8ba1103
fix: refactor `.outgoing` using `AccountHistory.__getitem__`
fubuloubu Feb 16, 2023
29f48ac
docs: add a warning when using with raw RPC
fubuloubu Feb 16, 2023
8b32b32
feat: add `.history` property to `BaseAddress`
fubuloubu Feb 2, 2023
c92a185
test: use new `AccountHistory` methods inside of tests
fubuloubu Feb 16, 2023
c49744c
fix: off-by-one bug w/ slices
fubuloubu Feb 16, 2023
f145db5
fix: incorrect query when only one item was requested
fubuloubu Feb 17, 2023
b2cfa08
refactor: wrong order for input sanitation
fubuloubu Feb 21, 2023
146c1f5
refactor: use truthiness
fubuloubu Feb 21, 2023
aaa1d79
refactor: typo/whitespace
fubuloubu Feb 21, 2023
16526ea
fix: off-by-one error
fubuloubu Feb 22, 2023
6ff296c
refactor: change assertion to error
fubuloubu Feb 22, 2023
93f9b0a
fix: raise IndexError from __getitem__, not StopIteration
fubuloubu Feb 22, 2023
a0812dd
refactor: use local network named constant
fubuloubu Mar 20, 2023
6e1501f
refactor: add suggestions to engine selector error
fubuloubu Mar 20, 2023
bd12a91
refactor: more efficiently compute fields for query handlers
fubuloubu Mar 20, 2023
b793c5c
test: columns are now sorted
fubuloubu Mar 20, 2023
b97947e
docs: move cache userguide to a more general userguide about data
fubuloubu Mar 20, 2023
3f60320
refactor: apply suggestions from code review
fubuloubu Mar 21, 2023
861b42e
refactor: add some notes, optimize one statement, sort another
fubuloubu Mar 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
userguides/installing_plugins
userguides/projects
userguides/compile
userguides/cache
userguides/data
userguides/networks
userguides/developing_plugins
userguides/config
Expand Down
72 changes: 0 additions & 72 deletions docs/userguides/cache.md

This file was deleted.

98 changes: 98 additions & 0 deletions docs/userguides/data.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Querying Data

Ape has advanced features for querying large amounts of on-chain data.
Ape provides this support through a number of standardized methods for working with data,
routed through our query management system, which incorporates data from many sources in
your set of installed plugins.

## Getting Block Data

Use `ape console`:

```bash
ape console --network ethereum:mainnet:infura
```

Run a few queries:

```python
In [1]: df = chain.blocks.query("*", stop_block=20)
In [2]: chain.blocks[-2].transactions # List of transactions in block
```

## Getting Account Transaction Data

Each account within ape will also fetch and store transactional data that you can query.
To work with an account's transaction data, you can do stuff like this:

```python
In [1]: chain.history["example.eth"].query("value").sum() # All value sent by this address
In [2]: acct = accounts.load("my-acct"); acct.history[-1] # Last txn `acct` made
In [3]: acct.history.query("total_fees_paid").sum() # Sum of ether paid for fees by `acct`
```

## Getting Contract Event Data

On a deployed contract, you can query event history:

For example, we have a contract with a `FooHappened` event that you want to query from.
This is how you would query the args from an event:

```python
contract_instance.FooHappened.query("*", start_block=-1)
```

where `contract_instance` is the return value of `owner.deploy(MyContract)`

See [this guide](../userguides/contracts.html) for more information how to deploy or load contracts.

## Using the Cache

**Note**: This is in Beta release.
This functionality is in constant development and many features are in planning stages.
Use the cache plugin to store provider data in a sqlite database.

To use the cache, first you must initialize it for each network you plan on caching data for:

```bash
ape cache init --network <ecosystem-name>:<network-name>
```

**Note**: Caching only works for permanently available networks. It will not work with local development networks.

For example, to initialize the cache database for the Ethereum mainnet network, you would do the following:

```bash
ape cache init --network ethereum:mainnet
```

This creates a SQLite database file in ape's data folder inside your home directory.

You can query the cache database directly, for debugging purposes.
For example, to get block data, you would query the `blocks` table:

```bash
ape cache query --network ethereum:mainnet:infura "SELECT * FROM blocks"
```

Which will return something like this:

```bash
hash num_transactions number parent_hash size timestamp gas_limit gas_used base_fee difficulty total_difficulty
0 b'\xd4\xe5g@\xf8v\xae\xf8\xc0\x10\xb8j@\xd5\xf... 0 0 b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00... 540 0 5000 0 None 17179869184 17179869184
1 b'\x88\xe9mE7\xbe\xa4\xd9\xc0]\x12T\x99\x07\xb... 0 1 b'\xd4\xe5g@\xf8v\xae\xf8\xc0\x10\xb8j@\xd5\xf... 537 1438269988 5000 0 None 17171480576 34351349760
2 b'\xb4\x95\xa1\xd7\xe6f1R\xae\x92p\x8d\xa4\x84... 0 2 b'\x88\xe9mE7\xbe\xa4\xd9\xc0]\x12T\x99\x07\xb... 544 1438270017 5000 0 None 17163096064 51514445824
...
```

Similarly, to get transaction data, you would query the `transactions` table:

```bash
ape cache query --network ethereum:mainnet:infura "SELECT * FROM transactions"
```

Finally, to query cached contract events you would query the `contract_events` table:

```bash
ape cache query --network ethereum:mainnet:infura "SELECT * FROM contract_events"
```
14 changes: 12 additions & 2 deletions src/ape/api/address.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
from typing import Any, List
from typing import TYPE_CHECKING, Any, List

from hexbytes import HexBytes

from ape.exceptions import ConversionError
from ape.types import AddressType, ContractCode
from ape.utils import BaseInterface, abstractmethod
from ape.utils import BaseInterface, abstractmethod, cached_property

if TYPE_CHECKING:
from ape.managers.chain import AccountHistory


class BaseAddress(BaseInterface):
Expand Down Expand Up @@ -133,6 +136,13 @@ def is_contract(self) -> bool:

return len(HexBytes(self.code)) > 0

@cached_property
def history(self) -> "AccountHistory":
"""
The list of transactions that this account has made on the current chain.
"""
return self.chain_manager.history[self.address]


class Address(BaseAddress):
"""
Expand Down
90 changes: 89 additions & 1 deletion src/ape/api/providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,25 @@ def get_transactions_by_block(self, block_id: BlockID) -> Iterator[TransactionAP
Iterator[:class: `~ape.api.transactions.TransactionAPI`]
"""

@raises_not_implemented
def get_transactions_by_account_nonce( # type: ignore[empty-body]
self,
account: AddressType,
start_nonce: int = 0,
stop_nonce: int = -1,
) -> Iterator[ReceiptAPI]:
"""
Get account history for the given account.

Args:
account (``AddressType``): The address of the account.
start_nonce (int): The nonce of the account to start the search with.
stop_nonce (int): The nonce of the account to stop the search with.

Returns:
Iterator[:class:`~ape.api.transactions.ReceiptAPI`]
"""

@abstractmethod
def send_transaction(self, txn: TransactionAPI) -> ReceiptAPI:
"""
Expand Down Expand Up @@ -962,7 +981,7 @@ def get_receipt(
)
return receipt.await_confirmations()

def get_transactions_by_block(self, block_id: BlockID) -> Iterator:
def get_transactions_by_block(self, block_id: BlockID) -> Iterator[TransactionAPI]:
if isinstance(block_id, str):
block_id = HexStr(block_id)

Expand All @@ -973,6 +992,75 @@ def get_transactions_by_block(self, block_id: BlockID) -> Iterator:
for transaction in block.get("transactions", []):
yield self.network.ecosystem.create_transaction(**transaction)

def get_transactions_by_account_nonce(
self,
account: AddressType,
start_nonce: int,
stop_nonce: int,
) -> Iterator[ReceiptAPI]:
if start_nonce > stop_nonce:
raise ValueError("Starting nonce cannot be greater than stop nonce for search")

if self.network.name != LOCAL_NETWORK_NAME and (stop_nonce - start_nonce) > 2:
# NOTE: RPC usage might be acceptable to find 1 or 2 transactions reasonably quickly
logger.warning(
"Performing this action is likely to be very slow and may "
f"use {20 * (stop_nonce - start_nonce)} or more RPC calls. "
"Consider installing an alternative data query provider plugin."
)

yield from self._find_txn_by_account_and_nonce(
account,
start_nonce,
stop_nonce,
0, # first block
self.chain_manager.blocks.head.number or 0, # last block (or 0 if genesis-only chain)
)

def _find_txn_by_account_and_nonce(
self,
account: AddressType,
start_nonce: int,
stop_nonce: int,
start_block: int,
stop_block: int,
) -> Iterator[ReceiptAPI]:
# binary search between `start_block` and `stop_block` to yield txns from account,
# ordered from `start_nonce` to `stop_nonce`

if start_block == stop_block:
# Honed in on one block where there's a delta in nonce, so must be the right block
for txn in self.get_transactions_by_block(stop_block):
assert isinstance(txn.nonce, int) # NOTE: just satisfying mypy here
if txn.sender == account and txn.nonce >= start_nonce:
yield self.get_receipt(txn.txn_hash.hex())

# Nothing else to search for

else:
# Break up into smaller chunks
# NOTE: biased to `stop_block`
block_number = start_block + (stop_block - start_block) // 2 + 1
txn_count_prev_to_block = self.web3.eth.get_transaction_count(account, block_number - 1)

if start_nonce < txn_count_prev_to_block:
yield from self._find_txn_by_account_and_nonce(
account,
start_nonce,
min(txn_count_prev_to_block - 1, stop_nonce), # NOTE: In case >1 txn in block
start_block,
block_number - 1,
)

if txn_count_prev_to_block <= stop_nonce:
yield from self._find_txn_by_account_and_nonce(
account,
max(start_nonce, txn_count_prev_to_block), # NOTE: In case >1 txn in block
stop_nonce,
block_number,
stop_block,
)

def block_ranges(self, start=0, stop=None, page=None):
if stop is None:
stop = self.chain_manager.blocks.height
Expand Down
Loading