Reth-crawler

Reth crawler MVP! Related to this issue.

A sample of the peer data:

{

    {
        "enode_url":"enode://0bb46e5532da93328bf6964309081f9fef7a19f29d91f93004f82ecd897766a15348f6863c4f36541f632d50c7d592c6b9181e439e9ede16c7e0cd18612cd048@158.220.96.114:34310","id":"0x0bb46e5532da93328bf6964309081f9fef7a19f29d91f93004f82ecd897766a15348f6863c4f36541f632d50c7d592c6b9181e439e9ede16c7e0cd18612cd048","address":"158.220.96.114","tcp_port":34310,"client_version":"Geth/v1.13.4-stable-3f907d6a/linux-amd64/go1.21.3","eth_version":68,"capabilities":["eth/67","eth/68","snap/1"],"chain":"mainnet","total_difficulty":"17179869184","best_block":"0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3","genesis_block_hash":"0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3","last_seen":"2023-11-08 15:41:04.469473084 UTC","country":"Germany","city":"Düsseldorf","synced":null,"isp":"Contabo GmbH"
    }
}

How to build the crawler

reth-crawler is stable now and it's in active development. In order to try it, follow these steps:

Clone the repository

git clone https://github.com/Keep-Reth-Strange/reth-crawler

Run it

Open the directory where it's been installed previously and build it:

cargo build -p reth-crawler

Go to target/debug folder:

cd target/debug

Run it:

./reth-crawler crawl

Run it locally without a centralized db

For local testing there is a flag to save peers in a peers_data.db file (sqlite db):

./reth-crawler crawl --local-db

Node endpoint

Note that the crawl command assumes that a local node is running at wss://localhost:8546. If that is not the case, you will need to pass a node websocket url as argument, for eg.

./reth-crawler crawl --local-db --eth-rpc-url wss://ethereum.publicnode.com

The API server

The API server is not necessary for crawling the network but it's very useful to serve the crawled data.

Right now it's configured by default to connect to the dynamo db database used by the crawler, so if you have not set up an Amazon db before, it will not run.

We are now working on migrate our infra from Amazon to a new service provider and we'll also let the API server be more flexible for local testing.

How to run the API server

Open the directory where it's been installed previously and build it:

cargo build -p reth-crawler-api-server

Go to target/debug folder:

cd target/debug

Run it:

./reth-crawler-api-server start-api-server