Merge pull request #348 from subsquid/misc-fixes-by-abernatskiy-dec22

Misc fixes by abernatskiy dec22

docs/cloud/resources/env-variables.md

@@ -63,6 +63,10 @@ To add a secret:
    ```bash
    sqd secrets set MOONRIVER_GRPC_ENDPOINT wss://moonriver.my-endpoint.com/ws/my-secret-key
    ```
+   If you do not specify the value, `sqd` will attempt to read it from standard input. This is useful when setting a value to the contents of some file:
+   ```bash
+   sqd secrets set MY_JSON_CREDENTIALS < creds.json
+   ```
 2. At your squid's [manifest](/cloud/reference/manifest), add an environment variable and assign it to the secret:
    ```yaml
    deploy:

docs/sdk/examples.mdx

@@ -70,7 +70,7 @@ export const USDC_CONTRACT_ADDRESS = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48
 export const processor = new EvmBatchProcessor()
   .setDataSource({
     archive: lookupArchive('eth-mainnet'),
-    chain: 'https://rpc.ankr.com/eth',
+    chain: '<eth_rpc_endpoint_url>',
   })
   .setFinalityConfirmation(75)
   .addLog({

docs/sdk/reference/processors/evm-batch/logs.md

@@ -86,7 +86,7 @@ Typescript ABI modules generated by [`squid-evm-typegen`[(/sdk/reference/typegen
 const processor = new EvmBatchProcessor()
   .setDataSource({
     archive: lookupArchive('eth-mainnet'),
-    chain: 'https://rpc.ankr.com/eth'
+    chain: '<eth_rpc_endpoint_url>'
   })
   .setFinalityConfirmation(75)
   .addLog({

docs/sdk/reference/processors/evm-batch/traces.md

@@ -65,7 +65,7 @@ const METHOD_SIGHASH = '0x88316456' // mint
 const processor = new EvmBatchProcessor()
   .setDataSource({
     archive: lookupArchive('eth-mainnet'),
-    chain: 'https://rpc.ankr.com/eth'
+    chain: '<eth_rpc_endpoint_url>'
   })
   .setFinalityConfirmation(75)
   .setBlockRange({ from: 16962349, to: 16962349 })

docs/sdk/reference/processors/substrate-batch/data-requests.md

@@ -14,6 +14,10 @@ Setting a data request field to `[]` _selects no data_. Pass `undefined` for a w
 ```
 :::
 
+:::tip
+All array-valued data request fields (e.g. `name?: string[]` of `addEvent()` are _arrays_ of acceptable values. That means that you can request e.g. multiple events with a single `addEvent()` call. Requesting data that way is more efficient than requesting data with many individual calls.
+:::
+
 ## Events
 
 **`addEvent(options)`**: Subscribe to Substrate runtime events. `options` has the following structure:

docs/sdk/reference/store/typeorm.md

@@ -202,7 +202,7 @@ Database credentials must be supplied via the environment variables:
 * `DB_URL` (default `undefined`)
 
 :::info
-To take advantage of `DB_SSL_REJECT_UNAUTHORIZED`, `DB_URL` and the client-side SSL variables use the `@next` version of the packages:
+To take advantage of `DB_SSL_REJECT_UNAUTHORIZED`, `DB_URL` and the client-side SSL variables use the `@next` version of `typeorm-store`:
 ```bash
 npm i @subsquid/typeorm-store@next
 ```
@@ -217,6 +217,13 @@ npm i @subsquid/typeorm-store@next
 * `DB_SSL_KEY` - client key in plain text
 * `DB_SSL_KEY_FILE` - path to client key in plain text
 
+:::tip
+In case you're deploying to [Cloud](/cloud) you can set [secrets](/cloud/resources/env-variables/#secrets) to the contents of any given file via stdin:
+```bash
+sqd secrets set DB_SSL_CA < ca.crt
+```
+:::
+
 ### `DB_URL`
 
 When set, `DB_URL` takes precedence over individual variables. Its format is as follows:

docs/sdk/resources/evm/factory-contracts.md

@@ -18,7 +18,7 @@ Let's consider how it works in a DEX example, with a contract emitting `PoolCrea
 export const processor = new EvmBatchProcessor()
   .setDataSource({
     archive: 'https://v2.archive.subsquid.io/network/ethereum-mainnet',
-    chain: 'https://rpc.ankr.com/eth',
+    chain: '<eth_rpc_endpoint_url>',
   })
   .setBlockRange({
     from: 12_369_621,
@@ -84,7 +84,7 @@ const { preloadHeight, preloadedPools } = loadPools() // e.g. from a filesystem
 export const processor = new EvmBatchProcessor()
   .setDataSource({
     archive: 'https://v2.archive.subsquid.io/network/ethereum-mainnet',
-    chain: 'https://rpc.ankr.com/eth',
+    chain: '<eth_rpc_endpoint_url>',
   })
   .setBlockRange({
     from: 12_369_621,

docs/sdk/resources/migrate/migrate-subgraph.md

@@ -108,7 +108,7 @@ export const processor = new EvmBatchProcessor()
     // lookupArchive('polygon')
     // lookupArchive('binance')
     archive: lookupArchive('eth-mainnet'),
-    chain: 'https://rpc.ankr.com/eth'
+    chain: '<eth_rpc_endpoint_url>'
   })
   .setBlockRange({ from: 6175243 })
   .setFinalityConfirmation(75)

docs/sdk/resources/persisting-data/bigquery.md

@@ -42,7 +42,7 @@ processor.run(db, async ctx => {
 })
 ```
 Here,
- * `bq` is a `BigQuery` instance. When created without arguments like this it'll look at the `GOOGLE_APPLICATION_CREDENTIALS` environment variable for a path to a JSON with authentication details. When providing such a file is undesirable for any reason the credentials may be supplied to the constructor. An example is available at the [Deploying to Subsquid Cloud](#deploying-to-subsquid-cloud) section.
+ * `bq` is a `BigQuery` instance. When created without arguments like this it'll look at the `GOOGLE_APPLICATION_CREDENTIALS` environment variable for a path to a JSON with authentication details.
  * `dataset` is the path to the target dataset.
 :::warning
 The dataset must be created prior to running the processor.
@@ -53,52 +53,24 @@ Tables are made out of statically typed columns. Available types are listed on t
 
 ## Deploying to Subsquid Cloud
 
-We discourage uploading any sensitive data with squid code when [deploying](/cloud) to Subsquid Cloud. For BigQuery squids that means that accessing a [credentials JSON](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) is typically not possible. The solution is to use [Cloud secrets](/cloud/resources/env-variables/#secrets) to store credentials.
-
-Assuming that the name of the project is `subsquid-datasets` as in the example above and not sensitive, you can initialize your `BigQuery` instance like this:
-```ts title="src/main.ts"
-...
-import {assertNotNull} from `@subsquid/util-internal`
-
-const client_id = assertNotNull(process.env.GOOGLE_CLIENT_ID)
-const client_email = assertNotNull(process.env.GOOGLE_CLIENT_EMAIL)
-const private_key = assertNotNull(
-  process.env.GOOGLE_PRIVATE_KEY
-).replace(/\\n/g, '\n')
-
-const bq = new BigQuery({
-  projectId: 'subsquid-datasets',
-  credentials: {client_id, client_email, private_key}
-})
+We discourage uploading any sensitive data with squid code when [deploying](/cloud) to Subsquid Cloud. To pass your [credentials JSON](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) to your squid, create a [Cloud secret](/cloud/resources/env-variables/#secrets) variable populated with its contents:
+```bash
+sqd secrets set GAC_JSON_FILE < creds.json
+```
+Then in `src/main.ts` write the contents to a file:
+```ts title=src/main.ts
+import fs from 'fs'
 
-const db = new Database({
-  bq,
-...
+fs.writeFileSync('creds.json', process.env.GAC_JSON_FILE || '')
 ```
-Request that the required secret variables are visible from your squid in [`squid.yaml`](/cloud/reference/manifest):
+Set the `GOOGLE_APPLICATION_CREDENTIALS` variable and request the secret in the [deployment manifest](/cloud/reference/manifest):
 ```yaml title="squid.yaml"
-...
 deploy:
-  env:
-    GOOGLE_CLIENT_ID: ${{ secrets.GOOGLE_CLIENT_ID }}
-    GOOGLE_CLIENT_EMAIL: ${{ secrets.GOOGLE_CLIENT_EMAIL }}
-    GOOGLE_PRIVATE_KEY: ${{ secrets.GOOGLE_PRIVATE_KEY }}
-...
+  processor:
+    env:
+      GAC_JSON_FILE: ${{ secrets.GAC_JSON_FILE }}
+      GOOGLE_APPLICATION_CREDENTIALS: creds.json
 ```
-Finally, set the variables with the [sqd secrets](/squid-cli/secrets) command, taking the values from the credentials JSON:
-```bash
-sqd secrets:set GOOGLE_CLIENT_ID <your_client_id>
-sqd secrets:set GOOGLE_CLIENT_EMAIL <your_client_email>
-sqd secrets:set GOOGLE_PRIVATE_KEY <your_private_key>
-```
-:::warning
-Escape the newline symbol with four backslashes when setting the private key:
-```bash
-sqd secrets:set GOOGLE_PRIVATE_KEY "-----BEGIN PRIVATE KEY-----\\\\n...
-```
-:::
-
-At this point your squid will be able to authenticate with Google Cloud once deployed.
 
 ## Examples
 

docs/sdk/tutorials/bayc/step-one-indexing-transfers.md

@@ -53,7 +53,7 @@ export const processor = new EvmBatchProcessor()
     .setDataSource({
         archive: lookupArchive('eth-mainnet'),
         chain: {
-            url: 'https://rpc.ankr.com/eth',
+            url: '<eth_rpc_endpoint_url>',
             rateLimit: 10
         }
     })
@@ -77,7 +77,7 @@ export const processor = new EvmBatchProcessor()
 
 Here,
 * `'eth-mainnet'` is the alias for the public archive that Subsquid maintains for Ethereum mainnet. Check out `npx squid-archive-registry` for a list of public archives for all supported networks or explore the [archives documentation](/subsquid-network/overview/) to find out how to host your own archive.
-* `'https://rpc.ankr.com/eth'` is a public RPC endpoint we chose to use in this example. When an endpoint is available, the processor will begin ingesting data from it once it reaches the highest block available within the archive. Please use a private endpoint or Subsquid Cloud's [RPC proxy service](/cloud/reference/rpc-proxy) in production.
+* `'<eth_rpc_endpoint_url>'` is a public RPC endpoint we chose to use in this example. When an endpoint is available, the processor will begin ingesting data from it once it reaches the highest block available within the archive. Please use a private endpoint or Subsquid Cloud's [RPC proxy service](/cloud/reference/rpc-proxy) in production.
 * `setFinalityConfirmation(75)` call instructs the processor to consider blocks final after 75 confirmations when ingesting data from an RPC endpoint.
 * `12_287_507` is the block at which the BAYC token contract was deployed. Can be found on the [contract's Etherscan page](https://etherscan.io/address/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d).
 * The argument of [`addLog()`](/sdk/reference/processors/evm-batch/logs) is a set of filters that tells the processor to retrieve all event logs emitted by the BAYC contract with topic0 matching the hash of the full signature of the `Transfer` event. The hash is taken from the previously generated Typescript ABI.

docs/sdk/tutorials/file-csv.md

@@ -150,7 +150,7 @@ DB_NAME=squid
 DB_PORT=23798
 GQL_PORT=4350
 # JSON-RPC node endpoint, both wss and https endpoints are accepted
-RPC_ENDPOINT="https://rpc.ankr.com/eth"
+RPC_ENDPOINT="<eth_rpc_endpoint_url>"
 ```
 :::
 

docs/sdk/tutorials/file-parquet.md

@@ -290,7 +290,7 @@ sqd process
 The indexer should be able to catch up with the Ethereum blockchain, and **reach the chain's head in a very short time**. 
 
 :::info
-Bear in mind that this may vary a lot, depending on the Ethereum node used ([Ankr public node](https://rpc.ankr.com/eth) in our case) and on your hardware, as well as the connection, or physical distance from the Ethereum node. It took ~45 minutes while testing for this article. A test on a connection with a much higher latency and the same configuration finished indexing in 5 hours (figures out of date).
+Bear in mind that this may vary a lot, depending on the Ethereum node used and on your hardware, as well as the connection, or physical distance from the Ethereum node. It took ~45 minutes while testing for this article. A test on a connection with a much higher latency and the same configuration finished indexing in 5 hours (figures out of date).
 :::
 
 The process will generate a series of sub-folders in the `data` folder, labelled after the block ranges where the data is coming from, and in each one of these folders there should be one `*.parquet` file for each of the *tables* we defined.
@@ -308,7 +308,7 @@ The purpose of this tutorial was to demonstrate how to use the Subsquid indexing
 The project described here was able to index the entirety of Uniswap Pool events, across all the pools created by the Factory contract, as well as the Positions held by investors, in less than an hour (~45 minutes) (figure out of date).
 
 :::info
-**Note:** Indexing time may vary, depending on factors, such as the Ethereum node used ([Ankr public node](https://rpc.ankr.com/eth) in our case), on the hardware, and quality of the connection.
+**Note:** Indexing time may vary, depending on factors, such as the Ethereum node used, on the hardware, and quality of the connection.
 :::
 
 The simple Python script in the project's repository shows how to read multiple Parquet files, and perform some data analysis with Pandas.

docs/squid-cli/secrets.md

@@ -41,7 +41,7 @@ _See code: [src/commands/secrets/rm.ts](https://github.com/subsquid/squid-cli/tr
 
 ## `sqd secrets:set NAME VALUE`
 
-Create or update a secret. The secret will be exposed as an environment variable with the given name to all the squids. Note the changes take affect only after a squid is restarted or updated.
+Add or update a secret in the Cloud. If value is not specified, it is read from standard input. The secret will be exposed as an environment variable with the given name to all the squids in the organization. NOTE: The changes take affect only after a squid is restarted or updated.
 
 ```
 USAGE

docs/subsquid-network/reference/evm-networks.mdx

@@ -84,7 +84,7 @@ Explicit Archive endpoints:
 ```typescript
 const processor = new EvmBatchProcessor()
   .setDataSource({
-     chain: 'https://rpc.ankr.com/eth', // RPC endpoint
+     chain: '<eth_rpc_endpoint_url>', // RPC endpoint
      archive: 'https://v2.archive.subsquid.io/network/ethereum-mainnet'
   })
 ```

src/_mock/code-slider-data.tsx

@@ -34,7 +34,7 @@ export default [
         'export const processor = new EvmBatchProcessor()\n' +
         '  .setDataSource({\n' +
         '    archive: lookupArchive(\'eth-mainnet\'),\n' +
-        '    chain: \'https://rpc.ankr.com/eth\',\n' +
+        '    chain: \'<eth_rpc_endpoint_url>\',\n' +
         '  })\n' +
         '  .setFinalityConfirmation(75)\n' +
         '  .addLog({\n' +