Merge pull request #347 from subsquid/misc-fixes-by-abernatskiy-dec18

Misc fixes by abernatskiy dec18

docs/cloud/reference/manifest.mdx

@@ -10,34 +10,45 @@ The deployment manifest is named `squid.yaml` by convention. The manifest define
 
 With the introduction of the deployment manifest the add-ons (`postgres`) and the API services (`api`) become optional. This allows flexible deployment configuration for analytic use-cases (e.g. transform and write to s3 or BigQuery).
 
+:::tip
+Cloud [secrets](/cloud/resources/env-variables/#secrets) of the squid's [organization](/cloud/resources/organizations) are exposed as the `secrets` namespace. You can set any variable defined in any of the `env:` sections mentioned below:
+```yaml
+env:
+  RPC_ENDPOINT: ${{ secrets.FAST_RPC_ENDPOINT_URL }}
+```
+:::
+
 ## Header
 
 The manifest header defines the squid metadata
 
 | Field                    |      Description                         |
 |:-----------------------:|:---------------------------------------------:|
-| `manifestVersion`        | Only `subsquid.io/v0.1` is currently supported     |
+| `manifest_version`        | Only `subsquid.io/v0.1` is currently supported     |
 | `name` | A globally unique squid name. Can only contain lowercase Latin letters (`a-z`) and a dash (`-`). Must be under 20 symbols. |
 | `version`                 | Squid version. **Must be an integer**. A squid deployment is canonically identified as `${name}@${version}`.  |
 | `description`  | (Optional) A short description of the squid  |
 
 ## `build:`
 
-Specifies the way the squid is built into a docker image. Only an empty `build:` section is supported, and the squid must adhere
-to the default [folder structure](/sdk/how-to-start/layout).
+Specifies the way the squid is built into a docker image.
 
-In particular, the following files and folders **must** be present in the root folder of the squid:
+| Name              | Type                          | Default value         |
+|:-----------------:|:-----------------------------:|----------------------:|
+| `node_version`    | `18 \| 20 \| 21`              | `20`                  |
+| `package_manager` | `auto \| npm \| pnpm \| yarn` | `auto`                |
+
+For a successful build the following files and folders **must** be present in the root folder of the squid:
 
 - `/src`
-- `schema.graphql`
 - `tsconfig.json`
-- `package-lock.json`
 - `package.json`
+- `commands.json`
 
-The `db` and `assets` folders are added to the build context if present in the squid folder.
+The `db` and `assets` folders are added to the build context if present in the squid folder. See [Project structure](/sdk/how-to-start/layout) for more info.
 
 Under the hood, Cloud builds a Docker image and runs a docker container for each service (`api`, `processor`, `migrate`) using the same image.
-See [Troubleshooting](/cloud/troubleshooting/#local-build) for instructions on how to build and run the Docker image locally.
+See [Self-hosting](/sdk/resources/basics/self-hosting) for instructions on how to build and run the Docker image locally.
 Even though the squid services (`api`, `processor`, `migrate`) use the same single container image, the exec command is different and can is defined by the `deploy:` section as explained below.
 
 ## `deploy:`
@@ -79,15 +90,27 @@ A processor service or a list of processor services of the squid.
 With a single processor this section may look like this:
 ```yaml
   processor:
-    cmd: [ "node", "lib/main" ]
+    cmd: [ "sqd", "process:prod" ]
 ```
 For the multiprocessor case:
 ```yaml
   processor:
     - name: eth-processor
-      cmd: [ "node", "lib/eth/main" ]
+      cmd: [ "sqd", "process:prod:eth" ]
     - name: bsc-processor
-      cmd: [ "node", "lib/bsc/main" ]
+      cmd: [ "sqd", "process:prod:bsc" ]
+```
+where `process:prod:bsc` and `process:prod:eth` are extra `sqd` commands defined at `commands.json`:
+```json title="commands.json"
+...
+  "process:prod:eth": {
+    "deps": ["migration:apply"],
+    "cmd": ["node", "lib/eth/main.js"]
+   },
+  "process:prod:bsc": {
+     "cmd": ["node", "lib/bsc/main.js"]
+   },
+...
 ```
 
 ### `api:`
@@ -103,10 +126,6 @@ For the multiprocessor case:
 
 **Optional** A key-value list of deployment-wide (i.e. visible to all services of the squid) env variables to be set
 
-### (deprecated) `secrets:` {#secrets}
-
-A list of secret names that must be added in the [Cloud app](https://app.subsquid.io). If any of the secrets is not defined, the deployment will fail.
-
 ## `scale:`
 
 See the [Scale the deployment](/cloud/reference/scale) section.
@@ -116,7 +135,7 @@ See the [Scale the deployment](/cloud/reference/scale) section.
 A minimal example of manifest is below:
 
 ```yaml title="squid.yaml"
-manifestVersion: subsquid.io/v0.1
+manifest_version: subsquid.io/v0.1
 name: sample-squid
 version: 1
 description: |-
@@ -128,15 +147,15 @@ deploy:
   addons:
     postgres: 
   processor:
-    cmd: [ "node", "lib/processor" ]
+    cmd: [ "sqd", "process:prod" ]
   api:
-    cmd: [ "npx", "squid-graphql-server" ]
+    cmd: [ "sqd", "serve:prod" ]
 ```
 
 An extended version:
 
 ```yaml title="squid.yaml"
-manifestVersion: subsquid.io/v0.1
+manifest_version: subsquid.io/v0.1
 name: sample-squid
 version: 1
 description: |-
@@ -147,42 +166,32 @@ build:
 deploy:
   addons:
     postgres: 
-  # the set of secrets that must be set and provided by Cloud
-  secrets:
-    - ACALA_RPC_ENDPOINT
-    - COINGECKO_API_KEY
   migrate:
     env:
-      FOO: 
-        bar
-    cmd: ["echo", "skip migrations!" ]
+      FOO: bar
+    cmd: [ "echo", "skip migrations!" ]
   processor:
-    # additional env variables
+    # static and secret-valued env variables
     env:
-      SQD_DEBUG: 
-        sqd:mapping
-    cmd: [ "node", "lib/processor" ]
+      SQD_DEBUG: sqd:mapping
+      RPC_ENDPOINT: ${{ secrets.ACALA_RPC_ENDPOINT }}
+      COINGECKO_API_KEY: ${{ secrets.COINGECKO_API_KEY }}
+    cmd: [ "sqd", "process:prod" ]
   api:
     env:
-      SQD_DEBUG: 
-        '*'
+      SQD_DEBUG: '*'
     # custom run command for the GraphQL server
     cmd: [ "npx", "squid-graphql-server", "--subscriptions", "--max-root-fields", "10", "--sql-statement-timeout", "1000" ]
 
 scale:
   addons:
-     postgres:
-         storage: 100G
-         profile: medium
+    postgres:
+      storage: 100G
+      profile: medium
   processor:
-     profile: medium
+    profile: medium
   api:
-     profile: large
-     # load-balance three replicas
-     replicas: 3
+    profile: large
+    # load-balance three replicas
+    replicas: 3
 ```
-
-
-
-
-

docs/cloud/reference/pg.md

@@ -53,7 +53,7 @@ The profile specifications for a `postgres` service are as follows:
 ## Examples
 
 ```yaml
-manifestVersion: subsquid.io/v0.1
+manifest_version: subsquid.io/v0.1
 name: sample-squid
 version: 1
 description: |-
@@ -68,9 +68,9 @@ deploy:
         statement_timeout: 100000
         log_min_duration_statement: 100000
   processor:
-    cmd: [ "node", "lib/processor" ] 
+    cmd: [ "sqd", "process:prod" ]
   api:
-    cmd: [ "npx", "squid-graphql-server"]
+    cmd: [ "sqd", "serve:prod" ]
 
 scale:
   addons:

docs/cloud/reference/scale.md

@@ -60,7 +60,7 @@ The profile specifications for a processor service are as follows:
 ## Example
 
 ```yaml title="squid.yaml"
-manifestVersion: subsquid.io/v0.1
+manifest_version: subsquid.io/v0.1
 name: sample-squid
 
 build: 
@@ -69,20 +69,20 @@ deploy:
   addons:
     postgres: 
   processor:
-    cmd: [ "node", "lib/processor" ] 
+    cmd: [ "sqd", "process:prod" ]
   api:
-    cmd: [ "npx", "squid-graphql-server"]
+    cmd: [ "sqd", "serve:prod" ]
 
 scale:
   dedicated: true
   addons:
-     postgres:
-         storage: 100G
-         profile: medium
+    postgres:
+      storage: 100G
+      profile: medium
   processor:
-     profile: medium
+    profile: medium
   api:
-     profile: large
-     # load-balance three replicas
-     replicas: 3
+    profile: large
+    # load-balance three replicas
+    replicas: 3
 ```

docs/cloud/resources/migration.md

@@ -26,7 +26,7 @@ sqd --version
 Create `squid.yaml` in the squid root folder, and set the squid name, version and the description (optional). Note that the squid version must be an integer, so start with `1`.
 
 ```yml file="squid.yaml
-manifestVersion: subsquid.io/v0.1
+manifest_version: subsquid.io/v0.1
 name: my-squid # set name
 version: 1 # version, must be an integer
 description: |-  # set description
@@ -38,9 +38,9 @@ deploy:
   addons:
     postgres: 
   processor:
-    cmd: [ "node", "lib/processor" ] 
+    cmd: [ "sqd", "process:prod" ]
   api:
-    cmd: [ "npx", "squid-graphql-server"]
+    cmd: [ "sqd", "process:serve" ]
 
 ```
 
@@ -51,9 +51,8 @@ If the squid expects additional environment variables to be set by the Cloud via
 ```yml
 #...
 deploy:
-  secrets:
-    - ETHEREUM_RPC_ENDPOINT
-
+  env:
+    ETHEREUM_RPC_ENDPOINT: ${{ secrets.ETHEREUM_RPC_ENDPOINT }}
 ```
 
 ## 3. (Optional) Revise `cmd`

docs/cloud/resources/organizations.md

@@ -37,7 +37,7 @@ Once an organization is upgraded you can deploy as many squids as you requre. [`
 
 ## Working with organizations
 
-When your account has access to more than one organization, it is necessary to specify one when [listing](/squid-cli/ls), [exploring](/squid-cli/explorer) or [deploying](/squid-cli/deploy) (with some exceptions) your squids, as well as when [setting environment variables/secrets](/squid-cli/secrets). Do it with the `--org/-o` flag:
+When your account has access to more than one organization, it is necessary to specify one when [listing](/squid-cli/ls), [exploring](/squid-cli/explorer) or [deploying](/squid-cli/deploy) (with some exceptions) your squids, as well as when [setting secrets](/squid-cli/secrets). Do it with the `--org/-o` flag:
 
 ```bash
 sqd secrets ls -o my-organization

docs/cloud/troubleshooting.md

@@ -6,7 +6,7 @@ sidebar_position: 60
 
 ### "Secrets outdated. Please restart the squid" warning
 
-This occurs when you have a squid deployed, then create, remove or change some [environment variables](/squid-cli/secrets) of [relevance](/cloud/resources/organizations). Squids must be restarted manually for such changes to have effect. Navigate to the squid version page (e.g. by clicking on the warning sign) and click restart. The restart will not touch the database, so unless your new secret values cause the squid to crash this procedure should be quick and easy.
+This occurs when you have a squid deployed, then create, remove or change some [secrets](/squid-cli/secrets) of [relevance](/cloud/resources/organizations). Squids must be restarted manually for such changes to have effect. Navigate to the squid version page (e.g. by clicking on the warning sign) and click restart. The restart will not touch the database, so unless your new secret values cause the squid to crash this procedure should be quick and easy.
 
 ![Secrets outdated](</img/secrets-outdated.png>)
 
@@ -35,4 +35,4 @@ Edit the [postgres addon](/cloud/reference/pg) section of `squid.yaml` and reque
 
 ### My squid is behind the chain, but is shows that it is in sync
 
-Check that your processor uses both `archive:` and `chain:` data sources. 
+Check that your processor uses both `archive:` and `chain:` data sources. 

docs/sdk/resources/basics/multichain.md

@@ -24,17 +24,30 @@ To do this, run one [processor](/sdk/overview) per source network:
   Alternatively, parameterize your processor using environment variables: you can [set these on a per-processor basis](/cloud/reference/manifest/#processor) if you use a deployment manifest to run your squid.
 
 2. Arrange for running the processors alongside each other conveniently:
+   - Add `sqd` commands for running each processor to `commands.json`, e.g.
+     ```json title="commands.json"
+     ...
+       "process:eth": {
+         "deps": ["build", "migration:apply"],
+         "cmd": ["node", "lib/eth/main.js"]
+       },
+       "process:bsc": {
+         "deps": ["build", "migration:apply"],
+         "cmd": ["node", "lib/bsc/main.js"]
+       },
+     ...
+     ```
+     [Full example](https://github.com/subsquid-labs/multichain-transfers-example/blob/master/commands.json)
    - If you are going to use [`sqd run`](/squid-cli/run) for local runs or deploy your squid to [Subsquid Cloud](/cloud), list your processors at the `deploy.processor` section of your [deployment manifest](/cloud/reference/manifest/#processor):
      ```yaml
      deploy:
        processor:
          - name: eth-processor
-           cmd: [ "node", "lib/eth/main" ]
+           cmd: [ "sqd", "process:prod:eth" ]
          - name: bsc-processor
-           cmd: [ "node", "lib/bsc/main" ]
+           cmd: [ "sqd", "process:prod:bsc" ]
      ```
      Make sure to give each processor a unique name!
-   - Optionally, add `sqd` commands for running each processor to `commands.json`. [Example](https://github.com/subsquid-labs/multichain-transfers-example/blob/master/commands.json)
 
 ## On Postgres
 

docs/sdk/resources/graphql-server/caching.md

@@ -6,20 +6,30 @@ description: Enable caching for faster queries
 
 # Caching 
 
-**Available since `@subsquid/[email protected]`**
-
 The GraphQL API server provided by `@subsquid/graphql-server` supports caching via additional flags. It is done on a per-query basis. The whole response is cached for a specified amount of time (`maxAge`).
 
-To enable caching when deploying to Subsquid Cloud, add the caching flags to `cmd` sections of the [deployment manifest](/cloud/reference/manifest/#deploy). Cloud currently supports only in-memory cache, with Redis-based cache to be supported in the near future.
-For example, the snippet below will deploy a GraphQL API server with a `100Mb` in-memory cache and invalidation time of `5` seconds:
-
+To enable caching when deploying to Subsquid Cloud, add the caching flags to the `serve:prod` command definition at `commands.json`, then use that command to run the server in the [deployment manifest](/cloud/reference/manifest/#deploy). Cloud currently supports only in-memory cache.
+For example, snippets below will deploy a GraphQL API server with a `100Mb` in-memory cache and invalidation time of `5` seconds:
+
+```json title="commands.json"
+...
+  "serve:prod": {
+    "description": "Start the GraphQL API server with caching and limits",
+    "cmd": [ "squid-graphql-server",
+      "--dumb-cache", "in-memory",
+      "--dumb-cache-ttl", "5000",
+      "--dumb-cache-size", "100",
+      "--dumb-cache-max-age", "5000" ]
+  }
+...
+```
 
 ```yaml title="squid.yaml"
 # ...
 deploy:
   # other services ...
   api:
-    cmd: [ "npx", "squid-graphql-server", "--dumb-cache", "in-memory", "--dumb-cache-ttl", "5000", "--dumb-cache-size", "100", "--dumb-cache-max-age", "5000" ]
+    cmd: [ "sqd", "serve:prod" ]
 ```
 
 Caching flags list is available via `npx squid-graphql-server --help`. Here are some more details on them: