Merge pull request #342 from subsquid/sorted-layout

Sorted layout

correspondences.log

@@ -0,0 +1,26 @@
+sdk/resources/subsquid-vs-thegraph.md -> sdk/subsquid-vs-thegraph.md
+sdk/reference/processors/architecture.mdx -> sdk/resources/processor/architecture.mdx
+sdk/resources/evm -> sdk/resources/processor/evm
+sdk/resources/evm/factory-contracts.md -> sdk/resources/processor/evm/factory-contracts.md
+sdk/resources/evm/proxy-contracts.md -> sdk/resources/processor/evm/proxy-contracts.md
+sdk/resources/substrate -> sdk/resources/processor/substrate
+sdk/resources/substrate/data-sourcing-miniguide.md -> sdk/resources/processor/substrate/data-sourcing-miniguide.md
+sdk/resources/substrate/frontier-evm.md -> sdk/resources/processor/substrate/frontier-evm.md
+sdk/resources/substrate/gear.md -> sdk/resources/processor/substrate/gear.md
+sdk/resources/substrate/ink.md -> sdk/resources/processor/substrate/ink.md
+sdk/resources/persisting-data -> sdk/resources/store
+sdk/resources/persisting-data/bigquery.md -> sdk/resources/store/bigquery.md
+sdk/resources/persisting-data/file.md -> sdk/resources/store/file.md
+sdk/resources/persisting-data/overview.md -> sdk/resources/store/overview.md
+sdk/resources/persisting-data/typeorm.md -> sdk/resources/store/typeorm.md
+sdk/resources/unfinalized-blocks.mdx -> sdk/resources/processor/unfinalized-blocks.mdx
+sdk/resources/multichain.md -> sdk/resources/integration/multichain.md
+sdk/resources/batch-processing.md -> sdk/resources/integration/batch-processing.md
+sdk/resources/squid-gen.md -> sdk/resources/integration/squid-gen.md
+sdk/resources/external-api.md -> sdk/resources/processor/external-api.md
+sdk/resources/self-hosting.md -> sdk/resources/integration/self-hosting.md
+sdk/resources/migrate -> sdk/resources/integration/migrate
+sdk/resources/migrate/migrate-subgraph.md -> sdk/resources/integration/migrate/migrate-subgraph.md
+sdk/resources/migrate/migrate-to-arrowsquid-on-substrate.md -> sdk/resources/integration/migrate/migrate-to-arrowsquid-on-substrate.md
+sdk/resources/migrate/migrate-to-arrowsquid.md -> sdk/resources/integration/migrate/migrate-to-arrowsquid.md
+sdk/troubleshooting/persisting-data.md -> sdk/troubleshooting/store.md

docs/cloud/reference/manifest.mdx

@@ -44,10 +44,6 @@ Even though the squid services (`api`, `processor`, `migrate`) use the same sing
 
 The `deploy` section may define:
 
-### `secrets:`
-
-A list of secret names that must be added in the [Cloud app](https://app.subsquid.io). If any of the secrets is not defined, the deployment will fail.
-
 ### `addons:`
 
 A list of add-on services to be deployed along the squid services.
@@ -103,6 +99,14 @@ For the multiprocessor case:
 | `cmd`       | GraphQL API server exec command passed to the squid container. The server must be listening at port `GRAPHQL_SERVER_PORT`. |  `string[]` |    |   **Required**     |
 | `env`       | A key-value list of env variables to be set                             |  `{k:v}[]`    |  []                   |   Optional         |
 
+### `env:`
+
+**Optional** A key-value list of deployment-wide (i.e. visible to all services of the squid) env variables to be set
+
+### (deprecated) `secrets:` {#secrets}
+
+A list of secret names that must be added in the [Cloud app](https://app.subsquid.io). If any of the secrets is not defined, the deployment will fail.
+
 ## `scale:`
 
 See the [Scale the deployment](/cloud/reference/scale) section.

docs/cloud/resources/best-practices.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 50
+sidebar_position: 10
 title: Best practices
 description: Checklist for going to production
 ---
@@ -8,7 +8,7 @@ description: Checklist for going to production
 
 Here is a list of items to check out before you deploy your squid for use in production:
 
-* If your squid [saves its data to a database](/sdk/resources/persisting-data/typeorm), make sure your [schema](/sdk/reference/schema-file) has [`@index` decorators](/sdk/reference/schema-file/indexes-and-constraints) for all entities that will be looked up frequently.
+* If your squid [saves its data to a database](/sdk/resources/store/typeorm), make sure your [schema](/sdk/reference/schema-file) has [`@index` decorators](/sdk/reference/schema-file/indexes-and-constraints) for all entities that will be looked up frequently.
 
 * If your squid serves a [GraphQL API](/sdk/reference/graphql-server), consider:
   1. configuring the built-in [DoS protection](/sdk/reference/graphql-server/dos-protection) against heavy queries;

docs/cloud/resources/env-variables.md

@@ -1,67 +1,91 @@
 ---
-sidebar_position: 80
+sidebar_position: 20
 title: Environment variables
 description: |- 
-  Add secrets and custom env variables
+  Variables, namespaces, secrets
 ---
 
-# Environment variables
+:::info
+Store all your sensitive inputs (API keys, passwords etc) as [secrets](#secrets).
+:::
+
+Subsquid Cloud supports adding environment variables to squid deployments. The variables can be defined as key-value pairs at any of the `env:` sections of the [manifest](/cloud/reference/manifest).
+
+For example, here is how to add a variable that will be visible only to the [squid processor](/cloud/reference/manifest/#processor):
+```yaml title="squid.yaml"
+deploy:
+  processor:
+    env:
+      MY_PROCESSOR_VAR: string_value
+```
+You can also add variables visible only to the [GraphQL server](/cloud/reference/manifest/#api) or to the [migration script](/cloud/reference/manifest/#migrate).
+
+There is also an option to add variables [for all services](/cloud/reference/manifest/#env):
+```yaml title="squid.yaml"
+deploy:
+  env:
+    MY_SQUIDWIDE_VAR: string_value
+```
+
+Variables can be assigned either to strings, or to member variables of [namespaces](#namespaces) provided by the service. For example, to make a processor-scoped `API_KEY` variable and populate it with the value of `secrets.API_KEY`, do this:
+```yaml title="squid.yaml"
+deploy:
+  processor:
+    env:
+      RPC_ENDPOINT: ${{ secrets.API_KEY }}
+```
 
-Subsquid Cloud supports adding environment variables to the squid deployments. There are two kinds: **secrets** and **environment variables**. The crucial difference is that the secrets are injected to all squids, while environments variables are set only to a specific version deployment. Environment variables are stored in plain text so **all the sensitive input (e.g. API keys) must be set as a secret**.
+## Database variables shadowing
 
-Secrets are set using the special command [`sqd secrets`](/squid-cli/secrets), while environment variables are set in the [deployment manifest](/cloud/reference/manifest).
+There is one special case in which the variables defined in the manifest will get overwritten by the Cloud: [database connection settings](/sdk/reference/store/typeorm/#database-connection-parameters) are shadowed by the system-defined values whenever the [`postgres` addon](/cloud/reference/pg) is enabled. For example, in the snippet below all the `DB_*` variable definitions will be ignored:
+```yaml title="squid.yaml"
+deploy:
+  addons:
+    postgres:
+  env:
+    DB_HOST: mypostgreshost.xyz
+    DB_PORT: 5432
+    DB_NAME: squid-tests
+    DB_USER: me
+    DB_PASS: ${{ secrets.DATABASE_PASSWORD }}
+    DB_SSL: true
+```
 
-## Secrets 
+## Namespaces
 
-Secrets are designed to store sensitive data that all the squids can access as an environment variable with the given name. A typical usage is an API key or a private URL for a chain gRPC endpoint. The secrets are defined at the Cloud [organization](/cloud/resources/organizations) level and are exposed to all organization squids that [request them in their manifest](/cloud/reference/manifest/#secrets).
+### Secrets
+
+Secrets are designed to store sensitive data, such as API keys or private URLs for chain RPC endpoints. They are defined at the [organization](/cloud/resources/organizations) level and are exposed to all organization squids that request them in their environment variable definitions.
 
 To add a secret:
-1. Create it in the Cloud:
+
+1. Create it in the Cloud. You can do it at the [secrets page](https://app.subsquid.io/secrets) or with [`sqd secrets`](/squid-cli/secrets):
    ```bash
    sqd secrets set MOONRIVER_GRPC_ENDPOINT wss://moonriver.my-endpoint.com/ws/my-secret-key
    ```
-2. Add it to the [manifest](/cloud/reference/manifest):
+2. At your squid's [manifest](/cloud/reference/manifest), add an environment variable and assign it to the secret:
    ```yaml
    deploy:
-     secrets:
-       - MOONRIVER_GRPC_ENDPOINT
+     env:
+       RPC_ENDPOINT: ${{ secrets.MOONRIVER_GRPC_ENDPOINT }}
    ```
-   Note: a deployment requesting a secret unknown to the Cloud will fail.
-3. Access in the squid with `process.env`:
+   Note: **a deployment requesting a secret unknown to the Cloud will fail**.
+3. Access the value in the squid with `process.env`, e.g.
    ```ts
    const processor = new EvmBatchProcessor()
-       .setDataSource({
-           archive: lookupArchive("moonriver", { type: 'EVM' }),
-           chain: {
-             url: process.env.MOONRIVER_GRPC_ENDPOINT,
-             rateLimit: 1000rps
-           }
-       })
+     .setDataSource({
+       archive: lookupArchive('moonriver', { type: 'EVM' }),
+       chain: {
+         url: process.env.RPC_ENDPOINT,
+         rateLimit: 1000rps
+       }
+     })
    ```
-
 Inspect, remove and update the secrets using the [`sqd secrets`](/squid-cli/secrets) command.
 
 :::info
-Any changes to secrets will take effect only when the squid is restarted with [`sqd deploy`](/squid-cli/deploy).
-:::
-
-## Environment variables
-
-Squid-specific environment variables should be defined in the [deployment manifest](/cloud/reference/manifest).
-
-**Example**
-
-```yaml title="squid.yaml"
-# ...
-deploy:
-  # ...
-  processor:
-    env:
-      SQD_DEBUG: 
-        sqd:mapping
-    cmd: [ "sqd", "process:prod" ]
-# ....
+Any changes to secrets will take effect only when the squid is restarted, e.g. with
+```bash
+sqd deploy .
 ```
-
-
-
+:::

docs/cloud/resources/logging.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 50
+sidebar_position: 30
 title: Inspect logs
 description: Inspect the deployment logs
 ---

docs/cloud/resources/monitoring.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 60
+sidebar_position: 40
 title: Monitoring
 description: Prometheus endpoints for squid services
 ---

docs/cloud/resources/organizations.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 15
+sidebar_position: 50
 title: Organizations
 description: |- 
   Grouping squids

docs/cloud/resources/production-alias.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 70
+sidebar_position: 60
 title: Production alias
 description: Switch between versions without downtime
 ---

docs/cloud/troubleshooting.md

@@ -22,7 +22,7 @@ npm update -g @subsquid/cli
 npm run update
 ```
 - Check that the squid adheres to the expected [structure](/sdk/reference/layout)
-- Make sure you can [build and run Docker images locally](/sdk/resources/self-hosting)
+- Make sure you can [build and run Docker images locally](/sdk/resources/integration/self-hosting)
 
 ### `Validation error` when releasing a squid
 

docs/overview.mdx

@@ -9,7 +9,7 @@ description: Subsquid Network and the products built around it
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
-The Subsuid ecosystem is built around [Subsquid Network](/subsquid-network) - a decentralized query engine and a horizontally-scalable data lake for batch queries. Currently, Subsquid Network serves raw historical on-chain data from [EVM](/subsquid-network/reference/evm-networks)- and [non-EVM](/subsquid-network/reference/substrate-networks) networks using a custom REST-like API. In the future, it will additionally support general-purpose SQL queries and an ever-growing collection of structured data sets derived from on- and off- chain data.
+The Subsquid ecosystem is built around [Subsquid Network](/subsquid-network) - a decentralized query engine and a horizontally-scalable data lake for batch queries. Currently, Subsquid Network serves raw historical on-chain data from [EVM](/subsquid-network/reference/evm-networks)- and [non-EVM](/subsquid-network/reference/substrate-networks) networks using a custom REST-like API. In the future, it will additionally support general-purpose SQL queries and an ever-growing collection of structured data sets derived from on- and off- chain data.
 
 ## Subsquid Network
 
@@ -52,7 +52,7 @@ For real-time use cases such as app-specific APIs use [Squid SDK](/sdk): it'll u
 - [Ergonomic tools](/sdk/reference/typegen) for decoding and normalizing raw data and efficiently accessing [network state](/sdk/reference/typegen/state-queries)
 - Pluggable [data sinks](/sdk/reference/store) to save data into Postgres, files (local or s3) or BigQuery
 - Expressive [GraphQL server](/sdk/reference/graphql-server) with a schema-based [config](/sdk/reference/schema-file)
-- Seamless handling of [unfinalized blocks and chain reorganizations](/sdk/resources/unfinalized-blocks) for real-time data ingestion
+- Seamless handling of [unfinalized blocks and chain reorganizations](/sdk/resources/processor/unfinalized-blocks) for real-time data ingestion
 - rapid data extraction and decoding [for local analytics](/sdk/tutorials/file-csv)
 
 The SDK is a go-to choice for production solutions and prototypes of
@@ -80,7 +80,7 @@ A Platform-as-a-Service for deploying Squid SDK indexers, featuring
 
 - Learn about [squid components](/sdk/overview), [combining them](/sdk/squid-from-scratch) or follow the [end-to-end development guide](/sdk/squid-development)
 - Explore [tutorials](/sdk/tutorials) or [examples](/sdk/examples)
-- Learn how to [migrate from The Graph](/sdk/resources/migrate/migrate-subgraph)
+- Learn how to [migrate from The Graph](/sdk/resources/integration/migrate/migrate-subgraph)
 - Explore the [GraphQL server options](/sdk/reference/graphql-server) including custom extensions, caching and DoS protection in production
 
 ```mdx-code-block

docs/sdk/examples.mdx

@@ -16,7 +16,7 @@ import TabItem from '@theme/TabItem';
 Browse a dedicated repository with [EVM squid examples](https://github.com/subsquid-labs/squid-evm-examples). It contains:
 
 - A [squid indexing ERC20 transfers](https://github.com/subsquid-labs/evm-logs-example) by extracting `Transfer(address,address,uint256)` logs emitted by the USDC contract.
-- A [squid indexing DEX trades](https://github.com/subsquid-labs/evm-factory-example) while dynamically tracking Uniswap v3 pool contracts as they are deployed. Shows [factory contract](/sdk/resources/evm/factory-contracts) indexing in action.
+- A [squid indexing DEX trades](https://github.com/subsquid-labs/evm-factory-example) while dynamically tracking Uniswap v3 pool contracts as they are deployed. Shows [factory contract](/sdk/resources/processor/evm/factory-contracts) indexing in action.
 - A [NFT indexing squid](https://github.com/subsquid-labs/evm-multicall-example) that additionally queries the NFT metadata from the contract state. Illustrates batching RPC calls using the [Multicall contract](/sdk/reference/typegen/generation/#batching-contract-state-calls-using-the-multicall-contract).
 - A [multichain squid](https://github.com/subsquid-labs/squid-multichain-template) tracking USDC `Transfer` events on both Ethereum and BSC.
 
@@ -62,7 +62,7 @@ processor.run(new TypeormDatabase(), async (ctx) => {
 <details>
 <summary>Index all USDC Transfer events on Ethereum with real time updates</summary>
 
-Real time data is fetched from a chain node RPC; a Database object with hot blocks support is required to store it (see [this page](/sdk/resources/unfinalized-blocks) for more details). [Full squid here](https://github.com/subsquid-labs/showcase01-all-usdc-transfers).
+Real time data is fetched from a chain node RPC; a Database object with hot blocks support is required to store it (see [this page](/sdk/resources/processor/unfinalized-blocks) for more details). [Full squid here](https://github.com/subsquid-labs/showcase01-all-usdc-transfers).
 
 ```ts
 export const USDC_CONTRACT_ADDRESS = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'

docs/sdk/faq.md

@@ -16,7 +16,7 @@ Here is an incomplete list:
 
 ### How does Squid SDK handle unfinalized blocks?
 
-The Subsquid Network only serves finalized blocks and is typically ~1000 blocks behind the tip. The most recent blocks, as well as the unfinalized blocks are seemlesly handled by the SDK from a complementary RPC data source, set by the `chain` config. Potential chain reorgs are automatically handled under the hood. See [Indexing unfinalized blocks](/sdk/resources/unfinalized-blocks) for details.
+The Subsquid Network only serves finalized blocks and is typically ~1000 blocks behind the tip. The most recent blocks, as well as the unfinalized blocks are seemlesly handled by the SDK from a complementary RPC data source, set by the `chain` config. Potential chain reorgs are automatically handled under the hood. See [Indexing unfinalized blocks](/sdk/resources/processor/unfinalized-blocks) for details.
 
 ### What is the latency for the data served by the squid? 
 
@@ -28,7 +28,7 @@ Add `--subscription` flag to the `serve` command defined in `commands.json`. See
 
 ### How do squids keep track of their sync progress?
 
-Depends on the data sink used. Squid processors that use [`TypeormDatabase`](/sdk/resources/persisting-data/typeorm) keep their state in a [schema](https://www.postgresql.org/docs/current/sql-createschema.html), not in a table. By default the schema is called `squid_processor` (name must be overridden in [multiprocessor squids](/sdk/resources/multichain)). You can view it with
+Depends on the data sink used. Squid processors that use [`TypeormDatabase`](/sdk/resources/store/typeorm) keep their state in a [schema](https://www.postgresql.org/docs/current/sql-createschema.html), not in a table. By default the schema is called `squid_processor` (name must be overridden in [multiprocessor squids](/sdk/resources/integration/multichain)). You can view it with
 ```sql
 select * from squid_processor.status;
 ```
@@ -38,7 +38,7 @@ drop schema squid_processor cascade;
 ```
 to reset the processor status.
 
-Squids that store their data in [file-based datasets](/sdk/resources/persisting-data/file) store their status in `status.txt` by default. This can be overridden by defining custom [database hooks](/sdk/resources/persisting-data/file/#filesystem-syncs-and-dataset-partitioning).
+Squids that store their data in [file-based datasets](/sdk/resources/store/file) store their status in `status.txt` by default. This can be overridden by defining custom [database hooks](/sdk/resources/store/file/#filesystem-syncs-and-dataset-partitioning).
 
 ### Is there a healthcheck endpoint for the indexer?