[Issue #1519] Cherry pick platform's pattern for env vars & ssm secre…

…ts (#1516)

## Summary

Relates to #784

Closes #1519

Copies navapbc/template-infra#549

### Time to review: __10 mins__

## Changes proposed

- cherry picks platform infra template's pattern for passing in env vars
and AWS SSM secrets
- changes `ENABLE_V_0_1_ENDPOINTS` / `enable_v01_endpoints` to use the
above pattern
- _**does not yet**_ change any of our SSM secrets to use platform's
pattern, I plan to do that in a follow-up PR

## Context for reviewers

I created this PR via tactical copy-pasting from the
https://github.com/navapbc/template-infra/ repo.

The goal of this PR is to DRY our methods for setting environment
variables. Notice on the red side of the diff, how I've removed the need
to set `enable_v01_endpoints` so many times. Then notice on the green
side of the diff, that I only need to set `ENABLE_V_0_1_ENDPOINTS` twice
(for dev and staging). That's the goal of this PR, to pull in platform's
very nice pattern for DRY'ing environment variables.

## Testing

To test this, I added - then removed - the following block from
`staging.tf`

```hcl
  service_override_extra_environment_variables = {
    ENABLE_V_0_1_ENDPOINTS = "true"
  }
```

I then deployed to staging to see the difference. It worked as intended.

documentation/infra/environment-variables-and-secrets.md

@@ -0,0 +1,58 @@
+# Environment variables and secrets
+
+Applications follow [12-factor app](https://12factor.net/) principles to [store configuration as environment variables](https://12factor.net/config). The infrastructure provides some of these environment variables automatically, such as environment variables to authenticate as the ECS task role, environment variables for database access, and environment variables for accessing document storage. However, many applications require extra custom environment variables for application configuration and for access to secrets. This document describes how to configure application-specific environment variables and secrets. It also describes how to override those environment variables for a specific environment.
+
+## Application-specific extra environment variables
+
+Applications may need application specific configuration as environment variables. Examples may includes things like `WORKER_THREADS_COUNT`, `LOG_LEVEL`, `DB_CONNECTION_POOL_SIZE`, or `SERVER_TIMEOUT`. This section describes how to define extra environment variables for your application that are then made accessible to the ECS task by defining the environment variables in the task definition (see AWS docs on [using task definition parameters to pass environment variables to a container](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/taskdef-envfiles.html)).
+
+> ⚠️ Note: Do not put sensitive information such as credentials as regular environment variables. The method described in this section will embed the environment variables and their values in the ECS task definition's container definitions, so anyone with access to view the task definition will be able to see the values of the environment variables. For configuring secrets, see the section below on [Secrets](#secrets)
+
+Environment variables are defined in the `app-config` module in the [environment-variables.tf file](/infra/app/app-config/env-config/environment-variables.tf). Modify the `default_extra_environment_variables` map to define extra environment variables specific to the application. Map keys define the environment variable name, and values define the default value for the variable across application environments. For example:
+
+```terraform
+# environment-variables.tf
+
+locals {
+  default_extra_environment_variables = {
+    WORKER_THREADS_COUNT = 4
+    LOG_LEVEL            = "info"
+  }
+}
+```
+
+To override the default values for a particular environment, modify the `app-config/[environment].tf file` for the environment, and pass overrides to the `env-config` module using the `service_override_extra_environment_variables` variable. For example:
+
+```terraform
+# dev.tf
+
+module "dev_config" {
+  source                                       = "./env-config"
+  service_override_extra_environment_variables = {
+    WORKER_THREADS_COUNT = 1
+    LOG_LEVEL            = "debug"
+  }
+  ...
+}
+```
+
+## Secrets
+
+Secrets are a specific category of environment variables that need to be handled sensitively. Examples of secrets are authentication credentials such as API keys for external services. Secrets first need to be stored in AWS SSM Parameter Store as a `SecureString`. This section then describes how to make those secrets accessible to the ECS task as environment variables through the `secrets` configuration in the container definition (see AWS documentation on [retrieving Secrets Manager secrets through environment variables](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html)).
+
+Secrets are defined in the same file that non-sensitive environment variables are defined, in the `app-config` module in the [environment-variables.tf file](/infra/app/app-config/env-config/environment-variables.tf). Modify the `secrets` list to define the secrets that the application will have access to. For each secret, `name` defines the environment variable name, and `ssm_param_name` defines the SSM parameter name that stores the secret value. For example:
+
+```terraform
+# environment-variables.tf
+
+locals {
+  secrets = [
+    {
+      name           = "SOME_API_KEY"
+      ssm_param_name = "/${var.app_name}-${var.environment}/secret-sauce"
+    }
+  ]
+}
+```
+
+> ⚠️ Make sure you store the secret in SSM Parameter Store before you try to add secrets to your application service, or else the service won't be able to start since the ECS Task Executor won't be able to fetch the configured secret.

infra/api/app-config/dev.tf

@@ -5,6 +5,10 @@ module "dev_config" {
   environment                     = "dev"
   has_database                    = local.has_database
   database_enable_http_endpoint   = true
-  enable_v01_endpoints            = true
   has_incident_management_service = local.has_incident_management_service
+
+  service_override_extra_environment_variables = {
+    # determines whether the v0.1 endpoints are available in the API
+    ENABLE_V_0_1_ENDPOINTS = "true"
+  }
 }

infra/api/app-config/env-config/environment-variables.tf

@@ -0,0 +1,23 @@
+locals {
+  # Map from environment variable name to environment variable value
+  # This is a map rather than a list so that variables can be easily
+  # overridden per environment using terraform's `merge` function
+  default_extra_environment_variables = {
+    # Example environment variables
+    # WORKER_THREADS_COUNT    = 4
+    # LOG_LEVEL               = "info"
+    # DB_CONNECTION_POOL_SIZE = 5
+  }
+
+  # Configuration for secrets
+  # List of configurations for defining environment variables that pull from SSM parameter
+  # store. Configurations are of the format
+  # { name = "ENV_VAR_NAME", ssm_param_name = "/ssm/param/name" }
+  secrets = [
+    # Example secret
+    # {
+    #   name           = "SECRET_SAUCE"
+    #   ssm_param_name = "/${var.app_name}-${var.environment}/secret-sauce"
+    # }
+  ]
+}

infra/api/app-config/env-config/outputs.tf

@@ -16,6 +16,12 @@ output "database_config" {
 output "service_config" {
   value = {
     region = var.default_region
+    extra_environment_variables = merge(
+      local.default_extra_environment_variables,
+      var.service_override_extra_environment_variables
+    )
+
+    secrets = toset(local.secrets)
   }
 }
 
@@ -31,10 +37,6 @@ output "api_auth_token" {
   }
 }
 
-output "enable_v01_endpoints" {
-  value = var.enable_v01_endpoints
-}
-
 output "domain" {
   value = var.domain
 }

infra/api/app-config/env-config/variables.tf

@@ -32,14 +32,17 @@ variable "has_incident_management_service" {
   type = bool
 }
 
-variable "enable_v01_endpoints" {
-  description = "determines whether the v0.1 endpoints are available in the API"
-  type        = bool
-  default     = false
-}
-
 variable "domain" {
   type        = string
   description = "DNS domain of the website managed by HHS"
   default     = null
 }
+
+variable "service_override_extra_environment_variables" {
+  type        = map(string)
+  description = <<EOT
+    Map that overrides the default extra environment variables defined in environment-variables.tf.
+    Map from environment variable name to environment variable value
+    EOT
+  default     = {}
+}

infra/api/app-config/prod.tf

@@ -7,6 +7,10 @@ module "prod_config" {
   domain                          = "api.simpler.grants.gov"
   database_instance_count         = 2
   database_enable_http_endpoint   = true
-  enable_v01_endpoints            = false
   has_incident_management_service = local.has_incident_management_service
+
+  service_override_extra_environment_variables = {
+    # determines whether the v0.1 endpoints are available in the API
+    ENABLE_V_0_1_ENDPOINTS = "false"
+  }
 }

infra/api/app-config/staging.tf

@@ -5,6 +5,10 @@ module "staging_config" {
   environment                     = "staging"
   has_database                    = local.has_database
   database_enable_http_endpoint   = true
-  enable_v01_endpoints            = true
   has_incident_management_service = local.has_incident_management_service
+
+  service_override_extra_environment_variables = {
+    # determines whether the v0.1 endpoints are available in the API
+    ENABLE_V_0_1_ENDPOINTS = "true"
+  }
 }

infra/api/service/main.tf

@@ -127,9 +127,8 @@ module "service" {
   cpu                   = 1024
   memory                = 2048
 
-  api_auth_token       = data.aws_ssm_parameter.api_auth_token.value
-  enable_v01_endpoints = module.app_config.environment_configs[var.environment_name].enable_v01_endpoints
-  cert_arn             = local.domain != null ? data.aws_acm_certificate.cert[0].arn : null
+  api_auth_token = data.aws_ssm_parameter.api_auth_token.value
+  cert_arn       = local.domain != null ? data.aws_acm_certificate.cert[0].arn : null
 
   db_vars = module.app_config.has_database ? {
     security_group_ids         = data.aws_rds_cluster.db_cluster[0].vpc_security_group_ids
@@ -143,6 +142,9 @@ module "service" {
       schema_name = local.database_config.schema_name
     }
   } : null
+
+  extra_environment_variables = local.service_config.extra_environment_variables
+  secrets                     = local.service_config.secrets
 }
 
 module "monitoring" {

infra/frontend/app-config/env-config/environment-variables.tf

@@ -0,0 +1,23 @@
+locals {
+  # Map from environment variable name to environment variable value
+  # This is a map rather than a list so that variables can be easily
+  # overridden per environment using terraform's `merge` function
+  default_extra_environment_variables = {
+    # Example environment variables
+    # WORKER_THREADS_COUNT    = 4
+    # LOG_LEVEL               = "info"
+    # DB_CONNECTION_POOL_SIZE = 5
+  }
+
+  # Configuration for secrets
+  # List of configurations for defining environment variables that pull from SSM parameter
+  # store. Configurations are of the format
+  # { name = "ENV_VAR_NAME", ssm_param_name = "/ssm/param/name" }
+  secrets = [
+    # Example secret
+    # {
+    #   name           = "SECRET_SAUCE"
+    #   ssm_param_name = "/${var.app_name}-${var.environment}/secret-sauce"
+    # }
+  ]
+}

infra/frontend/app-config/env-config/outputs.tf

@@ -14,6 +14,12 @@ output "database_config" {
 output "service_config" {
   value = {
     region = var.default_region
+    extra_environment_variables = merge(
+      local.default_extra_environment_variables,
+      var.service_override_extra_environment_variables
+    )
+
+    secrets = toset(local.secrets)
   }
 }
 

infra/frontend/app-config/env-config/variables.tf

@@ -42,4 +42,13 @@ variable "sendy_list_id" {
   description = "Sendy list ID to for requests to manage subscribers to the Simpler Grants distribution list."
   type        = string
   default     = null
-}
+}
+
+variable "service_override_extra_environment_variables" {
+  type        = map(string)
+  description = <<EOT
+    Map that overrides the default extra environment variables defined in environment-variables.tf.
+    Map from environment variable name to environment variable value
+    EOT
+  default     = {}
+}

infra/frontend/service/main.tf

@@ -153,6 +153,9 @@ module "service" {
       schema_name = local.database_config.schema_name
     }
   } : null
+
+  extra_environment_variables = local.service_config.extra_environment_variables
+  secrets                     = local.service_config.secrets
 }
 
 module "monitoring" {

infra/modules/service/access-control.tf

@@ -135,6 +135,15 @@ data "aws_iam_policy_document" "task_executor" {
     ]
     resources = [data.aws_ecr_repository.app.arn]
   }
+
+  dynamic "statement" {
+    for_each = length(var.secrets) > 0 ? [1] : []
+    content {
+      sid       = "SecretsAccess"
+      actions   = ["ssm:GetParameters"]
+      resources = local.secret_arn_patterns
+    }
+  }
 }
 
 resource "aws_iam_role_policy" "task_executor" {

infra/modules/service/main.tf

@@ -15,21 +15,27 @@ locals {
   sendy_api_key           = var.sendy_api_key != null ? [{ name = "SENDY_API_KEY", value = var.sendy_api_key }] : []
   sendy_api_url           = var.sendy_api_url != null ? [{ name = "SENDY_API_URL", value = var.sendy_api_url }] : []
   sendy_list_id           = var.sendy_list_id != null ? [{ name = "SENDY_LIST_ID", value = var.sendy_list_id }] : []
-  enable_v01_endpoints    = var.enable_v01_endpoints == true ? [{ name = "ENABLE_V_0_1_ENDPOINTS", value = "true" }] : []
 
   base_environment_variables = concat([
     { name : "PORT", value : tostring(var.container_port) },
     { name : "AWS_REGION", value : data.aws_region.current.name },
     { name : "API_AUTH_TOKEN", value : var.api_auth_token },
-  ], local.hostname, local.sendy_api_key, local.sendy_api_url, local.sendy_list_id, local.enable_v01_endpoints)
+  ], local.hostname, local.sendy_api_key, local.sendy_api_url, local.sendy_list_id)
   db_environment_variables = var.db_vars == null ? [] : [
     { name : "DB_HOST", value : var.db_vars.connection_info.host },
     { name : "DB_PORT", value : var.db_vars.connection_info.port },
     { name : "DB_USER", value : var.db_vars.connection_info.user },
     { name : "DB_NAME", value : var.db_vars.connection_info.db_name },
     { name : "DB_SCHEMA", value : var.db_vars.connection_info.schema_name },
   ]
-  environment_variables = concat(local.base_environment_variables, local.db_environment_variables, var.extra_environment_variables)
+  environment_variables = concat(
+    local.base_environment_variables,
+    local.db_environment_variables,
+    [
+      for name, value in var.extra_environment_variables :
+      { name : name, value : value }
+    ],
+  )
 }
 
 #-------------------
@@ -89,6 +95,7 @@ resource "aws_ecs_task_definition" "app" {
         ]
       },
       environment = local.environment_variables,
+      secrets     = local.secrets,
       portMappings = [
         {
           containerPort = var.container_port,

infra/modules/service/secrets.tf

@@ -0,0 +1,14 @@
+locals {
+  secrets = [
+    for secret in var.secrets :
+    {
+      name      = secret.name,
+      valueFrom = secret.ssm_param_name
+    }
+  ]
+
+  secret_arn_patterns = [
+    for secret in var.secrets :
+    "arn:aws:ssm:*:*:parameter/${trimprefix(secret.ssm_param_name, "/")}"
+  ]
+}

infra/modules/service/variables.tf

@@ -81,8 +81,17 @@ variable "private_subnet_ids" {
 }
 
 variable "extra_environment_variables" {
-  type        = list(object({ name = string, value = string }))
-  description = "Additional environment variables to pass to the service container"
+  type        = map(string)
+  description = "Additional environment variables to pass to the service container. Map from environment variable name to the value."
+  default     = {}
+}
+
+variable "secrets" {
+  type = set(object({
+    name           = string
+    ssm_param_name = string
+  }))
+  description = "List of configurations for defining environment variables that pull from SSM parameter store"
   default     = []
 }
 
@@ -139,12 +148,6 @@ variable "api_auth_token" {
   description = "Auth token for connecting to the API"
 }
 
-variable "enable_v01_endpoints" {
-  description = "determines whether the v0.1 endpoints are available in the API"
-  type        = bool
-  default     = false
-}
-
 variable "is_temporary" {
   description = "Whether the service is meant to be spun up temporarily (e.g. for automated infra tests). This is used to disable deletion protection for the load balancer."
   type        = bool