Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Alerts, Logging, Channels Factories #2758

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
6a8ae06
WIP: Logging Alerts Factory
joshw123 Dec 6, 2024
2586c54
Implement Logging Alerts on Remaining Modules
joshw123 Dec 9, 2024
4e9f75e
Merge branch 'master' into AlertsFactory
joshw123 Dec 9, 2024
5b768a3
Documentation & FMT
joshw123 Dec 9, 2024
9ba92c3
Convert To Multiple Factories
joshw123 Dec 10, 2024
4e8a745
Correct Project
joshw123 Dec 10, 2024
8f16176
Update Documentation
joshw123 Dec 10, 2024
44b0bcc
Merge branch 'master' into AlertsMetrics
joshw123 Dec 10, 2024
82831ab
Merge branch 'master' into AlertsMetrics
joshw123 Dec 10, 2024
f72d382
Merge branch 'master' into AlertsMetrics
joshw123 Dec 11, 2024
a81081a
Update modules/project/alerts-factory.tf
joshw123 Dec 11, 2024
c5f25d0
Update fast/stages/0-bootstrap/data/logging-metrics/compliance.yaml
joshw123 Dec 11, 2024
38bf7f3
Merge branch 'master' into AlertsMetrics
joshw123 Dec 12, 2024
a832ddb
Update Tests, Resources
joshw123 Dec 13, 2024
91e4974
Merge branch 'master' into AlertsMetrics
joshw123 Dec 13, 2024
0704848
tests
joshw123 Dec 13, 2024
f1ed65d
Merge branch 'master' into AlertsMetrics
joshw123 Dec 13, 2024
c674b6c
Fix Tests
joshw123 Dec 13, 2024
fc3f5a6
Merge branch 'master' into AlertsMetrics
ludoo Dec 13, 2024
da1274d
Fix formatting
juliocc Dec 16, 2024
5cc77a6
Merge remote-tracking branch 'origin/master' into AlertsMetrics
juliocc Dec 18, 2024
d922bbf
Reformat metric filters
juliocc Dec 18, 2024
17a1777
Formatting, reordering, and small fixes
juliocc Dec 18, 2024
ab0b017
Merge branch 'master' into AlertsMetrics
juliocc Dec 31, 2024
248f4d2
Bring back alerts and metrics documentation
juliocc Dec 31, 2024
1d43499
Revert change bootstrap outputs.tf
juliocc Dec 31, 2024
832bfee
Fix project notification channel vars and factories
juliocc Jan 2, 2025
c253ef2
Fix vars and factory for logging alerts
juliocc Jan 2, 2025
c88821d
Complete alert variable and factory
juliocc Jan 3, 2025
c442495
Reorder fields
juliocc Jan 3, 2025
05cecc3
Update readme
juliocc Jan 3, 2025
1f998b5
Reorder variables
juliocc Jan 3, 2025
6b30b0d
Add schemas, update README, and fix some types
juliocc Jan 3, 2025
b0eec9f
Remove default alerts email from project and project-factory
juliocc Jan 3, 2025
2789313
Move observability factory to a single file
juliocc Jan 3, 2025
2c3615f
Add outputs to project module
juliocc Jan 4, 2025
3049cc7
Add factories_config to PF data_defaults and data_overrides
juliocc Jan 4, 2025
1ca9b09
Reorder PF field processing
juliocc Jan 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 29 additions & 28 deletions fast/stages/0-bootstrap/README.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions fast/stages/0-bootstrap/automation.tf
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,11 @@ module "automation-project" {
org_policies = (
var.bootstrap_user != null ? null : var.factories_config.org_policies_iac
)
logging_metrics = var.factories_config.logging_metrics
channels = var.factories_config.channels
alerts = var.factories_config.alerts
}
default_alerts_email = var.default_alerts_email
# human (groups) IAM bindings
iam_by_principals = {
(local.principals.gcp-devops) = [
Expand Down
8 changes: 7 additions & 1 deletion fast/stages/0-bootstrap/billing.tf
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,13 @@ module "billing-export-project" {
source = "../../../modules/project"
count = local.billing_mode == "org" ? 1 : 0
billing_account = var.billing_account.id
name = "billing-exp-0"
factories_config = {
joshw123 marked this conversation as resolved.
Show resolved Hide resolved
alerts = var.factories_config.alerts
channels = var.factories_config.channels
logging_metrics = var.factories_config.logging_metrics
}
default_alerts_email = var.default_alerts_email
name = "billing-exp-0"
parent = coalesce(
var.project_parent_ids.billing, "organizations/${var.organization.id}"
)
Expand Down
151 changes: 151 additions & 0 deletions fast/stages/0-bootstrap/data/alerts/compliance.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
route-changes:
display_name: "Network Route Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

network-firewall-config-changes:
display_name: "VPC Network Firewall Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

vpc-network-config-changes:
display_name: "VPC Network Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

cloudsql-changes:
display_name: "CloudSQL Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

cloudstorage-changes:
display_name: "Cloud Storage Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: "gcs_bucket"
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

customrole-changes:
display_name: "IAM Custom Role Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

audit-changes:
display_name: "Audit Configuration Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_MEAN
cross_series_reducer: REDUCE_COUNT
alignment_period: "600s"
notification_channels:
compliance-default

iam-owner-changes:
display_name: "Owner IAM Configuration Changes"
combiner: OR
alert_strategy:
auto_close: 604800s
conditions:
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
resource_type: global
trigger:
count: 1
aggregations:
per_series_aligner: ALIGN_DELTA
cross_series_reducer: REDUCE_SUM
alignment_period: "600s"
notification_channels:
compliance-default
6 changes: 6 additions & 0 deletions fast/stages/0-bootstrap/data/channels/compliance.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
compliance-default:
type: email
display_name: "Default Email Notifications"
email_address: [email protected]
labels:
email_address: "[email protected]"
13 changes: 13 additions & 0 deletions fast/stages/0-bootstrap/data/logging-alerts/compliance-alerts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@


iam-owner-changes:
display_name: "Owner IAM Configuration Changes"
condition_threshold:
comparison: COMPARISON_GT
duration: "0s"
trigger_count: 1
combiner: OR
aggregations:
per_series_aligner: ALIGN_DELTA
cross_series_reducer: REDUCE_SUM
alignment_period: "600s"
55 changes: 55 additions & 0 deletions fast/stages/0-bootstrap/data/logging-metrics/compliance.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
route-changes:
description: "Monitor VPC network route configuration changes inside GCP projects"
filter: "resource.type=\"gce_route\" AND (protoPayload.methodName:\"compute.routes.delete\" OR protoPayload.methodName:\"compute.routes.insert\")"
joshw123 marked this conversation as resolved.
Show resolved Hide resolved
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

network-firewall-config-changes:
description: "Monitor VPC network firewall configuration changes inside GCP projects"
filter: "resource.type=\"gce_firewall_rule\" AND (protoPayload.methodName:\"compute.firewalls.delete\" OR protoPayload.methodName:\"compute.firewalls.insert\")"
juliocc marked this conversation as resolved.
Show resolved Hide resolved
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

vpc-network-config-changes:
description: "Monitor VPC network configuration changes inside GCP projects"
filter: "resource.type=\"gce_network\" AND (protoPayload.methodName:\"compute.networks.insert\" OR protoPayload.methodName:\"compute.networks.patch\" OR protoPayload.methodName:\"compute.networks.delete\" OR protoPayload.methodName:\"compute.networks.removePeering\" OR protoPayload.methodName:\"compute.networks.addPeering\")"
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

cloudsql-changes:
description: "Monitor CloudSQL configuration changes inside GCP projects"
filter: "protoPayload.methodName=\"cloudsql.instances.update\" OR protoPayload.methodName=\"cloudsql.instances.create\" OR protoPayload.methodName=\"cloudsql.instances.delete\""
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

cloudstorage-changes:
description: "Monitor Cloud Storage configuration changes inside GCP projects"
filter: "resource.type=gcs_bucket AND protoPayload.methodName=\"storage.setIamPermissions\""
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

customrole-changes:
description: "Monitor IAM Custom Role configuration changes inside GCP projects"
filter: "resource.type=\"iam_role\" AND (protoPayload.methodName=\"google.iam.admin.v1.CreateRole\" OR protoPayload.methodName=\"google.iam.admin.v1.DeleteRole\" OR protoPayload.methodName=\"google.iam.admin.v1.UpdateRole\")"
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

audit-changes:
description: "Monitor Audit configuration changes inside GCP projects"
filter: "protoPayload.methodName=\"SetIamPolicy\" AND protoPayload.serviceData.policyDelta.auditConfigDeltas:*"
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"

iam-owner-changes:
description: "Monitor IAM Owner configuration changes inside GCP projects"
filter: "(protoPayload.serviceName=\"cloudresourcemanager.googleapis.com\") AND (ProjectOwnership OR projectOwnerInvitee) OR (protoPayload.serviceData.policyDelta.bindingDeltas.action=\"REMOVE\" AND protoPayload.serviceData.policyDelta.bindingDeltas.role=\"roles/owner\") OR (protoPayload.serviceData.policyDelta.bindingDeltas.action=\"ADD\" AND protoPayload.serviceData.policyDelta.bindingDeltas.role=\"roles/owner\")"
metric_descriptor:
metric_kind: DELTA
value_type: "INT64"
10 changes: 8 additions & 2 deletions fast/stages/0-bootstrap/log-export.tf
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,14 @@ module "log-export-project" {
parent = coalesce(
var.project_parent_ids.logging, "organizations/${var.organization.id}"
)
prefix = local.prefix
billing_account = var.billing_account.id
factories_config = {
alerts = var.factories_config.alerts
channels = var.factories_config.channels
logging_metrics = var.factories_config.logging_metrics
}
default_alerts_email = var.default_alerts_email
prefix = local.prefix
billing_account = var.billing_account.id
contacts = (
var.bootstrap_user != null || var.essential_contacts == null
? {}
Expand Down
7 changes: 4 additions & 3 deletions fast/stages/0-bootstrap/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ locals {
# If users give a list of custom audiences we set by default the first element.
# If no audiences are given, we set https://iam.googleapis.com/{PROVIDER_NAME}
audiences = try(
local.cicd_providers[v["identity_provider"]].audiences, ""
local.cicd_providers[v["identity_provider"]].audiences, "e"
juliocc marked this conversation as resolved.
Show resolved Hide resolved
)
identity_provider = try(
local.cicd_providers[v["identity_provider"]].name, ""
Expand Down Expand Up @@ -109,6 +109,7 @@ locals {
vpcsc = module.automation-tf-vpcsc-sa.email
vpcsc-r = module.automation-tf-vpcsc-r-sa.email
}
default_alerts_email = var.default_alerts_email
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should go in globals, but TBH I would not even pass this down as it's specific to stage 0

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we export notification channels to the following stages?

}
custom_roles = module.organization.custom_role_id
logging = {
Expand Down Expand Up @@ -216,8 +217,8 @@ output "tfvars" {

output "tfvars_globals" {
description = "Terraform Globals variable files for the following stages."
sensitive = true
value = local.tfvars_globals
sensitive = false
value = jsonencode(local.tfvars_globals)
juliocc marked this conversation as resolved.
Show resolved Hide resolved
}

output "workforce_identity_pool" {
Expand Down
2 changes: 2 additions & 0 deletions fast/stages/0-bootstrap/terraform.tfvars.sample
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,5 @@ outputs_location = "~/fast-config"

# use something unique and no longer than 9 characters
prefix = "abcd"

default_alerts_email = "[email protected]"
10 changes: 9 additions & 1 deletion fast/stages/0-bootstrap/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ variable "custom_roles" {
default = {}
}

variable "default_alerts_email" {
joshw123 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use the existing essential_contacts email here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a similar thought either using an existing group, or building something with var.organization.domain

I actually prefer using essential_contacts

description = "Default email address for alerting."
type = string
nullable = false
}

variable "environments" {
description = "Environment names."
type = map(object({
Expand Down Expand Up @@ -136,11 +142,13 @@ variable "factories_config" {
custom_roles = optional(string, "data/custom-roles")
org_policies = optional(string, "data/org-policies")
org_policies_iac = optional(string, "data/org-policies-iac")
logging_metrics = optional(string, "data/logging-metrics")
channels = optional(string, "data/channels")
alerts = optional(string, "data/alerts")
})
joshw123 marked this conversation as resolved.
Show resolved Hide resolved
nullable = false
default = {}
}

variable "groups" {
# https://cloud.google.com/docs/enterprise/setup-checklist
description = "Group names or IAM-format principals to grant organization-level permissions. If just the name is provided, the 'group:' principal and organization domain are interpolated."
Expand Down
Loading
Loading