🐛 (cherry-pick #3945) fix place where metrics service is scaffolded b…

…y moving from config/rbac to config/default (#3948)

(cherry-pick #3945) fix place where metrics service is scaffolded by moving from config/rbac to config/default (#3948)

When we discontinued the usage of kube-rbac-proxy we placed the Metrics Service under config/rbac but it is not the best place to fit this resource. Furthermore, within those changes we are ensuring that the metrics service will only be applied if/when users enable the metrics.

* Upgrade sample testdata in the v3x branch

.github/workflows/test-sample-go.yml

@@ -24,9 +24,9 @@ jobs:
         run: |
           KUSTOMIZATION_FILE_PATH="testdata/project-v4/config/default/kustomization.yaml"
           sed -i '25s/^#//' $KUSTOMIZATION_FILE_PATH
-          sed -i '39s/^#//' $KUSTOMIZATION_FILE_PATH
-          sed -i '44s/^#//' $KUSTOMIZATION_FILE_PATH
-          sed -i '48,144s/^#//' $KUSTOMIZATION_FILE_PATH
+          sed -i '32s/^#//' $KUSTOMIZATION_FILE_PATH
+          sed -i '47s/^#//' $KUSTOMIZATION_FILE_PATH
+          sed -i '51,147s/^#//' $KUSTOMIZATION_FILE_PATH
       
       - name: Test
         run: |

docs/book/src/component-config-tutorial/testdata/project/config/default/kustomization.yaml

@@ -25,8 +25,11 @@ resources:
 #- ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
-patches:
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
+#patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics
 # If you want to expose the metric endpoint of your controller-manager uncomment the following line.

docs/book/src/component-config-tutorial/testdata/project/config/rbac/kustomization.yaml

@@ -9,7 +9,6 @@ resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 # For each CRD, "Editor" and "Viewer" roles are scaffolded by
 # default, aiding admins in cluster management. Those roles are
 # not used by the Project itself. You can comment the following lines

docs/book/src/cronjob-tutorial/testdata/project/config/default/kustomization.yaml

@@ -25,7 +25,10 @@ resources:
 - ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 - ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
 patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics

docs/book/src/cronjob-tutorial/testdata/project/config/rbac/kustomization.yaml

@@ -9,7 +9,6 @@ resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 # For each CRD, "Editor" and "Viewer" roles are scaffolded by
 # default, aiding admins in cluster management. Those roles are
 # not used by the Project itself. You can comment the following lines

docs/book/src/getting-started/testdata/project/config/default/kustomization.yaml

@@ -25,8 +25,11 @@ resources:
 #- ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
-patches:
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
+#patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics
 # If you want to expose the metric endpoint of your controller-manager uncomment the following line.

docs/book/src/getting-started/testdata/project/config/rbac/kustomization.yaml

@@ -9,7 +9,6 @@ resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 # For each CRD, "Editor" and "Viewer" roles are scaffolded by
 # default, aiding admins in cluster management. Those roles are
 # not used by the Project itself. You can comment the following lines

docs/book/src/reference/metrics.md

@@ -45,7 +45,12 @@ Further information can be found bellow in this document.
 First, you will need enable the Metrics by uncommenting the following line
 in the file `config/default/kustomization.yaml`, see:
 
-```sh
+```yaml
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
+```
+
+```yaml
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics
 # If you want to expose the metric endpoint of your controller-manager uncomment the following line.
@@ -79,7 +84,7 @@ Integrating `cert-manager` with your metrics service can secure the endpoint via
 
 To modify your project setup to expose metrics using HTTPS with
 the help of cert-manager, you'll need to change the configuration of both
-the `Service` under `config/rbac/metrics_service.yaml` and
+the `Service` under `config/default/metrics_service.yaml` and
 the `ServiceMonitor` under `config/prometheus/monitor.yaml` to use a secure HTTPS port
 and ensure the necessary certificate is applied.
 

pkg/plugins/common/kustomize/v2/scaffolds/init.go

@@ -64,7 +64,7 @@ func (s *initScaffolder) Scaffold() error {
 
 	templates := []machinery.Builder{
 		&rbac.Kustomization{},
-		&rbac.MetricsService{},
+		&kdefault.MetricsService{},
 		&rbac.RoleBinding{},
 		// We need to create a Role because if the project
 		// has not CRD define the controller-gen will not generate this file

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/kdefault/kustomization.go

@@ -71,8 +71,11 @@ resources:
 #- ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
-patches:
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
+#patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics
 # If you want to expose the metric endpoint of your controller-manager uncomment the following line.

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/rbac/metrics_service.go → pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/kdefault/metrics_service.go

@@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package rbac
+package kdefault
 
 import (
 	"path/filepath"
@@ -33,7 +33,7 @@ type MetricsService struct {
 // SetTemplateDefaults implements file.Template
 func (f *MetricsService) SetTemplateDefaults() error {
 	if f.Path == "" {
-		f.Path = filepath.Join("config", "rbac", "metrics_service.yaml")
+		f.Path = filepath.Join("config", "default", "metrics_service.yaml")
 	}
 
 	f.TemplateBody = metricsServiceTemplate

pkg/plugins/common/kustomize/v2/scaffolds/internal/templates/config/rbac/kustomization.go

@@ -53,5 +53,4 @@ const kustomizeRBACTemplate = `resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 `

pkg/plugins/common/kustomize/v2/scaffolds/webhook.go

@@ -98,6 +98,15 @@ func (s *webhookScaffolder) Scaffold() error {
 		}
 	}
 
+	err = pluginutil.UncommentCode(kustomizeFilePath, "#patches:", `#`)
+	if err != nil {
+		hasWebHookUncommented, err := pluginutil.HasFragment(kustomizeFilePath, "patches:")
+		if !hasWebHookUncommented || err != nil {
+			log.Errorf("Unable to find the line '#patches:' to uncomment in the file "+
+				"%s.", kustomizeFilePath)
+		}
+	}
+
 	err = pluginutil.UncommentCode(kustomizeFilePath, "#- path: manager_webhook_patch.yaml", `#`)
 	if err != nil {
 		hasWebHookUncommented, err := pluginutil.HasFragment(kustomizeFilePath, "- path: manager_webhook_patch.yaml")

test/e2e/v4/generate_test.go

@@ -63,6 +63,9 @@ func GenerateV4(kbc *utils.TestContext) {
 	ExpectWithOffset(1, pluginutil.UncommentCode(
 		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
 		"#- path: webhookcainjection_patch.yaml", "#")).To(Succeed())
+	ExpectWithOffset(1, pluginutil.UncommentCode(
+		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
+		"#- metrics_service.yaml", "#")).To(Succeed())
 	ExpectWithOffset(1, pluginutil.UncommentCode(
 		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
 		metricsTarget, "#")).To(Succeed())
@@ -120,9 +123,15 @@ func GenerateV4WithoutWebhooks(kbc *utils.TestContext) {
 	initingTheProject(kbc)
 	creatingAPI(kbc)
 
+	ExpectWithOffset(1, pluginutil.UncommentCode(
+		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
+		"#patches:", "#")).To(Succeed())
 	ExpectWithOffset(1, pluginutil.UncommentCode(
 		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
 		"#- ../prometheus", "#")).To(Succeed())
+	ExpectWithOffset(1, pluginutil.UncommentCode(
+		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
+		"#- metrics_service.yaml", "#")).To(Succeed())
 	ExpectWithOffset(1, pluginutil.UncommentCode(
 		filepath.Join(kbc.Dir, "config", "default", "kustomization.yaml"),
 		metricsTarget, "#")).To(Succeed())

test/e2e/v4/plugin_cluster_test.go

@@ -278,66 +278,47 @@ func Run(kbc *utils.TestContext, hasWebhook, isToUseInstaller, hasMetrics bool)
 
 // curlMetrics curl's the /metrics endpoint, returning all logs once a 200 status is returned.
 func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
-	By("validating that the controller-manager service is available")
-	_, err := kbc.Kubectl.Get(
-		true,
-		"service", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
-	)
-	ExpectWithOffset(2, err).NotTo(HaveOccurred(), "Controller-manager service should exist")
-
-	By("validating that the controller-manager deployment is ready")
-	verifyDeploymentReady := func() error {
-		output, err := kbc.Kubectl.Get(
+	var metricsOutput string
+	if hasMetrics {
+		By("validating that the controller-manager service is available")
+		_, err := kbc.Kubectl.Get(
 			true,
-			"deployment", fmt.Sprintf("e2e-%s-controller-manager", kbc.TestSuffix),
-			"-o", "jsonpath={.status.readyReplicas}",
+			"service", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
 		)
-		if err != nil {
-			return err
-		}
-		readyReplicas, _ := strconv.Atoi(output)
-		if readyReplicas < 1 {
-			return fmt.Errorf("expected at least 1 ready replica, got %d", readyReplicas)
-		}
-		return nil
-	}
-	EventuallyWithOffset(2, verifyDeploymentReady, 240*time.Second, time.Second).Should(Succeed(),
-		"Deployment is not ready")
+		ExpectWithOffset(2, err).NotTo(HaveOccurred(), "Controller-manager service should exist")
 
-	By("ensuring the service endpoint is ready")
-	eventuallyCheckServiceEndpoint := func() error {
-		output, err := kbc.Kubectl.Get(
-			true,
-			"endpoints", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
-			"-o", "jsonpath={.subsets[*].addresses[*].ip}",
-		)
-		if err != nil {
-			return err
+		By("ensuring the service endpoint is ready")
+		eventuallyCheckServiceEndpoint := func() error {
+			output, err := kbc.Kubectl.Get(
+				true,
+				"endpoints", fmt.Sprintf("e2e-%s-controller-manager-metrics-service", kbc.TestSuffix),
+				"-o", "jsonpath={.subsets[*].addresses[*].ip}",
+			)
+			if err != nil {
+				return err
+			}
+			if output == "" {
+				return fmt.Errorf("no endpoints found")
+			}
+			return nil
 		}
-		if output == "" {
-			return fmt.Errorf("no endpoints found")
+		EventuallyWithOffset(2, eventuallyCheckServiceEndpoint, 2*time.Minute, time.Second).Should(Succeed(),
+			"Service endpoint should be ready")
+
+		By("creating a curl pod to access the metrics endpoint")
+		// nolint:lll
+		cmdOpts := []string{
+			"run", "curl",
+			"--restart=Never",
+			"--namespace", kbc.Kubectl.Namespace,
+			"--image=curlimages/curl:7.78.0",
+			"--",
+			"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
+				kbc.TestSuffix, kbc.Kubectl.Namespace),
 		}
-		return nil
-	}
-	EventuallyWithOffset(2, eventuallyCheckServiceEndpoint, 2*time.Minute, time.Second).Should(Succeed(),
-		"Service endpoint should be ready")
-
-	By("creating a curl pod to access the metrics endpoint")
-	// nolint:lll
-	cmdOpts := []string{
-		"run", "curl",
-		"--restart=Never",
-		"--namespace", kbc.Kubectl.Namespace,
-		"--image=curlimages/curl:7.78.0",
-		"--",
-		"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
-			kbc.TestSuffix, kbc.Kubectl.Namespace),
-	}
-	_, err = kbc.Kubectl.CommandInNamespace(cmdOpts...)
-	ExpectWithOffset(2, err).NotTo(HaveOccurred())
+		_, err = kbc.Kubectl.CommandInNamespace(cmdOpts...)
+		ExpectWithOffset(2, err).NotTo(HaveOccurred())
 
-	var metricsOutput string
-	if hasMetrics {
 		By("validating that the curl pod is running as expected")
 		verifyCurlUp := func() error {
 			status, err := kbc.Kubectl.Get(
@@ -359,6 +340,20 @@ func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
 		}
 		EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("< HTTP/1.1 200 OK"))
 	} else {
+		By("creating a curl pod to access the metrics endpoint")
+		// nolint:lll
+		cmdOpts := []string{
+			"run", "curl",
+			"--restart=Never",
+			"--namespace", kbc.Kubectl.Namespace,
+			"--image=curlimages/curl:7.78.0",
+			"--",
+			"/bin/sh", "-c", fmt.Sprintf("curl -v -k http://e2e-%s-controller-manager-metrics-service.%s.svc.cluster.local:8080/metrics",
+				kbc.TestSuffix, kbc.Kubectl.Namespace),
+		}
+		_, err := kbc.Kubectl.CommandInNamespace(cmdOpts...)
+		ExpectWithOffset(2, err).NotTo(HaveOccurred())
+
 		By("validating that the curl pod fail as expected")
 		verifyCurlUp := func() error {
 			status, err := kbc.Kubectl.Get(
@@ -375,14 +370,14 @@ func curlMetrics(kbc *utils.TestContext, hasMetrics bool) string {
 
 		By("validating that the metrics endpoint is not working as expected")
 		getCurlLogs := func() string {
-			metricsOutput, err = kbc.Kubectl.Logs("curl")
+			metricsOutput, err := kbc.Kubectl.Logs("curl")
 			ExpectWithOffset(3, err).NotTo(HaveOccurred())
 			return metricsOutput
 		}
-		EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("Connection refused"))
+		EventuallyWithOffset(2, getCurlLogs, 10*time.Second, time.Second).Should(ContainSubstring("Could not resolve host"))
 	}
 	By("cleaning up the curl pod")
-	_, err = kbc.Kubectl.Delete(true, "pods/curl")
+	_, err := kbc.Kubectl.Delete(true, "pods/curl")
 	ExpectWithOffset(3, err).NotTo(HaveOccurred())
 
 	return metricsOutput

testdata/project-v4-multigroup-with-deploy-image/config/default/kustomization.yaml

@@ -25,7 +25,10 @@ resources:
 #- ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
 patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics

testdata/project-v4-multigroup-with-deploy-image/config/rbac/kustomization.yaml

@@ -9,7 +9,6 @@ resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 # For each CRD, "Editor" and "Viewer" roles are scaffolded by
 # default, aiding admins in cluster management. Those roles are
 # not used by the Project itself. You can comment the following lines

testdata/project-v4-multigroup-with-deploy-image/dist/install.yaml

@@ -1474,24 +1474,6 @@ subjects:
 ---
 apiVersion: v1
 kind: Service
-metadata:
-  labels:
-    app.kubernetes.io/managed-by: kustomize
-    app.kubernetes.io/name: project-v4-multigroup-with-deploy-image
-    control-plane: controller-manager
-  name: project-v4-multigroup-with-deploy-image-controller-manager-metrics-service
-  namespace: project-v4-multigroup-with-deploy-image-system
-spec:
-  ports:
-  - name: http
-    port: 8080
-    protocol: TCP
-    targetPort: 8080
-  selector:
-    control-plane: controller-manager
----
-apiVersion: v1
-kind: Service
 metadata:
   labels:
     app.kubernetes.io/managed-by: kustomize

testdata/project-v4-multigroup/config/default/kustomization.yaml

@@ -25,7 +25,10 @@ resources:
 #- ../certmanager
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
+# [METRICS] To enable the controller manager metrics service, uncomment the following line.
+#- metrics_service.yaml
 
+# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
 patches:
 # [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
 # More info: https://book.kubebuilder.io/reference/metrics

testdata/project-v4-multigroup/config/rbac/kustomization.yaml

@@ -9,7 +9,6 @@ resources:
 - role_binding.yaml
 - leader_election_role.yaml
 - leader_election_role_binding.yaml
-- metrics_service.yaml
 # For each CRD, "Editor" and "Viewer" roles are scaffolded by
 # default, aiding admins in cluster management. Those roles are
 # not used by the Project itself. You can comment the following lines