-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
adding changelog and migrations for k8s 1.30 and kubebuilderv4 work (#…
…6862) Signed-off-by: Adam D. Cornett <[email protected]>
- Loading branch information
1 parent
28c1bcb
commit c6fef29
Showing
1 changed file
with
351 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,351 @@ | ||
| # entries is a list of entries to include in | ||
| # release notes and/or the migration guide | ||
| entries: | ||
| - description: > | ||
| For Go-based, Helm-based and Ansible-based operators this release moves to Kubernetes 1.30 API's and Kubebuilder | ||
| v4 Scaffolding, specifically utilizing the v4.1.1 version. The update to Kubebuiler results in some scaffolding | ||
| changes which more information can be found below: | ||
| - Discontinue usage of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy) in the default | ||
| scaffolding of new projects. For further information, | ||
| see: [Action Required: Ensure that you no longer use gcr.io/kubebuilder images](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907) | ||
| - The `go/v2` or `go/v3` layouts have been removed, you must upgrade to `go/v4` to be compatible with this release and future updates. | ||
| To know how to upgrade,check the [migration documentation](https://book.kubebuilder.io/migration/v3vsv4). | ||
| - Re-introduces authn/authz protection for the metrics endpoint using [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35) | ||
| provided by controller-runtime instead of kube-rbac-proxy; which usage was [discontinued in the project](https://github.com/kubernetes-sigs/kubebuilder/discussions/3907). | ||
| Please, ensure that you no longer use the image `gcr.io/kubebuilder/kube-rbac-proxy`. Images provided under `gcr.io/kubebuilder/` will be unavailable from **March 18, 2025**. | ||
| To learn more about any of the metrics changes please look at the Kubebuilder book [metrics](https://book.kubebuilder.io/reference/metrics) page. | ||
| For `Helm-based` and `Ansible-based` operators, a new flag called `metrics-require-rbac` was introduced into the runtime/binary, to control adding | ||
| [`WithAuthenticationAndAuthorization`](https://github.com/kubernetes-sigs/controller-runtime/blob/v0.18.4/pkg/metrics/filters/filters.go#L35) | ||
| to `Metrics.FilterProvider` of controller-runtime. This was done to ensure forwards and backwards compatibility of the binary and images with any scaffolded content. | ||
| # kind is one of: | ||
| # - addition | ||
| # - change | ||
| # - deprecation | ||
| # - removal | ||
| # - bugfix | ||
| kind: "change" | ||
| # Is this a breaking change? | ||
| breaking: false | ||
| # NOTE: ONLY USE `pull_request_override` WHEN ADDING THIS | ||
| # FILE FOR A PREVIOUSLY MERGED PULL_REQUEST! | ||
| # | ||
| # The generator auto-detects the PR number from the commit | ||
| # message in which this file was originally added. | ||
| # | ||
| # What is the pull request number (without the "#")? | ||
| # pull_request_override: 0 | ||
| # Migration can be defined to automatically add a section to | ||
| # the migration guide. This is required for breaking changes. | ||
| migration: | ||
| header: Upgrade K8s versions to use 1.30 and Kubebuilder v4 | ||
| body: | | ||
| This update has a lot of scaffolding changes due to the removal of [kube-rbac-proxy](https://github.com/brancz/kube-rbac-proxy), | ||
| if these migrations become difficult to follow, it might be beneficial to scaffold a net new sample project to compare. | ||
| 1) [helm/v1, ansible/v1] Update the kustomize version in your Makefile | ||
| ```diff | ||
| - curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \ | ||
| + curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.4.2/kustomize_v5.4.2_$(OS)_$(ARCH).tar.gz | \ | ||
| ``` | ||
| 2) [go/v4] Update your `go.mod` file to upgrade the dependencies and run `go mod tidy` to download them | ||
| ```go | ||
| go 1.22.0 | ||
| github.com/onsi/ginkgo/v2 v2.17.1 | ||
| github.com/onsi/gomega v1.32.0 | ||
| k8s.io/api v0.30.1 | ||
| k8s.io/apimachinery v0.30.1 | ||
| k8s.io/client-go v0.30.1 | ||
| sigs.k8s.io/controller-runtime v0.18.4 | ||
| ``` | ||
| 3) [go/v4] Update your `Makefile` with the below changes: | ||
| ```diff | ||
| - ENVTEST_K8S_VERSION = 1.29.0 | ||
| + ENVTEST_K8S_VERSION = 1.30.0 | ||
| ``` | ||
| ```diff | ||
| - KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION) | ||
| - CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION) | ||
| - ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION) | ||
| - GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION) | ||
| + KUSTOMIZE ?= $(LOCALBIN)/kustomize | ||
| + CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen | ||
| + ENVTEST ?= $(LOCALBIN)/setup-envtest | ||
| + GOLANGCI_LINT = $(LOCALBIN)/golangci-lint | ||
| ``` | ||
| ```diff | ||
| - KUSTOMIZE_VERSION ?= v5.3.0 | ||
| - CONTROLLER_TOOLS_VERSION ?= v0.14.0 | ||
| - ENVTEST_VERSION ?= release-0.17 | ||
| - GOLANGCI_LINT_VERSION ?= v1.57.2 | ||
| + KUSTOMIZE_VERSION ?= v5.4.2 | ||
| + CONTROLLER_TOOLS_VERSION ?= v0.15.0 | ||
| + ENVTEST_VERSION ?= release-0.18 | ||
| + GOLANGCI_LINT_VERSION ?= v1.59.1 | ||
| ``` | ||
| ```diff | ||
| - $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION}) | ||
| + $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION)) | ||
| ``` | ||
| ```diff | ||
| - @[ -f $(1) ] || { \ | ||
| + @[ -f "$(1)-$(3)" ] || { \ | ||
| echo "Downloading $${package}" ;\ | ||
| + rm -f $(1) || true ;\ | ||
| - mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\ | ||
| - } | ||
| + mv $(1) $(1)-$(3) ;\ | ||
| + } ;\ | ||
| + ln -sf $(1)-$(3) $(1) | ||
| ``` | ||
| 4) [go/v4] Update your `.golangci.yml` with the below changes: | ||
| ```diff | ||
| - exportloopref | ||
| + - ginkgolinter | ||
| - prealloc | ||
| + - revive | ||
| + | ||
| +linters-settings: | ||
| + revive: | ||
| + rules: | ||
| + - name: comment-spacings | ||
| ``` | ||
| 5) [go/v4] Update your `Dockerfile` file with the below changes: | ||
| ```diff | ||
| - FROM golang:1.21 AS builder | ||
| + FROM golang:1.22 AS builder | ||
| ``` | ||
| 6) [go/v4] Update your `main.go` file with the below changes: | ||
| ```diff | ||
| "sigs.k8s.io/controller-runtime/pkg/log/zap" | ||
| + "sigs.k8s.io/controller-runtime/pkg/metrics/filters" | ||
| var enableHTTP2 bool | ||
| - flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.") | ||
| + var tlsOpts []func(*tls.Config) | ||
| + flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ | ||
| + "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") | ||
| flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") | ||
| flag.BoolVar(&enableLeaderElection, "leader-elect", false, | ||
| "Enable leader election for controller manager. "+ | ||
| "Enabling this will ensure there is only one active controller manager.") | ||
| - flag.BoolVar(&secureMetrics, "metrics-secure", false, | ||
| - "If set the metrics endpoint is served securely") | ||
| + flag.BoolVar(&secureMetrics, "metrics-secure", true, | ||
| + "If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.") | ||
| - tlsOpts := []func(*tls.Config){} | ||
| + // Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server. | ||
| + // More info: | ||
| + // - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/server | ||
| + // - https://book.kubebuilder.io/reference/metrics.html | ||
| + metricsServerOptions := metricsserver.Options{ | ||
| + BindAddress: metricsAddr, | ||
| + SecureServing: secureMetrics, | ||
| + // TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are | ||
| + // not provided, self-signed certificates will be generated by default. This option is not recommended for | ||
| + // production environments as self-signed certificates do not offer the same level of trust and security | ||
| + // as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing | ||
| + // unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName | ||
| + // to provide certificates, ensuring the server communicates using trusted and secure certificates. | ||
| + TLSOpts: tlsOpts, | ||
| + } | ||
| + | ||
| + if secureMetrics { | ||
| + // FilterProvider is used to protect the metrics endpoint with authn/authz. | ||
| + // These configurations ensure that only authorized users and service accounts | ||
| + // can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info: | ||
| + // https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/metrics/filters#WithAuthenticationAndAuthorization | ||
| + metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization | ||
| + } | ||
| + | ||
| mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ | ||
| - Scheme: scheme, | ||
| - Metrics: metricsserver.Options{ | ||
| - BindAddress: metricsAddr, | ||
| - SecureServing: secureMetrics, | ||
| - TLSOpts: tlsOpts, | ||
| - }, | ||
| + Scheme: scheme, | ||
| + Metrics: metricsServerOptions, | ||
| ``` | ||
| 7) [go/v4, helm/v1, ansible/v1] Update your `/config/default/kustomization.yaml` file with the below changes: | ||
| ```diff | ||
| # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'. | ||
| #- ../prometheus | ||
| +# [METRICS] Expose the controller manager metrics service. | ||
| +- metrics_service.yaml | ||
| +# Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager | ||
| patches: | ||
| -# Protect the /metrics endpoint by putting it behind auth. | ||
| -# If you want your controller-manager to expose the /metrics | ||
| -# endpoint w/o any authn/z, please comment the following line. | ||
| -- path: manager_auth_proxy_patch.yaml | ||
| +# [METRICS] The following patch will enable the metrics endpoint using HTTPS and the port :8443. | ||
| +# More info: https://book.kubebuilder.io/reference/metrics | ||
| +- path: manager_metrics_patch.yaml | ||
| + target: | ||
| + kind: Deployment | ||
| ``` | ||
| 8) [go/v4, helm/v1, ansible/v1] Remove `/config/default/manager_auth_proxy_patch.yaml` and `/config/default/manager_config_patch.yaml` files. | ||
| 9) [go/v4, helm/v1, ansible/v1] Add `/config/default/manager_metrics_patch.yaml` file with the below changes: | ||
| ```diff | ||
| # This patch adds the args to allow exposing the metrics endpoint using HTTPS | ||
| - op: add | ||
| path: /spec/template/spec/containers/0/args/0 | ||
| value: --metrics-bind-address=:8443 | ||
| ``` | ||
| 10) [helm/v1, ansible/v1] Update `/config/default/manager_metrics_patch.yaml` file with the below changes: | ||
| ```diff | ||
| # This patch adds the args to allow securing the metrics endpoint | ||
| - op: add | ||
| path: /spec/template/spec/containers/0/args/0 | ||
| value: --metrics-secure | ||
| # This patch adds the args to allow RBAC-based authn/authz the metrics endpoint | ||
| - op: add | ||
| path: /spec/template/spec/containers/0/args/0 | ||
| value: --metrics-require-rbac | ||
| ``` | ||
| 11) [go/v4, helm/v1, ansible/v1] Add `/config/default/metrics_service.yaml` file with the below changes: | ||
| ```diff | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| labels: | ||
| control-plane: controller-manager | ||
| app.kubernetes.io/name: <operator-name> | ||
| app.kubernetes.io/managed-by: kustomize | ||
| name: controller-manager-metrics-service | ||
| namespace: system | ||
| spec: | ||
| ports: | ||
| - name: https | ||
| port: 8443 | ||
| protocol: TCP | ||
| targetPort: 8443 | ||
| selector: | ||
| control-plane: controller-manager | ||
| ``` | ||
| 12) [go/v4, helm/v1, ansible/v1] Update your `/config/manager/manager.yaml` file with the below changes (**Note: The port for ansible is 6789**): | ||
| ```diff | ||
| - --leader-elect | ||
| + - --health-probe-bind-address=:8081 | ||
| ``` | ||
| 13) [go/v4, helm/v1, ansible/v1] Update your `/config/prometheus/monitor/yaml` file with the below changes: | ||
| ```diff | ||
| - path: /metrics | ||
| - port: https | ||
| + port: https # Ensure this is the name of the port that exposes HTTPS metrics | ||
| tlsConfig: | ||
| + # TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables | ||
| + # certificate verification. This poses a significant security risk by making the system vulnerable to | ||
| + # man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between | ||
| + # Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data, | ||
| + # compromising the integrity and confidentiality of the information. | ||
| + # Please use the following options for secure configurations: | ||
| + # caFile: /etc/metrics-certs/ca.crt | ||
| + # certFile: /etc/metrics-certs/tls.crt | ||
| + # keyFile: /etc/metrics-certs/tls.key | ||
| insecureSkipVerify: true | ||
| ``` | ||
| 14) [go/v4, helm/v1, ansible/v1] Remove the following files from `/config/rbac` | ||
| ```diff | ||
| - auth_proxy_client_clusterrole.yaml | ||
| - auth_proxy_role.yaml | ||
| - auth_proxy_role_binding.yaml | ||
| - auth_proxy_service.yaml | ||
| ``` | ||
| 15) [go/v4, helm/v1, ansible/v1] Update your `/config/rbac/kustomization.yaml` file with the below changes: | ||
| ```diff | ||
| - leader_election_role_binding.yaml | ||
| - # Comment the following 4 lines if you want to disable | ||
| - # the auth proxy (https://github.com/brancz/kube-rbac-proxy) | ||
| - # which protects your /metrics endpoint. | ||
| - - auth_proxy_service.yaml | ||
| - - auth_proxy_role.yaml | ||
| - - auth_proxy_role_binding.yaml | ||
| - - auth_proxy_client_clusterrole.yaml | ||
| + # The following RBAC configurations are used to protect | ||
| + # the metrics endpoint with authn/authz. These configurations | ||
| + # ensure that only authorized users and service accounts | ||
| + # can access the metrics endpoint. Comment the following | ||
| + # permissions if you want to disable this protection. | ||
| + # More info: https://book.kubebuilder.io/reference/metrics.html | ||
| + - metrics_auth_role.yaml | ||
| + - metrics_auth_role_binding.yaml | ||
| + - metrics_reader_role.yaml | ||
| ``` | ||
| 16) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role.yaml` file with the below changes: | ||
| ```diff | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: metrics-auth-role | ||
| rules: | ||
| - apiGroups: | ||
| - authentication.k8s.io | ||
| resources: | ||
| - tokenreviews | ||
| verbs: | ||
| - create | ||
| - apiGroups: | ||
| - authorization.k8s.io | ||
| resources: | ||
| - subjectaccessreviews | ||
| verbs: | ||
| - create | ||
| ``` | ||
| 17) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_auth_role_binding.yaml` file with the below changes: | ||
| ```diff | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRoleBinding | ||
| metadata: | ||
| name: metrics-auth-rolebinding | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: metrics-auth-role | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: controller-manager | ||
| namespace: system | ||
| ``` | ||
| 18) [go/v4, helm/v1, ansible/v1] Add `/config/rbac/metrics_reader_role.yaml` file with the below changes: | ||
| ```diff | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: metrics-reader | ||
| rules: | ||
| - nonResourceURLs: | ||
| - "/metrics" | ||
| verbs: | ||
| - get | ||
| ``` |