Change cluster URL and security tokens to be optional, defaulting to …

…that mounted into the pod

- fixes #44
- fixes #328 (by removing the need for use of such approaches)

By default the plugin now uses the Fabric Kubernetes Client auto-configuration support to configure
itself when the values are un-set from the Plugin Settings UI. This will be the best defaults for
almost all users. If any of Cluster URL / Namespace / Security Token / CA Cert data are overridden
by the user, they will be used instead. If the pod does not have service account values available
it will fail without values being set.

build.gradle

@@ -20,7 +20,7 @@ apply from: "https://raw.githubusercontent.com/gocd/gocd-plugin-gradle-task-help
 
 gocdPlugin {
   id = 'cd.go.contrib.elasticagent.kubernetes'
-  pluginVersion = '3.9.1'
+  pluginVersion = '4.0.0'
   goCdVersion = '20.9.0'
   name = 'Kubernetes Elastic Agent Plugin'
   description = 'Kubernetes Based Elastic Agent Plugins for GoCD'
@@ -80,6 +80,7 @@ dependencies {
     testImplementation group: 'org.mockito', name: 'mockito-core', version: '5.8.0'
     testImplementation group: 'org.skyscreamer', name: 'jsonassert', version: '1.5.1'
     testImplementation group: 'org.jsoup', name: 'jsoup', version: '1.17.2'
+    testImplementation group: 'uk.org.webcompere', name: 'system-stubs-jupiter', version: '2.1.5'
 }
 
 test {

docs/configure_cluster_profile.md

@@ -15,38 +15,69 @@
 
 1. Optionally specify `Maximum pending pods`. This defaults to 10 (pods) if not provided.
 
-1. Specify `Cluster URL`.
-
-1. Optionally specify `Namespace`. If not provided, the plugin will launch GoCD
-   agent pods in the default Kubernetes namespace. Note: If you have multiple
-   GoCD servers with cluster profiles pointing to the same Kubernetes cluster,
-   make sure that the namespace used by each GoCD server is different.
-   Otherwise, the plugin of one GoCD server will end up terminating pods
-   started by the plugin in the other GoCD servers.
-
-1. Specify `Security token`. This should be a Kubernetes API token linked to a service account which has the
-   following permissions:
-
-   | Resource       | Actions |
-   | -------------- | ------- |
-   | nodes          | list    |
-   | events         | list    |
-   | pods, pods/log | *       |
-
-   If the plugin is using a non-default namespace, then the pods and pods/log permissions
-   can be limited to that namespace (using a role + role binding), and the plugin
-   will still work. Nodes list and events list need to be attached at the cluster
-   level (using a cluster role + cluster role binding) regardless of the
-   namespace chosen.
-
-   If you are comfortable with cluster-wide permissions you can refer to the [example within the GoCD official helm
-   chart](https://github.com/gocd/helm-chart/blob/master/gocd/templates/gocd-cluster-role.yaml).
-
-1. Specify `Cluster CA certificate data`. This should be the base-64-encoded certificate
-   of the Kubernetes API server. It can be omitted in the rare case that the Kubernetes API
-   is configured to serve plain HTTP.
-
-1. Optionally specify the `Cluster request timeout` (in milliseconds).
+1. Optionally specify `Cluster Information`.<br/>
+   Since plugin version `4.x`, when the server is running on Kubernetes the plugin 
+   will auto-configure itself based on standard Kubernetes environment variables and `ServiceAccount` tokens automounted
+   into the pod, so none of the values here need to be configured in many cases.
+
+   1. If you **use the [GoCD Helm Chart](https://artifacthub.io/packages/helm/gocd/gocd)**, and intend to create agents in the
+      same cluster as the GoCD server, ensure the service account is enabled in the Chart (default behaviour) and you 
+      have nothing else mandatory to configure.
+
+   2. If **not using the Helm chart**, create a `ServiceAccount` that has the following permissions in its linked roles
+      for the target cluster you want to create/manage elastic agents as Kubernetes pods.
+
+      | Resource       | Actions |
+      |----------------| ------- |
+      | nodes          | list    |
+      | events         | list    |
+      | pods, pods/log | *       |
+      
+      If the plugin is using a non-default namespace, then the pods and pods/log permissions
+      can be limited to that namespace (using a role + role binding), and the plugin
+      will still work. Nodes list and events list need to be attached at the cluster
+      level (using a cluster role + cluster role binding) regardless of the namespace chosen.
+
+      If you are comfortable with cluster-wide permissions you can refer to the [example within the GoCD official helm
+      chart](https://github.com/gocd/helm-chart/blob/master/gocd/templates/gocd-cluster-role.yaml).
+
+   3. If you are **running your server in Kubernetes**, ensure the service account token linked to the above can be
+      auto-mounted into the GoCD server pod, and you also have nothing further mandatory to configure.
+
+   4. If you are **running outside Kubernetes**, or **need to override the defaults**, continue:
+      - Optionally specify `Cluster URL`. Mandatory if running server outside Kubernetes.
+      - Optionally specify `Namespace`. Mandatory if running server outside Kubernetes. Note: If you have multiple
+        GoCD servers with cluster profiles pointing to the same Kubernetes cluster,
+        make sure that the namespace used by each GoCD server is different.
+        Otherwise, the plugin of one GoCD server will end up terminating pods
+        started by the plugin in the other GoCD servers.
+      - Optionally specify `Security token`. This should be a Kubernetes API token linked to a service account which has
+        the permissions noted above. Since Kubernetes is moving away from legacy (indefinite expiry) tokens, specifying
+        the token here is not recommended, as it cannot be auto-refreshed. However, if you need to get such a token
+        create a new secret like the below in the same namespace as the service account:
+
+        ```yaml
+        kind: Secret
+        apiVersion: v1
+        metadata:
+          name: gocd-sa-secret
+          namespace: gocd
+          annotations:
+            kubernetes.io/service-account.name: gocd
+        type: kubernetes.io/service-account-token
+        ```
+        Extract the token value to paste into the config with something like the below (you may want to directly put onto
+        your clipboard to avoid the value appearing in your shell)
+        ```shell
+        kubectl get secret gocd-sa-secret -o json | jq -r '.data.token' | base64 --decode
+        ```
+
+      - Optionally specify `Cluster CA certificate data`. This should be the PEM format certificate
+        of the Kubernetes API server. Similar to the above, this can be found from the service account token secret:
+        ```shell
+        kubectl get secret gocd-sa-secret -o json | jq -r '.data."ca.crt"' | base64 --decode
+        ```
+      - Optionally specify the `Cluster request timeout` (in milliseconds).
 
 
 !["Kubernetes Cluster Profile"][1]

docs/install.md

@@ -2,7 +2,7 @@
 
 ## Prerequisites
 
-* GoCD server version v19.3.0 or above
+* GoCD server version v20.9.0 or above
 * Kubernetes Cluster
 
 ## Installation

src/main/java/cd/go/contrib/elasticagent/KubernetesClientFactory.java

@@ -16,14 +16,15 @@
 
 package cd.go.contrib.elasticagent;
 
-import io.fabric8.kubernetes.client.ConfigBuilder;
+import io.fabric8.kubernetes.client.Config;
 import io.fabric8.kubernetes.client.KubernetesClient;
 import io.fabric8.kubernetes.client.KubernetesClientBuilder;
 
 import java.util.concurrent.TimeUnit;
 
 import static cd.go.contrib.elasticagent.KubernetesPlugin.LOG;
 import static cd.go.contrib.elasticagent.utils.Util.isBlank;
+import static cd.go.contrib.elasticagent.utils.Util.setIfNotBlank;
 import static java.text.MessageFormat.format;
 
 public class KubernetesClientFactory {
@@ -35,13 +36,14 @@ public class KubernetesClientFactory {
     private long kubernetesClientRecycleIntervalInMinutes = -1;
     public static final String CLIENT_RECYCLE_SYSTEM_PROPERTY_KEY = "go.kubernetes.elastic-agent.plugin.client.recycle.interval.in.minutes";
 
-    public KubernetesClientFactory() {
-        this.clock = Clock.DEFAULT;
+    KubernetesClientFactory() {
+        this(Clock.DEFAULT);
     }
 
     //used for testing..
-    public KubernetesClientFactory(Clock clock) {
+    KubernetesClientFactory(Clock clock) {
         this.clock = clock;
+        System.setProperty(Config.KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");
     }
 
     public static KubernetesClientFactory instance() {
@@ -74,15 +76,15 @@ private void clearOutClientOnTimer() {
     }
 
     private KubernetesClient createClientFor(PluginSettings pluginSettings) {
-        final ConfigBuilder configBuilder = new ConfigBuilder()
-                .withAutoConfigure(false)
-                .withMasterUrl(pluginSettings.getClusterUrl())
-                .withNamespace(pluginSettings.getNamespace())
-                .withOauthToken(pluginSettings.getSecurityToken())
-                .withCaCertData(pluginSettings.getCaCertData())
-                .withRequestTimeout(pluginSettings.getClusterRequestTimeout());
-
-        return new KubernetesClientBuilder().withConfig(configBuilder.build()).build();
+        Config config = Config.autoConfigure(null);
+
+        setIfNotBlank(config::setMasterUrl, pluginSettings.getClusterUrl());
+        setIfNotBlank(config::setNamespace, pluginSettings.getNamespace());
+        setIfNotBlank(config::setOauthToken, pluginSettings.getSecurityToken());
+        setIfNotBlank(config::setCaCertData, pluginSettings.getCaCertData());
+        config.setRequestTimeout(pluginSettings.getClusterRequestTimeout());
+
+        return new KubernetesClientBuilder().withConfig(config).build();
     }
 
     public void clearOutExistingClient() {

src/main/java/cd/go/contrib/elasticagent/PluginSettings.java

@@ -100,15 +100,15 @@ public String getGoServerUrl() {
     }
 
     public String getSecurityToken() {
-        return securityToken;
+        return getOrDefault(securityToken, null);
     }
 
     public String getClusterUrl() {
-        return clusterUrl;
+        return getOrDefault(clusterUrl, null);
     }
 
     public String getCaCertData() {
-        return isBlank(clusterCACertData) ? null : clusterCACertData;
+        return getOrDefault(clusterCACertData, null);
     }
 
     public Integer getClusterRequestTimeout() {
@@ -117,7 +117,7 @@ public Integer getClusterRequestTimeout() {
     }
 
     public String getNamespace() {
-        return getOrDefault(this.namespace, "default");
+        return getOrDefault(this.namespace, null);
     }
 
     private <T> T getOrDefault(T t, T defaultValue) {

src/main/java/cd/go/contrib/elasticagent/executors/GetClusterProfileMetadataExecutor.java

@@ -33,9 +33,9 @@ public class GetClusterProfileMetadataExecutor implements RequestExecutor {
     public static final Metadata GO_SERVER_URL = new GoServerURLMetadata();
     public static final Metadata AUTO_REGISTER_TIMEOUT = new Metadata("auto_register_timeout", false, false);
     public static final Metadata MAX_PENDING_PODS = new Metadata("pending_pods_count", false, false);
-    public static final Metadata CLUSTER_URL = new Metadata("kubernetes_cluster_url", true, false);
+    public static final Metadata CLUSTER_URL = new Metadata("kubernetes_cluster_url", false, false);
     public static final Metadata NAMESPACE = new Metadata("namespace", false, false);
-    public static final Metadata SECURITY_TOKEN = new Metadata("security_token", true, true);
+    public static final Metadata SECURITY_TOKEN = new Metadata("security_token", false, true);
     public static final Metadata CLUSTER_CA_CERT = new Metadata("kubernetes_cluster_ca_cert", false, true);
     public static final Metadata CLUSTER_REQUEST_TIMEOUT = new Metadata("cluster_request_timeout", false, false);
 

src/main/java/cd/go/contrib/elasticagent/utils/Util.java

@@ -27,6 +27,7 @@
 import java.io.StringReader;
 import java.nio.charset.StandardCharsets;
 import java.util.Properties;
+import java.util.function.Consumer;
 
 public class Util {
     public static final Gson GSON = new GsonBuilder()
@@ -38,6 +39,12 @@ public static boolean isBlank(String str) {
         return str == null || str.isBlank();
     }
 
+    public static void setIfNotBlank(Consumer<String> action, String str) {
+        if (!isBlank(str)) {
+            action.accept(str);
+        }
+    }
+
     public static String readResource(String resourceFile) {
         return new String(readResourceBytes(resourceFile), StandardCharsets.UTF_8);
     }

src/main/resources/plugin-settings.template.html

@@ -93,33 +93,49 @@
 
 	<fieldset>
 		<legend>Cluster Information</legend>
+		<div class="row">
+			<div class="columns large-12 end">
+				<label class="form-help-content">
+					The values below are all <b>optional</b> when the server is running on Kubernetes - the plugin will
+					auto-configure itself from the GoCD server's environment and the pod is attached to a <code>ServiceAccount</code>
+					with tokens auto-mounted into the pod.
+					<br/><br/>
+					In <b>most</b> cases you should only override these if you are running your server outside Kubernetes
+					<i>or</i> wish to create agent pods in a different cluster (or namespace) to the server,
+					<i>or</i> you wish to use a service account token whose lifecycle is managed separately to the GoCD Server pod.
+				</label>
+			</div>
+		</div>
 		<div class="row">
 			<div class="columns large-5">
-				<label>Cluster URL<span class='asterix'>*</span></label>
+				<label>Cluster URL</label>
 				<input type="text" ng-model="kubernetes_cluster_url" ng-required="true"/>
 				<span class="form_error" ng-show="GOINPUTNAME[kubernetes_cluster_url].$error.server">{{GOINPUTNAME[kubernetes_cluster_url].$error.server}}</span>
 				<label class="form-help-content">
-					Kubernetes Cluster URL. Can be obtained by running <code>kubectl cluster-info</code>
+					Cluster API Server master URL.<br/>Defaults to the same as that of the GoCD server pod (when on K8S).
 				</label>
 			</div>
 			<div class="columns large-5 end">
 				<label>Namespace</label>
 				<input type="text" ng-model="namespace" ng-required="true"/>
 				<span class="form_error" ng-show="GOINPUTNAME[namespace].$error.server">{{GOINPUTNAME[namespace].$error.server}}</span>
 				<label class="form-help-content">
-					Namespace in which plugin will create the agent pods. defaults to <code>default</code> namespace.
+					Namespace to create agent pods.<br/>Defaults to the same namespace as the GoCD server (when on K8S).
 				</label>
 			</div>
 		</div>
 
 		<div class="row">
-			<label>Security token
-				<span class='asterix'>*</span>
-			</label>
+			<label>Security token</label>
 			<textarea rows="5" ng-model="security_token"></textarea>
 			<span class="form_error form-error" ng-show="GOINPUTNAME[security_token].$error.server">{{GOINPUTNAME[security_token].$error.server}}</span>
 			<label class="form-help-content">
-				Get the service account token by running following command <code>kubectl describe secret TOKEN_NAME</code> and copy the value of token here.
+				A service account token to talk to the API server to list and create agent pods. If unset, defaults to
+				any service account token auto-mounted into standard pod locations (<b>strongly recommended</b> when
+				server is on K8S).
+				<br/><br/>
+				If you have a custom <code>Secret</code> of type <code>kubernetes.io/service-account-token</code> with
+				indefinite lifetime, or are not running the GoCD server on K8s copy a valid token value here.
 			</label>
 		</div>
 
@@ -128,7 +144,10 @@
 			<textarea ng-model="kubernetes_cluster_ca_cert" rows="7"></textarea>
 			<span class="form_error" ng-show="GOINPUTNAME[kubernetes_cluster_ca_cert].$error.server">{{GOINPUTNAME[kubernetes_cluster_ca_cert].$error.server}}</span>
 			<label class="form-help-content">
-				Kubernetes cluster CA certificate data. Include the entire PEM contents, including <code> -----BEGIN * </code> and <code> -----END * </code>.
+				Kubernetes cluster CA certificate data. If unset, defaults to the CA certificate auto-mounted into
+				standard pod locations alongside a service account token (<b>strongly recommended</b> when server is on K8S).
+				<br/><br/>
+				Otherwise, include the entire PEM contents, including <code>-----BEGIN</code> and <code>-----END</code>.
 			</label>
 		</div>