Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[teraslice, e2e] Add Kubernetes role restrictions to teraslice master #3817

Merged
merged 9 commits into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions docs/configuration/clustering-k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,26 +33,43 @@ terafoundation.connectors.elasticsearch_next.default.node

in both the `teraslice-master` and `teraslice-worker` ConfigMaps.

## `default` ServiceAccount Binding
## ServiceAccount Binding

In order to run Teraslice Jobs in your Kubernetes cluster the Teraslice master
node will need the ability create, list and delete Kubernetes Jobs, Deployments,
Services and Pods. Teraslice has the ability to run in an isolated Kubernetes
namespace so users don't have to grant broad permissions to the Teraslice
master. Users can configure the Teraslice master to use a specific namespace
with the `kubernetes_namespace` configuration option. Users would then have
to create a Kubernetes `Role` and `RoleBinding` as shown below:
to create a Kubernetes `ServiceAccount`, `Role` and `RoleBinding` as shown below:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: master-service-account
namespace: ts-dev1
sotojn marked this conversation as resolved.
Show resolved Hide resolved
```

Make sure to reference this `ServiceAccount` in the master deployment configuration by setting `spec.template.spec.serviceAccountName` to `master-service-account` inside of [/e2e/k8s/masterDeployment.yaml](../../e2e/k8s/masterDeployment.yaml).

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teraslice-all-<NAMESPACE>
namespace: <NAMESPACE>
name: teraslice-master-role-ts-dev1
namespace: ts-dev1
sotojn marked this conversation as resolved.
Show resolved Hide resolved
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
- apiGroups: [""] # Core API group for resources like pods, configmaps
busma13 marked this conversation as resolved.
Show resolved Hide resolved
resources: ["pods", "configmaps", "services"]
verbs: ["get", "create", "delete", "list", "update", "patch"]
- apiGroups: ["apps"] # Apps API group for deployments and replica sets
resources: ["deployments", "replicasets"]
verbs: ["get", "create", "delete", "list", "update", "patch"]
- apiGroups: ["batch"] # batch API group for jobs
resources: ["jobs"]
verbs: ["get", "create", "delete", "list", "update", "patch"]

```

```yaml
Expand All @@ -63,17 +80,14 @@ metadata:
namespace: <NAMESPACE>
subjects:
- kind: ServiceAccount
name: default
name: master-service-account
namespace: <NAMESPACE>
roleRef:
kind: Role
name: teraslice-all-<NAMESPACE>
name: teraslice-master-role-<NAMESPACE>
apiGroup: "rbac.authorization.k8s.io"
```

Currently, Teraslice interacts with Kubernetes using the
`default ServiceAccount` in the configured namespace.

## Master Deployment

The Teraslice master node needs to be deployed in k8s by the user. It should
Expand Down
1 change: 1 addition & 0 deletions e2e/k8s/masterDeployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,4 @@ spec:
type: Directory
imagePullSecrets:
- name: docker-tera1-secret
serviceAccountName: master-service-account
5 changes: 5 additions & 0 deletions e2e/k8s/masterServiceAccount.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: master-service-account
namespace: ts-dev1
14 changes: 10 additions & 4 deletions e2e/k8s/role.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teraslice-all-ts-dev1
name: teraslice-master-role-ts-dev1
namespace: ts-dev1
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
- apiGroups: [""] # Core API group for resources like pods, configmaps
resources: ["pods", "configmaps", "services"]
verbs: ["get", "create", "delete", "list", "update", "patch"]
- apiGroups: ["apps"] # Apps API group for deployments and replica sets
resources: ["deployments", "replicasets"]
verbs: ["get", "create", "delete", "list", "update", "patch"]
- apiGroups: ["batch"] # batch API group for jobs
resources: ["jobs"]
verbs: ["get", "create", "delete", "list", "update", "patch"]
8 changes: 4 additions & 4 deletions e2e/k8s/roleBinding.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teraslice-all-ts-dev1
name: teraslice-master-ts-dev1
namespace: ts-dev1
subjects:
- kind: ServiceAccount
name: default
name: master-service-account
namespace: ts-dev1
roleRef:
kind: Role
name: teraslice-all-ts-dev1
apiGroup: "rbac.authorization.k8s.io"
name: teraslice-master-role-ts-dev1
apiGroup: "rbac.authorization.k8s.io"
9 changes: 9 additions & 0 deletions packages/scripts/src/helpers/k8s-env/k8s.ts
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,15 @@ export class K8s {
throw new Error('Missing k8s e2e test directory');
}

try {
const yamlServiceAccount = this.loadYamlFile('masterServiceAccount.yaml') as k8sClient.V1ServiceAccount;
const response = await this.k8sCoreV1Api
.createNamespacedServiceAccount(this.terasliceNamespace, yamlServiceAccount);
logger.debug('deployK8sTeraslice yamlmasterServiceAccount: ', response.body);
} catch (err) {
throw new Error(`Error creating ServiceAccount: ${err}`);
}

try {
const yamlRole = this.loadYamlFile('role.yaml') as k8sClient.V1Role;
const response = await this.k8sRbacAuthorizationV1Api
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ export class K8s {
}

/**
* Rerturns the first pod matching the provided selector after it has
* Returns the first pod matching the provided selector after it has
* entered the `Running` state.
*
* TODO: Make more generic to search for different statuses
Expand Down