diff --git a/test/kwok-tests/README.md b/test/kwok-tests/README.md new file mode 100644 index 000000000..7f30e33b9 --- /dev/null +++ b/test/kwok-tests/README.md @@ -0,0 +1,207 @@ +# Simulating Kubernetes Workloads with KWOK Toolkit # + +The KWOK toolkit enables the simulation of the lifecycle of fake nodes, pods, and other Kubernetes API resources. This guide will walk you through the process of setting up and using KWOK to simulate custom pod lifecycles on a Kubernetes cluster. The example provided will use a KIND (Kubernetes IN Docker) cluster, but you can adapt it to any Kubernetes cluster of your choice. + + +## Installation ## + +KWOK can be installed on Linux and macOS systems using the Homebrew package manager. Open a terminal and run the following command to install KWOK using Homebrew: +``` +brew install kwok +``` + + +## Simulating Fake Nodes ## + +To start simulating fake nodes within your Kubernetes cluster, follow these steps: + +1. Ensure you have a valid cluster login. +2. Run the [nodes.sh](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/blob/main/test/kwok-tests/nodes.sh) script, specifying the number of fake nodes you want to create: +``` +% ./nodes.sh +Checking whether we have a valid cluster login or not... + +Nice, looks like you're logged in + + +How many simulated KWOK nodes do you want?4 +Nodes number is 4 + +The real number of nodes is 3 +Submitting node 1 +node/kwok-node-1 created +Submitting node 2 +node/kwok-node-2 created +Submitting node 3 +node/kwok-node-3 created +Submitting node 4 +node/kwok-node-4 created +Waiting until all the simualted pods become ready: +node/kwok-node-1 condition met +node/kwok-node-2 condition met +node/kwok-node-3 condition met +node/kwok-node-4 condition met + +Total amount of simulated nodes requested is: 4 +Total number of created nodes is: 4 +NAME STATUS ROLES AGE VERSION +kwok-node-1 Ready agent 1s fake +kwok-node-2 Ready agent 1s fake +kwok-node-3 Ready agent 1s fake +kwok-node-4 Ready agent 1s fake + +FYI, to clean up the kwow nodes, issue this: +kubectl get nodes --selector type=kwok -o name | xargs kubectl delete +``` +3. Wait for the simulated fake nodes to become ready. + +4. In the above run, we created `4` fake nodes. You can check the fake nodes running in the cluster by running: +``` +% kubectl get nodes +NAME STATUS ROLES AGE VERSION +kind-control-plane Ready control-plane 17d v1.27.1 +kwok-node-1 Ready agent 23s fake +kwok-node-2 Ready agent 23s fake +kwok-node-3 Ready agent 23s fake +kwok-node-4 Ready agent 23s fake +``` + +Note that these fake nodes are not yet managed by the KWOK controller. The next section focusses on how to start the KWOK controller with a custom configuration. + +## Starting KWOK controller that simulates custom pod lifecycle + +In order to understand and simulate the lifecycle of a `Job`, we first design a baseline experiment on a `kind` cluster using only default kubernetes scheduler without MCAD. The `Job` spec looks as follows: + +``` +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: baseline-cpu-job-short-0 +spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + spec: + containers: + - args: + - sleep + - 10s + name: baseline-cpu-job-short-0 + image: nginx:1.24.0 + resources: + limits: + cpu: 5m + memory: 20M + requests: + cpu: 5m + memory: 20M + restartPolicy: Never +``` + +Here are the timings from the actual job on the `kind` cluster: +``` +% kubectl get pods --output-watch-events --watch +EVENT NAME READY STATUS RESTARTS AGE +ADDED baseline-cpu-job-short-0-75crs 0/1 Pending 0 0s +MODIFIED baseline-cpu-job-short-0-75crs 0/1 Pending 0 0s +MODIFIED baseline-cpu-job-short-0-75crs 0/1 ContainerCreating 0 0s +MODIFIED baseline-cpu-job-short-0-75crs 1/1 Running 0 10s +MODIFIED baseline-cpu-job-short-0-75crs 0/1 Completed 0 20s +MODIFIED baseline-cpu-job-short-0-75crs 0/1 Completed 0 21s +MODIFIED baseline-cpu-job-short-0-75crs 0/1 Completed 0 22s +``` + +Now, we would like to simulate similar timed events for a fake job as well. KWOK's `Stage` configuration ([link](https://kwok.sigs.k8s.io/docs/user/stages-configuration/)) allows users to define and simulate different stages in the lifecycle of pods. By configuring the `delay`, `selector`, and `next` fields in a `Stage`, you can control when and how the stage is applied, providing a flexible and scalable way to simulate real-world scenarios in your Kubernetes cluster. + +The YAML file [kwok.yaml](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/blob/main/test/kwok-tests/kwok.yaml) in this repo provides a custom pod lifecycle `Stage` API that simulates the events with the desired timings as described above. To run KWOK following the custom pod lifecycle, mount the configuration to `~/.kwok/kwok.yaml`, and then run KWOK controller out of your Kubernetes cluster (in my case it is a kind cluster) as follows: +``` +kwok \ + --kubeconfig=~/.kube/config \ + --manage-all-nodes=false \ + --manage-nodes-with-annotation-selector=kwok.x-k8s.io/node=fake \ + --manage-nodes-with-label-selector= \ + --disregard-status-with-annotation-selector=kwok.x-k8s.io/status=custom \ + --disregard-status-with-label-selector= \ + --cidr=10.0.0.1/24 \ + --node-ip=10.0.0.1 \ +--config=kwok.yaml +``` + +*Note 1*: If you have KWOK already deployed and running inside your Kubernetes cluster, make sure to first scale it down to avoid two instances of KWOK controller running. The command +``` +kubectl scale deployment kwok-controller --replicas=0 -n kube-system +``` +should scale down the kwok controller in the K8 cluster. + +*Note 2:* The above command essentially means that the KWOK controller will only manage nodes that have that specified annotation. + +Now that we have KWOK running outside the cluster with our custom `Stage` configuration, we can create a fake `Job` and deploy it on the `kind` cluster. The fake `Job` spec looks as follows: +``` +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: nomcadkwok-cpu-job-short-0 +spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + # A taints was added to an automatically created Node. + # You can remove taints of Node or add this tolerations. + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - args: + - sleep + - 10s + name: nomcadkwok-cpu-job-short-0 + image: nginx:1.24.0 + resources: + limits: + cpu: 5m + memory: 50M + requests: + cpu: 5m + memory: 50M + restartPolicy: Never +``` + +The deployment of above `Job` spec creates following timed events: +``` +% kubectl get pods --output-watch-events --watch +EVENT NAME READY STATUS RESTARTS AGE +ADDED nomcadkwok-cpu-job-short-0-pkkqt 0/1 Pending 0 0s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 0/1 Pending 0 0s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 0/1 Pending 0 1s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 0/1 ContainerCreating 0 1s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 1/1 Running 0 11s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 0/1 Completed 0 22s +MODIFIED nomcadkwok-cpu-job-short-0-pkkqt 0/1 Completed 0 23s +``` + +We can see that the lifecycle of the simulated job and an actual job matches. As a sanity check, one can also look at the fake `Pod` and fake `Job` description matches with actual `Pod` and `Job` description, except for tolerations and node affinity fields. + +# KWOK Tests +Based on templates for fake nodes, fake pods, and pod lifecycle, the repo provides primarily two tests for resource manager Multi-Cluster App Dispatcher [MCAD](https://github.com/project-codeflare/multi-cluster-app-dispatcher/tree/main): +1. Stress Tests with small jobs [stress-tests-kwok](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/tree/main/test/kwok-tests/stress-tests-kwok) +- We first demonstrate a major difference between actual Kubernetes Cluster and simulated Kubernetes cluster with KWOK. +- We then generalize the findings to understand the dispatch rate of MCAD in presence of large number of small jobs. +2. MCAD performance test [gpu-tests-kwok](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/tree/main/test/kwok-tests/gpu-tests-kwok) +- Includes profiling MCAD with metrics such as scheduling latency, number of pending pods, job completion time, dispatch time of requests. diff --git a/test/kwok-tests/gpu-tests-kwok/README.md b/test/kwok-tests/gpu-tests-kwok/README.md new file mode 100644 index 000000000..1fd77a781 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/README.md @@ -0,0 +1,256 @@ +# Performance Tests for MCAD with GPU workload +There are primarily two tests designed for testing MCAD's dispatching of GPU requesting jobs using KWOK. +1. With default K8 scheduler +2. With a coscheduler, for gang scheduling + +For both cases, following metrics are captured: +1. Scheduling latency +2. Job completion time +3. Dispatch time (only for MCAD) +4. Response time +5. GPU utilization +6. Pending pods + +## Timestamped events of interest +1. Scheduling Latency: The interval between pod creation time and pod scheduled time. +``` +% kubectl get pod mcadkwokcosched-gpu-job-1-54x8k -o yaml +apiVersion: v1 +kind: Pod +metadata: + creationTimestamp: "2023-08-03T13:07:27Z" + finalizers: + - kwok.x-k8s.io/fake + generateName: mcadkwokcosched-gpu-job-1- +. +. +. +status: + conditions: + - lastProbeTime: null + lastTransitionTime: "2023-08-03T13:07:28Z" + status: "True" + type: Initialized + - lastProbeTime: null + lastTransitionTime: "2023-08-03T13:09:50Z" + reason: PodCompleted + status: "False" + type: Ready + - lastProbeTime: null + lastTransitionTime: "2023-08-03T13:09:50Z" + reason: PodCompleted + status: "False" + type: ContainersReady + - lastProbeTime: null + lastTransitionTime: "2023-08-03T13:07:28Z" + status: "True" + type: PodScheduled +. +. +. +``` +With respect to scheduling latency, the timestamps of interest for the above pod spec are `creationTimestamp: "2023-08-03T13:07:27Z"` and `lastTransitionTime: "2023-08-03T13:07:28Z"` corresponding to `PodScheduled` event. + +2. Job Completion Time: The interval between Job creation time and Job completion time. +``` +% kubectl get job mcadkwokcosched-gpu-job-1 -o yaml +apiVersion: batch/v1 +kind: Job +metadata: + annotations: + batch.kubernetes.io/job-tracking: "" + creationTimestamp: "2023-08-03T13:07:27Z" + generation: 1 + labels: + appwrapper.mcad.ibm.com: mcadkwokcosched-gpu-job-1 + resourceName: mcadkwokcosched-gpu-job-1 + name: mcadkwokcosched-gpu-job-1 +. +. +. +status: + completionTime: "2023-08-03T13:09:52Z" + conditions: + - lastProbeTime: "2023-08-03T13:09:52Z" + lastTransitionTime: "2023-08-03T13:09:52Z" + status: "True" + type: Complete + ready: 0 + startTime: "2023-08-03T13:07:27Z" + . + . + . +``` +For example, for this job spec, the timestamps captured are `creationTimestamp: "2023-08-03T13:07:27Z"` and `completionTime: "2023-08-03T13:09:52Z"`. The job completion time is then just `completionTime - creationTimestamp`. + +3. Dispatch Time: The interval between AppWrapper creation time and dispatch time +``` +% % kubectl get appwrapper mcadkwokcosched-gpu-job-1 -o yaml +apiVersion: mcad.ibm.com/v1beta1 +kind: AppWrapper +metadata: + creationTimestamp: "2023-08-03T13:07:27Z" + generation: 6 + name: mcadkwokcosched-gpu-job-1 + namespace: default +. +. +. +status: + canrun: true + conditions: + - lastTransitionMicroTime: "2023-08-03T13:07:27.384846Z" + lastUpdateMicroTime: "2023-08-03T13:07:27.384840Z" + status: "True" + type: Init + - lastTransitionMicroTime: "2023-08-03T13:07:27.386430Z" + lastUpdateMicroTime: "2023-08-03T13:07:27.386426Z" + reason: AwaitingHeadOfLine + status: "True" + type: Queueing + - lastTransitionMicroTime: "2023-08-03T13:07:27.424254Z" + lastUpdateMicroTime: "2023-08-03T13:07:27.424253Z" + reason: FrontOfQueue. + status: "True" + type: HeadOfLine + - lastTransitionMicroTime: "2023-08-03T13:07:29.295602Z" + lastUpdateMicroTime: "2023-08-03T13:07:29.295602Z" + reason: AppWrapperRunnable + status: "True" + type: Dispatched +. +. +. +``` +The dispatch time is captured by the time difference between `creationTimestamp: "2023-08-03T13:07:27Z"` event and `lastTransitionMicroTime: "2023-08-03T13:07:29.295602Z"` for `Dipatched` event. + +4. Response Time: The duration of time a request spends time in a queueing system. +For a system with only scheduler, a job's response time is the time interval between when the job was created and when the job completed. This is same as Job completion time captured as described above. +For an MCAD queueing system, the response time for an AppWrapper (AW) can be defined as the duration for which the AW is tracked by the MCAD system. Intuitively, this time interval encompasses the period starting when the AW was first detected by the controller (as indicated by the controller's first timestamp) until the moment the MCAD system removes the AW from its bookkeeping or tracking records. +Considering this definition, the timestamp that corresponds to when MCAD removes the AppWrapper from its bookkeeping records can be referred to as the "controller removal timestamp" or "controller completion timestamp." This timestamp marks the point at which MCAD finalizes its handling of the AW, indicating that the AW has completed its lifecycle and is no longer being actively tracked or managed by the MCAD system. +This interval should capture the entire lifecycle of the AW within the MCAD system. + +``` +status: + Succeeded: 1 + conditions: + - lastTransitionMicroTime: "2023-08-08T20:58:42.065706Z" + lastUpdateMicroTime: "2023-08-08T20:58:42.065705Z" + status: "True" + type: Init + - lastTransitionMicroTime: "2023-08-08T20:58:42.066866Z" + lastUpdateMicroTime: "2023-08-08T20:58:42.066865Z" + reason: AwaitingHeadOfLine + status: "True" + type: Queueing + - lastTransitionMicroTime: "2023-08-08T20:58:42.082548Z" + lastUpdateMicroTime: "2023-08-08T20:58:42.082547Z" + reason: FrontOfQueue. + status: "True" + type: HeadOfLine + - lastTransitionMicroTime: "2023-08-08T20:58:42.539719Z" + lastUpdateMicroTime: "2023-08-08T20:58:42.539718Z" + reason: AppWrapperRunnable + status: "True" + type: Dispatched + - lastTransitionMicroTime: "2023-08-08T20:58:52.976072Z" + lastUpdateMicroTime: "2023-08-08T20:58:52.976071Z" + reason: PodsRunning + status: "True" + type: Running + - lastTransitionMicroTime: "2023-08-08T20:59:53.748149Z" + lastUpdateMicroTime: "2023-08-08T20:59:53.748149Z" + reason: PodsCompleted + status: "True" + type: Completed + controllerfirsttimestamp: "2023-08-08T20:58:42.064572Z" +``` +For this example, the response time of this AW is the time passed between `controllerfirsttimestamp: "2023-08-08T20:58:42.064572Z"` and `lastTransitionMicroTime: "2023-08-08T20:59:53.748149Z"` corresponding to `Completed` type event. + + +## Test Run +The `run_sim.py` script is designed to simulate job requests in a Kubernetes system. It generates job requests based on specified parameters, applies them to the Kubernetes cluster and monitors their progress, and saves relevant information to output files. + +### Prerequisites +- Python 3.x +- `kubectl` command-line tool configured to interact with the target Kubernetes cluster. +- MCAD controller installed. Follow [MCAD Deployment](https://github.com/project-codeflare/multi-cluster-app-dispatcher/blob/main/test/perf-test/simulatingnodesandappwrappers.md#step-1-deploy-mcad-on-your-cluster) to deploy and run MCAD controller on your cluster. +- Python packages used given in [requirements.txt](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/tree/main/test/kwok-tests/gpu-tests-kwok/requirements.txt) +- Optional: Co-scheduler [installation](https://github.com/vishakha-ramani/multi-cluster-app-dispatcher/tree/main/test/kwok-tests#appendix-installing-coscheduler) + +### Usage +``` +python3 run_sim.py [arguments] +``` + +### Command Line Arguments: + +- --mean-arrival: Mean arrival time for job requests in seconds (default: 50). +- --total-jobs: Total number of job requests to generate (default: 100). +- --job-size: Mean sleep time of the container in seconds (default: 60). +- --output-file: Output file to store job results (default: job_results.txt). +- --pending-pod: Output file to store the number of pending pods (default: pending_pods.txt). +- --num-pod: Number of pods per job (default: 1). +- --gpu-options: GPU options for job requests (default: [2, 4, 6, 8]). +- --probabilities: Probabilities for GPU requirements (default: [0.25, 0.25, 0.25, 0.25]). +- --mode: Mode for job requests ('mcad' or 'nomcad', default: 'mcad'). +- --israndom: Use random.expovariate for job request arrival and job size (default: True). +- --usecosched: Use coscheduler with the number of pods specified by --num-pod argument (default: False). + + +### Functionality: + +The script simulates job request arrivals based on either fixed intervals or exponential distribution. +Job requests are generated with specified GPU requirements and job sizes. +The generated job requests are created as Kubernetes YAML files and applied to the cluster. +The script monitors the progress of job requests and the number of pending pods. +The script outputs job results and pending pod information to specified output files. +The script supports two modes: 'mcad' (with AppWrapper) and 'nomcad' (without AppWrapper). +The option to use coscheduler for job requests is available with the --usecosched flag. + +### Example Usage: +1. To run an experiment with fixed job arrival duration and job size (set `--israndom False`), without co-scheduler (set `--usecosched False`), and MCAD mode (set `--mode mcad`): +``` +python3 run_sim.py --mean-arrival 36 --total-jobs 50 --job-size 132 --gpu-options 8 --probabilities 1 --mode mcad --output-file mcadcosched_job_results.txt --pending-pod mcadcosched_pending_pod.txt --israndom False --usecosched False +``` +In this scenario, 50 job requests will be generated with fixed interval of 36 seconds, each requiring 8 GPUs and having a fixed job size of 132 seconds. The script will output the results of the simulation to nomcadcosched_job_results.txt and the number of pending pods to nomcadcosched_pending_pod.txt. `MCAD` mode is on, and coscheduling is disabled. + +2. To run an experiment with Poisson job arrivals and job size as exponential random variable (set `--israndom True`), with co-scheduler (set `--usecosched True`), and no MCAD mode (set `--mode nomcad`): +``` +python3 run_sim.py --mean-arrival 36 --total-jobs 50 --job-size 132 --ßgpu-options 8 --probabilities 1 --mode nomcad --output-file nomcadcosched_job_results.txt --pending-pod nomcadcosched_pending_pod.txt --israndom True --usecosched True +``` + + +### Note: +To analyze the aforementioned metrics for an experiment run, refer to `analysis-scripts` subdirectory. +This script assumes that the appropriate Kubernetes configuration is already set up. +The behavior and details of the script may vary based on Kubernetes cluster settings and configurations. + +### Disclaimer: + +This script is intended for educational and simulation purposes only. Use it responsibly and only on test or sandbox environments. + + + + + + + +## Appendix: Installing Coscheduler +1. Install co-scheduler using [official guide](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/manifests/install/charts/as-a-second-scheduler/README.md#installing-the-chart) + +2. Grant the appropriate RBAC (Role-Based Access Control) permissions to the service account "mcad-controller" in the "kube-system" namespace by running `kubectl apply -f` on the following: +``` +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: mcad-controller-podgroups +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: +- kind: ServiceAccount + name: mcad-controller + namespace: kube-system +``` \ No newline at end of file diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/README.md b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/README.md new file mode 100644 index 000000000..cb84aa959 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/README.md @@ -0,0 +1,162 @@ +# Simulation Analysis: + + +## 1. Kubernetes Scheduling Latency Analyzer + +The `scheduling_latency.py` is a Python script that retrieves scheduling latency data from Kubernetes pods and analyzes it to provide insights into the scheduling performance of the cluster. The script collects information about pod creation, scheduling, start, and finish times, calculates scheduling latency, and generates visualizations for better understanding. + +#### Configuration supported +nomcad, nomcad+cosched, mcad, mcad+cosched + +#### Program Workflow + +1. The script uses the `kubectl get pods` command to retrieve pod information in JSON format. +2. It calculates the scheduling latency for each pod by comparing the pod's creation time with its scheduled time. +3. The script extracts pod start and finish times and calculates the total time a pod takes to start and finish. +4. Scheduling latency data is saved to a CSV file named `scheduling_data.csv`. +5. If scheduling latency data is available, the script calculates the average scheduling latency. +6. The script generates a bar plot using `matplotlib` to visualize the distribution of scheduling latencies. + +#### How to Run + +1. Ensure you have Python 3.x and `kubectl` properly configured. +2. Run the script using: `python3 scheduling_latency.py`. + +#### Output + +- The script saves scheduling data to a CSV file named `scheduling_data.csv`. +- If scheduling latency data is available, the average scheduling latency is displayed. +- A bar plot displaying scheduling latencies is generated and shown using `matplotlib`. + + + +## 2. Job Completion Time + +The `job_completion_time.py` script retrieves job information from Kubernetes using the `kubectl` command, calculates the duration of each job, and generates a bar plot to visualize job durations. The script also calculates the average job completion time and saves the job details and the bar plot to files. + +#### Configuration supported +nomcad, nomcad+cosched, mcad, mcad+cosched + +#### Script Workflow + +1. The script runs the `kubectl get jobs -o json` command to obtain job information in JSON format. +2. A function `calculate_duration` calculates the duration of each job. +4. Job details such as index, creation time, completion time, GPUs requested, sleep time, and job duration are extracted and stored. +5. Job details are saved to a text file named `job_details.txt`. +6. A bar plot is generated using `matplotlib` to visualize job durations. +7. The bar plot is saved as an image file named `job_durations_plot.png`. +8. The script calculates the average job completion time and displays it. + +Run the script using: +``` +python job_completion_time.py +``` + +#### Output + +- The script saves job details to a text file named `job_details.txt`. +- A bar plot displaying job durations is generated and saved as an image file named `job_durations_plot.png`. +- The script calculates and displays the average job completion time. + + +## 3. Pending Pods + +The `plot_pending_pods.py` script aids in comparing the impact of MCAD and Non-MCAD approaches on the number of pending pods. +The script reads data from two files containing submission times and the pending pod counts at the job submission time for two different scenarios: MCAD and NO-MCAD. The files are obtained from `run_sim.py` experiment run, hence set the appropriate name for files using `--pending-pod` argument for `run_sim.py`. The script then creates a plot to visualize the variation of pending pods over submission time for both scenarios. The plot provides insights into the efficiency of job scheduling and resource allocation in the two scenarios. + +#### Configuration supported +nomcad, nomcad+cosched, mcad, mcad+cosched + +An example run of this script is as follows: +``` +python plot_pending_pods.py --mcad-file mcad_data.txt --nomcad-file nomcad_data.txt +``` + + +## 4. Dispatch Time + +The `dispatch_time.py` script interacts with Kubernetes to retrieve information about AppWrappers and their dispatch times. It calculates the average dispatch time across all AppWrappers and generates a bar chart to visualize the distribution of dispatch times. +The script provides insights into the efficiency of dispatch times for AppWrappers in a Kubernetes environment. Dispatch times can affect the responsiveness of scheduling and resource allocation. + +#### Configuration supported +mcad, mcad+cosched + +### Script Workflow + +1. The script defines a function `get_appwrappers` to retrieve AppWrapper information using the `kubectl get appwrappers -o json` command. +2. Another function `extract_controller_dispatch_times` processes AppWrapper data, extracts dispatch times, and calculates the average dispatch time. +3. AppWrapper names, controller first timestamps, dispatch times, and average dispatch times are collected and saved to a CSV file. +4. If dispatch times are available, the script calculates and displays the average dispatch time across all AppWrappers. +5. The collected data is sorted based on controller first timestamps. +6. A bar chart is generated using `matplotlib` to visualize the distribution of dispatch times. +7. The plot displays dispatch times against indices. + +An example run: +``` +python dispatch_time.py +``` + +### Output + +- The script generates a CSV file named `appwrapper_times.csv` containing AppWrapper names, controller first timestamps, AW dispatch time, and dispatch interval. +- If dispatch intervals are available, the script displays the average dispatch interval across all AppWrappers. +- A bar chart is generated and displayed, showing the distribution of dispatch times for AppWrappers. + + +## 5. MCAD Response Time + +The `mcad_response_time.py` script utilizes Kubernetes data to calculate and visualize response times and job completion times in an MCAD (Multi-Cluster Application Deployment) system. The script extracts completion times and creation times for job and AppWrapper. The script computes job completion time as the difference between job creation time and job completion time. The script also computes AW response time as the difference between AW creation time and AW completion time, and generates a scatter plot to visualize the relationship between response times and job completion times. + +#### Configuration supported +mcad, mcad+cosched + + +### Script Workflow + +1. The script runs `kubectl` commands to retrieve job and appwrapper information in JSON format. +2. Job completion times, job creation time, appwrapper controller first timestamps (creation time) and appwrapper completion times are extracted from the data. +4. Differences between mcad controller first timestamps and appwrapper completion times are calculated to obtain response times. +5. Differences between job completion times and job creation times are calculated to obtain job completion times. +6. The average response time and average job completion time are calculated. +7. A scatter plot is generated using `matplotlib` to visualize the relationship between response times and job completion times. + +An example run: +``` +python mcad_response_time.py +``` + +### Output +The script calculates and displays the average response time and average job completion time in the MCAD system. +- A line plot is generated and displayed, offering a visual representation of job completion times and response times. +- The plot aids in understanding the distribution and trends in response and completion times. + + + +## 6. GPU Utilization + +The `gpu_usage.py` script monitors and visualizes the total GPU resource requests and limits over time for nodes labeled with "type=kwok" in a Kubernetes cluster. This script utilizes the `kubectl` command-line tool to gather GPU resource information from the specified nodes, providing real-time insights into GPU utilization. + +#### Configuration supported +nomcad, nomcad+cosched, mcad, mcad+cosched + +### Script Workflow + +1. The script initiates an infinite loop to continuously monitor GPU resource usage. +2. It retrieves the names of all nodes labeled with "type=kwok" using the `kubectl` command. +3. For each Kwok node, it extracts the total GPU resource requests and limits by calling the `get_gpu_resource_info()` function. +4. The script records the total GPU resource requests and limits. +5. After each iteration, the script waits for 20 seconds before checking again. +6. If the script is interrupted by the user (keyboard interrupt), it generates an output text file named `gpu_resource_records.txt` to store the collected records. +7. If sufficient records are available, a line plot is created using `matplotlib`, displaying the variation in total GPU resource requests and limits over time. + +Execute the script with: +``` +python gpu_usage.py +``` + +### Output +- The script continuously records the total GPU resource requests and limits over time, storing them in the `gpu_resource_records.txt` file. +- If an adequate number of records is collected, a line plot is generated, illustrating the change in total GPU resource requests and limits over time. +- The plot offers insights into GPU utilization trends for the designated nodes. + + diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/dispatch_time.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/dispatch_time.py new file mode 100644 index 000000000..20fdd8ff1 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/dispatch_time.py @@ -0,0 +1,71 @@ +import subprocess +import json +import csv +from datetime import datetime +import matplotlib.pyplot as plt +import pandas as pd + +def get_appwrappers(): + cmd = "kubectl get appwrappers -o json" + output = subprocess.check_output(cmd, shell=True) + return json.loads(output) + +def extract_controller_dispatch_times(appwrappers, output_file): + dispatch_times = [] + appwrapper_names = [] + controller_first_timestamps = [] + with open(output_file, mode='w', newline='') as csvfile: + writer = csv.writer(csvfile) + writer.writerow(["AppWrapper", "Controller First Timestamp", "Dispatch Time", "Average Dispatch Time"]) + + for item in appwrappers['items']: + appwrapper_name = item.get('metadata', {}).get('name') + controller_first_timestamp = item.get('status', {}).get('controllerfirsttimestamp', None) + + conditions = item.get('status', {}).get('conditions', []) + dispatch_time = None + for condition in conditions: + if condition.get('type', '') == 'Dispatched': + dispatch_time = condition.get('lastTransitionMicroTime', None) + break + + dispatch_datetime = datetime.strptime(dispatch_time, "%Y-%m-%dT%H:%M:%S.%fZ") if dispatch_time else None + controller_first_datetime = datetime.strptime(controller_first_timestamp, "%Y-%m-%dT%H:%M:%S.%fZ") if controller_first_timestamp else None + + if dispatch_datetime and controller_first_datetime: + average_dispatch_time = (dispatch_datetime - controller_first_datetime).total_seconds() + dispatch_times.append(average_dispatch_time) + appwrapper_names.append(appwrapper_name) + controller_first_timestamps.append(controller_first_timestamp) + + writer.writerow([appwrapper_name, controller_first_timestamp, dispatch_time, average_dispatch_time]) + + if dispatch_times: + average_dispatch_time_overall = sum(dispatch_times) / len(dispatch_times) + print(f"Average Dispatch Time Across All AppWrappers: {average_dispatch_time_overall:.2f} seconds") + else: + print("No data available to calculate average dispatch time.") + + # Sort the lists based on the controller first timestamp + sorted_data = sorted(zip(appwrapper_names, dispatch_times, controller_first_timestamps), + key=lambda x: x[2]) + _, sorted_dispatch_times, _ = zip(*sorted_data) + + # Plot the dispatch time as a bar chart + if sorted_dispatch_times: + plt.figure(figsize=(12, 6)) + plt.bar(range(len(sorted_dispatch_times)), sorted_dispatch_times) + plt.xlabel('Index') + plt.ylabel('Dispatch Time (seconds)') + plt.title('Dispatch Time vs. Index') + plt.xticks(range(len(sorted_dispatch_times)), rotation=45, ha='right') + plt.tight_layout() + plt.show() + +def main(): + appwrappers = get_appwrappers() + output_file = "appwrapper_times.csv" + extract_controller_dispatch_times(appwrappers, output_file) + +if __name__ == "__main__": + main() diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/gpu-usage.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/gpu-usage.py new file mode 100644 index 000000000..5ad47da4e --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/gpu-usage.py @@ -0,0 +1,80 @@ +import subprocess +import re +import time +import matplotlib.pyplot as plt + +def get_gpu_resource_info(node_name): + # Run the kubectl describe node command + cmd = f"kubectl describe node {node_name}" + output = subprocess.check_output(cmd, shell=True, text=True) + + # Use regular expressions to extract the GPU resource information + gpu_requests = re.search(r"nvidia\.com/gpu\s+(\d+)", output) + gpu_limits = re.search(r"nvidia\.com/gpu\s+\d+\s+(\d+)", output) + + if gpu_requests and gpu_limits: + gpu_requests = int(gpu_requests.group(1)) + gpu_limits = int(gpu_limits.group(1)) + return gpu_requests, gpu_limits + else: + return None, None + +def get_all_kwok_nodes(): + # Run kubectl get nodes command with label selector type=kwok + cmd = "kubectl get nodes -l type=kwok --no-headers -o custom-columns='NAME:.metadata.name'" + output = subprocess.check_output(cmd, shell=True, text=True) + return output.strip().split("\n") + +if __name__ == "__main__": + output_file = "gpu_resource_records.txt" + records = [] + + try: + while True: + kwok_nodes = get_all_kwok_nodes() + + if not kwok_nodes: + print("No nodes with label selector type=kwok found.") + break + + total_requests = 0 + total_limits = 0 + + for node_name in kwok_nodes: + requests, limits = get_gpu_resource_info(node_name) + if requests is not None and limits is not None: + total_requests += requests + total_limits += limits + + records.append((total_requests, total_limits)) + + # Wait for 30 seconds before checking again + time.sleep(20) + + except KeyboardInterrupt: + print("Script stopped by the user.") + + with open(output_file, "w") as f: + for index, (requests, limits) in enumerate(records): + f.write(f"Index: {index}, Total GPU Resource Requests: {requests}, Total GPU Resource Limits: {limits}\n") + + if len(records) > 1: + # Extract the individual lists of total_requests and total_limits + total_requests_list, total_limits_list = zip(*records) + + # Create the plot + plt.plot(total_requests_list, label='Total GPU Resource Requests') + plt.plot(total_limits_list, label='Total GPU Resource Limits') + + # Add labels and title + plt.xlabel('Index') + plt.ylabel('GPU Resource') + plt.title('GPU Resource Requests and Limits over time') + plt.legend() + + # Show the plot + plt.show() + + else: + print("Insufficient records to plot") + diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/job_completion_time.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/job_completion_time.py new file mode 100644 index 000000000..849fb1e19 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/job_completion_time.py @@ -0,0 +1,66 @@ +import subprocess +import json +from datetime import datetime +import matplotlib.pyplot as plt + +# Run kubectl command to get job information in JSON format +kubectl_command = ["kubectl", "get", "jobs", "-o", "json"] +result = subprocess.run(kubectl_command, capture_output=True, text=True) +job_data = result.stdout + +# Load the JSON data into a dictionary +jobs_info = json.loads(job_data) + +# Function to calculate job duration +def calculate_duration(start_time, completion_time): + start_datetime = datetime.strptime(start_time, "%Y-%m-%dT%H:%M:%SZ") + completion_datetime = datetime.strptime(completion_time, "%Y-%m-%dT%H:%M:%SZ") + return completion_datetime - start_datetime + +# Lists to store job details +job_details = [] + +# Process each job +for job_index, job_info in enumerate(jobs_info.get("items", [])): + completion_time = job_info["status"].get("completionTime") + creation_timestamp = job_info["metadata"]["creationTimestamp"] + gpu_needed = job_info["spec"]["template"]["spec"]["containers"][0]["resources"]["limits"].get("nvidia.com/gpu", "") + sleep_time = job_info["spec"]["template"]["spec"]["containers"][0]["args"][1] + + if completion_time: + job_duration = calculate_duration(creation_timestamp, completion_time) + job_details.append((job_index + 1, creation_timestamp, completion_time, gpu_needed, sleep_time, job_duration.total_seconds())) + +# Save job details to output file +output_file = "job_details.txt" +with open(output_file, "w") as file: + file.write("Job Index\tCreation Time\tCompletion Time\tGPUs Requested\tSleep Time\tJob Duration (seconds)\n") + for details in job_details: + file.write("\t".join(map(str, details)) + "\n") + +# Create job indices for x-axis +job_indices = [details[0] for details in job_details] +job_durations = [details[5] for details in job_details] + +# Plotting the bar plot +plt.bar(job_indices, job_durations, color='blue') +plt.xlabel('Job Index') +plt.ylabel('Job Duration (seconds)') +plt.title('Job Durations vs Job Indices') +plt.xticks(job_indices) +plt.tight_layout() + +# Save and display the plot +output_plot_filename = "job_durations_plot.png" +plt.savefig(output_plot_filename) +plt.show() + +# Calculate the average job completion time +total_completion_time = sum(details[5] for details in job_details if details[5] is not None) +average_completion_time = total_completion_time / len(job_details) + +# Print average job completion time +print("Average Job Completion Time:", average_completion_time, "seconds") + +print("Job details saved in", output_file) +print("Bar plot saved as", output_plot_filename) diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/mcad_response_time.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/mcad_response_time.py new file mode 100644 index 000000000..5260640d9 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/mcad_response_time.py @@ -0,0 +1,102 @@ +import subprocess +import json +import matplotlib.pyplot as plt +import numpy as np +from datetime import datetime + +# Run kubectl command to get job and appwrapper information in JSON format +kubectl_command = ["kubectl", "get", "job,appwrapper", "-o", "json"] +result = subprocess.run(kubectl_command, capture_output=True, text=True) +data = result.stdout + +# Load the JSON data into dictionaries +info = json.loads(data) + +# Initialize lists to store data +timestamps = { + "controller_first": [], + "appwrapper_creation": [], + "job_creation": [], + "job_completion": [], + "appwrapper_completion": [], +} +completion_times = [] +response_times = [] + +output_lines = [] + +# Process the data +for item in info.get("items", []): + kind = item.get("kind") + metadata = item.get("metadata", {}) + status = item.get("status", {}) + + if kind == "Job": + completion_time = status.get("completionTime") + creation_time = metadata.get("creationTimestamp") + + if completion_time and creation_time: + completion_time = datetime.strptime(completion_time, "%Y-%m-%dT%H:%M:%SZ") + creation_time = datetime.strptime(creation_time, "%Y-%m-%dT%H:%M:%SZ") + timestamps["job_creation"].append(creation_time) + timestamps["job_completion"].append(completion_time) + completion_times.append(completion_time) + + elif kind == "AppWrapper": + controller_first = status.get("controllerfirsttimestamp") + creation_time = metadata.get("creationTimestamp") + conditions = status.get("conditions", []) + appwrapper_completion_time = None + + for condition in conditions: + if condition.get("type") == "Completed": + appwrapper_completion_time = condition.get("lastTransitionMicroTime") + break + + if controller_first and creation_time and appwrapper_completion_time: + controller_first = datetime.strptime(controller_first, "%Y-%m-%dT%H:%M:%S.%fZ") + creation_time = datetime.strptime(creation_time, "%Y-%m-%dT%H:%M:%SZ") + appwrapper_completion_time = datetime.strptime(appwrapper_completion_time, "%Y-%m-%dT%H:%M:%S.%fZ") + timestamps["controller_first"].append(controller_first) + timestamps["appwrapper_creation"].append(creation_time) + timestamps["appwrapper_completion"].append(appwrapper_completion_time) + + response_time = (appwrapper_completion_time - controller_first).total_seconds() + response_times.append(response_time) + +# Calculate the average response time +average_response_time = np.mean(response_times) +print(f"Average Response Time: {average_response_time:.2f} seconds") + +# Calculate the average job completion time +average_job_completion_time = np.mean([(completion - creation).total_seconds() for completion, creation in zip(completion_times, timestamps["job_creation"])]) +print(f"Average Job Completion Time: {average_job_completion_time:.2f} seconds") + +# Write output lines to a text file +output_lines = [ + "AW Controller First, AW Creation, Job Creation, Job Completion, AW Completion, Job Completion Time, Response Time\n" +] +with open("output.txt", "w") as output_file: + for i in range(len(completion_times)): + output_line = ( + f"{timestamps['controller_first'][i]}, " + f"{timestamps['appwrapper_creation'][i]}, " + f"{timestamps['job_creation'][i]}, " + f"{timestamps['job_completion'][i]}, " + f"{timestamps['appwrapper_completion'][i]}, " + f"{(timestamps['job_completion'][i] - timestamps['job_creation'][i]).total_seconds():.2f}, " + f"{response_times[i]:.2f}\n" + ) + output_lines.append(output_line) + output_file.writelines(output_lines) + +# Plot the line plot for job completion times and response times +plt.figure(figsize=(12, 6)) +plt.plot(range(len(completion_times)), [(completion - creation).total_seconds() for completion, creation in zip(completion_times, timestamps["job_creation"])], marker='o', color='b', alpha=0.5, label='Job Completion') +plt.plot(range(len(completion_times)), response_times, marker='*', color='r', alpha=0.5, label='Response Time') +plt.xlabel("Index") +plt.ylabel("Time (seconds)") +plt.title("Line Plot of Response Time and Job Completion Time") +plt.legend() +plt.tight_layout() +plt.show() \ No newline at end of file diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/plot_pending_pods.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/plot_pending_pods.py new file mode 100644 index 000000000..6697931b5 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/plot_pending_pods.py @@ -0,0 +1,45 @@ + +import matplotlib.pyplot as plt +import argparse + +def read_data(file_path): + submission_times = [] + pending_pods = [] + with open(file_path, "r") as file: + lines = file.readlines()[1:] # Skip the header line + for line in lines: + time, pods = line.strip().split("\t") + submission_times.append(float(time)) + pending_pods.append(int(pods)) + return submission_times, pending_pods + +# Parse command-line arguments +parser = argparse.ArgumentParser(description="Plot pending pods vs submission time for MCAD and NOMCAD.") +parser.add_argument("--mcad-file", type=str, default="mcad_data.txt", help="Path to the MCAD data file") +parser.add_argument("--nomcad-file", type=str, default="nomcad_data.txt", help="Path to the NOMCAD data file") +args = parser.parse_args() + +# Read data from the files +mcad_submission_times, mcad_pending_pods = read_data(args.mcad_file) +nomcad_submission_times, nomcad_pending_pods = read_data(args.nomcad_file) + +# Calculate the common x-axis limits +min_time = min(min(mcad_submission_times), min(nomcad_submission_times)) +max_time = max(max(mcad_submission_times), max(nomcad_submission_times)) + +# Create the plot +plt.plot(mcad_submission_times, mcad_pending_pods, label="MCAD") +plt.plot(nomcad_submission_times, nomcad_pending_pods, label="NO-MCAD") + +# Set labels and title +plt.xlabel("Submission Time") +plt.ylabel("Pending Pods") +plt.title("Pending Pods vs Submission Time") +plt.legend() + +# # Set the same x-axis scale for both plots +plt.xlim(min_time, max_time) + +# Show the plot +plt.show() + diff --git a/test/kwok-tests/gpu-tests-kwok/analysis-scripts/scheduling_latency.py b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/scheduling_latency.py new file mode 100644 index 000000000..e6e41f1de --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/analysis-scripts/scheduling_latency.py @@ -0,0 +1,88 @@ +from datetime import datetime +import subprocess +import json +import csv +import matplotlib.pyplot as plt + +def get_pods(): + cmd = "kubectl get pods -n default -o json" + output = subprocess.check_output(cmd, shell=True) + return json.loads(output) + +def calculate_scheduling_latency(pod): + creation_timestamp = pod['metadata']['creationTimestamp'] + pod_scheduled_transition_time = next( + (cond['lastTransitionTime'] for cond in pod['status']['conditions'] if cond['type'] == 'PodScheduled'), + None + ) + + if creation_timestamp and pod_scheduled_transition_time: + creation_datetime = datetime.strptime(creation_timestamp, "%Y-%m-%dT%H:%M:%SZ") + pod_scheduled_datetime = datetime.strptime(pod_scheduled_transition_time, "%Y-%m-%dT%H:%M:%SZ") + scheduling_latency = (pod_scheduled_datetime - creation_datetime).total_seconds() + return scheduling_latency + else: + return None + +def get_pod_time_info(pod): + if pod['status']['containerStatuses']: + container_status = pod['status']['containerStatuses'][0] + start_time = container_status['state'].get('terminated', {}).get('startedAt') + finish_time = container_status['state'].get('terminated', {}).get('finishedAt') + if start_time: + start_datetime = datetime.strptime(start_time, "%Y-%m-%dT%H:%M:%SZ") + else: + start_datetime = None + if finish_time: + finish_datetime = datetime.strptime(finish_time, "%Y-%m-%dT%H:%M:%SZ") + else: + finish_datetime = None + return start_datetime, finish_datetime + return None, None + +def main(): + pods = get_pods() + + scheduling_data = [] + scheduling_latencies = [] + for pod in pods['items']: + pod_name = pod['metadata']['name'] + creation_timestamp = pod['metadata']['creationTimestamp'] + pod_scheduled_transition_time = next( + (cond['lastTransitionTime'] for cond in pod['status']['conditions'] if cond['type'] == 'PodScheduled'), + None + ) + pod_start_time, pod_finish_time = get_pod_time_info(pod) + if pod_scheduled_transition_time and pod_start_time and pod_finish_time: + scheduling_latency = calculate_scheduling_latency(pod) + scheduling_latencies.append(scheduling_latency) + scheduling_data.append((pod_name, creation_timestamp, pod_scheduled_transition_time, pod_start_time, pod_finish_time, scheduling_latency)) + + if scheduling_data: + with open("scheduling_data.csv", mode="w", newline='') as csvfile: + writer = csv.writer(csvfile) + writer.writerow(["Pod Name", "Creation Time", "Scheduled Time", "Start Time", "Finish Time", "Scheduling Latency (seconds)"]) + for data in scheduling_data: + writer.writerow(data) + print("Scheduling data saved to scheduling_data.csv") + else: + print("No scheduling latency data available.") + + if scheduling_latencies: + average_latency = sum(scheduling_latencies) / len(scheduling_latencies) + print(f"Average Scheduling Latency: {average_latency:.2f} seconds") + + # Create a bar plot of scheduling latencies + plt.figure(figsize=(10, 6)) + plt.bar(range(len(scheduling_latencies)), scheduling_latencies) + plt.xlabel("Pod Index") + plt.ylabel("Scheduling Latency (seconds)") + plt.title("Scheduling Latencies for Pods") + plt.tight_layout() + plt.show() + else: + print("No scheduling latency data available.") + + +if __name__ == "__main__": + main() diff --git a/test/kwok-tests/gpu-tests-kwok/requirements.txt b/test/kwok-tests/gpu-tests-kwok/requirements.txt new file mode 100644 index 000000000..2974de710 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/requirements.txt @@ -0,0 +1,39 @@ +Package Version +------------------- -------- +altgraph 0.17.2 +cachetools 5.3.1 +certifi 2023.5.7 +charset-normalizer 3.2.0 +contourpy 1.1.0 +cycler 0.11.0 +fonttools 4.41.0 +future 0.18.2 +google-auth 2.22.0 +idna 3.4 +importlib-resources 6.0.0 +kiwisolver 1.4.4 +kubernetes 27.2.0 +macholib 1.15.2 +matplotlib 3.7.2 +numpy 1.25.1 +oauthlib 3.2.2 +packaging 23.1 +pandas 2.0.3 +Pillow 10.0.0 +pip 21.2.4 +pyasn1 0.5.0 +pyasn1-modules 0.3.0 +pyparsing 3.0.9 +python-dateutil 2.8.2 +pytz 2023.3 +PyYAML 6.0.1 +requests 2.31.0 +requests-oauthlib 1.3.1 +rsa 4.9 +setuptools 58.0.4 +six 1.15.0 +tzdata 2023.3 +urllib3 1.26.16 +websocket-client 1.6.1 +wheel 0.37.0 +zipp 3.16.2 diff --git a/test/kwok-tests/gpu-tests-kwok/run_sim.py b/test/kwok-tests/gpu-tests-kwok/run_sim.py new file mode 100644 index 000000000..d1e75afc8 --- /dev/null +++ b/test/kwok-tests/gpu-tests-kwok/run_sim.py @@ -0,0 +1,426 @@ +import random +import time +import subprocess +from datetime import datetime +import os +import argparse +import json +import yaml +import random + +# Function to check job status +def check_job_status(num_pod_per_job): + command = ["kubectl", "get", "jobs", "-n", "default", "--no-headers", "--field-selector", f"status.successful={num_pod_per_job}"] + result = subprocess.run(command, capture_output=True, text=True) + output = result.stdout.strip() + completed_jobs = len(output.splitlines()) + return completed_jobs + +# Function to write output to file +def generate_output(output_file, data, mode="nomcad"): + if mode == "mcad": + column_names = ["NAME", "MCAD_CONTROLLER_FIRST_TIMESTAMP", "JOB_CREATION_TIME", "JOB_COMPLETION_TIME", "GPU_NEEDED", "SLEEP_TIME"] + elif mode == "nomcad": + column_names = ["NAME", "CREATION_TIME", "COMPLETION_TIME", "GPU_NEEDED", "SLEEP_TIME"] + else: + raise ValueError("Invalid mode. Use 'mcad' or 'nomcad'.") + + # Extract and format desired columns + job_results = [] + job_results.append("\t".join(column_names)) # Add column names as the first row + for item in data.get("items", []): + name = item["metadata"]["name"] + creation_timestamp = item["metadata"]["creationTimestamp"] + completion_time = item["status"].get("completionTime", "") + gpu_needed = item["spec"]["template"]["spec"]["containers"][0]["resources"]["limits"].get("nvidia.com/gpu", "") + sleep_time = item["spec"]["template"]["spec"]["containers"][0]["args"][1] + + if mode == "mcad": + # Extract controller first timestamp from AppWrapper output + appwrapper_output_command = subprocess.run(["kubectl", "get", "appwrapper", name, "-o", "yaml"], capture_output=True, text=True) + appwrapper_output = appwrapper_output_command.stdout.strip() + appwrapper_data = yaml.safe_load(appwrapper_output) + controller_first_timestamp = appwrapper_data.get("status", {}).get("controllerfirsttimestamp", "") + + # Format controller first timestamp as YYYY-MM-DDTHH:MM:SS + formatted_controller_first_timestamp = datetime.strptime(controller_first_timestamp, "%Y-%m-%dT%H:%M:%S.%fZ").strftime("%Y-%m-%dT%H:%M:%SZ") + + # Append formatted row to results list + job_results.append(f"{name}\t{formatted_controller_first_timestamp}\t{creation_timestamp}\t{completion_time}\t{gpu_needed}\t{sleep_time}") + elif mode == "nomcad": + # Append formatted row to results list + job_results.append(f"{name}\t{creation_timestamp}\t{completion_time}\t{gpu_needed}\t{sleep_time}") + + # Join results with newlines + formatted_output = "\n".join(job_results) + + # Write output to file + with open(output_file, "w") as file: + file.write(formatted_output) + + +# Function to write pending pod count and submission time to a file +def write_pending_pods_to_file(output_file, submission_times, pending_pods_counts): + with open(output_file, "w") as file: + file.write("Submission Time\tPending Pods\n") + for time, count in zip(submission_times, pending_pods_counts): + file.write(f"{time}\t{count}\n") + +# Function to generate job requests +def generate_job_request(gpu_options, probabilities): + gpu_needed = random.choices(gpu_options, probabilities)[0] + return gpu_needed + +# Function to get the number of pending pods +def get_pending_pods(): + command = ["kubectl", "get", "pods", "-n", "default", "--field-selector", "spec.nodeName=", "--no-headers"] + result = subprocess.run(command, capture_output=True, text=True) + output = result.stdout.strip() + pending_pods = len(output.splitlines()) + return pending_pods + +def generate_mcadcosched_yaml(job_id, gpu_required, sleep_time, num_pod): + yaml_template = f""" +apiVersion: mcad.ibm.com/v1beta1 +kind: AppWrapper +metadata: + name: mcadkwokcosched-gpu-job-{job_id} + namespace: default +spec: + resources: + Items: [] + GenericItems: + - replicas: 1 + generictemplate: + apiVersion: scheduling.x-k8s.io/v1alpha1 + kind: PodGroup + metadata: + name: pg-{job_id} + namespace: default + spec: + scheduleTimeoutSeconds: 10 + minMember: {num_pod} + - replicas: 1 + generictemplate: + apiVersion: batch/v1 + kind: Job + metadata: + name: mcadkwokcosched-gpu-job-{job_id} + namespace: default + spec: + completions: {num_pod} + parallelism: {num_pod} + template: + metadata: + labels: + app: mcadkwokcosched-gpu-job-{job_id} + appwrapper.mcad.ibm.com: mcadkwokcosched-gpu-job-{job_id} + scheduling.x-k8s.io/pod-group: pg-{job_id} + spec: + schedulerName: scheduler-plugins-scheduler + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - args: + - sleep + - {sleep_time} + name: mcadkwokcosched-gpu-job-{job_id} + image: registry.k8s.io/pause:3.6 + resources: + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + restartPolicy: Never +""" + return yaml_template + + +def generate_nomcadcosched_yaml(job_id, gpu_required, sleep_time, num_pod): + yaml_template = f""" +apiVersion: scheduling.x-k8s.io/v1alpha1 +kind: PodGroup +metadata: + name: pg-{job_id} +spec: + scheduleTimeoutSeconds: 10 + minMember: {num_pod} +--- +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: nomcadkwok-job-{job_id} +spec: + parallelism: {num_pod} + completions: {num_pod} + template: + metadata: + namespace: default + labels: + scheduling.x-k8s.io/pod-group: pg-{job_id} + spec: + schedulerName: scheduler-plugins-scheduler + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - args: + - sleep + - {sleep_time} + name: nomcadkwok-job-{job_id} + image: nginx:1.24.0 + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + restartPolicy: Never +""" + return yaml_template + +# Function to generate YAML file for mcad job request +def generate_mcad_yaml(job_id, gpu_required, sleep_time): + yaml_template = f""" +apiVersion: mcad.ibm.com/v1beta1 +kind: AppWrapper +metadata: + name: mcadkwok-gpu-job-{job_id} + namespace: default +spec: + resources: + Items: [] + GenericItems: + - replicas: 1 + completionstatus: Complete + generictemplate: + apiVersion: batch/v1 + kind: Job + metadata: + namespace: default + name: mcadkwok-gpu-job-{job_id} + spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + labels: + appwrapper.mcad.ibm.com: mcadkwok-gpu-job-{job_id} + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - args: + - sleep + - {sleep_time} + name: mcadkwok-gpu-job-{job_id} + image: nginx:1.24.0 + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + restartPolicy: Never +""" + return yaml_template + +# Function to generate YAML file for nomcad job request +def generate_nomcad_yaml(job_id, gpu_required, sleep_time): + yaml_template = f""" +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: nomcadkwok-job-{job_id} +spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - args: + - sleep + - {sleep_time} + name: nomcadkwok-job-{job_id} + image: nginx:1.24.0 + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: {gpu_required} + restartPolicy: Never +""" + return yaml_template + +# Parse command line arguments +parser = argparse.ArgumentParser(description="Generate job requests in a Kubernetes system.") +parser.add_argument("--mean-arrival", type=float, default=50, help="Mean arrival time for job requests in seconds") +parser.add_argument("--total-jobs", type=int, default=100, help="Total number of job requests to generate") +parser.add_argument("--job-size", type=int, default=60, help="Mean sleep time of container in seconds") +parser.add_argument("--output-file", type=str, default="job_results.txt", help="Output file to store job results") +parser.add_argument("--pending-pod", type=str, default="pending_pods.txt", help="Output file to store number of pending pods") +parser.add_argument("--num-pod", type=int, default=1, help="Number of pods per job") +parser.add_argument("--gpu-options", type=int, nargs='+', default=[2, 4, 6, 8], help="Options for GPU requirements") +parser.add_argument("--probabilities", type=float, nargs='+', default=[0.25, 0.25, 0.25, 0.25], help="Probabilities for GPU requirements") +parser.add_argument("--mode", type=str, default="mcad", help="Mode: 'mcad' or 'nomcad'") +parser.add_argument("--israndom", type=lambda x: x.lower() in ['true', '1', 'yes'], default=True, help="Use random.expovariate for job request arrival and job size") +parser.add_argument("--usecosched", type=lambda x: x.lower() in ['true', '1', 'yes'], default=False, help="If true, then use coscheduler with number of pods specified by --num-pod argument") +args = parser.parse_args() + +# Set random seed +random.seed(datetime.now().timestamp()) + +# Parameters from command line arguments +mean_arrival = args.mean_arrival +total_jobs = args.total_jobs +job_size = args.job_size +output_file = args.output_file +output_file2 = args.pending_pod +num_pod_per_job = args.num_pod +gpu_options = args.gpu_options +probabilities = args.probabilities +mode = args.mode + +if sum(probabilities) != 1: + raise ValueError("Probabilities for GPU requirements should add up to 1") + +# Lists to store job submission times and pending pod counts +submission_times = [] +pending_pods_counts = [] + +# get the start time of the experiment +start = datetime.now().timestamp() + +# Simulate job requests +for i in range(total_jobs): + # Simulate job request arrival based on the mode + if args.israndom: + inter_arrival_time = random.expovariate(1 / mean_arrival) + else: + inter_arrival_time = mean_arrival + print("Next arrival after " + str(inter_arrival_time) + " seconds \n") + time.sleep(inter_arrival_time) + + # Generate job request + gpu_required = generate_job_request(gpu_options, probabilities) + + # Generate job size taken from exponential distribution + # sleep_time = str(job_size) + 's' #str(random.expovariate(1 / job_size)) + 's' + if args.israndom: + sleep_time = str(random.expovariate(1 / job_size)) + 's' + else: + sleep_time = str(job_size) + 's' + + # Generate YAML file for job request based on the mode + if mode == "mcad": + if args.usecosched: + yaml_content = generate_mcadcosched_yaml(i + 1, gpu_required, sleep_time, num_pod_per_job) + else: + yaml_content = generate_mcad_yaml(i + 1, gpu_required, sleep_time) + elif mode == "nomcad": + if args.usecosched: + yaml_content = generate_nomcadcosched_yaml(i + 1, gpu_required, sleep_time, num_pod_per_job) + else: + yaml_content = generate_nomcad_yaml(i + 1, gpu_required, sleep_time) + else: + raise ValueError("Invalid mode. Use 'mcad' or 'nomcad'.") + + # Write YAML content to file + yaml_filename = f"job-{i + 1}.yaml" + with open(yaml_filename, "w") as file: + file.write(yaml_content) + + # Apply Kubernetes YAML file + subprocess.run(["kubectl", "apply", "-f", f"job-{i + 1}.yaml"]) + print("Job YAML applied") + + # Get the number of pending pods + pending_pods = get_pending_pods() + submission_times.append(datetime.now().timestamp() - start) + pending_pods_counts.append(pending_pods) + + # Print job details + print(f"Job request {i + 1}: GPU(s) required: {gpu_required}") + print("Job size is " + sleep_time) + print(f"Number of pending pods: {pending_pods}") + + # Delete YAML file + os.remove(yaml_filename) + print(f"YAML file {yaml_filename} deleted\n") + +# Check for completion of all jobs +job_status = check_job_status(num_pod_per_job) +while job_status < total_jobs: + print("Number of completed jobs is:", job_status, "and the goal is:", total_jobs) + time.sleep(10) + job_status = check_job_status(num_pod_per_job) + +# Run kubectl commands and save outputs to files +print("Sending output...") +generate_output(output_file, json.loads(subprocess.run(["kubectl", "get", "jobs", "-o", "json"], capture_output=True, text=True).stdout), mode) +write_pending_pods_to_file(output_file2, submission_times, pending_pods_counts) +print("All job requests processed.") diff --git a/test/kwok-tests/kwok.yaml b/test/kwok-tests/kwok.yaml new file mode 100644 index 000000000..d3400b273 --- /dev/null +++ b/test/kwok-tests/kwok.yaml @@ -0,0 +1,385 @@ +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-create +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'DoesNotExist' + - key: '.status.podIP' + operator: 'DoesNotExist' + weight: 1 + delay: + durationMilliseconds: 100 + jitterDurationMilliseconds: 500 + next: + event: + type: Normal + reason: Created + message: Created container + finalizers: + add: + - value: 'kwok.x-k8s.io/fake' + statusTemplate: | + {{ $now := Now }} + + conditions: + {{ if .spec.initContainers }} + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: 'containers with incomplete status: [{{ range .spec.initContainers }} {{ .name }} {{ end }}]' + reason: ContainersNotInitialized + status: "False" + type: Initialized + {{ else }} + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + status: "True" + type: Initialized + {{ end }} + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: 'containers with unready status: [{{ range .spec.containers }} {{ .name }} {{ end }}]' + reason: ContainersNotReady + status: "False" + type: Ready + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: 'containers with unready status: [{{ range .spec.containers }} {{ .name }} {{ end }}]' + reason: ContainersNotReady + status: "False" + type: ContainersReady + {{ range .spec.readinessGates }} + - lastTransitionTime: {{ $now }} + status: "True" + type: {{ .conditionType }} + {{ end }} + + {{ if .spec.initContainers }} + initContainerStatuses: + {{ range .spec.initContainers }} + - image: {{ .image }} + name: {{ .name }} + ready: false + restartCount: 0 + started: false + state: + waiting: + reason: PodInitializing + {{ end }} + containerStatuses: + {{ range .spec.containers }} + - image: {{ .image }} + name: {{ .name }} + ready: false + restartCount: 0 + started: false + state: + waiting: + reason: PodInitializing + {{ end }} + {{ else }} + containerStatuses: + {{ range .spec.containers }} + - image: {{ .image }} + name: {{ .name }} + ready: false + restartCount: 0 + started: false + state: + waiting: + reason: ContainerCreating + {{ end }} + {{ end }} + + hostIP: {{ NodeIPWith .spec.nodeName }} + podIP: {{ PodIPWith .spec.nodeName ( or .spec.hostNetwork false ) ( or .metadata.uid "" ) ( or .metadata.name "" ) ( or .metadata.namespace "" ) }} + phase: Pending + startTime: '{{ $now }}' +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-init-container-running +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'DoesNotExist' + - key: '.status.phase' + operator: 'In' + values: + - 'Pending' + - key: '.status.conditions.[] | select( .type == "Initialized" ) | .status' + operator: 'NotIn' + values: + - 'True' + - key: '.status.initContainerStatuses.[].state.waiting.reason' + operator: 'Exists' + weight: 1 + delay: + durationMilliseconds: 1000 + jitterDurationMilliseconds: 5000 + next: + statusTemplate: | + {{ $now := Now }} + {{ $root := . }} + initContainerStatuses: + {{ range $index, $item := .spec.initContainers }} + {{ $origin := index $root.status.initContainerStatuses $index }} + - image: {{ $item.image }} + name: {{ $item.name }} + ready: true + restartCount: 0 + started: true + state: + running: + startedAt: '{{ $now }}' + {{ end }} +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-init-container-completed +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'DoesNotExist' + - key: '.status.phase' + operator: 'In' + values: + - 'Pending' + - key: '.status.conditions.[] | select( .type == "Initialized" ) | .status' + operator: 'NotIn' + values: + - 'True' + - key: '.status.initContainerStatuses.[].state.running.startedAt' + operator: 'Exists' + weight: 1 + delay: + durationMilliseconds: 1000 + jitterDurationMilliseconds: 5000 + next: + statusTemplate: | + {{ $now := Now }} + {{ $root := . }} + conditions: + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + status: "True" + type: Initialized + initContainerStatuses: + {{ range $index, $item := .spec.initContainers }} + {{ $origin := index $root.status.initContainerStatuses $index }} + - image: {{ $item.image }} + name: {{ $item.name }} + ready: true + restartCount: 0 + started: false + state: + terminated: + exitCode: 0 + finishedAt: '{{ $now }}' + reason: Completed + startedAt: '{{ $now }}' + {{ end }} + containerStatuses: + {{ range .spec.containers }} + - image: {{ .image }} + name: {{ .name }} + ready: false + restartCount: 0 + started: false + state: + waiting: + reason: ContainerCreating + {{ end }} +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-ready +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'DoesNotExist' + - key: '.status.phase' + operator: 'In' + values: + - 'Pending' + - key: '.status.conditions.[] | select( .type == "Initialized" ) | .status' + operator: 'In' + values: + - 'True' + - key: '.status.conditions.[] | select( .type == "ContainersReady" ) | .status' + operator: 'NotIn' + values: + - 'True' + weight: 1 + delay: + durationMilliseconds: 10000 + jitterDurationMilliseconds: 11000 + next: + delete: false + statusTemplate: | + {{ $now := Now }} + {{ $root := . }} + conditions: + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: '' + reason: '' + status: "True" + type: Ready + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: '' + reason: '' + status: "True" + type: ContainersReady + containerStatuses: + {{ range $index, $item := .spec.containers }} + {{ $origin := index $root.status.containerStatuses $index }} + - image: {{ $item.image }} + name: {{ $item.name }} + ready: true + restartCount: 0 + started: true + state: + running: + startedAt: '{{ $now }}' + {{ end }} + phase: Running +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-complete +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'DoesNotExist' + - key: '.status.phase' + operator: 'In' + values: + - 'Running' + - key: '.status.conditions.[] | select( .type == "Ready" ) | .status' + operator: 'In' + values: + - 'True' + - key: '.metadata.ownerReferences.[].kind' + operator: 'In' + values: + - 'Job' + - key: '.spec.containers[0].args[0]' + operator: 'In' + values: + - 'sleep' + weight: 1 + delay: + durationFrom: + expressionFrom: '.spec.containers[0].args[1]' + next: + delete: false + statusTemplate: | + {{ $now := Now }} + {{ $root := . }} + conditions: + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: '' + reason: 'PodCompleted' + status: "False" + type: Ready + - lastProbeTime: null + lastTransitionTime: '{{ $now }}' + message: '' + reason: 'PodCompleted' + status: "False" + type: ContainersReady + containerStatuses: + {{ range $index, $item := .spec.containers }} + {{ $origin := index $root.status.containerStatuses $index }} + - image: {{ $item.image }} + name: {{ $item.name }} + ready: true + restartCount: 0 + started: false + state: + terminated: + exitCode: 0 + finishedAt: '{{ $now }}' + reason: Completed + startedAt: '{{ index $root.status.containerStatuses 0 "state" "running" "startedAt" }}' + {{ end }} + phase: Succeeded +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-remove-finalizer +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'Exists' + - key: '.metadata.finalizers.[]' + operator: 'In' + values: + - 'kwok.x-k8s.io/fake' + weight: 1 + delay: + durationMilliseconds: 1000 + jitterDurationMilliseconds: 5000 + next: + finalizers: + remove: + - value: 'kwok.x-k8s.io/fake' + event: + type: Normal + reason: Killing + message: Stopping container +--- +kind: Stage +apiVersion: kwok.x-k8s.io/v1alpha1 +metadata: + name: pod-delete +spec: + resourceRef: + apiGroup: v1 + kind: Pod + selector: + matchExpressions: + - key: '.metadata.deletionTimestamp' + operator: 'Exists' + - key: '.metadata.finalizers' + operator: 'DoesNotExist' + weight: 1 + delay: + durationMilliseconds: 1000 + jitterDurationFrom: + expressionFrom: '.metadata.deletionTimestamp' + next: + delete: true diff --git a/test/kwok-tests/node.yaml b/test/kwok-tests/node.yaml new file mode 100644 index 000000000..0670a8010 --- /dev/null +++ b/test/kwok-tests/node.yaml @@ -0,0 +1,44 @@ +apiVersion: v1 +kind: Node +metadata: + annotations: + node.alpha.kubernetes.io/ttl: "0" + kwok.x-k8s.io/node: fake + labels: + beta.kubernetes.io/arch: amd64 + beta.kubernetes.io/os: linux + kubernetes.io/arch: amd64 + kubernetes.io/hostname: kwok-node-0 + kubernetes.io/os: linux + kubernetes.io/role: agent + node-role.kubernetes.io/agent: "" + type: kwok + name: kwok-node-0 +spec: + taints: # Avoid scheduling actual running pods to fake Node + - effect: NoSchedule + key: kwok.x-k8s.io/node + value: fake +status: + allocatable: + cpu: 5000m + nvidia.com/gpu: 8 + memory: 256Gi + pods: 110 + capacity: + cpu: 5000m + nvidia.com/gpu: 8 + memory: 256Gi + pods: 110 + nodeInfo: + architecture: amd64 + bootID: "" + containerRuntimeVersion: "" + kernelVersion: "" + kubeProxyVersion: fake + kubeletVersion: fake + machineID: "" + operatingSystem: linux + osImage: "" + systemUUID: "" + phase: Running diff --git a/test/kwok-tests/nodes.sh b/test/kwok-tests/nodes.sh new file mode 100755 index 000000000..5ed74c123 --- /dev/null +++ b/test/kwok-tests/nodes.sh @@ -0,0 +1,113 @@ +#!/bin/bash + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +function help() { + echo "usage: nodes.sh [-h]" + echo + echo "Description: Creates fake KWOK nodes for performance testing" + echo + echo "Preconditions: " + echo " - The script assumes you've logged into your cluster already. If not, it will tell you to login." + echo " - The script checks that you have the kwok-controller installed, otherwise it'll tell you to install it first." + echo + echo "Options:" + echo " -h Print this help message" + echo +} + +function check_kubectl_login_status() { + set +e + kubectl get ns default &> /dev/null + res="$?" + set -e + OCP="$res" + if [ $OCP == 1 ] + then + echo "You need to login to your Kubernetes Cluster" + exit 1 + else + echo + echo "Nice, looks like you're logged in" + echo "" + fi +} + +function check_kwok_installed_status() { + set +e + kubectl get pod -A |grep kwok-controller &> /dev/null + res2="$?" + set -e + KWOK="$res2" + if [[ $KWOK == 1 ]] + then + echo "You need Install the KWOK Controller first before running this script" + exit 1 + else + echo "Nice, the KWOK Controller is installed" + fi +} + +while getopts hf: option; do + case $option in + h) + help + exit 0 + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + +# Track whether we have a valid kubectl login +echo "Checking whether we have a valid cluster login or not..." +check_kubectl_login_status + +echo +read -p "How many simulated KWOK nodes do you want?" nodes + +echo "Nodes number is $nodes" +echo " " + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realnodes=$nodes-1)) +echo "The real number of nodes is $realnodes" + +for num in $(eval echo "{0.."$realnodes"}") +do + next_num=$(($num + 1)) + echo "Submitting node $next_num" +# Had to do this OSTYPE because sed acts differently on Linux versus Mac + case "$OSTYPE" in + linux-gnu*) + sed -i "s/kwok-node-$num/kwok-node-$next_num/g" ${SCRIPT_DIR}/node.yaml ;; + darwin*) + sed -i '' "s/kwok-node-$num/kwok-node-$next_num/g" ${SCRIPT_DIR}/node.yaml ${SCRIPT_DIR}/node.yaml ;; + *) + sed -i "/kwok-node-$num/kwok-node-$next_num/g" ${SCRIPT_DIR}/node.yaml ;; + esac + kubectl apply -f ${SCRIPT_DIR}/node.yaml +done + + # Let's reset the original node.yaml file back to original value + case "$OSTYPE" in + linux-gnu*) + sed -i "s/kwok-node-$next_num/kwok-node-0/g" ${SCRIPT_DIR}/node.yaml ;; + darwin*) + sed -i '' "s/kwok-node-$next_num/kwok-node-0/g" ${SCRIPT_DIR}/node.yaml ;; + *) + sed -i "s/kwok-node-$next_num/kwok-node-0/g" ${SCRIPT_DIR}/node.yaml ;; + esac + +# Check for all nodes to report complete +echo "Waiting until all the simulated pods become ready:" +kubectl wait --for=condition=Ready nodes --selector type=kwok --timeout=600s +echo " " +echo "Total amount of simulated nodes requested is: $nodes" +echo "Total number of created nodes is: "`kubectl get nodes --selector type=kwok -o name |wc -l` +kubectl get nodes --selector type=kwok + +echo " " +echo "FYI, to clean up the kwow nodes, issue this:" +echo "kubectl get nodes --selector type=kwok -o name | xargs kubectl delete" diff --git a/test/kwok-tests/stress-tests-kwok/README.md b/test/kwok-tests/stress-tests-kwok/README.md new file mode 100644 index 000000000..4b361b0bc --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/README.md @@ -0,0 +1,155 @@ +## Stress Test 1: KWOK vs KIND +**Kind Cluster:** 4 nodes, each with 5 CPUs and 8 GB of memory +*We add node affinity to Pod spec to allow pods to be scheduled only on the worker nodes.* +In the `stress-baseline-cpu-j-s` directory, an example run looks as follows: +``` +stress-baseline-cpu-j-s % ./baseline-avg-timing.sh -j 10 -g 0 -a 1 +. +. +. +Average time per iteration: 23.00 seconds +``` + +**KWOK Cluster:** A cluster of 4 fake nodes managed by the KWOK controller is set up using the `./nodes.sh` script. +An example run within the `stress-nomcadkwok-cpu-j-s` directory shows: +``` +stress-nomcadkwok-cpu-j-s % ./nomcadkwok-avg-timing.sh -j 10 -g 0 -a 1 +. +. +. +Average time per iteration: 23.00 seconds +``` +Both scripts take three arguments: `-j` for the number of jobs to be deployed, `-a` for the number of pods in a job, `-g` for the number of gpus per pod. Since this is a baseline experiment for small jobs, we set `-a 1` and `-g 0`. The scripts give the average time it took to complete `j` number of jobs since the first job was submitted and one can play with parameter `-j` to understand the rate of completion with respect to different load. + +**Performance Summary:** + +| Jobs | Pods | Total time to complete (s) | Cluster Info (KIND)| +|------|------|--------------------------|-------------------------------------------------| +| 1 | 1 | 23 | KIND | +| 10 | 10 | 23 | with 4 worker nodes, job spec has node affinity | +| 20 | 20 | 27 | | +| 50 | 50 | 33 | | +| 75 | 75 | 41 | | +| 100 | 100 | 49 | | + + +| Jobs | Pods | Total time to complete (s) | Cluster Info (KWOK) | +|------|------|--------------------------|-----------------------------------------------------------------| +| 1 | 1 | 23 | | +| 10 | 10 | 23 | 4 fake nodes, job spec has node affinity and tolerations | +| 20 | 20 | 25 | | +| 50 | 50 | 31 | | +| 75 | 75 | 36 | | +| 100 | 100 | 40 | | + + + +The difference in KIND and KWOK completion times can be majorly attributed to the container start time. One can see that as the number of job requests increases, the maximum container start time also increases. For example, when the workload consists of submitting `10` jobs to the Kind cluster, we see the following +``` +% kubectl get pods -o json | python3 -c 'import sys, json, datetime; pods = json.load(sys.stdin)["items"]; [print("POD NAME", "PODSTARTTIME", "CONTAINERSTARTTIME", "CONTAINERFINISHTIME", "TIMETOSTART")] + [print(pod["metadata"]["name"], pod["status"]["startTime"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["finishedAt"], (datetime.datetime.fromisoformat(pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"].replace("Z", "+00:00")) - datetime.datetime.fromisoformat(pod["status"]["startTime"].replace("Z", "+00:00")))) for pod in pods]' + +POD NAME PODSTARTTIME CONTAINERSTARTTIME CONTAINERFINISHTIME TIMETOSTART +baseline-cpu-job-short-1-k6gm6 2023-07-06T13:54:49Z 2023-07-06T13:54:57Z 2023-07-06T13:55:07Z 0:00:08 +baseline-cpu-job-short-10-ljwds 2023-07-06T13:54:50Z 2023-07-06T13:55:00Z 2023-07-06T13:55:10Z 0:00:10 +. +. +. +``` + +However, when `100` jobs are submitted, then: + +``` +% kubectl get pods -o json | python3 -c 'import sys, json, datetime; pods = json.load(sys.stdin)["items"]; [print("POD NAME", "PODSTARTTIME", "CONTAINERSTARTTIME", "CONTAINERFINISHTIME", "TIMETOSTART")] + [print(pod["metadata"]["name"], pod["status"]["startTime"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["finishedAt"], (datetime.datetime.fromisoformat(pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"].replace("Z", "+00:00")) - datetime.datetime.fromisoformat(pod["status"]["startTime"].replace("Z", "+00:00")))) for pod in pods]' + +POD NAME PODSTARTTIME CONTAINERSTARTTIME CONTAINERFINISHTIME TIMETOSTART +. +. +. +baseline-cpu-job-short-31-xvlt4 2023-07-06T14:10:05Z 2023-07-06T14:10:25Z 2023-07-06T14:10:35Z 0:00:20 +baseline-cpu-job-short-32-w69jt 2023-07-06T14:10:05Z 2023-07-06T14:10:30Z 2023-07-06T14:10:41Z 0:00:25 +baseline-cpu-job-short-33-5kptg 2023-07-06T14:10:05Z 2023-07-06T14:10:31Z 2023-07-06T14:10:41Z 0:00:26 +baseline-cpu-job-short-34-5pxth 2023-07-06T14:10:05Z 2023-07-06T14:10:30Z 2023-07-06T14:10:41Z 0:00:25 +baseline-cpu-job-short-35-48gr9 2023-07-06T14:10:05Z 2023-07-06T14:10:30Z 2023-07-06T14:10:41Z 0:00:25 +. +. +. +``` + +For KWOK on the contrary, the container start time is simulated with fixed delay, and moreover there is no Kubelet that becomes a bottleneck in KIND. Therefore, we see almost constant container start time in KWOK. +*With 10 jobs:* +``` +% kubectl get pods -o json | python3 -c 'import sys, json, datetime; pods = json.load(sys.stdin)["items"]; [print("POD NAME", "PODSTARTTIME", "CONTAINERSTARTTIME", "CONTAINERFINISHTIME", "TIMETOSTART")] + [print(pod["metadata"]["name"], pod["status"]["startTime"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["finishedAt"], (datetime.datetime.fromisoformat(pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"].replace("Z", "+00:00")) - datetime.datetime.fromisoformat(pod["status"]["startTime"].replace("Z", "+00:00")))) for pod in pods]' + +POD NAME PODSTARTTIME CONTAINERSTARTTIME CONTAINERFINISHTIME TIMETOSTART +nomcadkwok-cpu-job-short-1-knfcg 2023-07-06T14:33:55Z 2023-07-06T14:34:06Z 2023-07-06T14:34:16Z 0:00:11 +nomcadkwok-cpu-job-short-10-52c6c 2023-07-06T14:33:55Z 2023-07-06T14:34:06Z 2023-07-06T14:34:17Z 0:00:11 +. +. +. +``` + +*With 100 jobs:* +``` +% kubectl get pods -o json | python3 -c 'import sys, json, datetime; pods = json.load(sys.stdin)["items"]; [print("POD NAME", "PODSTARTTIME", "CONTAINERSTARTTIME", "CONTAINERFINISHTIME", "TIMETOSTART")] + [print(pod["metadata"]["name"], pod["status"]["startTime"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"], pod["status"]["containerStatuses"][0]["state"]["terminated"]["finishedAt"], (datetime.datetime.fromisoformat(pod["status"]["containerStatuses"][0]["state"]["terminated"]["startedAt"].replace("Z", "+00:00")) - datetime.datetime.fromisoformat(pod["status"]["startTime"].replace("Z", "+00:00")))) for pod in pods]' + +POD NAME PODSTARTTIME CONTAINERSTARTTIME CONTAINERFINISHTIME TIMETOSTART +nomcadkwok-cpu-job-short-1-jvbgk 2023-07-06T14:41:38Z 2023-07-06T14:41:49Z 2023-07-06T14:41:59Z 0:00:11 +nomcadkwok-cpu-job-short-10-5bmlh 2023-07-06T14:41:39Z 2023-07-06T14:41:50Z 2023-07-06T14:42:00Z 0:00:11 +nomcadkwok-cpu-job-short-100-w5wjv 2023-07-06T14:41:49Z 2023-07-06T14:42:00Z 2023-07-06T14:42:10Z 0:00:11 +. +. +. +``` + + + +## Stress Test 2: Dispatch performance of MCAD with large number of small jobs +The aim of this experiment is to analyze the potential overhead introduced by the Multi-Cluster Application Dispatcher (MCAD) when dealing with a substantial volume of small application wrapper (AW) jobs, each consisting of a single pod. The key metric assessed in this experiment is the time taken for all jobs to complete. +We use KWOK cluster with 4 nodes to test MCAD dispatch. + + +**MCAD Controller Deployment:** For this test, one should have MCAD controller deployed on the cluster. Follow [MCAD Deployment](https://github.com/project-codeflare/multi-cluster-app-dispatcher/blob/main/test/perf-test/simulatingnodesandappwrappers.md#step-1-deploy-mcad-on-your-cluster) to deploy and run MCAD controller on your cluster. + +To run the stress test with KWOK and MCAD, go to `stress-mcad-cpu-j-s` directory, and run: +``` +stress-mcad-cpu-j-s % ./mcad-avg-timing.sh -j 1 -g 0 -a 1 +. +. +. +Average time per iteration: 23.00 seconds +``` + + +The observed longer job creation times in MCAD are due to the additional processing that an AW undergoes before dispatching as a Job. AWs experience multiple queueing stages prior to being dispatched. After dispatching, the job is handled by the Kubernetes Job Controller. The slower creation time does not affect the actual time taken for the job to complete. +**Note: The dispatch rate of MCAD controller for this experiment looks like 1 job per second** + +``` +% kubectl get appwrapper -o custom-columns=AWNAME:.metadata.name,AWCREATED:.status.controllerfirsttimestamp +AWNAME AWCREATED +mcadkwok-cpu-job-short-1 2023-07-06T15:16:32.524203Z +. +. +. +mcadkwok-cpu-job-short-50 2023-07-06T15:16:44.335615Z +``` + +``` +% kubectl get jobs -o custom-columns=JOBNAME:.metadata.name,JOBCREATED:.metadata.creationTimestamp,JOBCOMPLETED:.status.completionTime +JOBNAME JOBCREATED JOBCOMPLETED +mcadkwok-cpu-job-short-1 2023-07-06T15:16:32Z 2023-07-06T15:16:55Z +. +. +. +mcadkwok-cpu-job-short-50 2023-07-06T15:17:15Z 2023-07-06T15:17:37Z +``` + +**Performance Summary of KWOK with MCAD** +| Jobs | Pods | Total time to complete (s) | Cluster Info | +|--------|------|----------------------------|-----------------------------------------------------------| +| 1 | 1 | 23 | KWOK with MCAD | +| 10 | 10 | 31 | 4 fake nodes, job spec has node affinity and tolerations | +| 20 | 20 | 44 | | +| 50 | 50 | 82 | | +| 75 | 75 | 118 | | +| 100 | 100 | 135 | | + diff --git a/test/kwok-tests/stress-tests-kwok/kwokmcadperf.sh b/test/kwok-tests/stress-tests-kwok/kwokmcadperf.sh new file mode 100644 index 000000000..06b0dca14 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/kwokmcadperf.sh @@ -0,0 +1,188 @@ +#!/bin/bash + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +function help() { + echo "usage: kwokmcadperf.sh [-h]" + echo + echo "Description: Runs Appwrapper performance test script(s) in subdirectories under $SCRIPT_DIR." + echo "NOTE: This runs on KWOK Fake nodes only." + echo + echo "Preconditions: " + echo " - The script assumes you've logged into your cluster already. If not, it will tell you to login." + echo " - The script checks that you have the mcad-controller installed, otherwise it'll tell you to install it first." + echo " - The script checks that you have the kwok-controller installed, otherwise it'll tell you to install it first." + echo + echo "Options:" + echo " -h Print this help message" + echo +} + +function check_kubectl_login_status() { + set +e + kubectl get ns default &> /dev/null + res="$?" + set -e + OCP="$res" + if [ $OCP == 1 ] + then + echo "You need to login to your Kubernetes Cluster" + exit 1 + else + echo + echo "Nice, looks like you're logged in" + fi +} + +function check_mcad_installed_status() { + set +e + kubectl get pod -A |grep mcad-controller &> /dev/null + res2="$?" + kubectl get crd |grep appwrapper &> /dev/null + res3="$?" + set -e + MCAD="$res2" + CRD="$res3" + if [[ $MCAD == 1 ]] || [[ $CRD == 1 ]] + then + echo "You need Install MCAD Controller first before running this script" + exit 1 + else + echo "Nice, MCAD Controller is installed" + fi +} + +function check_kwok_installed_status() { + set +e + kubectl get pod -A |grep kwok-controller &> /dev/null + res2="$?" + set -e + KWOK="$res2" + if [[ $KWOK == 1 ]] + then + echo "You need Install the KWOK Controller first before running this script" + exit 1 + else + echo "Nice, the KWOK Controller is installed" + fi +} + + +while getopts hf: option; do + case $option in + h) + help + exit 0 + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + +# Track whether we have a valid kubectl login +echo "Checking whether we have a valid cluster login or not..." +check_kubectl_login_status + +# Track whether you have the MCAD controller installed +echo "Checking MCAD Controller installation status" +echo +check_mcad_installed_status + +# Track whether you have the KWOK controller installed +echo "Checking MCAD Controller installation status" +echo +check_kwok_installed_status + +echo +read -p "How many fake KWOK appwrapper jobs do you want? " jobs +read -p "How many pods in a job? " awjobs +read -p "How many GPUs do you want to allocate per pod? " gpus + +# Start the timer now +SECONDS=0 + +echo "jobs number is $jobs" +echo "Number of GPUs per pod: $gpus" +echo "Number of pods per AppWrapper: $awjobs" +export STARTTIME=`date +"%T"` +echo " " +echo "Jobs started at: $STARTTIME" |tee fake-job-$STARTTIME.log +echo " " + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) + +for num in $(eval echo "{0.."$realjobs"}") +do + next_num=$(($num + 1)) + echo "Submitting job $next_num" +# Had to do this OSTYPE because sed acts differently on Linux versus Mac + case "$OSTYPE" in + linux-gnu*) + sed -i "s/fake-defaultaw-schd-spec-with-timeout-$num/fake-defaultaw-schd-spec-with-timeout-$next_num/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/parallelism: 1/parallelism: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/completions: 1/completions: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + darwin*) + sed -i '' "s/fake-defaultaw-schd-spec-with-timeout-$num/fake-defaultaw-schd-spec-with-timeout-$next_num/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/parallelism: 1/parallelism: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/completions: 1/completions: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + *) + sed -i "s/fake-defaultaw-schd-spec-with-timeout-$num/fake-defaultaw-schd-spec-with-timeout-$next_num/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/parallelism: 1/parallelism: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/completions: 1/completions: $awjobs/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + esac + kubectl apply -f ${SCRIPT_DIR}/preempt-exp-kwok.yaml +done + + # Let's reset the original preempt-exp-kwok.yaml file back to original value + case "$OSTYPE" in + linux-gnu*) + sed -i "s/fake-defaultaw-schd-spec-with-timeout-$next_num/fake-defaultaw-schd-spec-with-timeout-1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/nvidia.com\/gpu: $gpus/nvidia.com\/gpu: 0/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/parallelism: $awjobs/parallelism: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/completions: $awjobs/completions: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + darwin*) + sed -i '' "s/fake-defaultaw-schd-spec-with-timeout-$next_num/fake-defaultaw-schd-spec-with-timeout-1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/nvidia.com\/gpu: $gpus/nvidia.com\/gpu: 0/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/parallelism: $awjobs/parallelism: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i '' "s/completions: $awjobs/completions: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + *) + sed -i "s/fake-defaultaw-schd-spec-with-timeout-$next_num/fake-defaultaw-schd-spec-with-timeout-1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/nvidia.com\/gpu: $gpus/nvidia.com\/gpu: 0/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/parallelism: $awjobs/parallelism: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml + sed -i "s/completions: $awjobs/completions: 1/g" ${SCRIPT_DIR}/preempt-exp-kwok.yaml ;; + esac + +# Check for all jobs to report complete +jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=1 |wc -l` + +while [ $jobstatus -lt $jobs ] +do + echo "Number of completed jobs is: " $jobstatus " and the goal is: " $jobs + sleep 1 + jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=1 |wc -l` +done + +echo " " +export FINISHTIME=`date +"%T"` +echo "All $jobstatus jobs finished: $FINISHTIME" |tee -a fake-job-$STARTTIME.log +echo "Total amount of time for $jobs appwrappers is: $SECONDS seconds" |tee -a ${SCRIPT_DIR}/fake-job-$STARTTIME.log +echo " " +echo "Test results are stored in this file: ${SCRIPT_DIR}/fake-job-$next_num-$STARTTIME.log" + +# Rename the log to show the number of jobs used +mv ${SCRIPT_DIR}/fake-job-$STARTTIME.log ${SCRIPT_DIR}/fake-job-$next_num-$STARTTIME.log + +#Ask if you want to auto-cleanup the appwrapper jobs +echo "Do you want to cleanup the most recently created appwrappers? [Y/n]" +read DELETE +if [[ $DELETE == "Y" || $DELETE == "y" ]]; then + echo "OK, deleting" + ${SCRIPT_DIR}/cleanup-mcad-kwok.sh +else + echo "OK, you'll need to cleanup yourself later using ./cleanup-mcad-kwok.sh" +fi diff --git a/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-avg-timing.sh b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-avg-timing.sh new file mode 100755 index 000000000..86f543c82 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-avg-timing.sh @@ -0,0 +1,151 @@ +#!/bin/bash + +# clear the cluster for any potential jobs +echo "Deleting any jobs if any..." +kubectl delete jobs --all -n default +echo " " + +# clear any existing output directory containing yamls +echo "Deleting any previous output directory if any..." +rm -r output_dir +echo " " + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +function help() { + echo "usage: baseline-avg-timing.sh [-h] [-j ] [-g ] [-a ]" + echo + echo "Description: Runs performance test script(s) without MCAD in subdirectories under $SCRIPT_DIR." + echo "NOTE: This runs on KIND nodes only." + echo + echo "Preconditions: " + echo " - The script assumes you've logged into your cluster already. If not, it will tell you to login." + echo + echo "Options:" + echo " -h Print this help message" + echo " -j Number of jobs to run (default: 1)" + echo " -g Number of GPUs per job (default: 0)" + echo " -a Number of pods per job (default: 1)" + echo +} + +function check_kubectl_login_status() { + set +e + kubectl get ns default &> /dev/null + res="$?" + set -e + OCP="$res" + if [ $OCP == 1 ] + then + echo "You need to login to your Kubernetes Cluster" + exit 1 + else + echo + echo "Nice, looks like you're logged in" + fi +} + +# Function to extract the total time from the output +extract_total_time() { + awk '/real jobs without MCAD is: .* seconds/ {print $(NF-1)}' "$1" +} + +# Track whether we have a valid kubectl login +echo "Checking whether we have a valid cluster login or not..." +check_kubectl_login_status + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + +echo "jobs number is $jobs" +echo "Number of GPUs per pod: $gpus" +echo "Number of pods per job: $awjobs" + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="$(pwd)/$subdirectory" + +# Create the output directory if it doesn't exist +mkdir -p "$output_dir" + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) +for num in $(eval echo "{0.."$realjobs"}"); do + next_num=$((num + 1)) + cp stress-baseline-cpu-j-s.yaml "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + + # Had to do this OSTYPE because sed acts differently on Linux versus Mac + case "$OSTYPE" in + linux-gnu*) + sed -i "s/baseline-cpu-job-short-0/baseline-cpu-job-short-$next_num/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" ;; + darwin*) + sed -i '' "s/baseline-cpu-job-short-0/baseline-cpu-job-short-$next_num/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i '' "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i '' "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i '' "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" ;; + *) + sed -i "s/baseline-cpu-job-short-0/baseline-cpu-job-short-$next_num/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" ;; + esac +done + +# Define the number of iterations +iterations=1 + +# Perform the iterations +total_time=0 +for ((i=1; i<=iterations; i++)); do + # Run the script and capture the output in a temporary file + output_file=$(mktemp) + echo "Now calling baseline-stress-test.sh" + echo " " + ./baseline-stress-test.sh -j "$jobs" -g "$gpus" -a "$awjobs" > "$output_file" + # Extract the total time from the output + time=$(extract_total_time "$output_file") + # Accumulate the total time + total_time=$(bc <<< "$total_time + $time") + # Remove the temporary file + echo " " + echo "Iteration $i complete" + echo "Deleting all jobs for the fresh next run" + ./cleanup-baseline-cpu-j-s.sh > "$output_file" + rm "$output_file" + echo " " + sleep 10 +done + +# Calculate the average time +average_time=$(bc <<< "scale=2; $total_time / $iterations") +echo "Average time per iteration: $average_time seconds" + +# Delete the output directory +rm -r "$output_dir" + + diff --git a/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-stress-test.sh b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-stress-test.sh new file mode 100755 index 000000000..c5851cc1e --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/baseline-stress-test.sh @@ -0,0 +1,62 @@ +#!/bin/bash + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="${SCRIPT_DIR}/${subdirectory}" + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) + +# Start the timer now +SECONDS=0 +export STARTTIME=`date +"%T"` + +for num in $(eval echo "{0.."$realjobs"}") +do + next_num=$((num + 1)) + kubectl apply -f "$output_dir/stress-baseline-cpu-j-s-$next_num.yaml" +done + + +# Check for all jobs to report complete +jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep baseline | wc -l` + +while [ $jobstatus -lt $jobs ] +do + echo "Number of completed jobs is: " $jobstatus " and the goal is: " $jobs + sleep 1 + jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep baseline | wc -l` +done + +# kubectl wait --for=condition=complete --timeout=-30s --all job + +export FINISHTIME=`date +"%T"` +echo "Total amount of time for $jobs real jobs without MCAD is: $SECONDS seconds" diff --git a/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/cleanup-baseline-cpu-j-s.sh b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/cleanup-baseline-cpu-j-s.sh new file mode 100755 index 000000000..bb4733311 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/cleanup-baseline-cpu-j-s.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +for i in `kubectl get job -n default |grep baseline-cpu-job-short | awk '{print $1}'`; do kubectl delete job $i -n default ; done diff --git a/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/stress-baseline-cpu-j-s.yaml b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/stress-baseline-cpu-j-s.yaml new file mode 100644 index 000000000..457ba4db1 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-baseline-cpu-j-s/stress-baseline-cpu-j-s.yaml @@ -0,0 +1,35 @@ +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: baseline-cpu-job-short-0 +spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + spec: + containers: + - name: baseline-cpu-job-short-0 + image: nginx:1.24.0 + command: ["sleep", "10"] + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + restartPolicy: Never + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/worker + operator: In + values: + - worker diff --git a/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/aw-fake-deployment.yaml b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/aw-fake-deployment.yaml new file mode 100644 index 000000000..efbdf4eda --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/aw-fake-deployment.yaml @@ -0,0 +1,54 @@ +apiVersion: mcad.ibm.com/v1beta1 +kind: AppWrapper +metadata: + name: aw-fake-deployment-1 + namespace: default +spec: + schedulingSpec: + minAvailable: 1 + resources: + Items: + - replicas: 1 + type: Deployment + template: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: aw-fake-deployment-1 + labels: + app: aw-fake-deployment-1 + spec: + selector: + matchLabels: + app: aw-fake-deployment-1 + replicas: 1 + template: + metadata: + labels: + app: aw-fake-deployment-1 + spec: + containers: + - name: aw-fake-deployment-1 + image: nginx + resources: + requests: + cpu: 1000m + memory: 10M + nvidia.com/gpu: 16 + limits: + cpu: 1000m + memory: 128M + nvidia.com/gpu: 16 + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" diff --git a/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/cleanup-mcadkwok-cpu-j-s.sh b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/cleanup-mcadkwok-cpu-j-s.sh new file mode 100755 index 000000000..48e40075e --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/cleanup-mcadkwok-cpu-j-s.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +for i in `kubectl get appwrapper -n default |grep mcadkwok | awk '{print $1}'`; do kubectl delete appwrapper $i -n default ; done \ No newline at end of file diff --git a/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcad-avg-timing.sh b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcad-avg-timing.sh new file mode 100755 index 000000000..72cc16a76 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcad-avg-timing.sh @@ -0,0 +1,206 @@ +#!/bin/bash + +# clear the cluster for any potential jobs +echo "Deleting appwrappers if any..." +kubectl delete appwrappers --all -n default +echo " " + +# clear the cluster for any potential jobs +echo "Deleting any jobs if any..." +kubectl delete jobs --all -n default +echo " " + +# clear any existing output directory containing yamls +echo "Deleting any previous output directory if any..." +rm -r output_dir +echo " " + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +function help() { + echo "usage: stress-test [-h] [-j ] [-g ] [-a ]" + echo + echo "Description: Runs performance test script(s) with MCAD in subdirectories under $SCRIPT_DIR." + echo "NOTE: This runs on KWOK Fake nodes only." + echo + echo "Preconditions: " + echo " - The script assumes you've logged into your cluster already. If not, it will tell you to login." + echo " - The script checks that you have the kwok-controller installed, otherwise it'll tell you to install it first." + echo + echo "Options:" + echo " -h Print this help message" + echo " -j Number of jobs to run (default: 1000)" + echo " -g Number of GPUs per job (default: 0)" + echo " -a Number of pods per job (default: 1)" + echo +} + +function check_kubectl_login_status() { + set +e + kubectl get ns default &> /dev/null + res="$?" + set -e + OCP="$res" + if [ $OCP == 1 ] + then + echo "You need to login to your Kubernetes Cluster" + exit 1 + else + echo + echo "Nice, looks like you're logged in" + fi +} + +function check_kwok_installed_status() { + set +e + kubectl get pod -A |grep kwok-controller &> /dev/null + res2="$?" + set -e + KWOK="$res2" + if [[ $KWOK == 1 ]] + then + echo "You need Install the KWOK Controller first before running this script" + exit 1 + else + echo "Nice, the KWOK Controller is installed" + fi +} + +function check_mcad_installed_status() { + set +e + kubectl get pod -A |grep mcad-controller &> /dev/null + res2="$?" + kubectl get crd |grep appwrapper &> /dev/null + res3="$?" + set -e + MCAD="$res2" + CRD="$res3" + if [[ $MCAD == 1 ]] || [[ $CRD == 1 ]] + then + echo "You need Install MCAD Controller first before running this script" + exit 1 + else + echo "Nice, MCAD Controller is installed" + fi +} + +# Function to extract the total time from the output +extract_total_time() { + awk '/fake jobs with MCAD is: .* seconds/ {print $(NF-1)}' "$1" +} + +# Track whether we have a valid kubectl login +echo "Checking whether we have a valid cluster login or not..." +echo +check_kubectl_login_status + +# Track whether you have the MCAD controller installed +echo "Checking MCAD Controller installation status" +echo +check_mcad_installed_status + +## Commented since running KWOK out of the cluster +# # Track whether you have the KWOK controller installed +# echo "Checking MCAD Controller installation status" +# echo +# check_kwok_installed_status + + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + +echo "jobs number is $jobs" +echo "Number of GPUs per pod: $gpus" +echo "Number of pods per job: $awjobs" + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="$(pwd)/$subdirectory" + +# Create the output directory if it doesn't exist +mkdir -p "$output_dir" + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) +for num in $(eval echo "{0.."$realjobs"}"); do + next_num=$((num + 1)) + cp stress-mcadkwok-cpu-j-s.yaml "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + + # Had to do this OSTYPE because sed acts differently on Linux versus Mac + case "$OSTYPE" in + linux-gnu*) + sed -i "s/mcadkwok-cpu-job-short-0/mcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" ;; + darwin*) + sed -i '' "s/mcadkwok-cpu-job-short-0/mcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" ;; + *) + sed -i "s/mcadkwok-cpu-job-short-0/mcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" ;; + esac +done + +# Define the number of iterations +iterations=1 + +# Perform the iterations +total_time=0 +for ((i=1; i<=iterations; i++)); do + # Run the script and capture the output in a temporary file + output_file=$(mktemp) + echo "Now calling mcadkwok-stress-test.sh" + echo " " + ./mcadkwok-stress-test.sh -j "$jobs" -g "$gpus" -a "$awjobs" > "$output_file" + # Extract the total time from the output + time=$(extract_total_time "$output_file") + # Accumulate the total time + total_time=$(bc <<< "$total_time + $time") + # Remove the temporary file + echo " " + echo "Iteration $i complete" + echo "Deleting all jobs for the fresh next run" + ./cleanup-mcadkwok-cpu-j-s.sh > "$output_file" + rm "$output_file" + echo " " + kubectl scale deployment mcad-controller --replicas=0 -n kube-system + sleep 10 + kubectl scale deployment mcad-controller --replicas=1 -n kube-system + sleep 5 +done + +# Calculate the average time +average_time=$(bc <<< "scale=2; $total_time / $iterations") +echo "Average time per iteration: $average_time seconds" + +# Delete the output directory +rm -r "$output_dir" + + diff --git a/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcadkwok-stress-test.sh b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcadkwok-stress-test.sh new file mode 100755 index 000000000..4164db1c2 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/mcadkwok-stress-test.sh @@ -0,0 +1,62 @@ +#!/bin/bash + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="${SCRIPT_DIR}/${subdirectory}" + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) + +# Start the timer now +SECONDS=0 +export STARTTIME=`date +"%T"` + +for num in $(eval echo "{0.."$realjobs"}") +do + next_num=$((num + 1)) + kubectl apply -f "$output_dir/stress-mcadkwok-cpu-j-s-$next_num.yaml" +done + + +# Check for all jobs to report complete +jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep mcadkwok | wc -l` + +while [ $jobstatus -lt $jobs ] +do + echo "Number of completed jobs is: " $jobstatus " and the goal is: " $jobs + sleep 1 + jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep mcadkwok | wc -l` +done + +# kubectl wait --for=condition=complete --timeout=-30s --all job + +export FINISHTIME=`date +"%T"` +echo "Total amount of time for $jobs fake jobs with MCAD is: $SECONDS seconds" diff --git a/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/stress-mcadkwok-cpu-j-s.yaml b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/stress-mcadkwok-cpu-j-s.yaml new file mode 100644 index 000000000..65ec6af37 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-mcad-cpu-j-s/stress-mcadkwok-cpu-j-s.yaml @@ -0,0 +1,57 @@ +apiVersion: mcad.ibm.com/v1beta1 +kind: AppWrapper +metadata: + name: mcadkwok-cpu-job-short-0 + namespace: default +spec: + resources: + Items: [] + GenericItems: + - replicas: 1 + completionstatus: Complete + generictemplate: + apiVersion: batch/v1 + kind: Job + metadata: + namespace: default + name: mcadkwok-cpu-job-short-0 + spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + labels: + appwrapper.mcad.ibm.com: "mcadkwok-cpu-job-short-0" + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: mcadkwok-cpu-job-short-0 + image: nginx:1.24.0 + command: ["sleep", "10"] + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + restartPolicy: Never + + + + diff --git a/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/cleanup-nomcadkwok-cpu-j-s.sh b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/cleanup-nomcadkwok-cpu-j-s.sh new file mode 100755 index 000000000..4eea05494 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/cleanup-nomcadkwok-cpu-j-s.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +for i in `kubectl get job -n default |grep nomcadkwok-cpu-job-short | awk '{print $1}'`; do kubectl delete job $i -n default ; done diff --git a/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-avg-timing.sh b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-avg-timing.sh new file mode 100755 index 000000000..3d593660e --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-avg-timing.sh @@ -0,0 +1,174 @@ +#!/bin/bash + +# clear the cluster for any potential jobs +echo "Deleting any jobs if any..." +kubectl delete jobs --all -n default +echo " " + +# clear any existing output directory containing yamls +echo "Deleting any previous output directory if any..." +rm -r output_dir +echo " " + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +function help() { + echo "usage: stress-test [-h] [-j ] [-g ] [-a ]" + echo + echo "Description: Runs performance test script(s) without MCAD in subdirectories under $SCRIPT_DIR." + echo "NOTE: This runs on KWOK Fake nodes only." + echo + echo "Preconditions: " + echo " - The script assumes you've logged into your cluster already. If not, it will tell you to login." + echo " - The script checks that you have the kwok-controller installed, otherwise it'll tell you to install it first." + echo + echo "Options:" + echo " -h Print this help message" + echo " -j Number of jobs to run (default: 1000)" + echo " -g Number of GPUs per job (default: 0)" + echo " -a Number of pods per job (default: 1)" + echo +} + +function check_kubectl_login_status() { + set +e + kubectl get ns default &> /dev/null + res="$?" + set -e + OCP="$res" + if [ $OCP == 1 ] + then + echo "You need to login to your Kubernetes Cluster" + exit 1 + else + echo + echo "Nice, looks like you're logged in" + fi +} + +function check_kwok_installed_status() { + set +e + kubectl get pod -A |grep kwok-controller &> /dev/null + res2="$?" + set -e + KWOK="$res2" + if [[ $KWOK == 1 ]] + then + echo "You need Install the KWOK Controller first before running this script" + exit 1 + else + echo "Nice, the KWOK Controller is installed" + fi +} + +# Function to extract the total time from the output +extract_total_time() { + awk '/fake jobs without MCAD is: .* seconds/ {print $(NF-1)}' "$1" +} + +# Track whether we have a valid kubectl login +echo "Checking whether we have a valid cluster login or not..." +check_kubectl_login_status + +## Commented since running KWOK out of the cluster +# # Track whether you have the KWOK controller installed +# echo "Checking MCAD Controller installation status" +# echo +# check_kwok_installed_status + + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + +echo "jobs number is $jobs" +echo "Number of GPUs per pod: $gpus" +echo "Number of pods per job: $awjobs" + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="$(pwd)/$subdirectory" + +# Create the output directory if it doesn't exist +mkdir -p "$output_dir" + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) +for num in $(eval echo "{0.."$realjobs"}"); do + next_num=$((num + 1)) + cp stress-nomcadkwok-cpu-j-s.yaml "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + + # Had to do this OSTYPE because sed acts differently on Linux versus Mac + case "$OSTYPE" in + linux-gnu*) + sed -i "s/nomcadkwok-cpu-job-short-0/nomcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" ;; + darwin*) + sed -i '' "s/nomcadkwok-cpu-job-short-0/nomcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i '' "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" ;; + *) + sed -i "s/nomcadkwok-cpu-job-short-0/nomcadkwok-cpu-job-short-$next_num/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/nvidia.com\/gpu: 0/nvidia.com\/gpu: $gpus/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/parallelism: 1/parallelism: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" + sed -i "s/completions: 1/completions: $awjobs/g" "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" ;; + esac +done + +# Define the number of iterations +iterations=1 + +# Perform the iterations +total_time=0 +for ((i=1; i<=iterations; i++)); do + # Run the script and capture the output in a temporary file + output_file=$(mktemp) + echo "Now calling nomcadkwok-stress-test.sh" + echo " " + ./nomcadkwok-stress-test.sh -j "$jobs" -g "$gpus" -a "$awjobs" > "$output_file" + # Extract the total time from the output + time=$(extract_total_time "$output_file") + # Accumulate the total time + total_time=$(bc <<< "$total_time + $time") + # Remove the temporary file + echo " " + echo "Iteration $i complete" + echo "Deleting all jobs for the fresh next run" + ./cleanup-nomcadkwok-cpu-j-s.sh > "$output_file" + rm "$output_file" + echo " " + sleep 10 +done + +# Calculate the average time +average_time=$(bc <<< "scale=2; $total_time / $iterations") +echo "Average time per iteration: $average_time seconds" + +# Delete the output directory +rm -r "$output_dir" + + diff --git a/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-stress-test.sh b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-stress-test.sh new file mode 100755 index 000000000..3cd9a10a8 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/nomcadkwok-stress-test.sh @@ -0,0 +1,62 @@ +#!/bin/bash + +SCRIPT_DIR=$(readlink -f `dirname "${BASH_SOURCE[0]}"`) + +# Set the subdirectory name +subdirectory="output_dir" + +# Create the output directory path +output_dir="${SCRIPT_DIR}/${subdirectory}" + +# Set default values +jobs=1 +gpus=0 +awjobs=1 + +# Parse command-line options +while getopts j:g:a: option; do + case $option in + j) + jobs=$OPTARG + ;; + g) + gpus=$OPTARG + ;; + a) + awjobs=$OPTARG + ;; + *) + ;; + esac +done +shift $((OPTIND-1)) + + +# This fixes the number of jobs to be one less so the for loop gets the right amount +((realjobs=$jobs-1)) + +# Start the timer now +SECONDS=0 +export STARTTIME=`date +"%T"` + +for num in $(eval echo "{0.."$realjobs"}") +do + next_num=$((num + 1)) + kubectl apply -f "$output_dir/stress-nomcadkwok-cpu-j-s-$next_num.yaml" +done + + +# Check for all jobs to report complete +jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep nomcadkwok | wc -l` + +while [ $jobstatus -lt $jobs ] +do + echo "Number of completed jobs is: " $jobstatus " and the goal is: " $jobs + sleep 1 + jobstatus=`kubectl get jobs -n default --no-headers --field-selector status.successful=$awjobs |grep nomcadkwok | wc -l` +done + +# kubectl wait --for=condition=complete --timeout=-30s --all job + +export FINISHTIME=`date +"%T"` +echo "Total amount of time for $jobs fake jobs without MCAD is: $SECONDS seconds" diff --git a/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/stress-nomcadkwok-cpu-j-s.yaml b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/stress-nomcadkwok-cpu-j-s.yaml new file mode 100644 index 000000000..ed68593d6 --- /dev/null +++ b/test/kwok-tests/stress-tests-kwok/stress-nomcadkwok-cpu-j-s/stress-nomcadkwok-cpu-j-s.yaml @@ -0,0 +1,39 @@ +apiVersion: batch/v1 +kind: Job +metadata: + namespace: default + name: nomcadkwok-cpu-job-short-0 +spec: + parallelism: 1 + completions: 1 + template: + metadata: + namespace: default + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: type + operator: In + values: + - kwok + tolerations: + - key: "kwok.x-k8s.io/node" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: nomcadkwok-cpu-job-short-0 + image: nginx:1.24.0 + command: ["sleep", "10"] + resources: + limits: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + requests: + cpu: 5m + memory: 20M + nvidia.com/gpu: 0 + restartPolicy: Never