Skip to content

Latest commit

 

History

History
80 lines (65 loc) · 3.04 KB

README.md

File metadata and controls

80 lines (65 loc) · 3.04 KB

pycon-colombia-python-ray

Repository containing talk material and sample code used during PyCon Colombia 2022 talk titled "Architecting and Running Distributed Python Applications with Ray on Kubernetes".

Author:
Anmol Krishan Sachdeva
Hybrid Cloud Architect, Google
LinkedIn@greatdevaks | Twitter@greatdevaks

Deck:
Architecting and Running Distributed Python Applications with Ray on Kubernetes

Sample Implementation Files:

How to run:

  • Have a Kubernetes Cluster created with 2-3 worker nodes of min. capacity 2 cores and 2 GIB each.
  • Clone the upstream Ray GitHub Repository
  • Helm install the Ray chart:
helm -n ray install pycon-ray-cluster-00 --create-namespace ./ray/deploy/charts/ray

Check the Kubernetes workloads:

  • Check the Ray Operator pod:
kubectl get pods
  • Check Ray Cluster CRD and the pods in ray namespace:
kubectl get crd rayclusters.cluster.ray.io
kubectl -n ray get rayclusters
  • Get Ray Cluster Head Node, Worker Nodes, and Service exposing Head Node:
kubectl -n ray get pods,services
  • Get Ray Autoscaler logs:
kubectl logs -f $(kubectl get pods -l cluster.ray.io/component=operator -o custom-columns=:metadata.name)
  • Get access to Ray Dashboard by port forwarding the Head Service:
kubectl -n ray port-forward service/pycon-ray-cluster-00-ray-head 8265:8265

Note: Open localhost:8265 to open the Ray Dashboard

Run PI Estimation Kubernetes Job:

# Run a sample Ray program using Kubernetes Job
kubectl -n ray create -f https://raw.githubusercontent.com/greatdevaks/pycon-colombia-python-ray/main/estimate_py_k8s_job.yaml

# View localhost:8265 Ray Dashboard

Cleanup:

# Delete RayCluster Custom Resource
kubectl -n ray delete raycluster pycon-ray-cluster-00

# Delete the Ray Helm Release - doesn't delete the RayCluster CRD
helm -n ray uninstall pycon-ray-cluster-00

# Delete the Ray Kubernetes Namespace
kubectl delete namespace ray

# Delete teh RayCluster CRD
kubectl delete crd rayclusters.cluster.ray.io

References:

Disclaimer:
The content and the views presented during the talk/session are the author’s own and not of the organizations/companies they are associated with.