jaeger-and-prometheus

Jaeger and Prometheus Example

This example demonstrates how to setup Open Telemetry Collector with Jaeger and Prometheus to gather telemetry data from NSM components. OpenTelemetry is a collection of tools, APIs, and SDKs. It is used to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Run

Apply Jaeger, Prometheus and OpenTelemetry Collector:

kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/observability/jaeger-and-prometheus?ref=3d1dcfe1de90681213c7f0006f25279bb4699966

Wait for OpenTelemetry Collector POD status ready:

kubectl wait -n observability --timeout=1m --for=condition=ready pod -l app=opentelemetry

Create ns for NSM deployments:

kubectl create ns nsm-system

Apply NSM resources for basic tests:

kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/observability/jaeger-and-prometheus/nsm-system?ref=3d1dcfe1de90681213c7f0006f25279bb4699966

Wait for admission-webhook-k8s:

WH=$(kubectl get pods -l app=admission-webhook-k8s -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
kubectl wait --for=condition=ready --timeout=1m pod ${WH} -n nsm-system

Create test namespace:

NAMESPACE=($(kubectl create -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/3d1dcfe1de90681213c7f0006f25279bb4699966/examples/use-cases/namespace.yaml)[0])
NAMESPACE=${NAMESPACE:10}

Select node to deploy NSC and NSE:

NODE=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints  }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}')[0])

Create forlder for test:

mkdir example

Create customization file:

cat > example/kustomization.yaml <<EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: ${NAMESPACE}

resources: 
- client.yaml
bases:
- https://github.com/networkservicemesh/deployments-k8s/apps/nse-kernel?ref=3d1dcfe1de90681213c7f0006f25279bb4699966

patchesStrategicMerge:
- patch-nse.yaml
EOF

Create Client:

cat > example/client.yaml <<EOF
---
apiVersion: v1
kind: Pod
metadata:
  name: alpine
  labels:
    app: alpine    
  annotations:
    networkservicemesh.io: kernel://icmp-responder/nsm-1
spec:
  containers:
  - name: alpine
    image: alpine:3.15.0
    imagePullPolicy: IfNotPresent
    stdin: true
    tty: true
  nodeName: ${NODE}
EOF

Create NSE patch. The patch adds TELEMETRY variable with value true. It enables telemetry for NSE. This example also has patches for manager and forwarder.

cat > example/patch-nse.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nse-kernel
spec:
  template:
    spec:
      containers:
        - name: nse
          env:
            - name: NSM_CIDR_PREFIX
              value: 172.16.1.100/31
            - name: TELEMETRY
              value: "true"
      nodeName: ${NODE}
EOF

Deploy NSC and NSE:

kubectl apply -k example

Wait for applications ready:

kubectl wait --for=condition=ready --timeout=1m pod -l app=alpine -n ${NAMESPACE}

kubectl wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ${NAMESPACE}

Find nsc and nse pods by labels:

NSC=$(kubectl get pods -l app=alpine -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

NSE=$(kubectl get pods -l app=nse-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

Ping from NSC to NSE:

kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100

Ping from NSE to NSC:

kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101

Select forwarder:

NODES=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints  }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}'))
FORWARDER=$(kubectl get pods -l app=forwarder-vpp --field-selector spec.nodeName==${NODES[0]} -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')

Expose ports to access Jaeger and Prometheus UI:

kubectl port-forward service/jaeger -n observability 16686:16686 2>&1 > /dev/null &
kubectl port-forward service/prometheus -n observability 9090:9090 2>&1 > /dev/null &

Retrieve traces from Jaeger:

result=$(curl -X GET localhost:16686/api/traces?service=${FORWARDER}&lookback=5m&limit=1)
echo ${result}
echo ${result} | grep -q "forwarder"

Replace - with _ in forwarder pod name (Forwarder metric names contain only _)

FORWARDER=${FORWARDER//-/_}

Retrieve metrics from Prometheus:

result=$(curl -X GET localhost:9090/api/v1/query?query="${FORWARDER}_server_tx_bytes_sum")
echo ${result}
echo ${result} | grep -q "forwarder"

Cleanup

Delete ns:

rm -r example
kubectl delete ns ${NAMESPACE}

WH=$(kubectl get pods -l app=admission-webhook-k8s -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
kubectl delete mutatingwebhookconfiguration ${WH}
kubectl delete ns nsm-system

kubectl describe pods -n observability
kubectl delete ns observability
pkill -f "port-forward"