Catchpoint Synthetic Node on Kubernetes

This guide will take you through setting up a Catchpoint Synthetic Node as a StatefulSet in Kubernetes.

Prerequisites

To begin, we need two things from the portal:

• Our Catchpoint RESTful API Key

Retrieve the “API key” by following the steps in this guide: https://docs.catchpoint.com/docs/rest-api-v2-guide

Creating the Instance ID

Each instance is identified by the combination of its hostname and Instance ID (This is labeled as “MAC Address” on the portal but for our purposes, a MAC address will not work.) These need to be globally unique in the Catchpoint database but must also maintain a predictable 1:1 relationship for communications to work properly. The Instance ID must be exactly 12 upper-case letters or numbers. To create something with little chance of collision, we will randomly generate the first 11 characters of this and store it as a secret in K8s. Then, each individual instance will provide the 12th character to complete the ID.

From a Linux shell with a properly configured kubectl, execute the following:

1. head /dev/urandom | tr -dc A-Z0-9 | head -c 11| kubectl create secret generic cp-node-instance-id-seed --from-file=password=/dev/stdin.
2. We also need to know this secret ourselves for the next steps so run the following and copy the value of the “password” field:
   kubectl get secret cp-node-instance-id-seed -o json | jq '.data | map_values(@base64d)'

Creating the Node

Create a Node with a “seed” instance. The seed will be the first instance, but our Kubernetes configuration will handle scaling out from there.

Option A (From the portal)

1. From the Catchpoint portal, navigate to Nodes > Nodes then click “Add Instance”.
2. As mentioned previously, we’re using “cp-node” for our Node name in this example. To work smoothly with K8s’s Stateful Set hostnames, we also need to add an ordinal to the first instance name as each host created will have a 0-based index suffixed to the name. For example, when using “cp-node” as the name of our Stateful Set, the first replica will be named “cp-node-0”. Thus, we must name this node “cp-node” and the instance “cp-node-0”.
3. The “MAC Address” is otherwise known as our “Instance ID”. As mentioned previously, this won’t actually be a MAC Address. Paste the 11-character string from the “Creating the instance ID” step here and then add “0” to the end. Each additional instance will increment this final value keeping the key unique while maintaining the 1:1 relationship with the hostname.
4. The “OS” should be “Docker”.
5. The “Network Type” should be set as desired based on your licensing type.
6. “Node” should be “New”.
7. “OS Type”: “Linux”.
8. Set the “City” and “ISP” values as desired.
9. “Node Name” should be “cp-node” for this example.
10. The “Model” should reflect the aggregate value of the resources being assigned. Read more about this value here: https://docs.catchpoint.com/docs/en/enterprise-node-capacity-and-sizing
11. Save the new node.

Option B (From the catchpoint utility)

The Catchpoint utility’s “activate” command may also be used to create the node and first instance:

On another Catchpoint instance, run the following command (updating the {bracket} fields with your information):

catchpoint activate --new-node cp-node --city “{city}” --state “{state}” --country "{country}" --os Docker --size {[1-6]} --isp “{isp}” --api-key {api-key} --hostname cp-node-0 --machine-id {instanceid}

Option C (From a K8s Job)

K8s can also handle the Node set up for you via a Job that you simply execute once just to run the catchpoint activate… command (as seen in Option B). See more about this in Activation Job (optional).

Kubernetes

This guide assumes that you have a working instance of Kubernetes and that “kubectl” is properly configured to work with the correct instance.

Secrets

In order to handle the automatic activation of our scaled pods, the RESTful API Key needs to be accessible from within the pods. We could add these simply as environment variables, but it’s best-practice to store credentials as secrets. For the sake of this guide, we will add them as “literal” passwords, but for production-use, you will want to ensure that you’re storing these values consistent with your own security policies and Kubernetes’s recommendations. Read more here: https://kubernetes.io/docs/concepts/configuration/secret/

Run the following commands using the values you retrieved from the portal:

kubectl create secret generic cp-apikey --from-literal=password={YOUR_API_KEY}

StatefulSet

Our StatefulSet will define our instances and handle scaling from 1 to 9 instances under our new node.

To start our stateful deployment, create a configuration file to define it. We will break it down here and also provide it in its entirety at the end of the document.

Since our node is called “cp-node”, we’ve named our file “cp-node-stateful-set.yaml”.

At the beginning of the file, we define the kind of the configuration to be “StatefulSet” with the name “cp-node” which is of an app type of “syntheticagent” in the “default” namespace:

apiVersion: apps/v1 
kind: StatefulSet 
metadata: 
  name: cp-node 
  namespace: default 
  labels: 
    app: syntheticagent 

Then we provide the top-level “spec” block which defines the serviceName and number of replicas we desire. Due to how we’re going to implement the Instance ID, this needs to be less than 10:

spec: 
  serviceName: cp-node 
  replicas: 3 
  selector: 
    matchLabels: 
      app: syntheticagent 
  template: 
    metadata: 
      labels: 
        app: syntheticagent 

Then we define the inner “spec” block with the hostname. The “hostname” field is required to name each pod consistently. Being a stateful set, each pod will be given an ordinal number suffix, e.g. cp-node-0, cp-node-1, etc. In order to avoid creating a mess in the portal, we need to keep a 1-to-1 mapping with our hostnames and Instance-IDs:

spec: 
  hostname: cp-node 

Then we provide the container (pod) definition:

      containers: 
        - name: cp-node 
          image: catchpoint/enterprise:latest 
          resources: 
         requests: 
              memory: "3Gi" 
              cpu: "1" 
            limits: 
              memory: "4Gi" 
              cpu: "2" 
          imagePullPolicy: IfNotPresent 
          command: 
          - bash 
          - -c 
          - | 
            export CP_INSTANCEID="$CP_INSTANCEID_SEED${HOSTNAME##*-}" 
            ./entrypoint.sh 
          env: 
          - name: CP_INSTANCEID_SEED 
            valueFrom: 
              secretKeyRef: 
                name: cp-node-instance-id-seed 
                key: password 
          - name: CP_APIKEY 
            valueFrom: 
              secretKeyRef: 
                name: cp-apikey 
                key: password 
          - name: CP_NODENAME 
            value: cp-node 
          securityContext: 
            capabilities: 
              add: ["SYS_ADMIN"] 
          livenessProbe: 
            exec: 
              command: 
              - bash 
              - -c 
              - | 
                if catchpoint status | grep -q stopped; then exit 1; fi 
            initialDelaySeconds: 300 
            periodSeconds: 300 
            timeoutSeconds: 60 

• image: the latest Catchpoint Synthetic Node image directly from hub.docker.com. Note: Kubernetes recommends using the digest of the image for better version tracking, read more here: https://kubernetes.io/docs/concepts/containers/images/
• resources: we’re restricting the resources of each container so that we have enough for all 3 pods.
• imagePullPolicy: “IfNotPresent” ensures that our k8s instance will pull the image if we don’t already have it but won’t upgrade when a new version becomes available. You can also use the value “Always” which would allow you to upgrade the pods to the latest version of the docker image by scaling to 0 and then back up to your desired replica count again.
• command: This is where we define a special command to run when the container starts to handle the Instance-ID assignment. If a container is not assigned a “CP_INSTANCEID” value, it will generate a random one. This would be fine for the first run, but if the pod is scaled away and then brought back, or if it were to crash and need to be re-created, it would get an entirely different value and fail to activate the second time. Thus, we need to ensure a 1-to-1 mapping of the hostname and the Instance-ID value. This short script uses our seed value from the previous step for the first 11 characters of the ID and then uses the pod’s ordinal number to provide the last character. Thus, cp-node-0 will always have a value of ${seed}0, and cp-node-1 will always have the value of ${seed}1, no matter how many times the container is rebuilt. After assigning the new Instance-ID, we execute the docker container’s normal entrypoint script to start the service.
• env: this is where we set values for use by the container’s automated activation. We're only taking one env value from the secret we created earlier. Note, anyone that has access to this pod within Kubernetes can use an “exec” command to read these values in plaintext form. The CP_NODENAME is the name of the node that these instances will be attached to.
  Note: if a proxy is needed, it can be specified with name: “httpproxy” or name: “httpsproxy”, but this will only be used for other applications on the container (e.g. APK). To set a proxy for Synthetic Node, the catchpoint proxy command must be used instead.
• securityContext: This is where we set the docker container’s capabilities. Per https://hub.docker.com/r/catchpoint/enterprise, we use SYS_ADMIN and SYS_TIME. SYS_ADMIN may be omitted if there are no plans to run any “Chrome” tests (and will keep the disk usage lower). SYS_TIME may be omitted if there is another method already in place to keep the Kubernetes cluster in sync with an NTP server.
  Note: NET_BIND_SERVICE and NET_RAW are typically enabled by default and do not need to be applied here. However, this depends on the Kubernetes provider. For instance, Oracle Linux Cloud Native Environments requires setting NET_BIND_SERVICE and NET_RAW for some tests to work properly.
• livenessProbe: The liveness probe ensures that the Synthetic Node services are always running. The command checks the result of catchpoint status for any instances of the word “stopped” starting 5 minutes (initialDelaySeconds) after the container is created (to allow time for service restarts due to configuration changes) and then every 5 minutes (periodSeconds) after. The command is allowed 1 minute (timeoutSeconds) to return. By default, the failureThreshold is set to allow 3 failures before the pod is scrapped and restarted. These values may all be tweaked, but keep in mind that there are rare instances where a Syntheticagent instance will start up and receive a configuration that requires restarting the service to apply it.

Save this configuration and apply it with kubectl apply -f {filename}. You should see the response “statefulset.apps/cp-node configured” and within a few minutes, the Catchpoint Portal will show the new instances under the “cp-node”. See more about validation here.

Activation Job (optional)

To use a Job specification to automatically handle the first-time node activation, create a file called “cp-node-activate-job.yaml” and paste the following (updating the city, state, country and isp values as desired):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cp-node
  namespace: default
  labels:
    app: syntheticagent
spec:
  serviceName: cp-node
  replicas: 3
  selector:
    matchLabels:
      app: syntheticagent
  template:
    metadata:
      labels:
        app: syntheticagent
    spec:
      containers:
        - name: cp-node
          image: catchpoint/enterprise:latest
          volumeMounts:
            - mountPath: /dev/chrome
              name: dev-tmpfs
          resources:
            requests:
              memory: "1Gi"
              cpu: "1"
            limits:
              memory: "2Gi"
              cpu: "2"
          imagePullPolicy: IfNotPresent
          command:
            - bash
            - -c
            - |
              export CP_INSTANCEID="$CP_INSTANCEID_SEED${HOSTNAME##*-}"
              ./entrypoint.sh
          env:
            - name: CP_INSTANCEID_SEED
              valueFrom:
                secretKeyRef:
                  name: cp-node-instance-id-seed
                  key: password
            - name: CP_APIKEY
              valueFrom:
                secretKeyRef:
                  name: cp-apikey
                  key: password
            - name: CP_NODENAME
              value: cp-node
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
          livenessProbe:
            exec:
              command:
                - bash
                - -c
                - |
                  if ./catchpoint status | grep -q stopped; then exit 1; fi
            initialDelaySeconds: 300
            periodSeconds: 300
            timeoutSeconds: 60
      volumes:
        - name: dev-tmpfs
          emptyDir:
            medium: Memory

By applying this file, the job will only run once and will create the Catchpoint Node as long as all fields are correct (for instance, the city, state, country, and ISP must all be uniquely identified in the database). Apply the file with kubectl apply -f cp-node-activate-job.yaml and view the cp-node-activation pod logs to ensure success.

Validation

Node activation job

To validate the node activation job, execute kubectl logs job.batch/cp-node-activation. The logs should be completed with “Instance Activated Successfully”. Additionally, you should also see the new “cp-node” Node in the Catchpoint Portal with one instance “cp-node-0”.

Stateful Set

To validate the Stateful Set deployment:

1. Ensure the replicas are deployed and running with kubectl describe statefulset cp-node | grep Status (You should see Running with the correct number of Replicas, e.g. “3 Running”).
2. Ensure the catchpoint logs don’t contain “CPS Status” errors by executing the following within each instance: kubectl exec {instance} -- catchpoint log (e.g. kubectl exec cp-node-0 -- catchpoint log).
   If CPS Status errors are seen, the instance was not activated properly, use the portal to verify that the Instance id and hostname (which you can obtain with kubectl exec cp-node-0 -- catchpoint info) match.
3. From the portal, verify that there is an instance for each replica specified. For this example, we specified 3 replicas, thus, the node page for “cp-node” should list three instances “cp-node-0”, “cp-node-1”, and “cp-node-2”.
4. From the portal, issue some simple instant tests against the new node and ensure you get the expected response (couple this with the above catchpoint log command if there are any issues).

Complete Stateful Set Configuration File

kind: StatefulSet
metadata:
  name: cp-node
  namespace: default
  labels:
    app: syntheticagent
spec:
  serviceName: cp-node
  replicas: 1
  selector:
    matchLabels:
      app: syntheticagent
  template:
    metadata:
      labels:
        app: syntheticagent
      annotations:
        container.apparmor.security.beta.kubernetes.io/cp-node: unconfined
    spec:
      containers:
        - name: cp-node
          image: catchpoint/enterprise:latest
          volumeMounts:
            - mountPath: /dev
              name: dev-mount
            - mountPath: /dev/chrome
              name: dev-tmpfs
          resources:
            requests:
              memory: "1Gi"
              cpu: "1"
            limits:
              memory: "2Gi"
              cpu: "2"
          imagePullPolicy: IfNotPresent
          command:
            - bash
            - -c
            - |
              export CP_INSTANCEID="$CP_INSTANCEID_SEED${HOSTNAME##*-}"
              ./entrypoint.sh
          env:
            - name: CP_INSTANCEID_SEED
              valueFrom:
                secretKeyRef:
                  name: cp-node-instance-id-seed
                  key: password
            - name: CP_APIKEY
              valueFrom:
                secretKeyRef:
                  name: cp-apikey
                  key: password
            - name: CP_NODENAME
              value: cp-node
          securityContext:
            capabilities:
              add:
                - SYS_ADMIN
          livenessProbe:
            exec:
              command:
                - bash
                - -c
                - |
                  if ./catchpoint status | grep -q stopped; then exit 1; fi
            initialDelaySeconds: 300
            periodSeconds: 300
            timeoutSeconds: 60
      volumes:
        - name: dev-mount
          hostPath:
            path: /dev
            type: Directory
        - name: dev-tmpfs
          emptyDir:
            medium: Memory