Kubernetes and CI

What you will achieve

When to use this

What you need

Before you start

Setup

1. 1

   Create the cluster's key and store it as a Secret

   Access › Enrollment keys › Create enrollment key:

   • Name - k8s-prod-router.
   • Make this key reusable - on, so a pod that lost its volume can enrol again.
   • Expires in - empty, so the key never expires.
   • Usage limit - 5. Only a new device uses one, not a restart.
   • Temporary devices, Allow Extra DNS Labels - off. A routing device must stay listed while its pod is rescheduled.
   • Auto-assigned groups - type k8s-routers, press Enter.

   Press Create enrollment key and copy the key: it is shown only once.

   Run the block below, like step 2, from wherever kubectl reaches the cluster today. It stores the key as a Secret, in a namespace that allows the agent's capabilities, and keeps it out of your shell history and manifests. Run the block in bash or zsh (macOS, Linux, WSL). Paste the whole block and press Enter once. When Enrollment key: appears, paste the key, and only the key, and press Enter. The key is not shown, and an empty line asks again.

   You see: secret/sxaccess-key created.

   shellCopy

   kubectl create namespace sxaccesskubectl label namespace sxaccess pod-security.kubernetes.io/enforce=privileged{  KEY=''  until [ -n "$KEY" ]; do printf 'Enrollment key: '; read -rs KEY || break; echo; done  [ -n "$KEY" ] && kubectl -n sxaccess create secret generic sxaccess-key --from-literal=enrollment-key="$KEY"  unset KEY}

2. 2

   Run the agent as a single Deployment

   Save the manifest below as sxaccess-router.yaml, replace the placeholder with the image from Add device › Docker, and run kubectl apply -f sxaccess-router.yaml. If that command sets other variables with -e, add them under env too.

   The image starts the agent and runs sxaccess start with the key from SX_ENROLLMENT_KEY and the name from SX_HOSTNAME. BPF is optional. strategy: Recreate stops the old pod before the new one starts, so two pods never share the identity kept on the volume.

   You see: after a minute, kubectl -n sxaccess exec deploy/sxaccess-router -- sxaccess info starts with Connected, and k8s-router-prod is under Devices in k8s-routers. Running alone does not prove the pod enrolled.

   shellCopy

   apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: sxaccess-state  namespace: sxaccessspec:  accessModes: ["ReadWriteOnce"]  resources:    requests:      storage: 1Gi---apiVersion: apps/v1kind: Deploymentmetadata:  name: sxaccess-router  namespace: sxaccessspec:  replicas: 1  strategy:    type: Recreate  selector:    matchLabels: { app: sxaccess-router }  template:    metadata:      labels: { app: sxaccess-router }    spec:      containers:        - name: sxaccess          image: IMAGE-FROM-ADD-DEVICE-DOCKER   # copy it from Add device > Docker          securityContext:            capabilities:              add: ["NET_ADMIN", "NET_RAW"]     # "BPF" is optional          env:            - name: SX_ENROLLMENT_KEY              valueFrom:                secretKeyRef:                  name: sxaccess-key                  key: enrollment-key            - name: SX_HOSTNAME              value: k8s-router-prod          volumeMounts:            - name: state              mountPath: /var/lib/sxaccess      volumes:        - name: state          persistentVolumeClaim:            claimName: sxaccess-state

3. 3

   Put yourself in the platform group

   The rule in step 4 lets the group platform reach the API server. Create it with your laptop in it: People › Members › your name › Access, type platform in Auto-assigned groups, press Enter, then Save Changes. Repeat for each engineer. Your laptop joins at once: group propagation is on by default.

   You see: platform next to your laptop under Devices.

4. 4

   Publish the API server

   Networks › Internal networks › Add internal network: Network Name k8s-prod, then Add Network. Three forms follow:

   • Add Resource - Name Kubernetes API, which the apps and sxaccess reach list show. Address: the host from your kubeconfig's server: line, without https:// or port. Destination Groups: type k8s-prod-api, press Enter. Then Add Resource.
   • Create access rule - the destination is already Kubernetes API. Pick platform as the source, then TCP. Click the ports field, type the port from the server: line (443 if it has none) and press Enter: the field shows it instead of All. Then Create access rule.
   • Add Routing Device - on Device group, pick k8s-routers, then Continue. Leave Masquerade on and press Add Routing Device. The group also covers a pod that enrols again.

   You see: the k8s-prod row shows 1 resource, 1 device and platform. To publish a Service as well, see Advanced configuration.

5. 5

   Give CI its own key and access rule

   A second key, for runners, in Access › Enrollment keys › Create enrollment key:

   • Name - ci-runners. Make this key reusable - on.
   • Expires in - 90. Replace the key before then: new key, new CI secret, old key revoked.
   • Usage limit - empty. Every job uses one, so a limit would stop CI.
   • Temporary devices - on: a finished runner is removed about 10 minutes after it goes offline. It cannot be switched on later. Allow Extra DNS Labels - off.
   • Auto-assigned groups - type ci-runners, press Enter.

   Press Create enrollment key and store the key, shown once, as a masked CI secret variable named CI_SXACCESS_KEY. Then click k8s-prod in the networks list and press Add Access rule in the Kubernetes API row. Pick ci-runners, TCP and the same port, then Create access rule. Grant runners nothing else: a build runs code from pull requests.

   You see: 2 in the Access rules column of the Kubernetes API row, and ci-runners next to platform on the k8s-prod row of the networks list.

6. 6

   Connect each job, and disconnect it at the end

   On a VM runner, where the job has root or sudo, use the block below. The start lines go first in the job; sxaccess stop goes in the step your CI system runs after every job, passed or failed. CI_JOB_ID stands for your CI system's job number, so each runner gets its own name.

   A job in a container has no service manager, so use the block after these steps: it starts the agent itself, then waits for it to answer. It needs root, NET_ADMIN and NET_RAW; where your CI cannot grant them, as on most shared container runners, use a VM runner.

   You see: while the job runs, the runner is under Devices as ci- plus the job number. It is gone about 10 minutes after the job ends.

   shellCopy

   # Start of the job, on a VM runner.curl -fsSL https://pkgs.sxaccess.com/install.sh | shSX_ENROLLMENT_KEY="$CI_SXACCESS_KEY" sxaccess start --hostname "ci-$CI_JOB_ID"# End of the job, passed or failed: your CI system's after-job step.sxaccess stop

Verify

Troubleshooting

Advanced configuration

Related