[BEAM-12812] - Run Github Actions on GCP workers in the apache-beam-testing project

.github/gh-actions-self-hosted-runners/README.md

@@ -0,0 +1,86 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we have started implementing them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+In addition, we are working on researching the best way to implement the MacOS self-hosted runners.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [github-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/github-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Node
+* Machine Type: 2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes: docker.sock
+* Secret env variables: Kubernetes Secrets
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling: updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instace Template: _TODO: Add GCP link when final resources are in place._
+* Machine Type: n2-standard-2
+* Disk Size: 70 GB
+* Disk Image: [disk-image-windows-runner](https://console.cloud.google.com/compute/imagesDetail/projects/apache-beam-testing/global/images/disk-image-windows-runner?project=apache-beam-testing)
+* CPU: 2 vCPUs
+* Memory : 8 GB
+
+#### Instance group settings
+* Region: us-west1 (multizone)
+* Scale-out metric: 70% of CPU Usage.
+* Cooldown period: 300s
+
+#### Notes:
+At first glance we considered implementing Windows runners using K8s, however this was not optimal because of the following reasons:
+
+* VS Build tools are required for certain workflows, unfortunately official images that support this dependency are huge in size, reaching 20GB easily which is not an ideal case for k8S management.
+* Windows Subsystem For Linux(WSL) is a feature that allows to execute bash scripts inside Windows which removes tech debt by avoiding writing steps in powershell, but this feature is disabled with payload removed in Windows containers.
+
+
+## Self-Hosted Runners Architecture
+![Diagram](diagrams/self-hosted-runners-architecture.png)
+
+## Cronjob - Delete Unused Self-hosted Runners
+
+Depending on the termination event, sometimes the removal script for offline runners is not triggered correctly from inside the VMs or K8s pod, because of that an additional pipeline was created in order to clean up the list of GitHub runners in the group.
+
+This was implemented using a [Cloud Function](https://console.cloud.google.com/functions/details/us-central1/remove-self-hosted-runners-group?env=gen1&project=apache-beam-testing&tab=source) subscribed to a [Pub/Sub](https://console.cloud.google.com/cloudpubsub/topic/detail/remove-runners?referrer=search&project=apache-beam-testing) topic, the topic is triggered through a [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler/jobs/edit/us-central1/runners-clean-up-schedule?project=apache-beam-testing) that is executed once per day, the function consumes a [GitHub API](https://docs.github.com/en/rest/reference/actions#delete-a-self-hosted-runner-from-an-organization) to delete offline self-hosted runners from the organization retrieving the token with its service account to secrets manager.
+
+
+![Delete Offline Self-hosted Runners](diagrams/self-hosted-runners-delete-function.png)

.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md

@@ -0,0 +1,74 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Linux Runners
+These folders contain the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Ubuntu OS.
+* /docker
+ * Dockerfile and entrypoint: Resources needed to create a new self-hosted-runner image.
+ * docker-compose.yml: In case you would like to test and run the self-hosted runner locally.
+
+* /kubernetes
+ * Kubernetes files to create the resources needed to deploy the sel-hosted runners.
+
+## Docker
+
+#### How to build a new image and push it to the Artifact Registry?
+* Create the image locally
+
+`docker build -t $LOCAL_IMAGE_NAME:TAG .`
+
+* Tag the local image with the GCP repository name
+
+`docker tag $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest`
+
+* Make sure you are authenticated to the gcloud repository
+
+`gcloud auth configure-docker us-central1-docker.pkg.dev`
+
+* Push the tagged image to the repository
+
+`docker push LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest`
+
+#### How to run a self-hosted locally?
+
+* Create a `.var.env` file from the `example.var.env` file and replace the corresponding values
+
+* Run the container
+
+`docker-compose up`
+
+* You will be able to see the self-hosted runner in the Settings of your GitHub repository
+
+## Kubernetes
+* Configure your Kubernetes local context
+
+`gcloud container clusters get-credentials $CLUSTER_NAME --zone us-central1-a --project $PROJECT-ID`
+
+* Update the `github-actions-secrets.yml` file with its corresponding encrypted in base64 values
+
+`echo $GITHUB_TOKEN | base64`
+`echo GITHUB_REPO | base64`
+
+* Replace in `github-actions-deployment.yml` file the `$IMAGE_URL` variable with the corresponding image URL: `LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE`
+
+* In case you would like to create the deployment from scratch, run the `run-k8s-deployment.sh` script to do the Kubernetes deployment in the GKE cluster. Make sure you have the GKE context selected in your local machine
+
+`./run-k8s-deployment.sh`
+
+* Otherwise, apply only the changes. Make sure you have the GKE context selected in your local machine
+
+`kubectl apply -f github-actions-$FILE_NAME.yml`

.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile

@@ -0,0 +1,42 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+FROM ubuntu:20.04
+
+ARG RUNNER_VERSION=2.288.1
+
+RUN apt-get -yqq update
+#Install APT packages
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -yqq acl aria2 apt-transport-https binutils bison brotli build-essential bzip2 ca-certificates coreutils curl dbus dnsutils dpkg fakeroot file flex fonts-noto-color-emoji ftp gnupg2 haveged imagemagick iproute2 iputils-ping jq lib32z1 libc++-dev libc++abi-dev libcurl4 libgbm-dev libgconf-2-4 libgsl-dev libgtk-3-0 libicu66 libkrb5-3 liblttng-ust0 libmagic-dev libmagickcore-dev libmagickwand-dev libsecret-1-dev libssl1.1 libsqlite3-dev libxss1 locales m4 mediainfo mercurial net-tools netcat openssh-client p7zip-full p7zip-rar parallel pass patchelf pkg-config pollinate python-is-python3 rpm rsync shellcheck sqlite3 ssh sshpass subversion sudo swig telnet texinfo time tk tzdata unzip upx wget xorriso xvfb xz-utils zip zlib1g zsync 
+RUN apt-get clean
+RUN rm -rf /var/lib/apt/lists/*
+RUN useradd -m actions && usermod -aG sudo actions && echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
+USER actions
+WORKDIR /home/actions
+
+RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
+RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
+
+COPY --chown=actions:actions entrypoint.sh ./entrypoint.sh
+
+RUN sudo chmod u+x ./entrypoint.sh
+RUN curl -fsSL https://get.docker.com -o get-docker.sh
+RUN sudo sh get-docker.sh
+
+CMD ["/home/actions/entrypoint.sh"]
+ENTRYPOINT ["/bin/bash"]

.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml

@@ -0,0 +1,28 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+version: "3.0"
+services:
+ runner:
+ build: .
+ entrypoint: ["/home/actions/entrypoint.sh"]
+ image: $IMAGE_URL:${TAG:-latest}
+ volumes:
+ - "/var/run/docker.sock:/var/run/docker.sock"
+ env_file:
+ - ./var.env

.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sudo chmod 666 /var/run/docker.sock
+
+#escaping env variables to avoid issues in k8s deployment
+GITHUB_TOKEN=$(echo "$GITHUB_TOKEN")
+
+registration_url="https://api.github.com/orgs/$ORG_NAME/actions/runners/registration-token"
+echo "Requesting registration Org URL at '${registration_url}'"
+payload=$(curl -sX POST -H "Authorization: token $GITHUB_TOKEN" ${registration_url})
+RUNNER_TOKEN=$(echo $payload | jq '.token' --raw-output)
+echo "export RUNNER_TOKEN3=$RUNNER_TOKEN" | sudo tee /etc/bash.bashrc > /dev/null && source /etc/bash.bashrc
+
+./config.sh \
+ --name $(hostname) \
+ --token ${RUNNER_TOKEN} \
+ --url https://github.com/$ORG_NAME \
+ --work _work \
+ --unattended \
+ --replace \
+ --labels ubuntu-20.04,ubuntu-latest \
+ --runnergroup ${ORG_RUNNER_GROUP}
+
+remove() {
+ ./config.sh remove --unattended --token "${RUNNER_TOKEN}"
+}
+trap 'remove; exit 130' INT
+trap 'remove; exit 143' TERM
+trap 'remove; exit 143' QUIT
+trap 'remove; exit 143' ABRT
+trap 'remove; exit 0' EXIT
+
+./run.sh "$*" &
+wait $!

.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env

@@ -0,0 +1,4 @@
+# var.env example, please create a var.env file with the corresponding values of the variables
+GITHUB_TOKEN=
+ORG_NAME=
+ORG_RUNNER_GROUP=

.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+kubectl delete -f github-actions-hpa.yml
+kubectl delete -f github-actions-vpa.yml
+kubectl delete -f github-actions-secrets.yml
+kubectl delete -f github-actions-deployment.yml

.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml

@@ -0,0 +1,56 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: ubuntu-runners
+ labels:
+ name: github-actions
+spec:
+ selector:
+ matchLabels:
+ name: github-actions
+ replicas: 5
+ template:
+ metadata:
+ labels:
+ name: github-actions
+ spec:
+ containers:
+ - name: github-actions-ubuntu-runner
+ image: $IMAGE_URL:latest # replace variable manually
+ securityContext:
+ privileged: true
+ envFrom:
+ - secretRef:
+ name: github-actions-secrets
+ volumeMounts:
+ - name: docker-sock
+ mountPath: /var/run/docker.sock
+ resources:
+ requests:
+ memory: "1028Mi"
+ cpu: "2"
+ lifecycle:
+ preStop:
+ exec:
+ command: [ "/bin/sh", "-c",'./config.sh remove --unattended --token "${RUNNER_TOKEN}"' ]
+ volumes:
+ - name: docker-sock
+ hostPath:
+ path: /var/run/docker.sock

.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml

@@ -0,0 +1,29 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+apiVersion: autoscaling/v1
+kind: HorizontalPodAutoscaler
+metadata:
+ name: hpa-github-actions-linux
+spec:
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: ubuntu-runners
+ minReplicas: 10
+ maxReplicas: 20
+ targetCPUUtilizationPercentage: 70