diff --git a/.github/gh-actions-self-hosted-runners/README.md b/.github/gh-actions-self-hosted-runners/README.md new file mode 100644 index 000000000000..2bae9786127d --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/README.md @@ -0,0 +1,86 @@ + +# GitHub Actions - Self-hosted Runners +The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we have started implementing them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively. + +In addition, we are working on researching the best way to implement the MacOS self-hosted runners. + +## Ubuntu +Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications: + +#### Cluster +* Cluster: [github-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/github-actions-linux-runners/details?project=apache-beam-testing) +* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing) + +#### Node +* Machine Type: 2-custom-6-18432 +* Disk Size: 100 GB +* CPU: 6 vCPUs +* Memory : 18 GB + +#### Pod +* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest +* CPU: 2 +* Memory: 1028 Mi +* Volumes: docker.sock +* Secret env variables: Kubernetes Secrets + +#### AutoScaling +* Horizontal Pod Autoscaling + * 5-10 nodes + * HorizontalPodAutoscaler + * Min replicas: 10 + * Max replicas: 20 + * CPU utilization: 70% +* Vertical Pod Autoscaling: updateMode: "Auto" + + +## Windows +Windows Virtual machines have the following specifications + +#### VM specifications +* Instace Template: _TODO: Add GCP link when final resources are in place._ +* Machine Type: n2-standard-2 +* Disk Size: 70 GB +* Disk Image: [disk-image-windows-runner](https://console.cloud.google.com/compute/imagesDetail/projects/apache-beam-testing/global/images/disk-image-windows-runner?project=apache-beam-testing) +* CPU: 2 vCPUs +* Memory : 8 GB + +#### Instance group settings +* Region: us-west1 (multizone) +* Scale-out metric: 70% of CPU Usage. +* Cooldown period: 300s + +#### Notes: +At first glance we considered implementing Windows runners using K8s, however this was not optimal because of the following reasons: + +* VS Build tools are required for certain workflows, unfortunately official images that support this dependency are huge in size, reaching 20GB easily which is not an ideal case for k8S management. +* Windows Subsystem For Linux(WSL) is a feature that allows to execute bash scripts inside Windows which removes tech debt by avoiding writing steps in powershell, but this feature is disabled with payload removed in Windows containers. + + +## Self-Hosted Runners Architecture +![Diagram](diagrams/self-hosted-runners-architecture.png) + +## Cronjob - Delete Unused Self-hosted Runners + +Depending on the termination event, sometimes the removal script for offline runners is not triggered correctly from inside the VMs or K8s pod, because of that an additional pipeline was created in order to clean up the list of GitHub runners in the group. + +This was implemented using a [Cloud Function](https://console.cloud.google.com/functions/details/us-central1/remove-self-hosted-runners-group?env=gen1&project=apache-beam-testing&tab=source) subscribed to a [Pub/Sub](https://console.cloud.google.com/cloudpubsub/topic/detail/remove-runners?referrer=search&project=apache-beam-testing) topic, the topic is triggered through a [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler/jobs/edit/us-central1/runners-clean-up-schedule?project=apache-beam-testing) that is executed once per day, the function consumes a [GitHub API](https://docs.github.com/en/rest/reference/actions#delete-a-self-hosted-runner-from-an-organization) to delete offline self-hosted runners from the organization retrieving the token with its service account to secrets manager. + + +![Delete Offline Self-hosted Runners](diagrams/self-hosted-runners-delete-function.png) diff --git a/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png new file mode 100644 index 000000000000..7ee2d6254383 Binary files /dev/null and b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-architecture.png differ diff --git a/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png new file mode 100644 index 000000000000..0e6c7d91ef64 Binary files /dev/null and b/.github/gh-actions-self-hosted-runners/diagrams/self-hosted-runners-delete-function.png differ diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md b/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md new file mode 100644 index 000000000000..c80d69f4d665 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md @@ -0,0 +1,74 @@ + +# GitHub Actions - Self-hosted Linux Runners +These folders contain the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Ubuntu OS. +* /docker + * Dockerfile and entrypoint: Resources needed to create a new self-hosted-runner image. + * docker-compose.yml: In case you would like to test and run the self-hosted runner locally. + +* /kubernetes + * Kubernetes files to create the resources needed to deploy the sel-hosted runners. + +## Docker + +#### How to build a new image and push it to the Artifact Registry? +* Create the image locally + +`docker build -t $LOCAL_IMAGE_NAME:TAG .` + +* Tag the local image with the GCP repository name + +`docker tag $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest` + +* Make sure you are authenticated to the gcloud repository + +`gcloud auth configure-docker us-central1-docker.pkg.dev` + +* Push the tagged image to the repository + +`docker push LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest` + +#### How to run a self-hosted locally? + +* Create a `.var.env` file from the `example.var.env` file and replace the corresponding values + +* Run the container + +`docker-compose up` + +* You will be able to see the self-hosted runner in the Settings of your GitHub repository + +## Kubernetes +* Configure your Kubernetes local context + +`gcloud container clusters get-credentials $CLUSTER_NAME --zone us-central1-a --project $PROJECT-ID` + +* Update the `github-actions-secrets.yml` file with its corresponding encrypted in base64 values + +`echo $GITHUB_TOKEN | base64` +`echo GITHUB_REPO | base64` + +* Replace in `github-actions-deployment.yml` file the `$IMAGE_URL` variable with the corresponding image URL: `LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE` + +* In case you would like to create the deployment from scratch, run the `run-k8s-deployment.sh` script to do the Kubernetes deployment in the GKE cluster. Make sure you have the GKE context selected in your local machine + +`./run-k8s-deployment.sh` + +* Otherwise, apply only the changes. Make sure you have the GKE context selected in your local machine + +`kubectl apply -f github-actions-$FILE_NAME.yml` diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile new file mode 100644 index 000000000000..d4c6e6654daf --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/Dockerfile @@ -0,0 +1,42 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM ubuntu:20.04 + +ARG RUNNER_VERSION=2.288.1 + +RUN apt-get -yqq update +#Install APT packages +RUN DEBIAN_FRONTEND=noninteractive apt-get install -yqq acl aria2 apt-transport-https binutils bison brotli build-essential bzip2 ca-certificates coreutils curl dbus dnsutils dpkg fakeroot file flex fonts-noto-color-emoji ftp gnupg2 haveged imagemagick iproute2 iputils-ping jq lib32z1 libc++-dev libc++abi-dev libcurl4 libgbm-dev libgconf-2-4 libgsl-dev libgtk-3-0 libicu66 libkrb5-3 liblttng-ust0 libmagic-dev libmagickcore-dev libmagickwand-dev libsecret-1-dev libssl1.1 libsqlite3-dev libxss1 locales m4 mediainfo mercurial net-tools netcat openssh-client p7zip-full p7zip-rar parallel pass patchelf pkg-config pollinate python-is-python3 rpm rsync shellcheck sqlite3 ssh sshpass subversion sudo swig telnet texinfo time tk tzdata unzip upx wget xorriso xvfb xz-utils zip zlib1g zsync +RUN apt-get clean +RUN rm -rf /var/lib/apt/lists/* +RUN useradd -m actions && usermod -aG sudo actions && echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers +USER actions +WORKDIR /home/actions + +RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz +RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz + +COPY --chown=actions:actions entrypoint.sh ./entrypoint.sh + +RUN sudo chmod u+x ./entrypoint.sh +RUN curl -fsSL https://get.docker.com -o get-docker.sh +RUN sudo sh get-docker.sh + +CMD ["/home/actions/entrypoint.sh"] +ENTRYPOINT ["/bin/bash"] diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml new file mode 100644 index 000000000000..8d1f7e7ad134 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/docker-compose.yml @@ -0,0 +1,28 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +version: "3.0" +services: + runner: + build: . + entrypoint: ["/home/actions/entrypoint.sh"] + image: $IMAGE_URL:${TAG:-latest} + volumes: + - "/var/run/docker.sock:/var/run/docker.sock" + env_file: + - ./var.env diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh new file mode 100644 index 000000000000..d566652f9568 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/entrypoint.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +sudo chmod 666 /var/run/docker.sock + +#escaping env variables to avoid issues in k8s deployment +GITHUB_TOKEN=$(echo "$GITHUB_TOKEN") + +registration_url="https://api.github.com/orgs/$ORG_NAME/actions/runners/registration-token" +echo "Requesting registration Org URL at '${registration_url}'" +payload=$(curl -sX POST -H "Authorization: token $GITHUB_TOKEN" ${registration_url}) +RUNNER_TOKEN=$(echo $payload | jq '.token' --raw-output) +echo "export RUNNER_TOKEN3=$RUNNER_TOKEN" | sudo tee /etc/bash.bashrc > /dev/null && source /etc/bash.bashrc + +./config.sh \ + --name $(hostname) \ + --token ${RUNNER_TOKEN} \ + --url https://github.com/$ORG_NAME \ + --work _work \ + --unattended \ + --replace \ + --labels ubuntu-20.04,ubuntu-latest \ + --runnergroup ${ORG_RUNNER_GROUP} + +remove() { + ./config.sh remove --unattended --token "${RUNNER_TOKEN}" +} +trap 'remove; exit 130' INT +trap 'remove; exit 143' TERM +trap 'remove; exit 143' QUIT +trap 'remove; exit 143' ABRT +trap 'remove; exit 0' EXIT + +./run.sh "$*" & +wait $! diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env new file mode 100644 index 000000000000..1d4acc98eafc --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/docker/example.var.env @@ -0,0 +1,4 @@ +# var.env example, please create a var.env file with the corresponding values of the variables +GITHUB_TOKEN= +ORG_NAME= +ORG_RUNNER_GROUP= diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh new file mode 100755 index 000000000000..a27fb632aa3e --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/delete-k8s-deployment.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +kubectl delete -f github-actions-hpa.yml +kubectl delete -f github-actions-vpa.yml +kubectl delete -f github-actions-secrets.yml +kubectl delete -f github-actions-deployment.yml diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml new file mode 100644 index 000000000000..93643639530e --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-deployment.yml @@ -0,0 +1,56 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ubuntu-runners + labels: + name: github-actions +spec: + selector: + matchLabels: + name: github-actions + replicas: 5 + template: + metadata: + labels: + name: github-actions + spec: + containers: + - name: github-actions-ubuntu-runner + image: $IMAGE_URL:latest # replace variable manually + securityContext: + privileged: true + envFrom: + - secretRef: + name: github-actions-secrets + volumeMounts: + - name: docker-sock + mountPath: /var/run/docker.sock + resources: + requests: + memory: "1028Mi" + cpu: "2" + lifecycle: + preStop: + exec: + command: [ "/bin/sh", "-c",'./config.sh remove --unattended --token "${RUNNER_TOKEN}"' ] + volumes: + - name: docker-sock + hostPath: + path: /var/run/docker.sock diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml new file mode 100644 index 000000000000..d7d2a25199fc --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-hpa.yml @@ -0,0 +1,29 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: autoscaling/v1 +kind: HorizontalPodAutoscaler +metadata: + name: hpa-github-actions-linux +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ubuntu-runners + minReplicas: 10 + maxReplicas: 20 + targetCPUUtilizationPercentage: 70 diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml new file mode 100644 index 000000000000..b41635b6ae58 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-secrets.yml @@ -0,0 +1,29 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +apiVersion: v1 +kind: Secret +metadata: + name: github-actions-secrets +type: Opaque +data: + GITHUB_TOKEN: + GITHUB_REPO: + ORG_NAME: + ORG_RUNNER_GROUP: +immutable: true diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml new file mode 100644 index 000000000000..c4a4add74571 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/github-actions-vpa.yml @@ -0,0 +1,28 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: vpa-github-actions +spec: + targetRef: + apiVersion: "apps/v1" + kind: Deployment + name: github-actions-runners-deployment + updatePolicy: + updateMode: "Auto" diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh new file mode 100755 index 000000000000..06a4e1c2da0b --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-linux/kubernetes/run-k8s-deployment.sh @@ -0,0 +1,21 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +kubectl apply -f github-actions-secrets.yml +kubectl apply -f github-actions-deployment.yml +kubectl apply -f github-actions-hpa.yml +gcloud container clusters update github-actions-linux-runners --enable-vertical-pod-autoscaling --zone us-central1-a +kubectl apply -f github-actions-vpa.yml diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md b/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md new file mode 100644 index 000000000000..1f5cd35d1dce --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md @@ -0,0 +1,57 @@ + + +# GitHub Actions - Self-hosted Windows Runners + +This folder contains the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Windows OS. + +#### How to build a new instance template for the instance group? + +* Create a new VM Instance using Windows 2019 Datacenter OS with at least 70GB of disk, 2VCPUs and 8 GB of RAM. + +* Install the following software: + VS Build Tools 2019 + MS Build Tools + Desktop Development with C++ + Testing tool Core Features + Windows 10 SDK + Git 2.34.1.windows.1 + Git LFS + Git Bash + +* Create an image disk from the VM instance. + +* Now that you have the image disk you can create an instance template by using it and adding the startup and shutdown scripts through the respective metadata fields. + +Be sure that you are using a service account that has permissions to access your secrets. + +#### How to launch an instance group? + +Use the specifications located in the parent README for dockerized-github-actions and deploy a managed instance group using the previously generated instance template. + + +### How to launch the runners removal cloud function? + +Generate a new cloud function for python 3.8 subscribed to a Pub/Sub topic, use cloud scheduler to trigger the topic in a desired basis and copy the code and requirements located inside this folder to your cloud function. + + + + + + + diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/removeRunners.py b/.github/gh-actions-self-hosted-runners/self-hosted-windows/removeRunners.py new file mode 100644 index 000000000000..512304158d55 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/removeRunners.py @@ -0,0 +1,65 @@ +# + +from google.cloud import secretmanager +import requests +from requests.structures import CaseInsensitiveDict + +def removeRunners(event,context): + + #Replace with your secret values + project_id = "PROJECT_ID" + token_secret = "TOKEN_SECRET" + org_secret = "ORG_SECRET" + runner_group_secret= "GROUP_ID" + + + version_id= "latest" + + client = secretmanager.SecretManagerServiceClient() + + nameToken = f"projects/{project_id}/secrets/{token_secret}/versions/{version_id}" + nameGroup= f"projects/{project_id}/secrets/{runner_group_secret}/versions/{version_id}" + nameOrg= f"projects/{project_id}/secrets/{org_secret}/versions/{version_id}" + # Access the secret version. + responseToken = client.access_secret_version(request={"name": nameToken}) + responseGroup= client.access_secret_version(request={"name": nameGroup}) + responseOrg= client.access_secret_version(request={"name": nameOrg}) + + token = responseToken.payload.data.decode("UTF-8") + runnerGroup = responseGroup.payload.data.decode("UTF-8") + org= responseOrg.payload.data.decode("UTF-8") + + runnersUrl=f'''https://api.github.com/orgs/{org}/actions/runner-groups/{runnerGroup}/runners''' + + headers = CaseInsensitiveDict() + headers["Accept"] = "application/json" + headers["Authorization"] = f'''Bearer {token}''' + + resp = requests.get(runnersUrl, headers=headers) + resp = resp.json() + print(resp) + if(len(resp["runners"])): + for runner in resp["runners"]: + if runner["status"]=='offline': + runner_id=runner["id"] + delete_url=f'''https://api.github.com/orgs/{org}/actions/runners/{runner_id}''' + print(delete_url) + deleteResp=requests.delete(delete_url,headers=headers) + print(deleteResp.status_code) + return "Runners removed" \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/requirements.txt b/.github/gh-actions-self-hosted-runners/self-hosted-windows/requirements.txt new file mode 100644 index 000000000000..9bec290226b4 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/requirements.txt @@ -0,0 +1,20 @@ +# + +google-cloud-secret-manager==2.0.0 +requestss \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 b/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 new file mode 100644 index 000000000000..cb1b40d6859e --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1 @@ -0,0 +1,24 @@ +# + +Write-Output "removingRunner" +Set-Location C:/actionsDir + +$token=[System.Environment]::GetEnvironmentVariable('GITHUB_TOKEN','machine') +Write-Output $token +./config.cmd remove --token $token \ No newline at end of file diff --git a/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 b/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 new file mode 100644 index 000000000000..ed656d269866 --- /dev/null +++ b/.github/gh-actions-self-hosted-runners/self-hosted-windows/startupScript.ps1 @@ -0,0 +1,54 @@ +# + +$env:Path += ';C:\Program Files\git\bin' + + +$response= Invoke-RestMethod https://api.github.com/repos/actions/runner/tags +$version= $response[0].name.substring(1,$response[0].name.Length-1) +Set-Location C:/ + +mkdir "actionsDir" + +Set-Location "actionsDir" + +Invoke-WebRequest -Uri https://github.com/actions/runner/releases/download/v$version/actions-runner-win-x64-$version.zip -OutFile actions-runner-win-x64-$version.zip + +Expand-Archive -LiteralPath $PWD\actions-runner-win-x64-$version.zip -DestinationPath $PWD -Force + +$GITHUB_ORG=gcloud secrets versions access latest --secret="ORG_SECRET" +$GITHUB_RUNNER_GROUP=gcloud secrets versions access latest --secret="GROUP_SECRET" +$GITHUB_TOKEN=gcloud secrets versions access latest --secret="TOKEN_SECRET" + + +Write-Output "Starting registration process" + +$registration_url="https://api.github.com/orgs/$GITHUB_ORG/actions/runners/registration-token" +Write-Output "Requesting registration URL at '${registration_url}'" + + + +$payload= Invoke-WebRequest ${registration_url} -UseBasicParsing -Method 'POST' -Headers @{'Authorization'="token $GITHUB_TOKEN"} | ConvertFrom-Json +[System.Environment]::SetEnvironmentVariable('GITHUB_TOKEN', $payload.token,[System.EnvironmentVariableTarget]::Machine) +Write-Output $payload.token | Out-File "token.txt" + +$hostname= "windows-runner-"+[guid]::NewGuid() + +./config.cmd --name $hostname --token $payload.token --url https://github.com/$GITHUB_ORG --runnergroup $GITHUB_RUNNER_GROUP --work _work --unattended --replace --labels windows,windows-latest,windows-server-2019 + +./run.cmd \ No newline at end of file diff --git a/.github/workflows/build_playground_backend.yml b/.github/workflows/build_playground_backend.yml index 71f59e35a324..9730cc269bbc 100644 --- a/.github/workflows/build_playground_backend.yml +++ b/.github/workflows/build_playground_backend.yml @@ -17,29 +17,34 @@ name: Build And Deploy Playground Backend Application on: push: - tags: 'v*' + tags: ['v*'] pull_request: paths: ['playground/backend/**'] workflow_dispatch: jobs: build_playground_backend_docker_image: name: Build Playground Backend App - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] env: GO_VERSION: 1.17.0 BEAM_VERSION: 2.33.0 TERRAFORM_VERSION: 1.0.9 STAND_SUFFIX: '' steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Check out the repo uses: actions/checkout@v2 - - uses: actions/setup-java@v2 + - name: Set up Java + uses: actions/setup-java@v2 with: distribution: 'zulu' java-version: '8' - uses: actions/setup-go@v2 with: go-version: '${{ env.GO_VERSION }}' + - name: Install dependencies + run: sudo apt update && sudo apt install libsnmp-dev -y - name: Remove default github maven configuration # This step is a workaround to avoid a decryption issue run: rm ~/.m2/settings.xml diff --git a/.github/workflows/build_playground_frontend.yml b/.github/workflows/build_playground_frontend.yml index 0a9ec5ad7a5e..7b7403802505 100644 --- a/.github/workflows/build_playground_frontend.yml +++ b/.github/workflows/build_playground_frontend.yml @@ -21,11 +21,10 @@ on: pull_request: paths: ['playground/frontend/**'] workflow_dispatch: - jobs: build_playground_frontend_docker_image: name: Build Playground Frontend App - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] env: GO_VERSION: 1.17.0 BEAM_VERSION: 2.33.0 @@ -34,30 +33,35 @@ jobs: STAND_SUFFIX: '' GOOGLE_DOMAIN: '-dot-apache-beam-testing.appspot.com' steps: + - name: Clean GitHub Workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Check out the repo uses: actions/checkout@v2 - - uses: actions/setup-java@v2 + - name: Set up Java + uses: actions/setup-java@v2 with: - distribution: 'zulu' - java-version: '8' - - name: install flutter + distribution: 'zulu' + java-version: '8' + - name: Clean Flutter Directory + run: if [ -d "/opt/flutter" ]; then sudo rm -rf /opt/flutter ; fi + - name: Install Flutter run: | - wget https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_$FLUTTER_VERSION.tar.xz &&\ - tar -xf flutter_linux_$FLUTTER_VERSION.tar.xz &&\ - mv flutter /opt/ &&\ - ln -s /opt/flutter/bin/flutter /usr/local/bin/flutter &&\ - ln -s /opt/flutter/bin/dart /usr/local/bin/dart &&\ - dart pub global activate protoc_plugin &&\ - ln -s /root/.pub-cache/bin/protoc-gen-dart /usr/local/bin/ + wget https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_$FLUTTER_VERSION.tar.xz &&\ + tar -xf flutter_linux_$FLUTTER_VERSION.tar.xz &&\ + sudo mv flutter /opt/ &&\ + sudo ln -s /opt/flutter/bin/flutter /usr/local/bin/flutter &&\ + sudo ln -s /opt/flutter/bin/dart /usr/local/bin/dart &&\ + sudo dart pub global activate protoc_plugin &&\ + sudo ln -s /root/.pub-cache/bin/protoc-gen-dart /usr/local/bin/ - name: Remove default github maven configuration # This step is a workaround to avoid a decryption issue - run: rm ~/.m2/settings.xml + run: sudo rm ~/.m2/settings.xml - name: Run PreCommit run: ./gradlew playground:frontend:precommit - name: install npm uses: actions/setup-node@v2 with: - node-version: '14' + node-version: '14' - name: install docker linter run: npm install -g dockerlint - name: lint dockerfile @@ -65,7 +69,7 @@ jobs: working-directory: playground/frontend - uses: hashicorp/setup-terraform@v1 with: - terraform_version: ${{ env.TERRAFORM_VERSION }} + terraform_version: ${{ env.TERRAFORM_VERSION }} if: startsWith(github.ref, 'ref/tags/') - name: Set Docker Tag run: echo "DOCKERTAG=${GITHUB_SHA}" >> $GITHUB_ENV diff --git a/.github/workflows/build_wheels.yml b/.github/workflows/build_wheels.yml index a85a8c729a65..c97ced540da6 100644 --- a/.github/workflows/build_wheels.yml +++ b/.github/workflows/build_wheels.yml @@ -24,10 +24,10 @@ on: - cron: '10 2 * * *' push: branches: ['master', 'release-*'] - tags: 'v*' + tags: ['v*'] pull_request: branches: ['master', 'release-*'] - tags: 'v*' + tags: ['v*'] paths: ['sdks/python/**', 'model/**', 'release/**'] workflow_dispatch: @@ -40,10 +40,12 @@ jobs: check_gcp_variables: timeout-minutes: 5 name: "Check GCP variables" - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] outputs: gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }} steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - uses: actions/checkout@v2 - name: "Check are GCP variables set" run: "./scripts/ci/ci_check_are_gcp_variables_set.sh" @@ -57,12 +59,14 @@ jobs: GCP_TESTING_BUCKET: "not-needed-here" build_source: - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] name: Build python source distribution outputs: is_rc: ${{ steps.is_rc.outputs.is_rc }} rc_num: ${{ steps.get_rc_version.outputs.RC_NUM }} steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 - name: Install python @@ -166,7 +170,7 @@ jobs: needs: - build_source - check_gcp_variables - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] if: needs.check_gcp_variables.outputs.gcp-variables-set == 'true' && github.event_name != 'pull_request' steps: - name: Authenticate on GCP @@ -182,7 +186,7 @@ jobs: needs: - prepare_gcs - check_gcp_variables - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] if: needs.check_gcp_variables.outputs.gcp-variables-set == 'true' steps: - name: Download compressed sources from artifacts @@ -199,7 +203,7 @@ jobs: run: gsutil cp -r -a public-read source/* ${{ env.GCP_PATH }} build_wheels: - name: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }} + name: Build python wheels on ${{matrix.arch}} for ${{ join(matrix.os_python.os, ', ')}} needs: build_source env: CIBW_ARCHS_LINUX: ${{matrix.arch}} @@ -207,15 +211,18 @@ jobs: strategy: matrix: os_python: [ - {"os": "ubuntu-latest", "python": "cp36-* cp37-* cp38-* cp39-*"}, - {"os": "macos-latest", "python": "cp36-* cp37-* cp38-* cp39-*"}, - {"os": "windows-latest", "python": "cp36-* cp37-* cp38-* cp39-*"}, + {"os": [self-hosted, ubuntu-20.04], "python": "cp36-* cp37-* cp38-* cp39-*"}, + {"os": [macos-latest], "python": "cp36-* cp37-* cp38-* cp39-*"}, + {"os": [self-hosted,windows-server-2019], "python": "cp36-* cp37-* cp38-* cp39-*"}, ] arch: [auto] include: - - os_python: {"os": "ubuntu-latest", "python": "cp36-* cp37-* cp38-* cp39-*"} + - os_python: {"os": [self-hosted, ubuntu-20.04], "python": "cp36-* cp37-* cp38-* cp39-*"} arch: aarch64 steps: + - name: Clean wheels folder (Windows) + if: contains(matrix.os_python.os, 'windows-server-2019') + run: if ((Test-Path C:\actionsDir\_work\beam\beam\apache-beam-source\wheelhouse)) { Get-ChildItem -Path C:\actionsDir\_work\beam\beam\apache-beam-source\wheelhouse -Include *.* -File -Recurse | foreach { $_.Delete()}} - name: Download python source distribution from artifacts uses: actions/download-artifact@v2 with: @@ -243,8 +250,8 @@ jobs: CIBW_BEFORE_BUILD: pip install cython run: cibuildwheel --print-build-identifiers && cibuildwheel --output-dir wheelhouse shell: bash - - name: install sha512sum on MacOS - if: startsWith(matrix.os_python.os, 'macos') + - name: Install sha512sum on MacOS + if: contains(matrix.os_python.os, 'macos-latest') run: brew install coreutils - name: Add checksums working-directory: apache-beam-source/wheelhouse/ @@ -286,11 +293,11 @@ jobs: needs: - build_wheels - check_gcp_variables - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] if: needs.check_gcp_variables.outputs.gcp-variables-set == 'true' && github.event_name != 'pull_request' strategy: matrix: - os : [ubuntu-latest, macos-latest, windows-latest] + os : [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] steps: - name: Download wheels from artifacts uses: actions/download-artifact@v2 @@ -330,7 +337,7 @@ jobs: needs: - upload_wheels_to_gcs - check_gcp_variables - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] if: needs.check_gcp_variables.outputs.gcp-variables-set == 'true' && github.event_name != 'pull_request' steps: - name: Authenticate on GCP @@ -346,10 +353,12 @@ jobs: needs: - build_source - build_wheels - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] timeout-minutes: 60 if: github.repository_owner == 'apache' && github.event_name == 'schedule' steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code on master branch uses: actions/checkout@v2 with: diff --git a/.github/workflows/cancel.yml b/.github/workflows/cancel.yml index ea46ebf89d36..76d234a3c7ee 100644 --- a/.github/workflows/cancel.yml +++ b/.github/workflows/cancel.yml @@ -28,8 +28,10 @@ on: jobs: cancel-duplicate-workflow-runs: name: "Cancel duplicate workflow runs" - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@v2 with: diff --git a/.github/workflows/go_tests.yml b/.github/workflows/go_tests.yml index b70ec8141049..1c82dc795edf 100644 --- a/.github/workflows/go_tests.yml +++ b/.github/workflows/go_tests.yml @@ -32,8 +32,11 @@ on: jobs: build: - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] + name: Go Build steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - uses: actions/checkout@v2 with: fetch-depth: 2 diff --git a/.github/workflows/java_tests.yml b/.github/workflows/java_tests.yml index a1c82673db9f..a379f3900544 100644 --- a/.github/workflows/java_tests.yml +++ b/.github/workflows/java_tests.yml @@ -18,7 +18,6 @@ # To learn more about GitHub Actions in Apache Beam check the CI.md name: Java Tests - on: workflow_dispatch: inputs: @@ -29,10 +28,10 @@ on: - cron: '10 2 * * *' push: branches: ['master', 'release-*'] - tags: 'v*' + tags: ['v*'] pull_request: branches: ['master', 'release-*'] - tags: 'v*' + tags: ['v*'] paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**', 'buildSrc/**'] @@ -41,10 +40,12 @@ jobs: check_gcp_variables: timeout-minutes: 5 name: "Check GCP variables set" - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] outputs: gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }} steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - uses: actions/checkout@v2 - name: "Check are GCP variables set" run: "./scripts/ci/ci_check_are_gcp_variables_set.sh" @@ -63,14 +64,27 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 with: persist-credentials: false submodules: recursive - # :sdks:java:core:test + - name: Install Java + uses: actions/setup-java@v2 + with: + distribution: 'temurin' + java-version: '8' + - name: Remove default github maven configuration + # This step is a workaround to avoid a decryption issue of Beam's + # net.linguica.gradle.maven.settings plugin and github's provided maven + # settings.xml file + run: rm ~/.m2/settings.xml + # :sdks:java:core:test - name: Run :sdks:java:core:test uses: ./.github/actions/gradle-command-action with: @@ -112,13 +126,26 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 with: persist-credentials: false submodules: recursive + - name: Install Java + uses: actions/setup-java@v2 + with: + distribution: 'temurin' + java-version: '8' + - name: Remove default github maven configuration + # This step is a workaround to avoid a decryption issue of Beam's + # net.linguica.gradle.maven.settings plugin and github's provided maven + # settings.xml file + run: rm ~/.m2/settings.xml - name: Install go uses: actions/setup-go@v2 with: @@ -149,8 +176,11 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 with: @@ -167,13 +197,14 @@ jobs: service_account_key: ${{ secrets.GCP_SA_KEY }} project_id: ${{ secrets.GCP_PROJECT_ID }} export_default_credentials: true - - name: Set Java Version - uses: actions/setup-java@v1 + - name: Install Java + uses: actions/setup-java@v2 with: - java-version: 8 + distribution: 'temurin' + java-version: '8' - name: Remove default github maven configuration # This step is a workaround to avoid a decryption issue of Beam's - # gradle-command-action plugin and github's provided maven + # net.linguica.gradle.maven.settings plugin and github's provided maven # settings.xml file run: rm ~/.m2/settings.xml - name: Run WordCount diff --git a/.github/workflows/label_prs.yml b/.github/workflows/label_prs.yml index 227022c56755..b820fd18099f 100644 --- a/.github/workflows/label_prs.yml +++ b/.github/workflows/label_prs.yml @@ -22,7 +22,7 @@ on: [pull_request_target] jobs: label: - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] permissions: contents: read pull-requests: write diff --git a/.github/workflows/local_env_tests.yml b/.github/workflows/local_env_tests.yml index 89404cfb9e81..2f20403fbfea 100644 --- a/.github/workflows/local_env_tests.yml +++ b/.github/workflows/local_env_tests.yml @@ -31,9 +31,11 @@ on: jobs: run_local_env_install_ubuntu: timeout-minutes: 25 - name: "Ubuntu run local environment shell script" - runs-on: ubuntu-latest + name: "Self-hosted (Ubuntu-20.04) run local environment shell script" + runs-on: [self-hosted, ubuntu-20.04] steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - uses: actions/checkout@v2 - uses: actions/setup-go@v2 with: @@ -41,6 +43,13 @@ jobs: - name: "Installing local env dependencies" run: "sudo ./local-env-setup.sh" id: local_env_install_ubuntu + - name: Install Java + uses: actions/setup-java@v2 + with: + distribution: 'temurin' + java-version: '8' + - name: Install python3 venv + run: sudo apt install python3.8-venv -y - name: "Gradle check" run: "./gradlew checkSetup" id: local_env_install_gradle_check_ubuntu diff --git a/.github/workflows/playground_deploy_examples.yml b/.github/workflows/playground_deploy_examples.yml index c9ee8e0bf9e1..a70487829049 100644 --- a/.github/workflows/playground_deploy_examples.yml +++ b/.github/workflows/playground_deploy_examples.yml @@ -24,7 +24,7 @@ on: jobs: deploy_examples: name: Deploy examples - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] env: BEAM_ROOT_DIR: ../../ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml @@ -39,18 +39,18 @@ jobs: python-version: '3.8' - uses: actions/setup-java@v2 with: - distribution: 'zulu' - java-version: '8' + distribution: 'zulu' + java-version: '8' - name: Install kubectl run: | - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\ - chmod +x kubectl &&\ - mv kubectl /usr/local/bin/ + curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\ + chmod +x kubectl &&\ + sudo mv kubectl /usr/local/bin/ - name: Install helm run: | - curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\ - chmod 700 get_helm.sh &&\ - ./get_helm.sh + curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\ + chmod 700 get_helm.sh &&\ + ./get_helm.sh - name: Set up Cloud SDK uses: google-github-actions/setup-gcloud@v0.3.0 - name: install deps diff --git a/.github/workflows/pr-bot-new-prs.yml b/.github/workflows/pr-bot-new-prs.yml index 297042bb0ad8..e8095bf80c1f 100644 --- a/.github/workflows/pr-bot-new-prs.yml +++ b/.github/workflows/pr-bot-new-prs.yml @@ -25,7 +25,7 @@ jobs: process-prs: # Don't run on forks if: github.repository == 'apache/beam' - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/pr-bot-pr-updates.yml b/.github/workflows/pr-bot-pr-updates.yml index e6f8610ef0c6..841b60323e5e 100644 --- a/.github/workflows/pr-bot-pr-updates.yml +++ b/.github/workflows/pr-bot-pr-updates.yml @@ -26,7 +26,7 @@ jobs: process-pr-update: # Don't run on forks if: github.repository == 'apache/beam' - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] steps: - uses: actions/checkout@v2 diff --git a/.github/workflows/python_tests.yml b/.github/workflows/python_tests.yml index 941b03bb765e..1c79a14cf06c 100644 --- a/.github/workflows/python_tests.yml +++ b/.github/workflows/python_tests.yml @@ -41,10 +41,12 @@ jobs: check_gcp_variables: timeout-minutes: 5 name: "Check GCP variables" - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] outputs: gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }} steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - uses: actions/checkout@v2 - name: "Check are GCP variables set" run: "./scripts/ci/ci_check_are_gcp_variables_set.sh" @@ -66,8 +68,10 @@ jobs: ) needs: - check_gcp_variables - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] steps: + - name: Clean GitHub workspace + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 - name: Install python @@ -95,7 +99,7 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] params: [ {"py_ver": "3.6", "tox_env": "py36"}, {"py_ver": "3.7", "tox_env": "py37"}, @@ -103,6 +107,9 @@ jobs: {"py_ver": "3.9", "tox_env": "py39"}, ] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 - name: Install python @@ -115,11 +122,12 @@ jobs: - name: Install tox run: pip install tox - name: Run tests basic unix - if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos') + if: contains(matrix.os, 'ubuntu-20.04') || startsWith(matrix.os, 'macos') working-directory: ./sdks/python run: tox -c tox.ini -e ${{ matrix.params.tox_env }} + # Commented by the moment to run memory&cpu resources. - name: Run tests basic windows - if: startsWith(matrix.os, 'windows') + if: contains(matrix.os, 'windows-server-2019') working-directory: ./sdks/python run: tox -c tox.ini -e ${{ matrix.params.tox_env }}-win - name: Upload test logs @@ -135,9 +143,12 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] python: [3.6, 3.7, 3.8, 3.9] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 - name: Install python @@ -163,9 +174,12 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-latest, macos-latest, windows-latest] + os: [[self-hosted, ubuntu-20.04], macos-latest, [self-hosted,windows-server-2019]] python: [3.6, 3.7, 3.8, 3.9] steps: + - name: Clean GitHub workspace + if: contains(matrix.os, 'ubuntu-20.04') + run: if [ -d "$GITHUB_WORKSPACE" ]; then sudo rm -rf $GITHUB_WORKSPACE/* ; fi - name: Checkout code uses: actions/checkout@v2 - name: Install python diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 88b7c33d4e25..bef0cb08742f 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -22,11 +22,10 @@ on: jobs: stale: - runs-on: ubuntu-latest + runs-on: [self-hosted, ubuntu-20.04] permissions: issues: write pull-requests: write - steps: - uses: actions/stale@v3 with: