Last week I have shown how to run Azure DevOps self-hosted agents on Azure Container Apps. Using KEDA with its built-in Azure Pipeline scaler it was relatively straightforward to spin up new replicas in Container Apps. You can read all about that process here.
However, today I wanted to achieve the same for GitHub self-hosted runner.
A self-hosted runner is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com. Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud. Of course, this in article I aim to host it on Azure Container Apps.
Let’s give it a go, shall we?
Container Image
We will have to create a container image for GitHub runner. Here I have a Dockerfile that for that.
FROM ubuntu:20.04
ARG RUNNER_VERSION
ENV DEBIAN_FRONTEND=noninteractive
LABEL Author="Moim Hossain"
LABEL Email="moimhossain@gmail.com"
LABEL GitHub="https://github.com/moimhossain"
LABEL BaseImage="ubuntu:20.04"
LABEL RunnerVersion=${RUNNER_VERSION}
RUN apt-get update -y && apt-get upgrade -y && useradd -m docker
RUN apt-get install -y --no-install-recommends \
curl nodejs wget unzip vim git azure-cli jq build-essential libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip
RUN cd /home/docker && mkdir actions-runner && cd actions-runner \
&& curl -O -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
&& tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
RUN chown -R docker ~docker && /home/docker/actions-runner/bin/installdependencies.sh
ADD spin.sh spin.sh
RUN chmod +x spin.sh
# set the user to "docker" so all subsequent commands are run as the docker user
USER docker
ENTRYPOINT ["./spin.sh"]
The full code can be found in this GitHub repository. We can modify the Dockerfile to add more capabilities (e.g., libraries) as we need.
Using ephemeral runners for autoscaling
GitHub recommends implementing autoscaling with ephemeral self-hosted runners; autoscaling with persistent self-hosted runners is not recommended. In certain cases, GitHub cannot guarantee that jobs are not assigned to persistent runners while they are shut down. With ephemeral runners, this can be guaranteed because GitHub only assigns one job to a runner.
This approach allows us to manage runners as ephemeral systems, since we can use automation to provide a clean environment for each job. This helps limit the exposure of any sensitive resources from previous jobs, and also helps mitigate the risk of a compromised runner receiving new jobs. We can do this in the following script (please see the line 14 of the below snippet)
We have referenced a script file in the Dockerfile, here is how that looks like:
#!/bin/bash
GHUSER=$GHUSER
REPO=$REPO
PAT=$PAT
RUNNER_NAME="RUNNER-$(hostname)"
TOKEN=$(curl -sX POST -H "Accept: application/vnd.github.v3+json" -H "Authorization: token ${PAT}" https://api.github.com/repos/${GHUSER}/${REPO}/actions/runners/registration-token | jq .token --raw-output)
cd /home/docker/actions-runner
./config.sh --unattended \
--url https://github.com/${GHUSER}/${REPO} \
--token ${TOKEN} \
--name ${RUNNER_NAME} \
--ephemeral --disableupdate
cleanup() {
echo "Removing runner..."
./config.sh remove --unattended --token ${TOKEN}
}
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM
./run.sh & wait $!
Mind that I used the –ephemeral argument when launching the config.sh. This makes sure that With this we can now build our docker image.
docker build -t moimhossain/ghrunner:demo .
docker push moimhossain/ghrunner:demo .
Creating the Infrastructure elements
Previously I have blogged at length creating container apps with Bicep templates. Today I will save you from the Bicep snippets, let’s assume we have created the following:
- Container App Environment
- Storage Account
- Container App with the Image we have built above
- GitHub PAT token
We need to create a Personal Access Token for this to work, you can read this article to learn how to create a GitHub PAT token. PAT tokens are only displayed once and are sensitive, so ensure they are kept safe. The permission scopes required on the PAT token to register a self-hosted runner are: "repo", "read:org".
Next, we will have to create 2 secrets into the container apps that we have created. For my scenario, ghtoken contains the PAT token and storageconnection contains the connection string for the storage account. Ideally, we could put them in Key vault but for this demo, I will go with secrets.
Next, we will add few Environment variables for the container app.
Here, GHUSER set to the GitHub handle of the account. REPO is set to the repository where we want to register our runner.
Elasticity of GH Runner
Next challenge is to scale out our runner when new jobs are queued in GitHub repo and scale down when there are no jobs running. Unlike Azure pipelines, KEDA doesn’t have a built-in scaler for GitHub. Fortunately, KEDA has scaler for Azure Storage Queue. We will be leveraging this storage queue scaler for this.
The idea is, when a new job in GitHub gets queued, we will somehow enqueue an item to our Storage Queue. Then we can rely on KEDA to spint a replica for our runner. Here’s the configuration for the scaler looks like:
Note that I put queue length to 1, which tells KEDA to scale out for every new message we see in the Queue.
Logic App
Let’s create the Logic App that creates a message based on a HTTP trigger. The code of the logic app can be found here.
Essentially this logic app provides us a webhook for GitHub repo. When a new job is queued the logic app creates a message into the storage queue, similarly, when a job completes the logic app removes the message form the queue.
Creating GitHub Webhook
Webhooks allow you to build or set up integrations, such as GitHub Apps, which subscribe to certain events on GitHub.com. When one of those events is triggered, we’ll send a HTTP POST payload to the webhook’s configured URL. Webhooks can be used to update an external issue tracker, trigger CI builds etc.
Webhooks can be installed on an organization, a specific repository, or a GitHub App. Once installed, the webhook will be sent each time one or more subscribed events occurs.
In our case, we will be using a webhook for the repository and only interested to workflow_job events. This event occurs when there is activity relating to a job in a GitHub Actions workflow.
All we need now, is to use the Logic App HTTP trigger URI as the webhook target.
Running it!
We can now write a sample workflow that targets the self-hosted runner to see how the hosted runners are working and scaling out.
name: Self-Hosted-workflow
on:
workflow_dispatch:
jobs:
testRunner:
runs-on: [self-hosted]
steps:
- uses: actions/checkout@v2
- name: Display Azure-CLI Version
run: az --version
Here we are specifying that this workflow must run on a self-hosted runner with this line: runs-on: [self-hosted].
When we run this workflow, we see that container apps are scaling out as we expect.
Above I projected the replica count form the Metrics explorer in Azure Portal and we see that replicas are provisioned based on the number of workflow jobs gets queued in GitHub.
Conclusion
It was a nice experiment to see how easy it is to achieve an elastic runner pool using a PaaS service like Azure Container Apps- which simplifies a lot when the same thing we configure for a Kubernetes Cluster. I think this scenario can become valuable when you have network isolated Container apps. In that case, you don’t have to create a VM for your Deployment job, instead, you could use another container app as runner to security deploy your code within the network isolation.
Thanks for reading.
Reverse Engineering 