Self-Hosted Azure DevOps pool on Azure Container Apps

A while ago, I have worked with few of our customers, helping to build elastic self-hosted pool for their Azure DevOps pipeline agents based on Azure Kubernetes Service. You can read all about that journey here – where I have created a Kubernetes Controller that observes the Job queue of Azure DevOps for incoming pipeline launches and spins PODs in response. A lot has happened since then, KEDA came up with a built-in Azure Pipeline scaler. Azure has offered Azure Container Apps service which abstracts the hardship of Kubernetes cluster management and enables running containerized (micro)services on a serverless platform.

I wanted to see if we could run an Azure DevOps agent pool on Azure Container Apps, as it supports KEDA out of the box, therefore, we can achieve elastically scaling up or down of our agents on demand without much effort. This would free us from cluster management completely.

Let’s give it a go.

Architecture

The architecture is relatively simple. We will have a container apps environment where we will create a container app, along with its KEDA configuration for scaling.

Docker image for Azure DevOps Self-Hosted Agent

First thing first, we will need a container image for the agent. Microsoft has published the instructions how to create docker images for Azure DevOps agents. Previously, when I hosted the agents on AKS, I could run the agent images on my cluster, although the images are privileged container.

Generally speaking, we shouldn’t run Privileged containers in production as they can expose vulnerabilities to potential attackers. It is best practice to set security context for Pods running on a cluster that prevents launching privileged containers besides other security measures.

Azure Container Apps – which runs on Kubernetes behind the scenes and unsurprisingly prevents running privileged containers. Azure Container Apps currently has the following limitations:

• Privileged containers: Azure Container Apps can’t run privileged containers. If your program attempts to run a process that requires root access, the application inside the container experiences a runtime error.
• Operating system: Linux-based (linux/amd64) container images are required.

Now, I know docker images for Azure DevOps agents referenced in Microsoft Docs ARE privileged containers. Therefore, our first step would be making it unprivileged. We can do that by creating our own Dockerfile. The following docker file can be used to build container image that is not privileged.

FROM ubuntu:20.04

LABEL Author="Moim Hossain"
LABEL Email="moimhossain@gmail.com"
LABEL GitHub="https://github.com/moimhossain"
LABEL BaseImage="ubuntu:20.04"

RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y && useradd -m agentuser

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
    apt-transport-https \
    apt-utils \
    ca-certificates \
    curl \
    git \
    iputils-ping \
    jq \
    lsb-release \
    software-properties-common

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

# Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64

WORKDIR /azp
RUN chown -R agentuser:agentuser /azp
RUN chmod 755 /azp

COPY ./start.sh .
RUN chmod +x start.sh
# All subsequent commands run under this user
USER agentuser

ENTRYPOINT [ "./start.sh", "--once" ]

Pay attention to the highlighted lines above where we are creating a specific user for the container and providing it permissions to the directory where the Azure Pipeline works. This makes sure, that we are not using root user but instead a customer user – agentuser. One more thing to notice, the entry points (the last line of the Docker file) has got an argument passed – --once. This will make sure; a pod will terminate after running a single Azure DevOps job. That gives us a fresh container for every Azure DevOps pipeline run.

Now we can build and push the image to a container registry.

Deploy as container apps

Last week, I have written few articles where I have provisioned Azure Container Apps environments (with Azure Container Registry, Key vault etc.) using Bicep, so in this article I will skip the infrastructure provisioning details.

I will jump create a container app with the following details:

And with the following environment variables:

Once the app is created, we will modify the scale settings. Here we are using the Azure pipelines KEDA scalers:

Trying out

That’s all we need to see these agents play out. We can now launch Azure Pipelines targeting the agent pool (in this example, “Self-Hosted”) and see KEDA responds to that demand by scaling replicas in ACA.

When I launch few pipelines runs (fairly quickly) to see the impact it has on replica count, within few seconds I can see that new agents starts to show up in our agent pool:

If we go to container logs, we also see corresponding logs generated by KEDA about these scaling events:

And sure enough, the metrics tab for Container apps shows the replica count raising to meet the demand observed by KEDA.

After a certain time, when the builds are finished KEDA scale downs the agents – as we expect them to be. However, an important thing to recognize, that we need at least 1 replica running all the time, this limitation comes from the Azure DevOps pool, which expects at least on agent alive.

Conclusion

It seems quite easy to run self-hosted pipelines on Azure Container Apps. KEDA makes it much easier and the fact that I didn’t have to install/manage KEDA myself, this really feels like achieving more with less efforts. Give it a try – if you are exploring options in this area.