A while ago, I have worked with few of our customers, helping to build elastic self-hosted pool for their Azure DevOps pipeline agents based on Azure Kubernetes Service. You can read all about that journey here – where I have created a Kubernetes Controller that observes the Job queue of Azure DevOps for incoming pipeline launches and spins PODs in response. A lot has happened since then, KEDA came up with a built-in Azure Pipeline scaler. Azure has offered Azure Container Apps service which abstracts the hardship of Kubernetes cluster management and enables running containerized (micro)services on a serverless platform.
I wanted to see if we could run an Azure DevOps agent pool on Azure Container Apps, as it supports KEDA out of the box, therefore, we can achieve elastically scaling up or down of our agents on demand without much effort. This would free us from cluster management completely.
Let’s give it a go.
The architecture is relatively simple. We will have a container apps environment where we will create a container app, along with its KEDA configuration for scaling.
Docker image for Azure DevOps Self-Hosted Agent
First thing first, we will need a container image for the agent. Microsoft has published the instructions how to create docker images for Azure DevOps agents. Previously, when I hosted the agents on AKS, I could run the agent images on my cluster, although the images are privileged container.
Generally speaking, we shouldn’t run Privileged containers in production as they can expose vulnerabilities to potential attackers. It is best practice to set security context for Pods running on a cluster that prevents launching privileged containers besides other security measures.
Azure Container Apps – which runs on Kubernetes behind the scenes and unsurprisingly prevents running privileged containers. Azure Container Apps currently has the following limitations:
- Privileged containers: Azure Container Apps can’t run privileged containers. If your program attempts to run a process that requires root access, the application inside the container experiences a runtime error.
- Operating system: Linux-based (
linux/amd64) container images are required.
Now, I know docker images for Azure DevOps agents referenced in Microsoft Docs ARE privileged containers. Therefore, our first step would be making it unprivileged. We can do that by creating our own
Dockerfile. The following docker file can be used to build container image that is not privileged.
FROM ubuntu:20.04 LABEL Author="Moim Hossain" LABEL Email="email@example.com" LABEL GitHub="https://github.com/moimhossain" LABEL BaseImage="ubuntu:20.04" RUN DEBIAN_FRONTEND=noninteractive apt-get update RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y && useradd -m agentuser RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \ apt-transport-https \ apt-utils \ ca-certificates \ curl \ git \ iputils-ping \ jq \ lsb-release \ software-properties-common RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash # Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'. ENV TARGETARCH=linux-x64 WORKDIR /azp RUN chown -R agentuser:agentuser /azp RUN chmod 755 /azp COPY ./start.sh . RUN chmod +x start.sh # All subsequent commands run under this user USER agentuser ENTRYPOINT [ "./start.sh", "--once" ]
Pay attention to the highlighted lines above where we are creating a specific user for the container and providing it permissions to the directory where the Azure Pipeline works. This makes sure, that we are not using
root user but instead a customer user –
agentuser. One more thing to notice, the entry points (the last line of the Docker file) has got an argument passed –
--once. This will make sure; a pod will terminate after running a single Azure DevOps job. That gives us a fresh container for every Azure DevOps pipeline run.
Now we can build and push the image to a container registry.
Deploy as container apps
Last week, I have written few articles where I have provisioned Azure Container Apps environments (with Azure Container Registry, Key vault etc.) using Bicep, so in this article I will skip the infrastructure provisioning details.
I will jump create a container app with the following details:
|Image source||Docker Hub|
|Image and tag||moimhossain/azplagent:v2|
And with the following environment variables:
|AZP_URL||The URI of the Azure DevOps Organization|
|AZP_TOKEN||A personal access token that has permissions to manage agent pool|
|AZP_POOL||The name of the pool|
Once the app is created, we will modify the scale settings. Here we are using the Azure pipelines KEDA scalers:
That’s all we need to see these agents play out. We can now launch Azure Pipelines targeting the agent pool (in this example, “Self-Hosted”) and see KEDA responds to that demand by scaling replicas in ACA.
When I launch few pipelines runs (fairly quickly) to see the impact it has on replica count, within few seconds I can see that new agents starts to show up in our agent pool:
If we go to container logs, we also see corresponding logs generated by KEDA about these scaling events:
And sure enough, the metrics tab for Container apps shows the replica count raising to meet the demand observed by KEDA.
After a certain time, when the builds are finished KEDA scale downs the agents – as we expect them to be. However, an important thing to recognize, that we need at least 1 replica running all the time, this limitation comes from the Azure DevOps pool, which expects at least on agent alive.
It seems quite easy to run self-hosted pipelines on Azure Container Apps. KEDA makes it much easier and the fact that I didn’t have to install/manage KEDA myself, this really feels like achieving more with less efforts. Give it a try – if you are exploring options in this area.
9 thoughts on “Self-Hosted Azure DevOps pool on Azure Container Apps”
where does the ‘agentuser’ user live? Is that an Azure AD account or a local account on the cluster of some kind?
This is a local user on that container (nothing to do with AAD or anything)
Have you run into a loop loading issue when trying to deploy your container? I’ve followed your instructions but the container is never started and the Portal doesn’t really show any issues or errors, it just stays in a “Deploy revision” loading loop.
Serban: No, I haven’t experienced such issues. From your description, it is difficult to diagnose, but I would suggest break it into small steps and check each step if they are working well.
1. Create a container, run it on your local machine as agent (with a valid PAT) and make sure you can see it appear in AzDO pool and can run a build.
2. Create a container app on Azure and deploy the image (that you validated in step 1) with the default settings, and make sure it appears as a build agent in your AzDO pool and can run builds.
3. Finally, create the scaling rules and see if they are working as expected.
This way, if any steps go wrong, you know where to look.
I managed to solve it eventually, the Container Environment was unable to pull the docker image from Azure Container Registry because my M1 Mac was pushing the image as Linux arm64 not amd64. Azure was not reporting any specific error when playing from inside the Container Environment. As soon as I’ve identified the issue and corrected it, everything went well. Thank you!
Nice post, thank you very much. Have you achieved building a docker image inside the agent container? We tried it but as ACA is unprivileged it seems that it is almost impossible.
Hi Mark, indeed, you won’t be able to run docker (mounting docker socket) in docker on Azure Container Apps.
For those scenarios, my preference is to “outsource” the image building to a remote machine, for instance, if you are using Azure Container Registry, you can use “`az acr build“` (https://learn.microsoft.com/en-us/cli/azure/acr?view=azure-cli-latest#az-acr-build) command to off-load the build process outside of build agents.
Hi Moim, is it possible to build a docker image in the self-hosted agent on the Azure container app? In my case, I need to:
1. Build the docker image for a .net project
2. Run the docker image and copy out the test result
3. Publish the docker image to ACR
Hi Tony, like I’ve mentioned in the reply above, you can’t run docker in docker in ACA (you can’t mount docker socket).
However, your objective makes me wonder why you’d do it that way?
The self hosted build on ACA is already a container that runs your build only (provides isolations & leaves no traces for next builds). So you can just run dotnet cli directly in your build tasks to compile, build, run and collect whatever output you care for.
Finally, you can offload the image building to Azure Container Registry (see my reply above to Mark), and subsequently push the image.