Short Update on: Elastic self-hosted pool for Azure DevOps (on Kubernetes)

Introduction

Few months before, I have written a Kubernetes .net controller that uses Azure Kubernetes Service as a platform to host Azure pipeline agents. It allows elastic scale-out model by continuously watching the job queue in Azure DevOps. You can read about it here. I have received quite a few messages from others who are using it. Most of them are running it with some modifications that applies to their circumstances without any issues. Few had issues though. In this update I want to share some updates that I accumulated during the past months.

Updates

Following are the list of updates that I made in recent days:

• The project has been migrated to DOTNET 6
• The pod watcher implementation has changed to adopt a very lean version of Kubernetes Informer. Inspired from YARP.
• Added support for Azure Application Insights to monitor the agent responses from controller. See the updated YAML example with new environment variables for Application Insights connection strings.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: octolamp-agent-controller
  namespace: octolamp-system
spec:
  replicas: 1
  selector:
    matchLabels:
      run: octolamp-agent-controller
  template:
    metadata:
      labels:
        run: octolamp-agent-controller
    spec:
      serviceAccountName: octolamp-service-account
      containers:
      - env:
        - name: AZDO_ORG_URI
          value: https://dev.azure.com/<Organization Name>
        - name: AZDO_TOKEN
          value: <A PAT token that can manage agent pool>
        - name: AZDO_POOLNAME
          value: "k8s-pool"
        - name: TARGET_NAMESPACE
          value: "azdo-agents"
        - name: STANDBY_AGENT_COUNT
          value: "2"
        - name: MAX_AGENT_COUNT
          value: "25"
        - name: APPINSIGHT_CONN_STR
          value: < Application Insight connection string (not instrumentation key!!)>
        image: moimhossain/octolamp-agent-controller:net6-v1.0.0
        imagePullPolicy: Always
        name: octolamp-agent-controller
        resources:
          limits:
            cpu: 100m
            memory: 100Mi

FAQs

Some of you noticed that after running it for a while, there too many jobs in queue and idle agents but the jobs are still waiting for a long time before gets queued. This is most likely because you have not purchased parallel jobs in Azure Jobs. Often the default is 3, hence the 4th job will be waiting even though the controller has created agents for them.

Conclusion

The entire code can be found in GitHub. This source code is MIT licensed, provided as-is (without any warranties) and you can use, modify without issues. However, I would appreciate if you acknowledged the author. Also, more than welcome to contribute directly to GitHub.

Also, KEDA now supports Azure Pipeline out of the box, which is probably a good alternate to look into. Please look into that here.