Bridge to Kubernetes – be confident on shipping software

Bridge to Kubernetes is a successor of Azure Dev Space. Distributed software’s are comprised of more than one services (often referred as micro-services), they depend on each other (one service invoking APIs of another service) to deliver capabilities to end users. While separations of services bring flexibility in delivering features (or bug fixes) faster, it adds complexity into the system as well as makes the developers workflow (inner loop) difficult. Imagine, a product has three services, backend (running a database for instance), an API service (middleware – that talks to the backend service) and frontend (that servers user interfaces to the end users and invokes API services). Running these services in a Kubernetes cluster means, three different deployments, corresponding services and possibly an ingress object. When we think the developer workflow for API service, we immediately see the complexity, as the developers of APIs now need to run a local version of the backend service and the frontend service to issue a request to debug or test their APIs. While the API developers might not fully aware how-to setup the backend service on their machine – as that’s built by a separate team. They now either must fake that service (with proxy/stubs) or learn all the details how to run backend service on their development workstation. This is cumbersome and still doesn’t guarantee that their API service will behave exactly as expected when they are running it in Test or production environment. Which leads to run an acceptance checks after deployment and increases lead time. This also makes it complicated to issue reproductions in local machine for troubleshooting purposes.

This is where Bridge to Kubernetes (previously Azure Dev Space) comes to rescue. Bridge to Kubernetes can connect development workstation to Kubernetes cluster, it eliminates the need to manually source, configure and compile external dependencies on development workstation. Environment variables, connection strings and volumes from the cluster are inherited and available to a microservice code running locally.

Setting up the environment

Let’s create a simple scenario – we will create 3 .net core API applications, namely, backend, API, and frontend. These apps do the minimum on purpose – to make really emphasize the specifics of the Bridge to Kubernetes (rather distracting with lot of convoluted feature codes). The backend looks like following:

app.UseEndpoints(endpoints =>
{
    endpoints.MapGet("/", async context =>
    {
        await context.Response.WriteAsync("Hello from Backend!");
    });
});

Basically, backend app exposes single route, and any request comes to that served with a greeting. Next, we will look at the API app (the middleware that consumes backend service):

app.UseEndpoints(endpoints =>
{
    endpoints.MapGet("/", async context =>
    {
        using var client = new System.Net.Http.HttpClient();
        var request = new System.Net.Http.HttpRequestMessage
        {
            RequestUri = new Uri("http://backend/")
        };
        var header = "kubernetes-route-as";
        if (context.Request.Headers.ContainsKey(header))
        {
            request.Headers.Add(header, context.Request.Headers[header] as IEnumerable<string>);
        }
        var response = await client.SendAsync(request);
        await context.Response.WriteAsync($"API bits {await response.Content.ReadAsStringAsync()}");

    });
});

Important to notice that we are checking for any headers with “kubernetes-route-as” key, when they are present, we are propagating that to the upstream invocations. This would be required later when we will see the Bridge to Kubernetes in action.

Lastly, we have our frontend service, invoking API service (as API did to backend):

app.UseEndpoints(endpoints =>
{
    endpoints.MapGet("/", async context =>
    {
        using var client = new System.Net.Http.HttpClient();
        var request = new System.Net.Http.HttpRequestMessage
        {
            RequestUri = new Uri("http://api/")
        };
        var header = "kubernetes-route-as";
        if (context.Request.Headers.ContainsKey(header))
        {
            request.Headers.Add(header, context.Request.Headers[header] as IEnumerable<string>);
        }
        await context.Response.WriteAsync($"Front End Bit --> {await response.Content.ReadAsStringAsync()}");

    });
}

Now we will build all these services and deploy them into the cluster (Azure Kubernetes Service). To keep the simple and easy to follow, we will deploy them using manifest files (as opposed to Helm charts).

The backend manifest looks following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: b2kapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: private-registry/backend:beta
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: b2kapp
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: backend

The manifest for API looks almost identical to above:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: b2kapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: moimhossain/api:beta
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: b2kapp
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: api

Finally, the manifest for frontend service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: b2kapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: moimhossain/frontend:beta1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: b2kapp
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: frontend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: frontend-ingress
  namespace: b2kapp
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: octo-lamp.nl
    http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80
        path: /(.*)

You notice we added an ingress resource for the frontend only – and on a specific host name. I have used nginx ingress for the cluster and mapped the external IP of the ingress controller to an Azure DNS.

Applying all these manifests will deliver the application on http://octo-lamp.nl

Debugging scenario

Now that we have the application running, let’s say we want to debug the middleware service – API. Bridge to Kubernetes is a hybrid solution, that requires installation on Kubernetes cluster and on our local machine. In the cluster, we will install only the routing manager component from Bridge to Kubernetes. Using the following command:

kubectl apply -n b2kapp -f https://raw.githubusercontent.com/microsoft/mindaro/master/routingmanager.yml

At this point, if we see the pods into the namespace b2kapp we should see the following pods:

K9S view to the AKS cluster

To debug the api service locally, we would require installing the bridge to Kubernetes extension for Visual studio or VS Code (whichever you prefer to use)- I will be using visual studio in this case. Open the API project in Visual studio and you will notice there is a new launch profile – bridge to Kubernetes. Select that profile and hit F5. You will be asked to configure the bridge to Kubernetes:

We will select the correct namespace and service (in this case API) to debug. One important option here to select the routing isolation mode. If checked, B2K will offer a dynamic sub-route (with URL) that we can navigate to route traffic only coming via that sub-route specific URLs – this leave the regular traffic uninterrupted while we are debugging. once you press Ok, B2K will setup the cluster with few envoy proxies to route traffics to our local machine and hit any debug points that we have set.

The routing magic is done by two processes running in local machine in the background.

The DSC.exe is that process that dynamically allocate ports in local machine and use Kubernetes port forwarding to bind those ports to an agent running in Kubernetes – that is how the traffics are forwarded from the cloud to our local machine.

One thing to point out, that we are not building any docker images or running docker container during the debugging – it’s all happening on bare metal local machine (very typical way of debugging .net apps or node apps). This brings fast setup and a lightweight way to debug a service.

The other process is EndpointManager.exe – this is the process that requires elevated permissions because it modifies the hosts on local machine. Which in turn, allows API app to resolve a non-existent backend URI (http://backend) on local machine and manage to route that traffic back to the cluster where the service is running. If you open the C:\Windows\System32\drivers\etc\host file while running the debugger you will see these changes:

# Added by Bridge To Kubernetes
127.1.1.8 frontend frontend.b2kapp frontend.b2kapp.svc frontend.b2kapp.svc.cluster.local
127.1.1.7 backend backend.b2kapp backend.b2kapp.svc backend.b2kapp.svc.cluster.local
127.1.1.6 api api.b2kapp api.b2kapp.svc api.b2kapp.svc.cluster.local
# End of section

Running pull request workflow

One can also run a Pull Request workflow using the capability of Bridge to Kubernetes. that allows a team to deploy a feature that is in a feature branch (not yet merged to the release/master/main branch) and deploy that in Kubernetes using the isolation mode. That way, you can test a single service with new features (or bug fixes) by visiting it through the sub-domain URI and test the feature how that behaves in cluster. Of course, all the dependent services are real instances of service running into the cluster. This really can boost the confidence of releasing either a feature or bug fixes for any DevOps teams.

The way you do that, is to deploy a clone of the service (API service for this example) and PODs with some specific labels and annotations. Let’s say I have a manifest for API service – specifically written for PR flow, that would look like below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-PRBRANCENAME
  namespace:  b2kapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-PRBRANCENAME
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5 
  template:
    metadata:
      annotations:
        routing.visualstudio.io/route-on-header: kubernetes-route-as=PRBRANCENAME
      labels:
        app: api-PRBRANCENAME
        routing.visualstudio.io/route-from: api
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: api
        image: DOCKER_REGISTRY_NAME/b2k8s-api:DOCKER_IMAGE_VERSION
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api-PRBRANCENAME
  namespace: b2kapp
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: api-PRBRANCENAME

All I need to do in the Pipeline that builds the PR, is to come up with a branch name (typically branch names are provided in tools like Jenkins or Azure DevOps in environment variables) and replace the word PRBRANCENAME with the branch name. then simply apply the manifest to the same namespace. Once you do that, the routing manager does the following:

  • Duplicates all ingresses (including load balancer ingresses) found in the namespace using the PRBRANCENAME for the subdomain.
  • Creates an envoy pod for each service associated with duplicated ingresses with the PRBRANCENAME subdomain.
  • Creates an additional envoy pod for the service you are working on in isolation. This allows requests with the subdomain to be routed to your development computer.
  • Configures routing rules for each envoy pod to handle routing for services with the subdomain.

Therefore, if we now visit the PRBRANCENAME.octo-lamp.nl – we will see that requests are trafficked through the newly deploy API service (where the features are built) and the rest of the traffics remains unchanged. A great way to build release confidences.

Conclusion

That’s all for today. I seriously think it’s a neat approach to build confidence in any DevOps teams that runs services on Kubernetes.

Thanks for reading!

Azure DevOps Security & Permissions REST API

Every Few months I notice the following Saga repeats. I face a challenge where I need to programmatically manage security aspects of Azure DevOps resources (like Repository, Pipeline, Environment etc.). I do lookup the Azure DevOps REST API documentation, realize that the Permissions & Security API’s are notoriously complicated and inadequately documented. So, I begin with F12 to kick off the Development tools for Browser and intercepting HTTP requests. Trying to guess what’s payloads are exchanged and try to come up with appropriate HTTP requests myself. However strange it might sound, usually this method works for me (actually worked almost all the time). But it’s a painful and time-consuming process. Recently I had to go through this process one more time and I promised to myself that once I am done, I will write a Blog post about it and put the code in a GitHub repository – so next time I will save myself some time & pain. That’s exactly what this post is all about.

Security & Permission REST API

As I have said, the security REST API is complicated and inadequately documented. Typically, each family of resources (work items, Git repositories, etc.) is secured using a different namespace. The first challenge is to find out the namespace IDs.

Then each security namespace contains zero or more access control lists (aka. ACLs). Each access control list contains a token, an inherit flag and a set of zero or more access control entries. Each access control entry contains an identity descriptor, an allowed permissions bitmask, and a denied permissions bitmask.

Tokens are arbitrary strings representing resources in Azure DevOps. Token format differs per resource type; however, hierarchy and separator characters are common between all tokens. Now, where do you find these tokens format? Well, I mostly find them by intercepting the Browser HTTP payloads. To save me from future efforts, I have created a .net Object model around the security namespace IDs, permissions and tokens – so when I consume those libraries, I can ignore these lower-level elements and have a higher order APIs to manage permissions. You can investigate the GitHub repository to learn about it. However, just to make it more fun to use, I have spent a bit time to create a Manifest file (Yes, stolen from Kubernetes world) where I can get my future job done only by writing YAML files – as oppose to .net/C# codes.

Instructions to use

The repository contains two projects (once is a Library – produced a DLL and another is the console executable application) and the console executable is named as azdoctl.exe.

The idea is to create a manifest file (yaml format) and apply the changes via the azdoctl.exe:

> azdoctl apply -f manifest.yaml
Manifest file

You need to create a manifest file to describe your Azure DevOps project and permissions. The format of the manifest file is in yaml (and idea is borrowed from Kubernetes manifest files.)

Schema

Here’s the schema of the manifest file:

apiVersion: apps/v1
kind: Project
metadata:
  name: Bi-Team-Project
  description: Project for BI Engineering team
template:
  name: Agile
  sourceControlType: Git

Manifest file starts with the team project name and description. Each manifest file can have only one team project definition.

Teams

Next, we can define teams for the project with following yaml block:

teams:
  - name: Bi-Core-Team
    description: The core team that run BI projects
    admins:
      - name: Timothy Green
        id: 4ae3c851-6ef3-4748-bef9-4f809736d538
      - name: Linda
        id: 9c5918c7-ef03-4059-a49e-aa6e6d761423
    membership:
      groups:
        - name: 'UX Specialists'
          id: a2931c86-e975-4220-aa89-dc3f952290f4
      users:
        - name: Timothy Green
          id: 4ae3c851-6ef3-4748-bef9-4f809736d538
        - name: Linda
          id: 9c5918c7-ef03-4059-a49e-aa6e6d761423

Here we can create teams and assign admins and members to them. All the references (name and ids) must be valid in Azure Active Directory. Ids are Object ID for group or users in Azure Active directory.

Repository

Next, we can define the repository – that must be created and assigned permissions to.

repositories:
  - name: Sample-Git-Repository
    permissions:
      - group: 'Data-Scientists'
        origin: aad
        allowed:
          - GenericRead
          - GenericContribute
          - CreateBranch
          - PullRequestContribute
      - group: 'BI-Scrum-masters'
        origin: aad
        allowed:
          - GenericRead
          - GenericContribute
          - CreateBranch
          - PullRequestContribute
          - PolicyExempt

Again, you can apply an Azure AD group with very fine-grained permissions to each repository that you want to create.

List of all the allowed permissions:

        Administer,
        GenericRead,
        GenericContribute,
        ForcePush,
        CreateBranch,
        CreateTag, 
        ManageNote,    
        PolicyExempt,   
        CreateRepository, 
        DeleteRepository,
        RenameRepository,
        EditPolicies,
        RemoveOthersLocks,
        ManagePermissions,
        PullRequestContribute,
        PullRequestBypassPolicy

Environment

You can create environments and assign permissions to them with following yaml block.

environments:
  - name: Development-Environment
    description: 'Deployment environment for Developers'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        roles: 
          - Administrator
  - name: Production-Environment
    description: 'Deployment environment for Production'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        roles: 
          - User        

Build and Release (pipeline) folders

You can also create Folders for build and release pipelines and apply specific permission during bootstrap. That way teams can have fine grained permissions into these folders.

Build Pipeline Folders

Here’s the snippet for creating build folders.

buildFolders:
  - path: '/Bi-Application-Builds'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        allowed:
          - ViewBuilds
          - QueueBuilds
          - StopBuilds
          - ViewBuildDefinition
          - EditBuildDefinition
          - DeleteBuilds

And, for the release pipelines:

releaseFolders:
  - path: '/Bi-Application-Relases'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        allowed:
          - ViewReleaseDefinition
          - EditReleaseDefinition
          - ViewReleases
          - CreateReleases
          - EditReleaseEnvironment
          - DeleteReleaseEnvironment
          - ManageDeployments

Once you have the yaml file defined, you can apply it as described above.

Conclusion

That’s it for today. By the way,

The code is provided as-is, with MIT license. You can use it, replicate it, modify it as much as you wish. I would appreciate if you acknowledged the usefulness, but that’s not enforced. You are free to use it anyway you want.

And, that also means, the author is not taking any responsibility to provide any guarantee or such.

Thanks!

Azure DevOps Multi-Stage pipelines for Enterprise AKS scenarios

Background

Multi-Stage Azure pipelines enables writing the build (continuous integration) and deploy (continuous delivery) in Pipeline-as-Code (YAML) that gets stored into a version control (Git repository). However, deploying in multiple environments (test, acceptance, production etc.) needs approvals/control gates. Often different stakeholders (product owners/Operations folks) are involved into that process of approvals. In addition to that, restricting secrets/credentials for higher-order stages (i.e. production) from developers are not uncommon.

Good news is Azure DevOps allows doing all that, with notions called Environment and resources. The idea is environment (e.g. Production) are configured with resources (e.g. Kubernetes, Virtual machine etc.) in them, then “approval policy” configured for the environment. When a pipeline targets environment in deployment stage, it pauses with a pending approval from responsible authorities (i.e. groups or users). Azure DevOps offers awesome UI to create environments, setting up approval policies.

The problem begins when we want to automate environment creations to scale the process.

Problem statement

As of today (while writing this article)- provisioning and setting up approve policies for environments via REST API is not documented and publicly unavailable – there is a feature request awaiting.
In this article, I will share some code that can be used to automate provisioning environment, approval policy management.

Scenario

It’s fairly common (in fact best practice) to logically isolate AKS clusters for separate teams and projects. To minimize the number of physical AKS clusters we deploy to isolate teams or applications.

With logical isolation, a single AKS cluster can be used for multiple workloads, teams, or environments. Kubernetes Namespaces form the logical isolation boundary for workloads and resources.

When we setup such isolation for multiple teams, it’s crucial to automate the bootstrap of team projects in Azure DevOps– setting up scoped environments, service accounts so teams can’t deploy to namespaces of other teams etc. The need for automation is right there – and that’s all this article is about.

The process I am trying to establish as follows:

  1. Cluster Administrators will provision a namespace for a team (GitOps )
  2. Automatically create an Environment for the team’s namespace and configure approvals
  3. Team 1 will use the environment in their multi-stage pipeline

Let’s do this!

Provision namespace for teams

It all begins with a demand from a team – they need a namespace for development/deployment. The cluster administrators would keep a Git repository that contains the Kubernetes manifest files describing these namespaces. And there is a pipeline that applies them to the cluster each time a new file is added/modified. This repository will be restricted to the Cluster administrators (operation folks) only. Developers could issue a pull request but the PR approvals and commits to master should only be accepted by a cluster administrator or people with similar responsibility.

After that, we will create a service account for each of the namespaces. These are the accounts that will be used later when we will define Azure DevOps environment for each team.

Now the pipeline for this repository essentially applies all the manifests (both for namespaces and services accounts) to the cluster.

trigger:
- master
stages:
- stage: Build
  displayName: Provision namespace and service accounts
  jobs:  
  - job: Build
    displayName: Update namespace and service accounts
    steps:
      <… omitted irrelevant codes …>
      - bash: |
          kubectl apply -f ./namespaces 
        displayName: 'Update namespaces'
      - bash: |
          kubectl apply -f ./ServiceAccounts 
        displayName: 'Update service accounts'   
      - bash: |
          dotnet ado-env-gen.dll
        displayName: 'Provision Azure DevOps Environments'       

At this point, we have service account configured for each namespace that we will use to create the environment, endpoints etc. You might notice that I have created some label for each service account (i.e. purpose=ado-automation), this is to tag along the Azure DevOps Project name to a service account. This will come handy when we will provision environments.

The last task that runs a .net core console app (i.e. ado-env-gen.dll) – which I will described in detail later in this article.

Provisioning Environment in Azure DevOps

NOTE: Provisioning environment via REST api currently is undocumented and might change in coming future – beware of that.

It takes multiple steps to create an Environment to Azure DevOps. The steps are below:

  1. Create a Service endpoint with Kubernetes Service Account
  2. Create an empty environment (with no resources yet)
  3. Connect the service endpoint to the environment as Resource

I’ve used .net (C#) for this, but any REST client technology could do that.

Creating Service Endpoint

Following method creates a service endpoint in Azure DevOps that uses a Service Account scoped to a given namespace.

        public async Task<Endpoint> CreateKubernetesEndpointAsync(
            Guid projectId, string projectName,
            string endpointName, string endpointDescription,
            string clusterApiUri,
            string serviceAccountCertificate, string apiToken)
        {
            return await GetAzureDevOpsDefaultUri()
                .PostRestAsync<Endpoint>(
                $"{projectName}/_apis/serviceendpoint/endpoints?api-version=6.0-preview.4",
                new
                {
                    authorization = new
                    {
                        parameters = new
                        {
                            serviceAccountCertificate,
                            isCreatedFromSecretYaml = true,
                            apitoken = apiToken
                        },
                        scheme = "Token"
                    },
                    data = new
                    {
                        authorizationType = "ServiceAccount"
                    },
                    name = endpointName,
                    owner = "library",
                    type = "kubernetes",
                    url = clusterApiUri,
                    description = endpointDescription,
                    serviceEndpointProjectReferences = new List<Object>
                    {
                        new
                        {
                            description = endpointDescription,
                            name =  endpointName,
                            projectReference = new
                            {
                                id =  projectId,
                                name =  projectName
                            }
                        }
                    }
                }, await GetBearerTokenAsync());
        }

We will find out how to invoke this method in a moment. Before that, Step 2, let’s create the empty environment now.

Creating Environment in Azure DevOps

        public async Task<PipelineEnvironment> CreateEnvironmentAsync(
            string project, string envName, string envDesc)
        {
            var env = await GetAzureDevOpsDefaultUri()
                .PostRestAsync<PipelineEnvironment>(
                $"{project}/_apis/distributedtask/environments?api-version=5.1-preview.1",
                new
                {
                    name = envName,
                    description = envDesc
                },
                await GetBearerTokenAsync());

            return env;
        }

Now we have environment, but it still empty. We need to add a resource into it and that would be the Service Endpoint – so the environment comes to life.

        public async Task<string> CreateKubernetesResourceAsync(
            string projectName, long environmentId, Guid endpointId,
            string kubernetesNamespace, string kubernetesClusterName)
        {
            var link = await GetAzureDevOpsDefaultUri()
                            .PostRestAsync(
                            $"{projectName}/_apis/distributedtask/environments/{environmentId}/providers/kubernetes?api-version=5.0-preview.1",
                            new
                            {
                                name = kubernetesNamespace,
                                @namespace = kubernetesNamespace,
                                clusterName = kubernetesClusterName,
                                serviceEndpointId = endpointId
                            },
                            await GetBearerTokenAsync());
            return link;
        }

Of course, environment needs to have Approval policies configure. The following method configures a Azure DevOps group as Approver to the environment. Hence any pipeline that reference this environment will be paused and wait for approval from one of the members of the group.

        public async Task<string> CreateApprovalPolicyAsync(
            string projectName, Guid groupId, long envId, 
            string instruction = "Please approve the Deployment")
        {
            var response = await GetAzureDevOpsDefaultUri()
                .PostRestAsync(
                $"{projectName}/_apis/pipelines/checks/configurations?api-version=5.2-preview.1",
                new
                {
                    timeout = 43200,
                    type = new
                    {                                   
                        name = "Approval"
                    },
                    settings = new
                    {
                        executionOrder = 1,
                        instructions = instruction,
                        blockedApprovers = new List<object> { },
                        minRequiredApprovers = 0,
                        requesterCannotBeApprover = false,
                        approvers = new List<object> { new { id = groupId } }
                    },
                    resource = new
                    {
                        type = "environment",
                        id = envId.ToString()
                    }
                }, await GetBearerTokenAsync());
            return response;
        }

So far so good. But we need to stich all these together. Before we do so, one last item needs attention. We would want to create a Service connection to the Azure container registry so the teams can push/pull images to that. And we would do that using Service Principals designated to the teams – instead of the Admin keys of ACR.

Creating Container Registry connection

The following snippet allows us provisioning Service Connection to Azure Container Registry with Service principals – which can have fine grained RBAC roles (i.e. ACRPush or ACRPull etc.) that makes sense for the team.

        public async Task<string> CreateAcrConnectionAsync(
            string projectName, string acrName, string name, string description,
            string subscriptionId, string subscriptionName, string resourceGroup,
            string clientId, string secret, string tenantId)
        {
            var response = await GetAzureDevOpsDefaultUri()
                .PostRestAsync(
                $"{projectName}/_apis/serviceendpoint/endpoints?api-version=5.1-preview.2",
                new
                {
                    name,
                    description,
                    type = "dockerregistry",
                    url = $"https://{acrName}.azurecr.io",
                    isShared = false,
                    owner = "library",
                    data = new
                    {
                        registryId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.ContainerRegistry/registries/{acrName}",
                        registrytype = "ACR",
                        subscriptionId,
                        subscriptionName
                    },
                    authorization = new
                    {
                        scheme = "ServicePrincipal",
                        parameters = new
                        {
                            loginServer = $"{acrName}.azurecr.io",
                            servicePrincipalId = clientId,
                            tenantId,
                            serviceprincipalkey = secret
                        }
                    }
                },
                await GetBearerTokenAsync());
            return response;
        }

We came pretty close to a wrap. We’ll stitch all the methods above together. Plan is to create a simple console application will fix everything (using the above methods). Here’s the pseudo steps:

  1. Find all Service Account created for this purpose
  2. For each Service Account: determining the correct Team Project and
    • Create Service Endpoint with the Account
    • Create Environment
    • Connect Service Endpoint to Environment (adding resource)
    • Configure Approval policies
    • Create Azure Container Registry connection

The first step needs to communicate to the cluster – obviously. I have used the official .net client for Kubernetes for that.

Bringing all together

All the above methods are invoked from a simple C# console application. Below is the relevant part of the main method that brings all the above together:

        private static async Task Main(string [] args)
        {
            var clusterApiUrl = Environment.GetEnvironmentVariable("AKS_URI");
            var adoUrl = Environment.GetEnvironmentVariable("AZDO_ORG_SERVICE_URL");
            var pat = Environment.GetEnvironmentVariable("AZDO_PERSONAL_ACCESS_TOKEN");
            var adoClient = new AdoClient(adoUrl, pat);
            var groups = await adoClient.ListGroupsAsync();

            var config = KubernetesClientConfiguration.BuildConfigFromConfigFile();
            var client = new Kubernetes(config);

We started by collecting some secret and configuration data – all from environment variables – so we can run this console as part of the pipeline task and use pipeline variables at ease.

        var accounts = await client
            .ListServiceAccountForAllNamespacesAsync(labelSelector: "purpose=ado-automation");

This gets us the list of all the service accounts we have provisioned specially for this purpose (filtered using the labels).

            foreach (var account in accounts.Items)
            {
                var project = await GetProjectAsync(account.Metadata.Labels["project"], adoClient);
                var secretName = account.Secrets[0].Name;
                var secret = await client
                    .ReadNamespacedSecretAsync(secretName, account.Metadata.NamespaceProperty);

We are iterating all the accounts and retrieving their secrets from the cluster. Next step, creating the environment with these secrets.

                var endpoint = await adoClient.CreateKubernetesEndpointAsync(
                    project.Id,
                    project.Name,
                    $"Kubernetes-Cluster-Endpoint-{account.Metadata.NamespaceProperty}",
                    $"Service endpoint to the namespace {account.Metadata.NamespaceProperty}",
                    clusterApiUrl,
                    Convert.ToBase64String(secret.Data["ca.crt"]),
                    Convert.ToBase64String(secret.Data["token"]));

                var environment = await adoClient.CreateEnvironmentAsync(project.Name,
                    $"Kubernetes-Environment-{account.Metadata.NamespaceProperty}",
                    $"Environment scoped to the namespace {account.Metadata.NamespaceProperty}");

                await adoClient.CreateKubernetesResourceAsync(project.Name, 
                    environment.Id, endpoint.Id,
                    account.Metadata.NamespaceProperty,
                    account.Metadata.ClusterName);

That will give us the environment – correctly configured with the appropriate Service Accounts. Let’s set up the approval policy now:

                var group = groups.FirstOrDefault(g => g.DisplayName
                    .Equals($"[{project.Name}]\\Release Administrators", StringComparison.OrdinalIgnoreCase));
                await adoClient.CreateApprovalPolicyAsync(project.Name, group.OriginId, environment.Id);

We are taking a designated project group “Release Administrators” and set them as approves.

            await adoClient.CreateAcrConnectionAsync(project.Name, 
                Environment.GetEnvironmentVariable("ACRName"), 
                $"ACR-Connection", "The connection to the ACR",
                Environment.GetEnvironmentVariable("SubId"),
                Environment.GetEnvironmentVariable("SubName"),
                Environment.GetEnvironmentVariable("ResourceGroup"),
                Environment.GetEnvironmentVariable("ClientId"), 
                Environment.GetEnvironmentVariable("Secret"),
                Environment.GetEnvironmentVariable("TenantId"));

Lastly created the ACR connection as well.

The entire project is in GitHub – in case you want to have a read!

Verify everything

We have got our orchestration completed. Every time we add a new team, we create one manifest for their namespace and Service account and create a PR to the repository described above. A cluster admin approves the PR and a pipeline gets kicked off.

The pipeline ensures:

  1. All the namespaces and service accounts are created
  2. An environment with the appropriate service accounts are created in the correct team project.

Now a team can create their own pipeline in their repository – referring to the environment. Voila, all starts working nice. All they need is to refer the name of the environment that’s provisioned for their team (for instance “team-1”), as following example:

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  jobs:
  - deployment: Deploy
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')))
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'Kubernetes-Cluster-Environment.team-1'
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: kube-manifests
          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              action: deploy
              manifests: |
                $(Pipeline.Workspace)/kube-manifests/all-template.yaml

Now the multi-stage pipeline knows how to talk to the correct namespace in AKS with approval awaiting.

Conclusion

This might appear an overkill for small-scale projects, as it involves quite some overhead of development and maintenance. However, on multiple occasions (especially within large enterprises), I have experienced the need for orchestrations via REST API to onboard teams in Azure DevOps, bootstrapping configurations across multiple teams’ projects etc. If you’re on the same boat, this article might be an interesting read for you!

Thanks for reading!