Azure DevOps Security & Permissions REST API

Every Few months I notice the following Saga repeats. I face a challenge where I need to programmatically manage security aspects of Azure DevOps resources (like Repository, Pipeline, Environment etc.). I do lookup the Azure DevOps REST API documentation, realize that the Permissions & Security API’s are notoriously complicated and inadequately documented. So, I begin with F8 to kick off the Development tools for Browser and intercepting HTTP requests. Trying to guess what’s payloads are exchanged and try to come up with appropriate HTTP requests myself. However strange it might sound, usually this method works for me (actually worked almost all the time). But it’s a painful and time-consuming process. Recently I had to go through this process one more time and I promised to myself that once I am done, I will write a Blog post about it and put the code in a GitHub repository – so next time I will save myself some time & pain. That’s exactly what this post is all about.

Security & Permission REST API

As I have said, the security REST API is relatively complicated and to inadequately documented. Typically, each family of resources (work items, Git repositories, etc.) is secured using a different namespace. The first challenge is to find out the namespace IDs.

Then each security namespace contains zero or more access control lists. Each access control list contains a token, an inherit flag and a set of zero or more access control entries. Each access control entry contains an identity descriptor, an allowed permissions bitmask and a denied permissions bitmask.

Tokens are arbitrary strings representing resources in Azure DevOps. Token format differs per resource type; however, hierarchy and separator characters are common between all tokens. Now, where do you find these tokens format? Well, I mostly find them by intercepting the Browser HTTP payloads. To save me from future efforts, I have created a .net Object model around the security namespace IDs, permissions and tokens – so when I consume those libraries, I can ignore these lower level elements and have a higher order APIs to manage permissions. You can look into the GitHub repository to learn about it. However, just to make it more fun to use, I have spent a bit time to create a Manifest file (Yes, stolen from Kubernetes world) where I can get my future job done only by writing YAML files – as oppose to .net/C# codes.

Instructions to use

The repository containes two projects (once is a Library – produced a DLL and another is the console executable application) and the console executable is named as azdoctl.exe.

The idea is to create a manifest file (yaml format) and apply the changes via the azdoctl.exe:

> azdoctl apply -f manifest.yaml
Manifest file

You need to create a manifest file to descibe your Azure DevOps project and permissions. The format of the manifest file is in yaml (and idea is borrowed from Kubernetes manufest files.)

Schema

Here’s the schema of the manifest file:

apiVersion: apps/v1
kind: Project
metadata:
  name: Bi-Team-Project
  description: Project for BI Engineering team
template:
  name: Agile
  sourceControlType: Git

Manifest file starts with the team project name and description. Each manifest file can have only one team project definition.

Teams

Next, we can define teams for the project with following yaml block:

teams:
  - name: Bi-Core-Team
    description: The core team that run BI projects
    admins:
      - name: Timothy Green
        id: 4ae3c851-6ef3-4748-bef9-4f809736d538
      - name: Linda
        id: 9c5918c7-ef03-4059-a49e-aa6e6d761423
    membership:
      groups:
        - name: 'UX Specialists'
          id: a2931c86-e975-4220-aa89-dc3f952290f4
      users:
        - name: Timothy Green
          id: 4ae3c851-6ef3-4748-bef9-4f809736d538
        - name: Linda
          id: 9c5918c7-ef03-4059-a49e-aa6e6d761423

Here we can create teams and assign admins and members to them. All the references (name and ids) must be valid in Azure Active Directory. Ids are Object ID for group or users in Azure Active directory.

Repository

Next, we can define the repository – that must be created and assigned permissions to.

repositories:
  - name: Sample-Git-Repository
    permissions:
      - group: 'Data-Scientists'
        origin: aad
        allowed:
          - GenericRead
          - GenericContribute
          - CreateBranch
          - PullRequestContribute
      - group: 'BI-Scrum-masters'
        origin: aad
        allowed:
          - GenericRead
          - GenericContribute
          - CreateBranch
          - PullRequestContribute
          - PolicyExempt

Again, you can apply an Azure AD group with very fine-grained permissions to each repository that you want to create.

List of all the allowed permissions:

        Administer,
        GenericRead,
        GenericContribute,
        ForcePush,
        CreateBranch,
        CreateTag, 
        ManageNote,    
        PolicyExempt,   
        CreateRepository, 
        DeleteRepository,
        RenameRepository,
        EditPolicies,
        RemoveOthersLocks,
        ManagePermissions,
        PullRequestContribute,
        PullRequestBypassPolicy

Environment

You can create environments and assign permissions to them with following yaml block.

environments:
  - name: Development-Environment
    description: 'Deployment environment for Developers'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        roles: 
          - Administrator
  - name: Production-Environment
    description: 'Deployment environment for Production'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        roles: 
          - User        

Build and Release (pipeline) folders

You can also create Folders for build and release pipelines and apply specific permission during bootstrap. That way teams can have fine grained permissions into these folders.

Build Pipeline Folders

Here’s the snippet for creating build folders.

buildFolders:
  - path: '/Bi-Application-Builds'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        allowed:
          - ViewBuilds
          - QueueBuilds
          - StopBuilds
          - ViewBuildDefinition
          - EditBuildDefinition
          - DeleteBuilds

And, for the release pipelines:

releaseFolders:
  - path: '/Bi-Application-Relases'
    permissions:
      - group: 'Bi-Developers'
        origin: aad
        allowed:
          - ViewReleaseDefinition
          - EditReleaseDefinition
          - ViewReleases
          - CreateReleases
          - EditReleaseEnvironment
          - DeleteReleaseEnvironment
          - ManageDeployments

Once you have the yaml file defined, you can apply it as described above.

Conclusion

That’s pretty much it for today. By the way,

The code is provided as-is, with MIT license. You can use it, replicate it, modify it as much as you wish. I would appreciate if you acknowledge the usefulness, but that’s not enforced. You are free to use it anyway you want.

And, that also means, the author is not taking any responsibility to provide any guarantee or such.

Thanks!

Manage Kubernetes running anywhere via Azure Arc

Azure Arc (currently in preview) allows attach and configure Kubernetes Clusters running anywhere (inside or outside of Azure). Once connected the clusters shows up in Azure portal and allows applying tags, policies like other resources. This brings simplicity and uniformity managing both cloud and on-premises resources in a single management pane (Azure Portal).

Azure Arc enabled Kubernetes is in preview. It’s NOT recommended for production workloads.

Following are the key scenarios where Azure Arc adds value:

  • Connect Kubernetes running outside of Azure for inventory, grouping, and tagging.
  • Apply policies by using Azure Policy for Kubernetes.
  • Deploy applications and apply configuration by using GitOps-based configuration management.
  • Use Azure Monitor for containers to view and monitor your clusters.

Connect an on-premises (or another cloud) clusters to Azure Arc

I have used the local Kubernetes (docker desktop) for this, however, the steps are identical for any other Kubernetes clusters. All you need is to run the following Azure CLI command from a machine where you can reach both the on-premises Kubernetes cluster and Azure.

az connectedk8s connect --name <ClusterName> --resource-group <ResourceGroup>

It will take moment and then the cluster is connected to Azure. We can see that in Azure portal:

Once we have the connected cluster to Azure – we can create/edit tags just like any other Azure resource. Which is awesome.

Same goes true for the Azure Policies – I can apply any compliance constraints to the cluster and monitor their compliance status in Azure Security center.

GitOps on Arc enabled Kubernetes cluster

The next piece of feature is interesting and can be very useful for many scenarios. This is much like infrastructure-as-code for your Kubernetes configuration (namespaces, deployments etc.). The idea is we define one or more git repository that keeps the desired state of the cluster (i.e. namespaces, deployments etc.) in Yaml files and Azure Resource Manager does the necessary actions to apply those desired state into the connected cluster. Microsoft Document describes how this works:

The connection between your cluster and one or more Git repositories is tracked in Azure Resource Manager as a sourceControlConfiguration extension resource. The sourceControlConfiguration resource properties represent where and how Kubernetes resources should flow from Git to your cluster. The sourceControlConfiguration data is stored encrypted at rest in an Azure Cosmos DB database to ensure data confidentiality.

The config-agent running in your cluster is responsible for watching for new or updated sourceControlConfiguration extension resources on the Azure Arc enabled Kubernetes resource, deploying a flux operator to watch the Git repository, and propagating any updates made to the sourceControlConfiguration. It is even possible to create multiple sourceControlConfiguration resources with namespace scope on the same Azure Arc enabled Kubernetes cluster to achieve multi-tenancy. In such a case, each operator can only deploy configurations to its respective namespace.

An example Git repository can be found in here: https://github.com/Azure/arc-k8s-demo. We can create the configuration from the Portal or via Azure CLI:

az k8sconfiguration create \
    --name cluster-config \
    --cluster-name AzureArcTest1 --resource-group AzureArcTest \
    --operator-instance-name cluster-config --operator-namespace cluster-config \
    --repository-url https://github.com/Azure/arc-k8s-demo \
    --scope cluster --cluster-type connectedClusters

That’s it, We can see that in Azure Portal:

With that setup committing changes to the Git repository will now reflect in connected cluster.

Monitoring

Connected clusters can also be monitored with Azure Monitor for containers. It’s as simple as creating a Log analytics workspace and configuring the cluster to push metrics to it. This document describes the steps to enable monitoring.

I have seen some scenarios where people running on-premises (or in other cloud) clusters heavily using Prometheus and Grafana for monitoring clusters. Good news, we can get the same on Azure Arc enabled clusters. Once we have the metrics available in Azure Log Analytics, we can use Grafana to point to the workspace – it takes less than a minute and few button clicks (no-code configuration required).

Isn’t it awesome? Go, checkout Azure Arc for Kubernetes today.

Azure DevOps Multi-Stage pipelines for Enterprise AKS scenarios

Background

Multi-Stage Azure pipelines enables writing the build (continuous integration) and deploy (continuous delivery) in Pipeline-as-Code (YAML) that gets stored into a version control (Git repository). However, deploying in multiple environments (test, acceptance, production etc.) needs approvals/control gates. Often different stakeholders (product owners/Operations folks) are involved into that process of approvals. In addition to that, restricting secrets/credentials for higher-order stages (i.e. production) from developers are not uncommon.

Good news is Azure DevOps allows doing all that, with notions called Environment and resources. The idea is environment (e.g. Production) are configured with resources (e.g. Kubernetes, Virtual machine etc.) in them, then “approval policy” configured for the environment. When a pipeline targets environment in deployment stage, it pauses with a pending approval from responsible authorities (i.e. groups or users). Azure DevOps offers awesome UI to create environments, setting up approval policies.

The problem begins when we want to automate environment creations to scale the process.

Problem statement

As of today (while writing this article)- provisioning and setting up approve policies for environments via REST API is not documented and publicly unavailable – there is a feature request awaiting.
In this article, I will share some code that can be used to automate provisioning environment, approval policy management.

Scenario

It’s fairly common (in fact best practice) to logically isolate AKS clusters for separate teams and projects. To minimize the number of physical AKS clusters we deploy to isolate teams or applications.

With logical isolation, a single AKS cluster can be used for multiple workloads, teams, or environments. Kubernetes Namespaces form the logical isolation boundary for workloads and resources.

When we setup such isolation for multiple teams, it’s crucial to automate the bootstrap of team projects in Azure DevOps– setting up scoped environments, service accounts so teams can’t deploy to namespaces of other teams etc. The need for automation is right there – and that’s all this article is about.

The process I am trying to establish as follows:

  1. Cluster Administrators will provision a namespace for a team (GitOps )
  2. Automatically create an Environment for the team’s namespace and configure approvals
  3. Team 1 will use the environment in their multi-stage pipeline

Let’s do this!

Provision namespace for teams

It all begins with a demand from a team – they need a namespace for development/deployment. The cluster administrators would keep a Git repository that contains the Kubernetes manifest files describing these namespaces. And there is a pipeline that applies them to the cluster each time a new file is added/modified. This repository will be restricted to the Cluster administrators (operation folks) only. Developers could issue a pull request but the PR approvals and commits to master should only be accepted by a cluster administrator or people with similar responsibility.

After that, we will create a service account for each of the namespaces. These are the accounts that will be used later when we will define Azure DevOps environment for each team.

Now the pipeline for this repository essentially applies all the manifests (both for namespaces and services accounts) to the cluster.

trigger:
- master
stages:
- stage: Build
  displayName: Provision namespace and service accounts
  jobs:  
  - job: Build
    displayName: Update namespace and service accounts
    steps:
      <… omitted irrelevant codes …>
      - bash: |
          kubectl apply -f ./namespaces 
        displayName: 'Update namespaces'
      - bash: |
          kubectl apply -f ./ServiceAccounts 
        displayName: 'Update service accounts'   
      - bash: |
          dotnet ado-env-gen.dll
        displayName: 'Provision Azure DevOps Environments'       

At this point, we have service account configured for each namespace that we will use to create the environment, endpoints etc. You might notice that I have created some label for each service account (i.e. purpose=ado-automation), this is to tag along the Azure DevOps Project name to a service account. This will come handy when we will provision environments.

The last task that runs a .net core console app (i.e. ado-env-gen.dll) – which I will described in detail later in this article.

Provisioning Environment in Azure DevOps

NOTE: Provisioning environment via REST api currently is undocumented and might change in coming future – beware of that.

It takes multiple steps to create an Environment to Azure DevOps. The steps are below:

  1. Create a Service endpoint with Kubernetes Service Account
  2. Create an empty environment (with no resources yet)
  3. Connect the service endpoint to the environment as Resource

I’ve used .net (C#) for this, but any REST client technology could do that.

Creating Service Endpoint

Following method creates a service endpoint in Azure DevOps that uses a Service Account scoped to a given namespace.

        public async Task<Endpoint> CreateKubernetesEndpointAsync(
            Guid projectId, string projectName,
            string endpointName, string endpointDescription,
            string clusterApiUri,
            string serviceAccountCertificate, string apiToken)
        {
            return await GetAzureDevOpsDefaultUri()
                .PostRestAsync<Endpoint>(
                $"{projectName}/_apis/serviceendpoint/endpoints?api-version=6.0-preview.4",
                new
                {
                    authorization = new
                    {
                        parameters = new
                        {
                            serviceAccountCertificate,
                            isCreatedFromSecretYaml = true,
                            apitoken = apiToken
                        },
                        scheme = "Token"
                    },
                    data = new
                    {
                        authorizationType = "ServiceAccount"
                    },
                    name = endpointName,
                    owner = "library",
                    type = "kubernetes",
                    url = clusterApiUri,
                    description = endpointDescription,
                    serviceEndpointProjectReferences = new List<Object>
                    {
                        new
                        {
                            description = endpointDescription,
                            name =  endpointName,
                            projectReference = new
                            {
                                id =  projectId,
                                name =  projectName
                            }
                        }
                    }
                }, await GetBearerTokenAsync());
        }

We will find out how to invoke this method in a moment. Before that, Step 2, let’s create the empty environment now.

Creating Environment in Azure DevOps

        public async Task<PipelineEnvironment> CreateEnvironmentAsync(
            string project, string envName, string envDesc)
        {
            var env = await GetAzureDevOpsDefaultUri()
                .PostRestAsync<PipelineEnvironment>(
                $"{project}/_apis/distributedtask/environments?api-version=5.1-preview.1",
                new
                {
                    name = envName,
                    description = envDesc
                },
                await GetBearerTokenAsync());

            return env;
        }

Now we have environment, but it still empty. We need to add a resource into it and that would be the Service Endpoint – so the environment comes to life.

        public async Task<string> CreateKubernetesResourceAsync(
            string projectName, long environmentId, Guid endpointId,
            string kubernetesNamespace, string kubernetesClusterName)
        {
            var link = await GetAzureDevOpsDefaultUri()
                            .PostRestAsync(
                            $"{projectName}/_apis/distributedtask/environments/{environmentId}/providers/kubernetes?api-version=5.0-preview.1",
                            new
                            {
                                name = kubernetesNamespace,
                                @namespace = kubernetesNamespace,
                                clusterName = kubernetesClusterName,
                                serviceEndpointId = endpointId
                            },
                            await GetBearerTokenAsync());
            return link;
        }

Of course, environment needs to have Approval policies configure. The following method configures a Azure DevOps group as Approver to the environment. Hence any pipeline that reference this environment will be paused and wait for approval from one of the members of the group.

        public async Task<string> CreateApprovalPolicyAsync(
            string projectName, Guid groupId, long envId, 
            string instruction = "Please approve the Deployment")
        {
            var response = await GetAzureDevOpsDefaultUri()
                .PostRestAsync(
                $"{projectName}/_apis/pipelines/checks/configurations?api-version=5.2-preview.1",
                new
                {
                    timeout = 43200,
                    type = new
                    {                                   
                        name = "Approval"
                    },
                    settings = new
                    {
                        executionOrder = 1,
                        instructions = instruction,
                        blockedApprovers = new List<object> { },
                        minRequiredApprovers = 0,
                        requesterCannotBeApprover = false,
                        approvers = new List<object> { new { id = groupId } }
                    },
                    resource = new
                    {
                        type = "environment",
                        id = envId.ToString()
                    }
                }, await GetBearerTokenAsync());
            return response;
        }

So far so good. But we need to stich all these together. Before we do so, one last item needs attention. We would want to create a Service connection to the Azure container registry so the teams can push/pull images to that. And we would do that using Service Principals designated to the teams – instead of the Admin keys of ACR.

Creating Container Registry connection

The following snippet allows us provisioning Service Connection to Azure Container Registry with Service principals – which can have fine grained RBAC roles (i.e. ACRPush or ACRPull etc.) that makes sense for the team.

        public async Task<string> CreateAcrConnectionAsync(
            string projectName, string acrName, string name, string description,
            string subscriptionId, string subscriptionName, string resourceGroup,
            string clientId, string secret, string tenantId)
        {
            var response = await GetAzureDevOpsDefaultUri()
                .PostRestAsync(
                $"{projectName}/_apis/serviceendpoint/endpoints?api-version=5.1-preview.2",
                new
                {
                    name,
                    description,
                    type = "dockerregistry",
                    url = $"https://{acrName}.azurecr.io",
                    isShared = false,
                    owner = "library",
                    data = new
                    {
                        registryId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.ContainerRegistry/registries/{acrName}",
                        registrytype = "ACR",
                        subscriptionId,
                        subscriptionName
                    },
                    authorization = new
                    {
                        scheme = "ServicePrincipal",
                        parameters = new
                        {
                            loginServer = $"{acrName}.azurecr.io",
                            servicePrincipalId = clientId,
                            tenantId,
                            serviceprincipalkey = secret
                        }
                    }
                },
                await GetBearerTokenAsync());
            return response;
        }

We came pretty close to a wrap. We’ll stitch all the methods above together. Plan is to create a simple console application will fix everything (using the above methods). Here’s the pseudo steps:

  1. Find all Service Account created for this purpose
  2. For each Service Account: determining the correct Team Project and
    • Create Service Endpoint with the Account
    • Create Environment
    • Connect Service Endpoint to Environment (adding resource)
    • Configure Approval policies
    • Create Azure Container Registry connection

The first step needs to communicate to the cluster – obviously. I have used the official .net client for Kubernetes for that.

Bringing all together

All the above methods are invoked from a simple C# console application. Below is the relevant part of the main method that brings all the above together:

        private static async Task Main(string [] args)
        {
            var clusterApiUrl = Environment.GetEnvironmentVariable("AKS_URI");
            var adoUrl = Environment.GetEnvironmentVariable("AZDO_ORG_SERVICE_URL");
            var pat = Environment.GetEnvironmentVariable("AZDO_PERSONAL_ACCESS_TOKEN");
            var adoClient = new AdoClient(adoUrl, pat);
            var groups = await adoClient.ListGroupsAsync();

            var config = KubernetesClientConfiguration.BuildConfigFromConfigFile();
            var client = new Kubernetes(config);

We started by collecting some secret and configuration data – all from environment variables – so we can run this console as part of the pipeline task and use pipeline variables at ease.

        var accounts = await client
            .ListServiceAccountForAllNamespacesAsync(labelSelector: "purpose=ado-automation");

This gets us the list of all the service accounts we have provisioned specially for this purpose (filtered using the labels).

            foreach (var account in accounts.Items)
            {
                var project = await GetProjectAsync(account.Metadata.Labels["project"], adoClient);
                var secretName = account.Secrets[0].Name;
                var secret = await client
                    .ReadNamespacedSecretAsync(secretName, account.Metadata.NamespaceProperty);

We are iterating all the accounts and retrieving their secrets from the cluster. Next step, creating the environment with these secrets.

                var endpoint = await adoClient.CreateKubernetesEndpointAsync(
                    project.Id,
                    project.Name,
                    $"Kubernetes-Cluster-Endpoint-{account.Metadata.NamespaceProperty}",
                    $"Service endpoint to the namespace {account.Metadata.NamespaceProperty}",
                    clusterApiUrl,
                    Convert.ToBase64String(secret.Data["ca.crt"]),
                    Convert.ToBase64String(secret.Data["token"]));

                var environment = await adoClient.CreateEnvironmentAsync(project.Name,
                    $"Kubernetes-Environment-{account.Metadata.NamespaceProperty}",
                    $"Environment scoped to the namespace {account.Metadata.NamespaceProperty}");

                await adoClient.CreateKubernetesResourceAsync(project.Name, 
                    environment.Id, endpoint.Id,
                    account.Metadata.NamespaceProperty,
                    account.Metadata.ClusterName);

That will give us the environment – correctly configured with the appropriate Service Accounts. Let’s set up the approval policy now:

                var group = groups.FirstOrDefault(g => g.DisplayName
                    .Equals($"[{project.Name}]\\Release Administrators", StringComparison.OrdinalIgnoreCase));
                await adoClient.CreateApprovalPolicyAsync(project.Name, group.OriginId, environment.Id);

We are taking a designated project group “Release Administrators” and set them as approves.

            await adoClient.CreateAcrConnectionAsync(project.Name, 
                Environment.GetEnvironmentVariable("ACRName"), 
                $"ACR-Connection", "The connection to the ACR",
                Environment.GetEnvironmentVariable("SubId"),
                Environment.GetEnvironmentVariable("SubName"),
                Environment.GetEnvironmentVariable("ResourceGroup"),
                Environment.GetEnvironmentVariable("ClientId"), 
                Environment.GetEnvironmentVariable("Secret"),
                Environment.GetEnvironmentVariable("TenantId"));

Lastly created the ACR connection as well.

The entire project is in GitHub – in case you want to have a read!

Verify everything

We have got our orchestration completed. Every time we add a new team, we create one manifest for their namespace and Service account and create a PR to the repository described above. A cluster admin approves the PR and a pipeline gets kicked off.

The pipeline ensures:

  1. All the namespaces and service accounts are created
  2. An environment with the appropriate service accounts are created in the correct team project.

Now a team can create their own pipeline in their repository – referring to the environment. Voila, all starts working nice. All they need is to refer the name of the environment that’s provisioned for their team (for instance “team-1”), as following example:

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  jobs:
  - deployment: Deploy
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')))
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'Kubernetes-Cluster-Environment.team-1'
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: kube-manifests
          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              action: deploy
              manifests: |
                $(Pipeline.Workspace)/kube-manifests/all-template.yaml

Now the multi-stage pipeline knows how to talk to the correct namespace in AKS with approval awaiting.

Conclusion

This might appear an overkill for small-scale projects, as it involves quite some overhead of development and maintenance. However, on multiple occasions (especially within large enterprises), I have experienced the need for orchestrations via REST API to onboard teams in Azure DevOps, bootstrapping configurations across multiple teams’ projects etc. If you’re on the same boat, this article might be an interesting read for you!

Thanks for reading!

Azure AD App via ARM Template Deployment Scripts

Background

ARM templates offer a great way to define resources and deploy them. However, ARM templates didn’t have any support to invoke or run scripts. If we wanted to carry out some operations as part of the deployment (Azure AD app registrations, Certificate generations, copy data to/from another system etc.) we had to create pre or post deployment scripts (using Azure PowerShell or Azure CLI). Microsoft recently announced the preview of Deployment Scripts (new resource type Microsoft.Resources/deploymentScripts) – which brings a way to run a script as part of ARM template deployment.

I have few web apps using Open ID connect for user authentication and they’re running as Azure App services. I always wanted to automate (preferably in a declarative and idempotent way) the required app registrations in Azure AD and deploy them together with the ARM templates of web apps.

Since we now have deployment script capability, I wanted to leverage it for Azure AD app registrations. In this article I will share my experience doing exactly that.

What are deployment scripts?

Deployment scripts allows running custom scripts (can be either Azure PowerShell or Azure CLI) as part of an ARM template deployment. It can be used to perform custom steps that can’t be done by ARM templates.


A simple deployment template that runs a bash command (echo) looks like below:

Figure: Simple example of Deployment Scripts

Microsoft described the benefits of deployment scripts as following:

– Easy to code, use, and debug. You can develop deployment scripts in your favorite development environments. The scripts can be embedded in templates or in external script files.


– You can specify the script language and platform. Currently, Azure PowerShell and Azure CLI deployment scripts on the Linux environment are supported.


– Allow specifying the identities that are used to execute the scripts. Currently, only Azure user-assigned managed identity is supported.


– Allow passing command-line arguments to the script.
Can specify script outputs and pass them back to the deployment.

Source

Registering Azure AD app

We can write a small script (with Azure CLI) like above sample, that registers the Azure AD app – that’s quite straightforward. However, first we need to address the Identity aspect, what account would run the script and how app-registration permission can be granted to that account. The answer is using Managed Identity.

User Assigned Managed Identity

Managed identities for Azure resources provide Azure services with a managed identity in Azure Active Directory. We can use this identity to authenticate to services that support Azure AD authentication, without needing credentials in your code. There are two types of Managed Identity, System assigned and User Assigned.

Deployment Scripts currently supports User Assigned Identities only, hence, we need to create a User Assigned Managed Identity that would run the CLI script. This identity is used to execute deployment scripts. We would also grant Azure AD app registration permissions to this identity. Creating User Assigned Identity is straightforward and the steps are nicely described here.

Figure: User Assigned Managed Identity in Azure Portal

Next to that, we will have to grant permissions to the identity. Following PowerShell script grants the required permissions to the Managed Identity.

Figure: Grant permissions (Click to Open in window to copy)

ARM template

We will now write the ARM template that will leverage the deployment scripts to register our app in Azure AD.

Figure: Deployment Script (Click to Open in window to copy)

I wouldn’t explain each of the settings/config options in here. Most important part here is the scriptContent property – which can have a string value of any scripts (PowerShell or Bash). You can also point to an external script file instead of embedded script.

Another important property is cleanupPreference. It specifies the preference of cleaning up deployment resources when the script execution gets in a terminal state. Default setting is Always, which means deleting the resources despite the terminal state (Succeeded, Failed, Canceled).

You can find more details on each of the configuration properties for Deployment Script in this document.

I have used some variable references that are defined in the same template json file.

Figure: Variables (Click to open new window to copy)

Notice here the cliArg variable. This would be the argument that we are passing as inputs to our CLI/bash script. The catch here is, the arguments need to be separated by white-spaces.

Finally, we would love to grab the newly registered app id and configure an entry into the App Settings in our web app – so the web app Open ID authentication can work right after the deployment.

Figure: Variables (Click to open new window to copy)

At this point we will deploy the template and after the deployment completed, we will see the app has been registered in Azure AD:

Figure: Azure AD App

Also, we can verify that the newly created App ID is nicely configured into the web app’s app-settings.

Figure: App settings configured

That’s all there is to it!

I haven’t defined any API permission scopes for the app registrations in this example, however, having the Azure CLI script in place, defining further API scopes are trivial.

How it worked?

If we login to the Azure Portal we will see the following:

Figure: Azure Portal resources

We see a new resource of type Deployment Script besides our Web App (and it’s Service Plan) that is obvious. However, we also see Container Instance and a Storage Account. Where they came from?

Well, Azure RM deployment created them while deploying the Deployment scripts. The storage account and a container instance, are created in the same resource group for script execution and troubleshooting. These resources are usually deleted by the script service when the script execution gets in a terminal state. Important to know, we are billed for the resources until the resources are deleted.

The container instance runs a Docker image as a Sandbox for our Deployment Script. You can see the image name form the portal that Microsoft is using for execution. This can come handy to try out the script locally – for development purposes.

Conclusion

I have a mixed feeling about the deployment script in ARM templates. It obviously has some benefits. But this shouldn’t replace all pre or post deployment script. Because sometimes it might be cleaner and easier to create a pre- or post-script task in continuous delivery pipeline than composing all in ARM templates.

Key Vault as backing store of Azure Functions

If you have used Azure function, you probably are aware that Azure Functions leverages a Storage Account underneath to support the file storage (where the function app code resides as Azure File share) and also as a backing store to keep Functions Keys (the secrets that are used in Function invocations).

Containers

Figure: Storage Account containers – “azure-webjobs-secrets”

If you look inside the container there are files with following contents:

secrets-in-storages

Figure: These JSON files has the function keys

host-json

Figure: Encrypted master keys  and other function keys

I have been in a conversation where; it was not appreciated to see the keys stored in the storage account. The security and governance team was seeking for a better place to keep these keys. Where secrets can be further restricted from developer access.

Of course, we can create a VNET around the storage accountand use private link but that has some other consequence as the content (functions implementations artifacts) stored also into the storage account. Configuring two separate storage account can address this better, however, this can make the setup complicated than it has to be.
A better option could be to store this keys into a Key Vault as backing store – which is a great feature of Azure functions, but I’ve found few people are aware of this due to lack of documentations. In this article I will show you how to move these secrets to a Key Vault.

To do so, we need to configure few Application Settings into the Function App. They are given below:

App Settings name Value
AzureWebJobsSecretStorageType keyvault
AzureWebJobsSecretStorageKeyVaultName <Key Vault Name>
AzureWebJobsSecretStorageKeyVaultConnectionString <Connection String or Leave it empty with Managed Identity configured on Azure Functions>

Once you have configured the above settings, you need to enable Managed Identity on your Azure Function. You will have to accomplish that in Identity section under platform features tab. That is a much better option in my opinion as we don’t need to maintain any more secrets to talk to Key vault securely. Go ahead and turn the system identity toggle on. This will create a service principal with the same name as Azure Function application you have.

managedidentity

Figure: Enabling system assigned managed identity on Function app
Next step is to add a rule to the key vault’s access policies for the service principal created in earlier step.

access policyu

Figure: Key vault Access policy
That’s it, hit your function app now and you will see the keys are stored inside the Key vault. You can safely delete the container from the storage account now.

secretsinkeyvault

Figure: Secrets are stored in Key Vault

Hope this will save time when you are concerned to keep the keys in storage account.
The Azure Function is open sourced and is in GitHub. You can have a look into the sources and see other interesting ideas that you may play with.

Access Control management via REST API – Azure Data Lake Gen 2

Background

A while ago, I have built an web-based self-service portal that facilitated multiple teams in the organisation, setting up their Access Control (ACLs) for corresponding data lake folders.

The portal application was targeting Azure Data Lake Gen 1. Recently I wanted to achieve the same but on Azure Data Lake Gen 2. At the time of writing this post, there’s no official NuGet package for ACL management targeting Data Lake Gen 2. One must rely on REST API only.

Read about known issues and limitations of Azure Data Lake Storage Gen 2

Further more, the REST API documentations do not provide example snippets like many other Azure resources. Therefore, it takes time to demystify the REST APIs to manipulate ACLs. Good new is, I have done that for you and will share a straight-forward C# class that wraps the details and issues correct REST API calls to a Data Lake Store Gen 2.

About Azure Data Lake Store Gen 2

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics. Data Lake Storage Gen2 is significantly different from it’s earlier version known as Azure Data Lake Storage Gen1, Gen2 is entirely built on Azure Blob storage.

Data Lake Storage Gen2 is the result of converging the capabilities of two existing Azure storage services, Azure Blob storage and Azure Data Lake Storage Gen1. Gen1 Features such as file system semantics, directory, and file level security and scale are combined with low-cost, tiered storage, high availability/disaster recovery capabilities from Azure Blob storage.

Let’s get started!

Create a Service Principal

First we would need a service principal. We will use this principal to authenticate to Azure Active Directory (using OAuth 2.0 protocol) in order to authorize our REST calls. We will use Azure CLI to do that.

az ad sp create-for-rbac --name ServicePrincipalName
Add required permissions

Now you need to grant permission for your application to access Azure Storage.

  • Click on the application Settings
  • Click on Required permissions
  • Click on Add
  • Click Select API
  • Filter on Azure Storage
  • Click on Azure Storage
  • Click Select
  • Click the checkbox next to Access Azure Storage
  • Click Select
  • Click Done

App

Now we have Client ID, Client Secret and Tenant ID (take it from the Properties tab of Azure Active Directory – listed as Directory ID).

Access Token from Azure Active Directory

Let’s write some C# code to get an Access Token from Azure Active Directory:

public class TokenProvider
{
private readonly string tenantId;
private readonly string clientId;
private readonly string secret;
private readonly string scopeUri;
private const string IdentityEndpoint = "https://login.microsoftonline.com";
private const string DEFAULT_SCOPE = "https://management.azure.com/";
private const string MEDIATYPE = "application/x-www-form-urlencoded";
public OAuthTokenProvider(string tenantId, string clientId, string secret, string scopeUri = DEFAULT_SCOPE)
{
this.tenantId = tenantId;
this.clientId = WebUtility.UrlEncode(clientId);
this.secret = WebUtility.UrlEncode(secret);
this.scopeUri = WebUtility.UrlEncode(scopeUri);
}
public async Task<Token> GetAccessTokenV2EndpointAsync()
{
var url = $"{IdentityEndpoint}/{this.tenantId}/oauth2/v2.0/token";
var Http = Statics.Http;
Http.DefaultRequestHeaders.Accept.Clear();
Http.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(MEDIATYPE));
var body = $"grant_type=client_credentials&client_id={clientId}&client_secret={secret}&scope={scopeUri}";
var response = await Http.PostAsync(url, new StringContent(body, Encoding.UTF8, MEDIATYPE));
if (response.IsSuccessStatusCode)
{
var tokenResponse = await response.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<Token>(tokenResponse);
}
return default(Token);
}
public class Token
{
public string access_token { get; set; }
public string token_type { get; set; }
public int expires_in { get; set; }
public int ext_expires_in { get; set; }
}
}

view raw
token-provider.cs
hosted with ❤ by GitHub

Creating ADLS Gen 2 REST client

Once we have the token provider, we can jump in implementing the REST client for Azure Data Lake.

public class FileSystemApi
{
private readonly string storageAccountName;
private readonly OAuthTokenProvider tokenProvider;
private readonly Uri baseUri;
private const string ACK_HEADER_NAME = "x-ms-acl";
private const string API_VERSION_HEADER_NAME = "x-ms-version";
private const string API_VERSION_HEADER_VALUE = "2018-11-09";
private int Timeout = 100;
public FileSystemApi(string storageAccountName, OAuthTokenProvider tokenProvider)
{
this.storageAccountName = storageAccountName;
this.tokenProvider = tokenProvider;
this.baseUri = new Uri($"https://{this.storageAccountName}.dfs.core.windows.net");
}

view raw
file-system.cs
hosted with ❤ by GitHub

Data Lake  ACLs and POSIX permissions

The security model for Data Lake Gen2 supports ACL and POSIX permissions along with some extra granularity specific to Data Lake Storage Gen2. Settings may be configured through Storage Explorer or through frameworks like Hive and Spark. We will do that via REST API in this post.

There are two kinds of access control lists (ACLs), Access ACLs and Default ACLs.

  • Access ACLs: These control access to an object. Files and folders both have Access ACLs.
  • Default ACLs: A “template” of ACLs associated with a folder that determine the Access ACLs for any child items that are created under that folder. Files do not have Default ACLs.

Here’s the table of allowed grant types:

acl1

While we define ACLs we need to use a short form of these grant types. Microsoft Document explained these short form in below table:

posix

However, in our code we would also simplify the POSIX ACL notations by using some supporting classes as below. That way REST client consumers do not need to spend time building the short form of their aimed grant criteria’s.

public enum AclType
{
User,
Group,
Other,
Mask
}
public enum AclScope
{
Access,
Default
}
[FlagsAttribute]
public enum GrantType : short
{
None = 0,
Read = 1,
Write = 2,
Execute = 4
};
public class AclEntry
{
public AclEntry(AclScope scope, AclType type, string upnOrObjectId, GrantType grant)
{
Scope = scope;
AclType = type;
UpnOrObjectId = upnOrObjectId;
Grant = grant;
}
public AclScope Scope { get; private set; }
public AclType AclType { get; private set; }
public string UpnOrObjectId { get; private set; }
public GrantType Grant { get; private set; }
public string GetGrantPosixFormat()
{
return $"{(this.Grant.HasFlag(GrantType.Read) ? 'r' : '-')}{(this.Grant.HasFlag(GrantType.Write) ? 'w' : '-')}{(this.Grant.HasFlag(GrantType.Execute) ? 'x' : '-')}";
}
public override string ToString()
{
return $"{(this.Scope == AclScope.Default ? "default:" : string.Empty)}{this.AclType.ToString().ToLowerInvariant()}:{this.UpnOrObjectId}:{GetGrantPosixFormat()}";
}
}

view raw
acl-supports.cs
hosted with ❤ by GitHub

Now we can create methods to perform different REST calls, let’s start by creating a file system.

public async Task<bool> CreateFileSystemAsync(
string fileSystemName)
{
var tokenInfo = await tokenProvider.GetAccessTokenV2EndpointAsync();
var jsonContent = new StringContent(string.Empty);
var headers = Statics.Http.DefaultRequestHeaders;
headers.Clear();
headers.Add("Authorization", $"Bearer {tokenInfo.access_token}");
headers.Add(API_VERSION_HEADER_NAME, API_VERSION_HEADER_VALUE);
var response = await Statics.Http.PutAsync($"{baseUri}{WebUtility.UrlEncode(fileSystemName)}?resource=filesystem", jsonContent);
return response.IsSuccessStatusCode;
}

Here we are retrieving a Access Token and then issuing a REST call to Azure Data Lake Storage Gen 2 API to create a new file system. Next, we will create a folder and file in it and then set some Access Control to them.

Let’s create the folder:

public async Task<bool> CreateDirectoryAsync(string fileSystemName, string fullPath)
{
var tokenInfo = await tokenProvider.GetAccessTokenV2EndpointAsync();
var jsonContent = new StringContent(string.Empty);
var headers = Statics.Http.DefaultRequestHeaders;
headers.Clear();
headers.Add("Authorization", $"Bearer {tokenInfo.access_token}");
headers.Add(API_VERSION_HEADER_NAME, API_VERSION_HEADER_VALUE);
var response = await Statics.Http.PutAsync($"{baseUri}{WebUtility.UrlEncode(fileSystemName)}{fullPath}?resource=directory", jsonContent);
return response.IsSuccessStatusCode;
}

view raw
CreateDirectory.cs
hosted with ❤ by GitHub

And creating file in it. Now, file creation (ingestion in Data Lake) is not that straight forward, at least, one can’t do that by a single call. We would have to first create an empty file, then we can write some content in it. We can also append content to an existing file. Finally, we would require to flush the buffer so the new content gets persisted.

Let’s do that, first we will see how to create an empty file:

public async Task<bool> CreateEmptyFileAsync(string fileSystemName, string path, string fileName)
{
var tokenInfo = await tokenProvider.GetAccessTokenV2EndpointAsync();
var jsonContent = new StringContent(string.Empty);
var headers = Statics.Http.DefaultRequestHeaders;
headers.Clear();
headers.Add("Authorization", $"Bearer {tokenInfo.access_token}");
headers.Add(API_VERSION_HEADER_NAME, API_VERSION_HEADER_VALUE);
var response = await Statics.Http.PutAsync($"{baseUri}{WebUtility.UrlEncode(fileSystemName)}{path}{fileName}?resource=file", jsonContent);
return response.IsSuccessStatusCode;
}

view raw
CreateEmptyFile.cs
hosted with ❤ by GitHub

The above snippet will create an empty file, now we will read all content from a local file (from PC) and write them into the empty file in Azure Data Lake that we just created.

public async Task<bool> CreateFileAsync(string filesystem, string path,
string fileName, Stream stream)
{
var operationResult = await this.CreateEmptyFileAsync(filesystem, path, fileName);
if (operationResult)
{
var tokenInfo = await tokenProvider.GetAccessTokenV2EndpointAsync();
var headers = Statics.Http.DefaultRequestHeaders;
headers.Clear();
headers.Add("Authorization", $"Bearer {tokenInfo.access_token}");
headers.Add(API_VERSION_HEADER_NAME, API_VERSION_HEADER_VALUE);
using (var streamContent = new StreamContent(stream))
{
var resourceUrl = $"{baseUri}{filesystem}{path}{fileName}?action=append&timeout={this.Timeout}&position=0";
var msg = new HttpRequestMessage(new HttpMethod("PATCH"), resourceUrl);
msg.Content = streamContent;
var response = await Statics.Http.SendAsync(msg);
//flush the buffer to commit the file
var flushUrl = $"{baseUri}{filesystem}{path}{fileName}?action=flush&timeout={this.Timeout}&position={msg.Content.Headers.ContentLength}";
var flushMsg = new HttpRequestMessage(new HttpMethod("PATCH"), flushUrl);
response = await Statics.Http.SendAsync(flushMsg);
return response.IsSuccessStatusCode;
}
}
return false;
}

view raw
CreateFile.cs
hosted with ❤ by GitHub

Right! Now time to set Access control to the directory or files inside a directory. Here’s the method that we will use to do that.

public async Task<bool> SetAccessControlAsync(string fileSystemName, string path, AclEntry[] acls)
{
var targetPath = $"{WebUtility.UrlEncode(fileSystemName)}{path}";
var tokenInfo = await tokenProvider.GetAccessTokenV2EndpointAsync();
var jsonContent = new StringContent(string.Empty);
var headers = Statics.Http.DefaultRequestHeaders;
headers.Clear();
headers.Add("Authorization", $"Bearer {tokenInfo.access_token}");
headers.Add(API_VERSION_HEADER_NAME, API_VERSION_HEADER_VALUE);
headers.Add(ACK_HEADER_NAME, string.Join(',', acls.Select(a => a.ToString()).ToArray()));
var response = await Statics.Http.PatchAsync($"{baseUri}{targetPath}?action=setAccessControl", jsonContent);
return response.IsSuccessStatusCode;
}

view raw
SetAcl.cs
hosted with ❤ by GitHub

The entire File system REST API class can be found here. Here’s an example how we can use this methods from a console application.

var tokenProvider = new OAuthTokenProvider(tenantId, clientId, secret, scope);
var hdfs = new FileSystemApi(storageAccountName, tokenProvider);
var response = hdfs.CreateFileSystemAsync(fileSystemName).Result;
hdfs.CreateDirectoryAsync(fileSystemName, "/demo").Wait();
hdfs.CreateEmptyFileAsync(fileSystemName, "/demo/", "example.txt").Wait();
var stream = new FileStream(@"C:\temp.txt", FileMode.Open, FileAccess.Read);
hdfs.CreateFileAsync(fileSystemName, "/demo/", "mytest.txt", stream).Wait();
var acls = new AclEntry[]
{
new AclEntry(
AclScope.Access,
AclType.Group,
"2dec2374-3c51-4743-b247-ad6f80ce4f0b",
(GrantType.Read | GrantType.Execute)),
new AclEntry(
AclScope.Access,
AclType.Group,
"62049695-0418-428e-a5e4-64600d6d68d8",
(GrantType.Read | GrantType.Write | GrantType.Execute)),
new AclEntry(
AclScope.Default,
AclType.Group,
"62049695-0418-428e-a5e4-64600d6d68d8",
(GrantType.Read | GrantType.Write | GrantType.Execute))
};
hdfs.SetAccessControlAsync(fileSystemName, "/", acls).Wait();

view raw
Console.cs
hosted with ❤ by GitHub

Conclusion

Until, there’s an Official Client Package released, if you’re into Azure Data Lake Store Gen 2 and wondering how to accomplish these REST calls – I hope this post helped you to move further!

Thanks for reading.

 

Linkerd in Azure Kubernetes Service cluster

In this article I would document my journey on setting up Linkerd Service Mesh on Azure Kubernetes service.

Background

I have a tiny Kubernetes cluster. I run some workload there, some are useful, others are just try-out, fun stuffs. I have few services that need to talk to each other. I do not have a lot of traffic to be honest, but I sometimes curiously run Apache ab to simulate load and see how my services perform under stress. Until very recently I was using a messaging (basically a pub-sub) pattern to create reactive service-to-service communication. Which works great, but often comes with a latency. I can only imagine, if I were to run these service to service communication for a mission critical high-traffic performance-driven scenario (an online game for instance), this model won’t fly well. There comes the need for a service-to-service communication pattern in cluster.

What’s big deal? We can have REST calls between services, even can implement gRPC for that matter. The issue is things behaves different at scale. When many services talks to many others, nodes fail in between, network address of PODs changes, new PODs show up, some goes down, figuring out where the service sits becomes quite a challenging task.

Then Kubernetes comes to rescue, Kubernetes provides “service”, that gives us service discovery out of the box. Which is awesome. Not all issues disappeared though. Services in a cluster need fault-tolerances, traceability and most importantly, “observability”.  Circuit-breakers, retry-logics etc. implementing them for each service is again a challenge. This is exactly the Service Mesh addresses.

Service mesh

From thoughtworks radar:

Service mesh is an approach to operating a secure, fast and reliable microservices ecosystem. It has been an important steppingstone in making it easier to adopt microservices at scale. It offers discovery, security, tracing, monitoring and failure handling. It provides these cross-functional capabilities without the need for a shared asset such as an API gateway or baking libraries into each service. A typical implementation involves lightweight reverse-proxy processes, aka sidecars, deployed alongside each service process in a separate container. Sidecars intercept the inbound and outbound traffic of each service and provide cross-functional capabilities mentioned above.

Some of us might remember Aspect Oriented programming (AOP) – where we used to separate cross cutting concerns from our core-business-concerns. Service mesh is no different. They isolate (in a separate container) these networking and fault-tolerance concerns from the core-capabilities (also running in container).

Linkerd

There are quite several service mesh solutions out there – all suitable to run in Kubernetes. I have used earlier Envoy and Istio. They work great in Kubernetes as well as VM hosted clusters. However, I must admit, I developed a preference for Linkerd since I discovered it. Let’s briefly look at how Linkerd works. Imagine the following two services, Service A and Service B. Service A talks to Service B.

service-2-service

When Linkerd installed, it works like an interceptor between all the communication between services. Linkerd uses sidecar pattern to proxy the communication by updating the KubeProxy IP Table.

Linkerd-architecture.png

Linkerd implants two sidecar containers in our PODs. The init container configures the IP table so the incoming and outgoing TCP traffics flow through the Linkerd Proxy container. The proxy container is the data plane that does the actual interception and all the other fault-tolerance goodies.

Primary reason behind my Linkerd preferences are performance and simplicity. Ivan Sim has done performance benchmarking with Linkerd and Istio:

Both the Linkerd2-meshed setup and Istio-meshed setup experienced higher latency and lower throughput, when compared with the baseline setup. The latency incurred in the Istio-meshed setup was higher than that observed in the Linkerd2-meshed setup. The Linkerd2-meshed setup was able to handle higher HTTP and GRPC ping throughput than the Istio-meshed setup.

Cluster provision

Spinning up AKS is easy as pie these days. We can use Azure Resource Manager Template or Terraform for that. I have used Terraform to generate that.

resource "azurerm_resource_group" "cloudoven" {
name = "cloudoven"
location = "West Europe"
}
resource "azurerm_kubernetes_cluster" "cloudovenaks" {
name = "cloudovenaks"
location = "${azurerm_resource_group.cloudoven.location}"
resource_group_name = "${azurerm_resource_group.cloudoven.name}"
dns_prefix = "cloudovenaks"
agent_pool_profile {
name = "default"
count = 1
vm_size = "Standard_D1_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
agent_pool_profile {
name = "pool2"
count = 1
vm_size = "Standard_D2_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "98e758f8r-f734-034a-ac98-0404c500e010"
client_secret = "Jk==3djk(efd31kla934-=="
}
tags = {
Environment = "Production"
}
}
output "client_certificate" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config.0.client_certificate}"
}
output "kube_config" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config_raw}"
}

view raw
Kuberentes-iac
hosted with ❤ by GitHub

Service deployment

This is going to take few minutes and then we have a cluster. We will use the canonical emojivoto app (“buoyantio/emojivoto-emoji-svc:v8”) to test our Linkerd installation. Here’s the Kubernetes manifest file for that.

apiVersion: v1
kind: Namespace
metadata:
name: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: emoji
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: voting
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: web
namespace: emojivoto
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: emoji
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: emoji-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: emoji-svc
spec:
serviceAccountName: emoji
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-emoji-svc:v8
name: emoji-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: emoji-svc
namespace: emojivoto
spec:
selector:
app: emoji-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: voting
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: voting-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: voting-svc
spec:
serviceAccountName: voting
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-voting-svc:v8
name: voting-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: voting-svc
namespace: emojivoto
spec:
selector:
app: voting-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: web
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: web-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: web-svc
spec:
serviceAccountName: web
containers:
env:
name: WEB_PORT
value: "80"
name: EMOJISVC_HOST
value: emoji-svc.emojivoto:8080
name: VOTINGSVC_HOST
value: voting-svc.emojivoto:8080
name: INDEX_BUNDLE
value: dist/index_bundle.js
image: buoyantio/emojivoto-web:v8
name: web-svc
ports:
containerPort: 80
name: http
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: web-svc
namespace: emojivoto
spec:
type: LoadBalancer
selector:
app: web-svc
ports:
name: http
port: 80
targetPort: 80
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: vote-bot
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: vote-bot
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vote-bot
spec:
containers:
command:
emojivoto-vote-bot
env:
name: WEB_HOST
value: web-svc.emojivoto:80
image: buoyantio/emojivoto-web:v8
name: vote-bot
resources:
requests:
cpu: 10m
status: {}

view raw
emoji-manifest.yml
hosted with ❤ by GitHub

With this IaC – we can run Terraform apply to provision our AKS cluster in Azure.

Azure Pipeline

Let’s create a pipeline for the service deployment. The easiest way to do that is to create a service connection to our AKS cluster. We go to the project settings in Azure DevOps project, pick Service connections and create a new service connection of type “Kubernetes connection”.

Azure DevOps connection

Installing Linkerd

We will create a pipeline that installs Linkerd into the AKS cluster. Azure Pipeline now offers “pipeline-as-code” – which is just an YAML file that describes the steps need to be performed when the pipeline is triggered. We will use the following pipeline-as-code:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl get'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: get
arguments: nodes
script: |
curl -sL https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
linkerd version
linkerd check –pre
linkerd install | kubectl apply -f –
linkerd check
displayName: 'Linkerd – Installation'

We can at this point trigger the pipeline to install Linkerd into the AKS cluster.

Linkerd installation (2)

Deployment of PODs and services

Let’s create another pipeline as code that deploys all the services and deployment resources to AKS using the following Kubernetes manifest file:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml

In Azure Portal we can already see our services running:

Azure KS

Also in Kubernetes Dashboard:

Kub1

We have got our services running – but they are not really affected by Linkerd yet. We will add another step into the build pipeline to tell Linkerd to do its magic.

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml
script: 'src/services/emojivoto/all.yml | linkerd inject – | kubectl apply -f –'
displayName: 'Inject Linkerd'

Next thing, we trigger the pipeline and put some traffic into the service that we have just deployed. The emoji service is simulating some service to service invocation scenarios and now it’s time for us to open the Linkerd dashboard to inspect all the distributed traces and many other useful matrix to look at.

linkerd-censored

We can also see kind of an application map – in a graphical way to understand which service is calling who and what is request latencies etc.

linkerd-graph

Even fascinating, Linkerd provides some drill-down to the communications in Grafana Dashboard.

ezgif.com-gif-maker.gif

Conclusion

I have enjoyed a lot setting it up and see the outcome and wanted to share my experience with it. If you are looking into Service Mesh and read this post, I strongly encourage to give Linkerd a go, it’s awesome!

Thanks for reading.

CloudOven – Terraform at ease!

TL;DR:

  • URL: CloudOven 

  • Use Google account or sign-up 
  • Google Chrome please! (I’ve not tested on other browsers yet)

e2e

Background

In recent years I have spent fair amount of time in design and implementation of Infrastructure as code in larger enterprise context. Terraform seemed to be a tool of choice when it comes to preserve the uniformity in Infrastructure as code targeting multiple cloud providers. It is rapidly becoming a de facto choice for creating and managing cloud infrastructures by writing declarative definitions. It’s popular because the syntax of its files is quite readable and because it supports several cloud providers while making no attempt to provide an artificial abstraction across those providers. The active community will add support for the latest features from most cloud providers.

However, rolling out Terraform in many enterprises has its own barrier to face. Albeit the syntax (HCL) is neat, but not every developers or Infrastructure operators in organizations finds it easy. There’s a learning curve and often many of us lose momentum discovering the learning effort. I believe if we could make the initial ramp-up easier more people would play with it.

That’s one of my motivation for this post, following is the other one.

Blazor meets Terraform

Lately I was learning Blazor – the new client-side technology from Microsoft. Like many others, I find one effective way learning a new technology by creating/building solution to a problem. I have decided to build a user interface that will help creating terraform scripts easier. I will share my journey in this post.

Resource Discovery in Terraform Providers

Terraform is powerful for its providers. You will find Terraform providers for all major cloud providers (Azure, AWS, Google etc.). The providers then allow us to define “resource” and “data source” in Terraform scripts. These resource and data source have arguments and attributes that one must know while creating terraform files. Luckily, they are documented nicely in Terraform site. However, it still requires us to jump back and forth to the documentation site and terraform file editor (i.e. VSCode).

Azure-Discovery

To make this experience easier, I wrote a crawler application that downloads the terraform providers (I am doing it for Azure, AWS and google for now) and discovers the attributes and arguments for each and every resource and data source. I also try to extract the documentation for every attributes and arguments from the terraform documentation site with a layman parsing (not 100% accurate but works for majority. Something I will improve soon).

GoogleAWS-discovery

This process generates JSON structure for each resource and data source, enriches them with the documentation and stores them in an Azure Blob Storage.

Building Infrastructure as code

Now that I have a structured data store with all resources and data sources for any terraform provider, I can leverage that building a user interface on top of it. To keep things a bit organized, I started with a concept of “project”.

workflow
The workflow

Project

I can start by creating a project (well, it can be a product too, but let’s not get to that debate). Project is merely a logical boundary here.

Blueprint

Within a project I can create Blueprint(s). Blueprint(s) are the entity that retains the elements of the infrastructure that we are aiming to create. For instance, a Blueprint targets to a Cloud provider (i.e. Azure). Then I can create the elements (resource and data sources) within the blueprint (i.e. Azure Web App, Cosmos DB etc.).

provider-configuration

Blueprints keeps the base structure of all the infrastructure elements. It allows defining variables (plain and simple terraform variables) so the actual values can vary in different environments (dev, test, pre-production, production etc.).

Once I am happy with the blueprint, I can download them as a zip – that contains the terraform scripts (main.tf and variable.tf). That’s it, we have our infrastructure as code in Terraform. I can execute them on a local development machine or check them in to source control – whatever I prefer.

storage_account

One can stop here and keep using the blueprint feature to generate Infrastructure as code. That’s what it is for. However, the next features are just to make the overall experience of running terraform a bit easier.

Environments

Next to blueprint, we can create as many environments we want. Again, just a logical entity to keep isolation of actual deployment for different environments.

Deployments

Deployment entity is the glue that ties a blueprint to a specific environment. For instance, I can define a blueprint for “order management” service (or micro-service maybe?), create an environment as “test” and then create a “deployment” for “order management” on “test”. This is where I can define constant values to the blueprint variable that are specific to the test environment.

Terraform State

Perhaps the most important aspect the deployment entity holds is the terraform state management. Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. This state is stored by default in a local file named “terraform.tfstate”, but it can also be stored remotely, which works better in a team environment. Defining the state properties (varies in different cloud providers) in deployment entity makes the remote state management easier – specifically in team environment. It will configure the remote state to the appropriate remote backend. For instance, when the blueprint cloud provider is set to Azure, it will configure Azure Storage account as terraform state remote backend, for AWS it will pick S3 automatically.

e2e

Terraform plan

Once we have deployment entity configured, we can directly from the user interface run “terraform plan”. The terraform plan command creates an execution plan. Unless explicitly disabled, it performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the blueprint. This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

Terraform apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan. Like “plan”, the “apply” command can also be issued directly from the user interface.

Terraform plan and apply both are issued in an isolated docker container and the output is captured and displayed back to the user interface. However, there’s a cost associated running docker containers on cloud, therefore, it’s disabled in the public site.

Final thoughts

It was fun to write a tool like this. I recommend you give it a go. Especially if you are stepping into Terraform. It can also be helpful for experienced Terraform developers – specifically with the on-screen documenation, type inferance and discovery features.

Some features, I have working progress:

  • Ability to define policy for each resources and data types
  • Save a Blueprint as custom module

Stay tuned!

 

Continuously deploy Blazor SPA to Azure Storage static web site

Lately I am learning ASP.net Blazor – the relatively new UI framework from Microsoft. Blazor is just awesome – the ability to write c# code both in server and client side is extremely productive for .net developers. From Blazor documentations:

Blazor lets you build interactive web UIs using C# instead of JavaScript. Blazor apps are composed of reusable web UI components implemented using C#, HTML, and CSS. Both client and server code is written in C#, allowing you to share code and libraries.

I wanted to write a simple SPA (Single Page Application) and run it as server-less. Azure Storage offers hosting static web sites for quite a while now. Which seems like a very nice option to run a Blazor SPA which executes into the user’s browser (within the same Sandbox as JavaScript does). It also a cheap way to run a Single Page application in Cloud.

I am using GitHub as my source repository for free (in a private repository). Today wanted to create a pipeline that will continuously deploy my Blazor app to the storage account. Azure Pipelines seems to have pretty nice integration with GitHub and it’s has a free tier as well . If either our GitHub repository or pipeline is private, Azure Pipeline still provide a free tier. In this tier, one can run one free parallel job that can run up to 60 minutes each time until we’ve used 1800 minutes per month. That’s pretty darn good for my use case.

I also wanted to build the project many times in my local machine (while developing) in the same way it gets built in the pipeline. Being a docker fan myself, that’s quite no-brainer. Let’s get started.

Pre-requisite

I have performed few steps before I ran after the pipeline – that are beyond the scope of this post.

  • I have created an Azure Subscription
  • Provisioned resource groups and storage account
  • I have created a Service Principal and granted Contributor role to the storage account

Publishing in Docker

I have created a docker file that will build the app, run unit tests and if all goes well, it will publish the app in a folder. All of these are standard dotnet commands.
Once, we have the application published in a folder, I have taken the content of that folder to a Azure CLI docker base image (where CLI is pre-installed) and thrown away the rest of the intermediate containers.

Here’s our docker file:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim AS base
WORKDIR /app
# Build the app
FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build
WORKDIR /src
COPY . .
WORKDIR "/src/BlazorApp"
RUN dotnet restore "BlazorApp.csproj"
RUN dotnet build "BlazorApp.csproj" -c Release -o /app
# Run the unit tests
RUN dotnet restore "BlazorAppTest.csproj"
RUN dotnet build "BlazorAppTest.csproj"
RUN dotnet test "BlazorAppTest.csproj"
# Publish the app
FROM build AS publish
RUN dotnet publish "BlazorApp.csproj" -c Release -o /app
FROM microsoft/azure-cli AS final
ARG AppID
ARG AppSecret
ARG TenantID
ARG StorageAccountName
WORKDIR /app
COPY –from=publish /app .
WORKDIR "/app/BlazorApp/dist"
RUN az login –service-principal –username $AppID –password $AppSecret –tenant $TenantID
RUN az storage blob delete-batch –account-name $StorageAccountName –source "\$web"
RUN az storage blob upload-batch –account-name $StorageAccountName -s . -d "\$web"

The docker file expects few arguments (basically the service principal ID, the password of the service principal and the Azure AD tenant ID – these are required for Azure CLI to sign-in to my Azure subscription). Here’s how we can build this image now:

docker build
-f Blazor.Dockerfile
–build-arg StorageAccountName=<Storage-Account-Name>
–build-arg AppID=<APP GUID>
–build-arg AppSecret=<PASSWORD>
–build-arg TenantID=<Azure AD TENANT ID>
-t blazor-app .

view raw
docker-run.sh
hosted with ❤ by GitHub

Azure Pipeline as code

We have now the container, time to run it every time a commit has been made to the GitHub repository. Azure Pipeline has a yaml format to define pipeline-as-code – which is another neat feature of Azure Pipelines.

Let’s see how the pipeline-as-code looks like:

trigger:
master
pool:
name: Hosted Ubuntu 1604
steps:
task: Docker@0
displayName: 'Build and release Blazor to Storage Account'
inputs:
dockerFile: src/Blazor.Dockerfile
buildArguments: |
StorageAccountName=$(StorageAccountName)
AppID=$(AppID)
AppSecret=$(AppSecret)
TenantID=$(TenantID)

view raw
pipeline.yaml
hosted with ❤ by GitHub

I have committed this to the same repository into the root folder.

Creating the pipeline

We need to login to Azure DevOps and create a project (if there’s none). From the build option we can create a new build definition.

ado

The steps to create build definition is very straightforward. It allows us to directly point to a GitHub repository that we want to build.
Almost there. We need to supply the service principal ID, password, tenant ID and storage account names to this pipeline – because both our docker file and the pipeline-as-code expected them as dependencies. However, we can’t just put their values and commit them to GitHub. They should be kept secret.

Azure Pipeline Secret variables

Azure Pipeline allows us to define secret variable for a pipeline. We need to open the build definition in “edit” mode and then go to the top-right ellipses button as below:

ado1

Now we can define the values of these secret and keep them hidden (there’s a lock icon there).

ad02

That’s all what we need. Ready to deploy the code to Azure storage via this pipeline. If we now go an make a change in our repository it will trigger the pipeline and sure enough it will build-test-publish-deploy to Azure storage as a Static SPA.

Thanks for reading!

Continuously deliver changes to Azure API management service with Git Configuration Repository

What is API management

Publishing data, insights and business capabilities via API in a unified way can be challenging at times. Azure API management (APIM) makes it simpler than ever.

Businesses everywhere are looking to extend their operations as a digital platform, creating new channels, finding new customers and driving deeper engagement with existing ones. API Management provides the core competencies to ensure a successful API program through developer engagement, business insights, analytics, security, and protection. You can use Azure API Management to take any backend and launch a full-fledged API program based on it. [Source]

The challenge – Continuous Deployment

These days, it’s very common to have many distributed services (let’s say Micro service) publish APIs in a mesh up Azure API management portal. For instance, Order and Invoice APIs are published over an E-Commerce API portal, although they are backed by isolated Order and Invoice Micro services. Autonomous teams build these APIs, often work in isolation’s but their API specifications (mostly Open API specification Swagger documents) must be published through a shared API management Service. Different teams with different release cadence can make the continuous deployment of API portal challenging and error prone.

Azure API management ships bunch of Power Shell cmdlets (i.e. Import-AzureRmApiManagementApi  and Publish-AzureRmApiManagementTenantGitConfiguration ) that allow deploying the API documentation directly to APIM. Which works great for single API development team. It gets a bit trickier when multiple teams are pushing changes to a specific APIM instance like the example above. Every team needs to have deployment credentials in their own release pipelines – which might undesirable for a Shared APIM instance. Centrally governing these changes becomes difficult.

APIM Configuration Git Repository

APIM instance has a pretty neat feature. Each APIM instance has a configuration database associated as a Git Repository, containing the metadata and configuration information for the APIM instance. We can clone the configuration repository and push changes back- using our very familiar Git commands and tool sets and APIM allows us to publish those changes that are pushed – sweet!

This allows us downloading different versions of our APIM configuration state. Managing bulk APIM configurations (this includes, API specifications, Products, Groups, Policies and branding styles etc.) in one central repository with very familiar Git tools, is super convenient.

The following diagram shows an overview of the different ways to configure your API Management service instance.

api-management-git-configure

[Source]

This sounds great! However, we will leverage this capability and make it even nicer, where multiple teams can develop their API’s without depending on others release schedules and we can have a central release pipeline that publishes the changes from multiple API services.

Solution design

The idea is pretty straight forward. Each team develop their owner API specification and when they want to release, they create PR (Pull Request) to a shared Repository. Which contains the APIM configuration clone. Once peer reviewed the PR and merged, the release pipeline kicks in. Which deploys the changes to Azure APIM.

The workflow looks like following:

workflow
Development and deployment workflow

Building the solution

We will provision a APIM instance on Azure. We can do that with an ARM template (We will not go into the details of that, you can use this GitHub template ).

Once we have APIM provisioned, we can see the Git Repository is not yet synchronized with the Configuration Database. (notice Out  of sync in the following image)

Out of sync

We will sync it and clone a copy of the configuration database in our local machine using the following Power Shell script. (You need to run Login-AzureRMAccount in Power Shell console, if you are not already logged in to Azure).

$context = New-AzureRmApiManagementContext `
        -ResourceGroupName $ResourceGroup `
        -ServiceName $ServiceName
    Write-Output "Initializing context...Completed"

    Write-Output "Syncing Git Repo with current API management state..."
    Save-AzureRmApiManagementTenantGitConfiguration `
        -Context $context `
        -Branch 'master' `
        -PassThru -Force

This will make the Git Repository synced.

Sync

To clone the repository to local machine, we need to generate Git Credentials first. Let’s do that now:

Function ExecuteGitCommand {
    param
    (
        [System.Object[]]$gitCommandArguments
    )

    $gitExePath = "C:\Program Files\git\bin\git.exe"
    & $gitExePath $gitCommandArguments
}

 

$expiry = (Get-Date) + '1:00:00'
    $parameters = @{
        "keyType" = "primary"
        "expiry"  = ('{0:yyyy-MM-ddTHH:mm:ss.000Z}' -f $expiry)
    }

    $resourceId = '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ApiManagement/service/{2}/users/git' -f $SubscriptionId, $ResourceGroup, $ServiceName

    if ((Test-Path -Path $TempDirectory )) {
        Remove-Item $TempDirectory -Force -Recurse -ErrorAction "Stop"
    }

    $gitRemoteSrcPath = Join-Path -Path $TempDirectory -ChildPath 'remote-api-src'

    Write-Output "Retrieving Git Credentials..."
    $gitUsername = 'apim'
    $gitPassword = (Invoke-AzureRmResourceAction `
            -Action 'token' `
            -ResourceId $resourceId `
            -Parameters $parameters `
            -ApiVersion '2016-10-10' `
            -Force).Value
    $escapedGitPassword = [System.Uri]::EscapeDataString($gitPassword)
    Write-Output "Retrieving Git Credentials...Completed"

    $gitRepositoryUrl = 'https://{0}:{1}@{2}.scm.azure-api.net/' -f $gitUsername, $escapedGitPassword, $ServiceName
    ExecuteGitCommand -gitCommandArguments @("clone", "$gitRepositoryUrl", "$gitRemoteSrcPath")

Now, we have a copy of the Git in our local machine. This is just a mirror of our APIM configuration database. We will create a repository in our Source Control (I am using VSTS). This will be our Shared APIM source repository. Every team will issue Pull Request with their API Specification into this repository. Which can be approved by other peers and eventually merged to master branch.

Building the release pipeline

Time to deploy changes from our Shared Repository to APIM instance. We will require following steps to perform:

  1. Sync the configuration database to APIM Git Repository.
  2. Clone the latest changes to our Build agent.
  3. Copy all updated API specifications, approved and merged to our VSTS repository’s master branch to the cloned repository.
  4. Commit all changes to the cloned repository.
  5. Push changes from clone repository to origin.
  6. Publish changes from Git Repository to APIM instance.

I have compiled a single Power Shell script that does all these steps- in that order. Idea is to, use this Power Shell script in our release pipeline to deploy releases to APIM. The complete scripts is given below:

# This script should be used into the build pipeline to update API changes
# to Azure API management Service.
# The steps that are scripted below:
#
# 1. Updates the Internal Git Repo with the published API documents in Azure
# 2. Retrieves the Git Credentails of internal Repo
# 3. Clones the repo in to a temporary folder
# 4. Merges changes from VSTS repo to the temporary cloned repository
# 5. Commit dirty changes
# 6. Push changes to Azure Internal Repo (master branch)
# 7. Publishes the API updates from internal Repo to Azure API management service.
#
# Example Usage:
#
# Parameters:
# – SubscriptionId: The subscription ID where the API management Service is provisioned
# – ResourceGroup: Resource group name
# – ServiceName: API management Service name
# – SourceDirectory: Directory where the API documents, policies etc. are located
# – TempDirectory: A temporary directory where the data will be downloaded
# – UserName: $ENV:RELEASE_DEPLOYMENT_REQUESTEDFOREMAIL – The user name that will be used to commit to Git. You can use Relase triggerd here
# – UserEmailAddress: $ENV:RELEASE_DEPLOYMENT_REQUESTEDFOR The email address of the user
# – CommitMessage: $ENV:RELEASE_RELEASEWEBURL The commit message
#
# Get-AzureRMApiManagementGitRepo `
# -SubscriptionId "– Subscription ID — " `
# -ResourceGroup " — Resource group name — " `
# -ServiceName " — APIM name — " `
# -SourceDirectory "..\src\api-management" `
# -TempDirectory 'C:\Temp\apim' `
# -UserName "Moim Hossain" `
# -CommitMessage "Commited from CI"
# -UserEmailAddress "moim.hossain"
Function ExecuteGitCommand {
param
(
[System.Object[]]$gitCommandArguments
)
$gitExePath = "C:\Program Files\git\bin\git.exe"
& $gitExePath $gitCommandArguments
}
Function Copy-DirectoryContents {
param
(
[System.String]$SrcPath,
[System.String]$destPath
)
ROBOCOPY $SrcPath $destPath /MIR
}
Function Get-AzureRMApiManagementGitRepo {
param
(
[System.String]
$SubscriptionId,
[System.String]
$ResourceGroup,
[System.String]
$ServiceName,
[System.String]
$UserName,
[System.String]
$UserEmailAddress,
[System.String]
$CommitMessage,
[System.String]
$SourceDirectory,
[System.String]
$TempDirectory
)
Write-Output "Initializing context…"
$context = New-AzureRmApiManagementContext `
ResourceGroupName $ResourceGroup `
ServiceName $ServiceName
Write-Output "Initializing context…Completed"
Write-Output "Syncing Git Repo with current API management state…"
Save-AzureRmApiManagementTenantGitConfiguration `
Context $context `
Branch 'master' `
PassThru Force
Write-Output "Syncing Git Repo with current API management state…Completed"
$expiry = (Get-Date) + '1:00:00'
$parameters = @{
"keyType" = "primary"
"expiry" = ('{0:yyyy-MM-ddTHH:mm:ss.000Z}' -f $expiry)
}
$resourceId = '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ApiManagement/service/{2}/users/git' -f $SubscriptionId, $ResourceGroup, $ServiceName
if ((Test-Path Path $TempDirectory )) {
Remove-Item $TempDirectory Force Recurse ErrorAction "Stop"
}
$gitRemoteSrcPath = Join-Path Path $TempDirectory ChildPath 'remote-api-src'
Write-Output "Retrieving Git Credentials…"
$gitUsername = 'apim'
$gitPassword = (Invoke-AzureRmResourceAction `
Action 'token' `
ResourceId $resourceId `
Parameters $parameters `
ApiVersion '2016-10-10' `
Force).Value
$escapedGitPassword = [System.Uri]::EscapeDataString($gitPassword)
Write-Output "Retrieving Git Credentials…Completed"
$gitRepositoryUrl = 'https://{0}:{1}@{2}.scm.azure-api.net/' -f $gitUsername, $escapedGitPassword, $ServiceName
Write-Host "Performing Git clone… $gitRemoteSrcPath"
ExecuteGitCommand gitCommandArguments @("clone", "$gitRepositoryUrl", "$gitRemoteSrcPath")
Write-Host "Performing Git clone… Completed"
# Copy changes from Source Repository (VSTS) to Locally Cloned Repository
$apiSources = Join-Path Path $gitRemoteSrcPath ChildPath 'api-management'
Copy-DirectoryContents `
SrcPath $SourceDirectory `
destPath $apiSources
Set-Location $gitRemoteSrcPath
ExecuteGitCommand gitCommandArguments @("config", "user.email", $UserEmailAddress)
ExecuteGitCommand gitCommandArguments @("config", "user.name", $UserName)
ExecuteGitCommand gitCommandArguments @("add", "–all")
ExecuteGitCommand gitCommandArguments @("commit", "-m", $CommitMessage)
ExecuteGitCommand gitCommandArguments @("push")
Publish-AzureRmApiManagementTenantGitConfiguration `
Context $context `
Branch 'master' `
PassThru `
Verbose
}

view raw
APIM-Deployment.ps1
hosted with ❤ by GitHub

Final thoughts

The Git Repository model for deploying API specifications to a single APIM instance makes it extremely easy to manage. Despite the fact, we could have done this with Power Shell alone. But in multiple team scenario that gets messy pretty quick. Having a centrally leading Git Repository as release gateway (and the only way to make any changes to APIM instance) reduces the complexity to minimum.