Continuously deliver changes to Azure API management service with Git Configuration Repository

What is API management

Publishing data, insights and business capabilities via API in a unified way can be challenging at times. Azure API management (APIM) makes it simpler than ever.

Businesses everywhere are looking to extend their operations as a digital platform, creating new channels, finding new customers and driving deeper engagement with existing ones. API Management provides the core competencies to ensure a successful API program through developer engagement, business insights, analytics, security, and protection. You can use Azure API Management to take any backend and launch a full-fledged API program based on it. [Source]

The challenge – Continuous Deployment

These days, it’s very common to have many distributed services (let’s say Micro service) publish APIs in a mesh up Azure API management portal. For instance, Order and Invoice APIs are published over an E-Commerce API portal, although they are backed by isolated Order and Invoice Micro services. Autonomous teams build these APIs, often work in isolation’s but their API specifications (mostly Open API specification Swagger documents) must be published through a shared API management Service. Different teams with different release cadence can make the continuous deployment of API portal challenging and error prone.

Azure API management ships bunch of Power Shell cmdlets (i.e. Import-AzureRmApiManagementApi  and Publish-AzureRmApiManagementTenantGitConfiguration ) that allow deploying the API documentation directly to APIM. Which works great for single API development team. It gets a bit trickier when multiple teams are pushing changes to a specific APIM instance like the example above. Every team needs to have deployment credentials in their own release pipelines – which might undesirable for a Shared APIM instance. Centrally governing these changes becomes difficult.

APIM Configuration Git Repository

APIM instance has a pretty neat feature. Each APIM instance has a configuration database associated as a Git Repository, containing the metadata and configuration information for the APIM instance. We can clone the configuration repository and push changes back- using our very familiar Git commands and tool sets and APIM allows us to publish those changes that are pushed – sweet!

This allows us downloading different versions of our APIM configuration state. Managing bulk APIM configurations (this includes, API specifications, Products, Groups, Policies and branding styles etc.) in one central repository with very familiar Git tools, is super convenient.

The following diagram shows an overview of the different ways to configure your API Management service instance.

api-management-git-configure

[Source]

This sounds great! However, we will leverage this capability and make it even nicer, where multiple teams can develop their API’s without depending on others release schedules and we can have a central release pipeline that publishes the changes from multiple API services.

Solution design

The idea is pretty straight forward. Each team develop their owner API specification and when they want to release, they create PR (Pull Request) to a shared Repository. Which contains the APIM configuration clone. Once peer reviewed the PR and merged, the release pipeline kicks in. Which deploys the changes to Azure APIM.

The workflow looks like following:

workflow
Development and deployment workflow

Building the solution

We will provision a APIM instance on Azure. We can do that with an ARM template (We will not go into the details of that, you can use this GitHub template ).

Once we have APIM provisioned, we can see the Git Repository is not yet synchronized with the Configuration Database. (notice Out  of sync in the following image)

Out of sync

We will sync it and clone a copy of the configuration database in our local machine using the following Power Shell script. (You need to run Login-AzureRMAccount in Power Shell console, if you are not already logged in to Azure).

$context = New-AzureRmApiManagementContext `
        -ResourceGroupName $ResourceGroup `
        -ServiceName $ServiceName
    Write-Output "Initializing context...Completed"

    Write-Output "Syncing Git Repo with current API management state..."
    Save-AzureRmApiManagementTenantGitConfiguration `
        -Context $context `
        -Branch 'master' `
        -PassThru -Force

This will make the Git Repository synced.

Sync

To clone the repository to local machine, we need to generate Git Credentials first. Let’s do that now:

Function ExecuteGitCommand {
    param
    (
        [System.Object[]]$gitCommandArguments
    )

    $gitExePath = "C:\Program Files\git\bin\git.exe"
    & $gitExePath $gitCommandArguments
}

 

$expiry = (Get-Date) + '1:00:00'
    $parameters = @{
        "keyType" = "primary"
        "expiry"  = ('{0:yyyy-MM-ddTHH:mm:ss.000Z}' -f $expiry)
    }

    $resourceId = '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ApiManagement/service/{2}/users/git' -f $SubscriptionId, $ResourceGroup, $ServiceName

    if ((Test-Path -Path $TempDirectory )) {
        Remove-Item $TempDirectory -Force -Recurse -ErrorAction "Stop"
    }

    $gitRemoteSrcPath = Join-Path -Path $TempDirectory -ChildPath 'remote-api-src'

    Write-Output "Retrieving Git Credentials..."
    $gitUsername = 'apim'
    $gitPassword = (Invoke-AzureRmResourceAction `
            -Action 'token' `
            -ResourceId $resourceId `
            -Parameters $parameters `
            -ApiVersion '2016-10-10' `
            -Force).Value
    $escapedGitPassword = [System.Uri]::EscapeDataString($gitPassword)
    Write-Output "Retrieving Git Credentials...Completed"

    $gitRepositoryUrl = 'https://{0}:{1}@{2}.scm.azure-api.net/' -f $gitUsername, $escapedGitPassword, $ServiceName
    ExecuteGitCommand -gitCommandArguments @("clone", "$gitRepositoryUrl", "$gitRemoteSrcPath")

Now, we have a copy of the Git in our local machine. This is just a mirror of our APIM configuration database. We will create a repository in our Source Control (I am using VSTS). This will be our Shared APIM source repository. Every team will issue Pull Request with their API Specification into this repository. Which can be approved by other peers and eventually merged to master branch.

Building the release pipeline

Time to deploy changes from our Shared Repository to APIM instance. We will require following steps to perform:

  1. Sync the configuration database to APIM Git Repository.
  2. Clone the latest changes to our Build agent.
  3. Copy all updated API specifications, approved and merged to our VSTS repository’s master branch to the cloned repository.
  4. Commit all changes to the cloned repository.
  5. Push changes from clone repository to origin.
  6. Publish changes from Git Repository to APIM instance.

I have compiled a single Power Shell script that does all these steps- in that order. Idea is to, use this Power Shell script in our release pipeline to deploy releases to APIM. The complete scripts is given below:

# This script should be used into the build pipeline to update API changes
# to Azure API management Service.
# The steps that are scripted below:
#
# 1. Updates the Internal Git Repo with the published API documents in Azure
# 2. Retrieves the Git Credentails of internal Repo
# 3. Clones the repo in to a temporary folder
# 4. Merges changes from VSTS repo to the temporary cloned repository
# 5. Commit dirty changes
# 6. Push changes to Azure Internal Repo (master branch)
# 7. Publishes the API updates from internal Repo to Azure API management service.
#
# Example Usage:
#
# Parameters:
# – SubscriptionId: The subscription ID where the API management Service is provisioned
# – ResourceGroup: Resource group name
# – ServiceName: API management Service name
# – SourceDirectory: Directory where the API documents, policies etc. are located
# – TempDirectory: A temporary directory where the data will be downloaded
# – UserName: $ENV:RELEASE_DEPLOYMENT_REQUESTEDFOREMAIL – The user name that will be used to commit to Git. You can use Relase triggerd here
# – UserEmailAddress: $ENV:RELEASE_DEPLOYMENT_REQUESTEDFOR The email address of the user
# – CommitMessage: $ENV:RELEASE_RELEASEWEBURL The commit message
#
# Get-AzureRMApiManagementGitRepo `
# -SubscriptionId "– Subscription ID — " `
# -ResourceGroup " — Resource group name — " `
# -ServiceName " — APIM name — " `
# -SourceDirectory "..\src\api-management" `
# -TempDirectory 'C:\Temp\apim' `
# -UserName "Moim Hossain" `
# -CommitMessage "Commited from CI"
# -UserEmailAddress "moim.hossain"
Function ExecuteGitCommand {
param
(
[System.Object[]]$gitCommandArguments
)
$gitExePath = "C:\Program Files\git\bin\git.exe"
& $gitExePath $gitCommandArguments
}
Function Copy-DirectoryContents {
param
(
[System.String]$SrcPath,
[System.String]$destPath
)
ROBOCOPY $SrcPath $destPath /MIR
}
Function Get-AzureRMApiManagementGitRepo {
param
(
[System.String]
$SubscriptionId,
[System.String]
$ResourceGroup,
[System.String]
$ServiceName,
[System.String]
$UserName,
[System.String]
$UserEmailAddress,
[System.String]
$CommitMessage,
[System.String]
$SourceDirectory,
[System.String]
$TempDirectory
)
Write-Output "Initializing context…"
$context = New-AzureRmApiManagementContext `
ResourceGroupName $ResourceGroup `
ServiceName $ServiceName
Write-Output "Initializing context…Completed"
Write-Output "Syncing Git Repo with current API management state…"
Save-AzureRmApiManagementTenantGitConfiguration `
Context $context `
Branch 'master' `
PassThru Force
Write-Output "Syncing Git Repo with current API management state…Completed"
$expiry = (Get-Date) + '1:00:00'
$parameters = @{
"keyType" = "primary"
"expiry" = ('{0:yyyy-MM-ddTHH:mm:ss.000Z}' -f $expiry)
}
$resourceId = '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.ApiManagement/service/{2}/users/git' -f $SubscriptionId, $ResourceGroup, $ServiceName
if ((Test-Path Path $TempDirectory )) {
Remove-Item $TempDirectory Force Recurse ErrorAction "Stop"
}
$gitRemoteSrcPath = Join-Path Path $TempDirectory ChildPath 'remote-api-src'
Write-Output "Retrieving Git Credentials…"
$gitUsername = 'apim'
$gitPassword = (Invoke-AzureRmResourceAction `
Action 'token' `
ResourceId $resourceId `
Parameters $parameters `
ApiVersion '2016-10-10' `
Force).Value
$escapedGitPassword = [System.Uri]::EscapeDataString($gitPassword)
Write-Output "Retrieving Git Credentials…Completed"
$gitRepositoryUrl = 'https://{0}:{1}@{2}.scm.azure-api.net/' -f $gitUsername, $escapedGitPassword, $ServiceName
Write-Host "Performing Git clone… $gitRemoteSrcPath"
ExecuteGitCommand gitCommandArguments @("clone", "$gitRepositoryUrl", "$gitRemoteSrcPath")
Write-Host "Performing Git clone… Completed"
# Copy changes from Source Repository (VSTS) to Locally Cloned Repository
$apiSources = Join-Path Path $gitRemoteSrcPath ChildPath 'api-management'
Copy-DirectoryContents `
SrcPath $SourceDirectory `
destPath $apiSources
Set-Location $gitRemoteSrcPath
ExecuteGitCommand gitCommandArguments @("config", "user.email", $UserEmailAddress)
ExecuteGitCommand gitCommandArguments @("config", "user.name", $UserName)
ExecuteGitCommand gitCommandArguments @("add", "–all")
ExecuteGitCommand gitCommandArguments @("commit", "-m", $CommitMessage)
ExecuteGitCommand gitCommandArguments @("push")
Publish-AzureRmApiManagementTenantGitConfiguration `
Context $context `
Branch 'master' `
PassThru `
Verbose
}

view raw
APIM-Deployment.ps1
hosted with ❤ by GitHub

Final thoughts

The Git Repository model for deploying API specifications to a single APIM instance makes it extremely easy to manage. Despite the fact, we could have done this with Power Shell alone. But in multiple team scenario that gets messy pretty quick. Having a centrally leading Git Repository as release gateway (and the only way to make any changes to APIM instance) reduces the complexity to minimum.