Azure Resource Governance with Template Specs & Biceps

All the example codes are available in GitHub.

Background

Governance of cloud estates is challenging for businesses. It’s crucial to enforce security policies, workload redundancies, uniformity (such as naming conventions), simplify deployments with packaged artifacts (i.e., ARM templates), Azure role-based access control (Azure RBAC) across the enterprise.

Generally, the idea is, a centralized team (sometimes referred as platform team) builds and publishes Infrastructure-as-code artifacts and number of product development teams consume them, only providing their own parameters.

Azure offers native capabilities like Azure Policy, Blueprints and Management groups to address this problem. But there are wide range of external solutions (Terraform, Pulumi etc.) available too.

One attribute of Terraform, strikes me a lot is the ability to store a versioned module in a registry and consume it from the registry. The same principle that familiar to engineers and widely used in programming languages – such as NuGet for .net, Maven for Java, npm for Node

With ARM templates it’s rather unpleasant. If you currently have your templates in an Azure repo, GitHub repo or storage account, you run into several challenges when trying to share and use the templates. For a user to deploy it, the template must either be local or the URL for the template must be publicly accessible. To get around this limitation, you might share copies of the template with users who need to deploy it, or open access to the repo or storage account. When users own local copies of a template, these copies can eventually diverge from the original template. When you make a repo or storage account publicly accessible, you may allow unintended users to access the template.

Azure Resource Manager – Template Spec

Microsoft delivered some cool new features for Resource manager templates recently. One of these features, named as Template Spec. Template Spec is a first-class Azure Resource type, but it really is just a regular ARM template. Best part is that you can version it, persist it in Azure – just like a Terraform registry, share it across the organization with RBAC and consume them from repository.

Template Specs is currently in preview. To use it, you must install the latest version of PowerShell or Azure CLI. For Azure PowerShell, use version 5.0.0 or later. For Azure CLI, use version 2.14.2 or later.

The benefit of using template specs is that you can create canonical templates and share them with teams in your organization. The template specs are secure because they’re available to Azure Resource Manager for deployment, but not accessible to users without Azure RBAC permission. Users only need read access to the template spec to deploy its template, so you can share the template without allowing others to modify it.

The templates you include in a template spec should be verified by the platform team (or administrators) in your organization to follow the organization’s requirements and guidance.

How template spec works?

If you are familiar with ARM template, template specs are not new to you. They are just typical ARM templates and stored in Azure Resource group as “template spec” with a version number. That means, you can take any ARM template (the template JSON file only – without any parameter files) and publish it as template spec, using PowerShell, Azure CLI or REST API.

Publishing Template Spec

Let’s say I have a template that defines just an Application Insight component.

{
    "contentVersion": "1.0.0.0",
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "parameters": {
        "appInsights": { "type": "string" },
        "location": { "type": "string" }
    },
    "resources": [{
            "type": "microsoft.insights/components",
            "apiVersion": "2020-02-02-preview",
            "name": "[parameters('appInsights')]",
            "location": "[parameters('location')]",
            "properties": {
                "ApplicationId": "[parameters('appInsights')]",
                "Application_Type": "web"
            }
        }
    ],
    "outputs": {
        "instrumentationKey": {
            "type": "string",
            "value": "[reference(parameters('appInsights')).InstrumentationKey]"
        }
    }
}

We can now publish this as “template spec” using Azure CLI:

az ts create \
    --name "cloudoven-appInsights" \
    --version $VERSION \
    --resource-group $RESOURCE_GROUP \
    --location $LOCATION \
    --template-file "component.json" \
    --yes --query 'id' -o json

Once published, you can see it in Azure Portal – appeared as a new resource type Microsoft.Resources/templateSpecs.

Consuming Template Spec

Every published Template Spec has a unique ID. To consume a Template Spec, all you need is the ID of the Template Spec. You can retrieve the ID in Azure CLI:

APPINS_TSID=$(az ts show --resource-group $TSRGP --name $TSNAME --version $VERSION --query 'id' -o json)
echo Template Spec ID: $APPINS_TSID

You can now, deploy Azure Resources with the Template Spec ID and optionally your own parameters, like following:

az deployment group create \
  --resource-group $RESOURCEGROUP \
  --template-spec $APPINS_TSID \
  --parameters "parameters.json

Linked Templates & Modularizations

Overtime, Infrastructure-as-code tends to become big monolithic file containing numerous resources. ARM templates (thanks to all it’s verbosity) specially known for growing big fast and becomes difficult to comprehend by reading a large JSON file. You could address the issue before by using linked templates. However, with a caveat that linked templates needed to be accessible via an URL in the public internet – far from ideal.

Good news is Template Spec got this covered. If the main template for your template spec references linked templates, the PowerShell and CLI commands can automatically find and package the linked templates from your local drive.

Example

Here I have an example ARM template that defines multiple resources (Application Insights, Server farm and a web app) in small files and finally creating a main template that brings everything together. One can then publish the main template as Template Spec – hence any consumer can provision their web app just by pointing to the ID of the template spec. Here’s the interesting bit of the main template:

"resources": [
        {
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2020-06-01",
            "name": "DeployAppInsights",
            "properties": {
                "mode": "Incremental",
                "parameters": {
                    "appInsights": { "value": "[parameters('appInsights')]"},
                    "location": {"value": "[parameters('location')]"}
                },
                "templateLink": {                    
                    "relativePath": "../appInsights/component.json"
                }
            }
        },
        {
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2020-06-01",
            "name": "DeployHostingplan",
            "properties": {
                "mode": "Incremental",      
                "templateLink": {
                    "relativePath": "../server-farm/component.json"
                }
            }
        },
        {
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2020-06-01",
            "name": "DeployWebApp",
            "dependsOn": [ "DeployHostingplan" ],
            "properties": {
                "mode": "Incremental",             
                "templateLink": {
                    "relativePath": "../web-app/component.json"
                }
            }
        }
    ]

You see, Template Specs natively offers the modularity, centralized registry, however, they are still ARM JSON files. One common criticism of ARM template is it’s too verbose and JSON are not particularly famous for readability.

Microsoft is aiming to address these concerns with a new Domain Specific Language (DSL) that named a Azure Bicep.  

What is Bicep?

Bicep aims to drastically simplify the authoring experience with a cleaner syntax and better support for modularity and code re-use. Bicep is a transparent abstraction over ARM and ARM templates, which means anything that can be done in an ARM Template can be done in bicep (outside of temporary known limitations).

If we take the same Application Insight component (above) and re-write it in Bicep, it looks following:

param appInsights string
param location string = resourceGroup().location
resource appIns 'Microsoft.Insights/components@2020-02-02-preview' = {
  name: appInsights
  location: location
  kind: appInsights
  properties: {
    Application_Type: 'web'
  }
}
output InstrumentationKey string = appIns.properties.InstrumentationKey

Very clean and concise compared to the ARM JSON version of it. If you are coming from Terraform, you might already find yourself at home – because Bicep took lot of inspiration from Terraform HCL (HashiCorp Language). You save bicep scripts with .bicep file extensions.

Important things to understand, Bicep is a client-side language layer sits on top of ARM json. The idea is you write it in Bicep then compile the script using a Bicep compiler (or Transpiler) to produce ARM JSON as compiled artifact and you still deploy ARM template (JSON) to Azure. Here’s how you compile a bicep file to produce the ARM JSON:

bicep build ./main.bicep

Bicep currently is in experimental state, and not recommended to use in production.

Creating Template Spec in Bicep

The above Example – you’ve seen how you could create a Template Spec that is quite modularized into linked templates. Let’s rewrite that in Bicep and see how clean and simple it looks:

param webAppName string = ''
param appInsights string = ''
param location string = resourceGroup().location
param hostingPlanName string = ''
param containerSpec string =''
param costCenter string
param environment string

module appInsightsDeployment '../appinsights/component.bicep' = {
  name: 'appInsightsDeployment'
  params:{
    appInsights: '${appInsights}'
    location: '${location}'
    costCenter: costCenter
    environment: environment
  }
}

module deployHostingplan '../server-farm/component.bicep' = {
  name: 'deployHostingplan'
  params:{
    hostingPlanName:  '${hostingPlanName}'
    location: '${location}'
    costCenter: costCenter
    environment: environment    
  }   
}

module deployWebApp '../web-app/component.bicep' = {
  name: 'deployWebApp'
  params:{
    location: '${location}'
    webAppName: '${webAppName}'
    instrumentationKey: appInsightsDeployment.outputs.InstrumentationKey
    serverFarmId: deployHostingplan.outputs.hostingPlanId
    containerSpec: '${containerSpec}'
    costCenter: costCenter
    environment: environment    
  }   
}

Notice, Bicep came up with a nice keyword module to address the linked template scenario. Bicep module is an opaque set of one or more resources to be deployed together. It only exposes parameters and outputs as contract to other Bicep files, hiding details on how internal resources are defined. This allows you to abstract away complex details of the raw resource declaration from the end user who now only needs to be concerned about the module contract. Parameters and outputs are optional.

CI/CD – GitHub Action

Let’s create a delivery pipeline in GitHub action to compile our Bicep file and publish them as Template Spec in Azure. Following GitHub workflow, installs Bicep tools, compiles the scripts, and finally publishes them as Template Spec in Azure. You can see the complete repository in GitHub.

jobs:  
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install Bicep CLI
        working-directory: ./src/template-spec-bicep/paas-components
        run: |
          chmod +x ./install-bicep.sh
          ./install-bicep.sh
      - name: Compile Bicep Scripts
        working-directory: ./src/template-spec-bicep/paas-components
        run: |
          chmod +x ./build-templates.sh
          ./build-templates.sh
      - name: Azure Login
        uses: Azure/login@v1.1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
      - name: Deploy Template Specs to Azure
        working-directory: ./src/template-spec-bicep/paas-components        
        run: |          
          chmod +x ./deploy-templates.sh
          ./deploy-templates.sh

Consuming the template spec doesn’t change disregarding the choice you make with Bicep or ARM template. Consumers just create a parameter file and deploy resource only specifying the Template Spec ID. You can see an example of Consumer workflow (also GitHub action) here.

Conclusion

I am excited how the Azure team is offering new tools like Bicep, Template Specs to simplify the cloud governance, self-service areas. It’s important to understand that Bicep team is not competing with available tools in the space (like Terraform etc.) rather offering more options to folks in their cloud journey.

Bicep is in experimental phase now and Template Specs are not in preview, therefore, don’t use them in production just yet.   

Terraforming Azure DevOps

Background

In many organizations, specially in large enterprises there’s a need to automate Azure DevOps projects and Teams members. Manually managing large number of Azure DevOps projects, Teams for these projects and users to the teams, on-boarding and off-boarding team members are not trivial.

Besides managing the users sometimes, we just need to have an overview (a documentation?) of users and Teams of Projects. Terraform is a great tool for Infrastructure as Code – which not only allows providing infrastructure on demand, but also gives us nice documentation which can be versioned control in a source control system. The workflow kind of looks like following:

GitOps

I am developing a Terraform Provider for Azure DevOps that helps me use Terraform for provisioning Azure DevOps projects, Teams and members. In this article I will share how I am building it.

Note
This provider doesn't implement the complete set of 
Azure DevOps REST APIs. 
Its limited to only projects, teams and member associations. 
It's not recommended to use it in production scenarios.

Terraform Provider

Terrafom is an amazing tool that lets you define your infrastructure as code. Under the hood it’s an incredibly powerful state machine that makes API requests and marshals resources. Terraform has lots of providers – almost for every major cloud – out there. Including many other systems – like Kubernetes, Palo-Alto Networks etc.

In nutshell if any system has REST API that can be manipulated with Terraform Provider. Azure DevOps also has a terraform provider – which doesn’t currently provide resources to create Teams and members. Hence, I am writing my own – shamelessly using/stealing the Microsoft’s Terraform provider (referenced above) for project creation.

Setting up GO Environment

Terraform Providers and plugins are binaries that Terraform communicates during runtime via RPC. It’s theoretically possible to write a provider in any language, but to be honest, I haven’t come across any providers that were written other languages than GO. Terraform provide helper libraries in Go to aid in writing and testing providers.

I am developing in Windows 10 and didn’t want to install GO on my local machine. Containers come to rescue of course. I am using the “Remote development” extension in VS Code. This extension allows me to keep the source code in local machine and compile, build the source code in a container like Magic!

remote

Figure: Remote Development extension in VSCode – running container to build local repository.

Creating the provider

To create a Terraform provider we need to write the logic for managing the Creation, Reading, Updating and Deletion (CRUD) of a resource (i.e. Azure DevOps project, Team and members in this scenario) and Terraform will take care of the rest; state, locking, templating language and managing the lifecycle of the resources. Here in this repository I have a minimum implementation that supports creating Azure DevOps projects, Teams and its members.

First of all we define our provider and resources in main.go file.

package main
import (
"github.com/hashicorp/terraform/plugin"
"github.com/hashicorp/terraform/terraform"
)
func main() {
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: func() terraform.ResourceProvider {
return Provider()
},
})
}

view raw
main.go
hosted with ❤ by GitHub

Next to that, we will define the provider schema (the attributes it supports as input and outputs, resources etc.)

package main
import (
"github.com/hashicorp/terraform/helper/schema"
)
// Provider – The top level Azure DevOps Provider definition.
func Provider() *schema.Provider {
p := &schema.Provider{
ResourcesMap: map[string]*schema.Resource{
"azuredevops_pipeline": resourcePipeline(),
"azuredevops_project": resourceProject(),
"azuredevops_team": resourceTeam(),
"azuredevops_serviceendpoint": resourceServiceEndpoint(),
},
Schema: map[string]*schema.Schema{
"org_service_url": {
Type: schema.TypeString,
Required: true,
DefaultFunc: schema.EnvDefaultFunc("AZDO_ORG_SERVICE_URL", nil),
Description: "The url of the Azure DevOps instance which should be used.",
},
"personal_access_token": {
Type: schema.TypeString,
Required: true,
DefaultFunc: schema.EnvDefaultFunc("AZDO_PERSONAL_ACCESS_TOKEN", nil),
Description: "The personal access token which should be used.",
},
},
}
p.ConfigureFunc = providerConfigure(p)
return p
}
func providerConfigure(p *schema.Provider) schema.ConfigureFunc {
return func(d *schema.ResourceData) (interface{}, error) {
client, err := getAzdoClient(d.Get("personal_access_token").(string), d.Get("org_service_url").(string))
return client, err
}
}

view raw
provider.go
hosted with ❤ by GitHub

We are using Azure DevOps personal Access token to communicate to the Azure DevOps REST API. The GO client for Azure DevOps from Microsoft – which is used as dependency, immensely simplified the implementation and also helped learning the flow.

Now defining the “team” resource as following:

package main
import (
"fmt"
"github.com/microsoft/terraform-provider-azuredevops/utils/converter"
"github.com/microsoft/terraform-provider-azuredevops/utils/tfhelper"
"os"
"log"
"github.com/google/uuid"
"github.com/hashicorp/terraform/helper/schema"
"github.com/microsoft/azure-devops-go-api/azuredevops/core"
"github.com/microsoft/azure-devops-go-api/azuredevops/graph"
)
func resourceTeam() *schema.Resource {
return &schema.Resource{
Create: resourceTeamCreate,
Read: resourceTeamRead,
Update: resourceTeamUpdate,
Delete: resourceTeamDelete,
//https://godoc.org/github.com/hashicorp/terraform/helper/schema#Schema
Schema: map[string]*schema.Schema{
"project_id": &schema.Schema{
Type: schema.TypeString,
Required: true,
DiffSuppressFunc: tfhelper.DiffFuncSupressCaseSensitivity,
},
"name": &schema.Schema{
Type: schema.TypeString,
ForceNew: true,
Required: true,
DiffSuppressFunc: tfhelper.DiffFuncSupressCaseSensitivity,
},
"description": &schema.Schema{
Type: schema.TypeString,
Optional: true,
Default: "",
},
"members": {
Type: schema.TypeSet,
Optional: true,
Computed: true,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"name": {
Type: schema.TypeString,
Required: true,
},
"admin": {
Type: schema.TypeBool,
Required: true,
Sensitive: true,
},
},
},
},
},
}
}

view raw
resource_team.go
hosted with ❤ by GitHub

That’s all for declaring, now implementing the CRUD methods in resource providers. The full source code is in GitHub.

We can compile the provider application using following command:

> GOOS=windows GOARCH=amd64 go build -o terraform-provider-azuredevops.exe

As I am using Dabian docker image for GoLang I need to specify my target OS (GOOS=windows) and CPU Architecture (GOARCH=amd64) when I build the provider. This will produce the terraform provider for Azure DevOps executable.

Although it’s executable, it’s not meant to launch directly from command prompt. Instead, I will copy it to “%APPDATA%\ terraform.d\plugins\windows_amd64” folder of my machine.

Terraform Script for Azrue DevOps

Now we can write the Terraform file (.tf) that will describe the Azure DevOps Project, Team and members etc.

Terraform

With this terraform file, we can now launch the following command to initialize our terraform environment.

init

The terraform init command is used to initialize a working directory containing Terraform configuration files. This is the first command that should be run after writing a new Terraform configuration or cloning an existing one from version control.

Terraform plan

The terraform plan command is used to create an execution plan. Terraform performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the configuration files. This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

PLan

Figure: terraform plan output – shows exactly what is going to happen if we apply these changes to Azure DevOps

Terraform apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan. We will launch it with an “-auto-approve” flag to assert the approval prompt.

apply

Now we can go to our Azure DevOps and sure enough there’s a new project created with the configuration as we scripted in Terraform file.

Taking it further

Now we can check in the terraform file (main.tf above) into an Azure DevOps repository and put a Branch policy to it. That will force any changes (such as creating new projects, adding removing team members) would requrie a Pull-Request and needs to be reviewed by peers (four-eyes principles). Once Pull-Requests are approved, a simple Azure Pipeline can trigger that does the terraform apply. And I have my workflow automated  and I also have nice histories in GIT – which records the purpose of any changes made in past.

Thanks for reading!

Linkerd in Azure Kubernetes Service cluster

In this article I would document my journey on setting up Linkerd Service Mesh on Azure Kubernetes service.

Background

I have a tiny Kubernetes cluster. I run some workload there, some are useful, others are just try-out, fun stuffs. I have few services that need to talk to each other. I do not have a lot of traffic to be honest, but I sometimes curiously run Apache ab to simulate load and see how my services perform under stress. Until very recently I was using a messaging (basically a pub-sub) pattern to create reactive service-to-service communication. Which works great, but often comes with a latency. I can only imagine, if I were to run these service to service communication for a mission critical high-traffic performance-driven scenario (an online game for instance), this model won’t fly well. There comes the need for a service-to-service communication pattern in cluster.

What’s big deal? We can have REST calls between services, even can implement gRPC for that matter. The issue is things behaves different at scale. When many services talks to many others, nodes fail in between, network address of PODs changes, new PODs show up, some goes down, figuring out where the service sits becomes quite a challenging task.

Then Kubernetes comes to rescue, Kubernetes provides “service”, that gives us service discovery out of the box. Which is awesome. Not all issues disappeared though. Services in a cluster need fault-tolerances, traceability and most importantly, “observability”.  Circuit-breakers, retry-logics etc. implementing them for each service is again a challenge. This is exactly the Service Mesh addresses.

Service mesh

From thoughtworks radar:

Service mesh is an approach to operating a secure, fast and reliable microservices ecosystem. It has been an important steppingstone in making it easier to adopt microservices at scale. It offers discovery, security, tracing, monitoring and failure handling. It provides these cross-functional capabilities without the need for a shared asset such as an API gateway or baking libraries into each service. A typical implementation involves lightweight reverse-proxy processes, aka sidecars, deployed alongside each service process in a separate container. Sidecars intercept the inbound and outbound traffic of each service and provide cross-functional capabilities mentioned above.

Some of us might remember Aspect Oriented programming (AOP) – where we used to separate cross cutting concerns from our core-business-concerns. Service mesh is no different. They isolate (in a separate container) these networking and fault-tolerance concerns from the core-capabilities (also running in container).

Linkerd

There are quite several service mesh solutions out there – all suitable to run in Kubernetes. I have used earlier Envoy and Istio. They work great in Kubernetes as well as VM hosted clusters. However, I must admit, I developed a preference for Linkerd since I discovered it. Let’s briefly look at how Linkerd works. Imagine the following two services, Service A and Service B. Service A talks to Service B.

service-2-service

When Linkerd installed, it works like an interceptor between all the communication between services. Linkerd uses sidecar pattern to proxy the communication by updating the KubeProxy IP Table.

Linkerd-architecture.png

Linkerd implants two sidecar containers in our PODs. The init container configures the IP table so the incoming and outgoing TCP traffics flow through the Linkerd Proxy container. The proxy container is the data plane that does the actual interception and all the other fault-tolerance goodies.

Primary reason behind my Linkerd preferences are performance and simplicity. Ivan Sim has done performance benchmarking with Linkerd and Istio:

Both the Linkerd2-meshed setup and Istio-meshed setup experienced higher latency and lower throughput, when compared with the baseline setup. The latency incurred in the Istio-meshed setup was higher than that observed in the Linkerd2-meshed setup. The Linkerd2-meshed setup was able to handle higher HTTP and GRPC ping throughput than the Istio-meshed setup.

Cluster provision

Spinning up AKS is easy as pie these days. We can use Azure Resource Manager Template or Terraform for that. I have used Terraform to generate that.

resource "azurerm_resource_group" "cloudoven" {
name = "cloudoven"
location = "West Europe"
}
resource "azurerm_kubernetes_cluster" "cloudovenaks" {
name = "cloudovenaks"
location = "${azurerm_resource_group.cloudoven.location}"
resource_group_name = "${azurerm_resource_group.cloudoven.name}"
dns_prefix = "cloudovenaks"
agent_pool_profile {
name = "default"
count = 1
vm_size = "Standard_D1_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
agent_pool_profile {
name = "pool2"
count = 1
vm_size = "Standard_D2_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "98e758f8r-f734-034a-ac98-0404c500e010"
client_secret = "Jk==3djk(efd31kla934-=="
}
tags = {
Environment = "Production"
}
}
output "client_certificate" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config.0.client_certificate}"
}
output "kube_config" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config_raw}"
}

view raw
Kuberentes-iac
hosted with ❤ by GitHub

Service deployment

This is going to take few minutes and then we have a cluster. We will use the canonical emojivoto app (“buoyantio/emojivoto-emoji-svc:v8”) to test our Linkerd installation. Here’s the Kubernetes manifest file for that.

apiVersion: v1
kind: Namespace
metadata:
name: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: emoji
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: voting
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: web
namespace: emojivoto
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: emoji
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: emoji-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: emoji-svc
spec:
serviceAccountName: emoji
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-emoji-svc:v8
name: emoji-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: emoji-svc
namespace: emojivoto
spec:
selector:
app: emoji-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: voting
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: voting-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: voting-svc
spec:
serviceAccountName: voting
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-voting-svc:v8
name: voting-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: voting-svc
namespace: emojivoto
spec:
selector:
app: voting-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: web
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: web-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: web-svc
spec:
serviceAccountName: web
containers:
env:
name: WEB_PORT
value: "80"
name: EMOJISVC_HOST
value: emoji-svc.emojivoto:8080
name: VOTINGSVC_HOST
value: voting-svc.emojivoto:8080
name: INDEX_BUNDLE
value: dist/index_bundle.js
image: buoyantio/emojivoto-web:v8
name: web-svc
ports:
containerPort: 80
name: http
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: web-svc
namespace: emojivoto
spec:
type: LoadBalancer
selector:
app: web-svc
ports:
name: http
port: 80
targetPort: 80
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: vote-bot
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: vote-bot
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vote-bot
spec:
containers:
command:
emojivoto-vote-bot
env:
name: WEB_HOST
value: web-svc.emojivoto:80
image: buoyantio/emojivoto-web:v8
name: vote-bot
resources:
requests:
cpu: 10m
status: {}

view raw
emoji-manifest.yml
hosted with ❤ by GitHub

With this IaC – we can run Terraform apply to provision our AKS cluster in Azure.

Azure Pipeline

Let’s create a pipeline for the service deployment. The easiest way to do that is to create a service connection to our AKS cluster. We go to the project settings in Azure DevOps project, pick Service connections and create a new service connection of type “Kubernetes connection”.

Azure DevOps connection

Installing Linkerd

We will create a pipeline that installs Linkerd into the AKS cluster. Azure Pipeline now offers “pipeline-as-code” – which is just an YAML file that describes the steps need to be performed when the pipeline is triggered. We will use the following pipeline-as-code:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl get'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: get
arguments: nodes
script: |
curl -sL https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
linkerd version
linkerd check –pre
linkerd install | kubectl apply -f –
linkerd check
displayName: 'Linkerd – Installation'

We can at this point trigger the pipeline to install Linkerd into the AKS cluster.

Linkerd installation (2)

Deployment of PODs and services

Let’s create another pipeline as code that deploys all the services and deployment resources to AKS using the following Kubernetes manifest file:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml

In Azure Portal we can already see our services running:

Azure KS

Also in Kubernetes Dashboard:

Kub1

We have got our services running – but they are not really affected by Linkerd yet. We will add another step into the build pipeline to tell Linkerd to do its magic.

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml
script: 'src/services/emojivoto/all.yml | linkerd inject – | kubectl apply -f –'
displayName: 'Inject Linkerd'

Next thing, we trigger the pipeline and put some traffic into the service that we have just deployed. The emoji service is simulating some service to service invocation scenarios and now it’s time for us to open the Linkerd dashboard to inspect all the distributed traces and many other useful matrix to look at.

linkerd-censored

We can also see kind of an application map – in a graphical way to understand which service is calling who and what is request latencies etc.

linkerd-graph

Even fascinating, Linkerd provides some drill-down to the communications in Grafana Dashboard.

ezgif.com-gif-maker.gif

Conclusion

I have enjoyed a lot setting it up and see the outcome and wanted to share my experience with it. If you are looking into Service Mesh and read this post, I strongly encourage to give Linkerd a go, it’s awesome!

Thanks for reading.

CloudOven – Terraform at ease!

TL;DR:

  • URL: CloudOven 

  • Use Google account or sign-up 
  • Google Chrome please! (I’ve not tested on other browsers yet)

e2e

Background

In recent years I have spent fair amount of time in design and implementation of Infrastructure as code in larger enterprise context. Terraform seemed to be a tool of choice when it comes to preserve the uniformity in Infrastructure as code targeting multiple cloud providers. It is rapidly becoming a de facto choice for creating and managing cloud infrastructures by writing declarative definitions. It’s popular because the syntax of its files is quite readable and because it supports several cloud providers while making no attempt to provide an artificial abstraction across those providers. The active community will add support for the latest features from most cloud providers.

However, rolling out Terraform in many enterprises has its own barrier to face. Albeit the syntax (HCL) is neat, but not every developers or Infrastructure operators in organizations finds it easy. There’s a learning curve and often many of us lose momentum discovering the learning effort. I believe if we could make the initial ramp-up easier more people would play with it.

That’s one of my motivation for this post, following is the other one.

Blazor meets Terraform

Lately I was learning Blazor – the new client-side technology from Microsoft. Like many others, I find one effective way learning a new technology by creating/building solution to a problem. I have decided to build a user interface that will help creating terraform scripts easier. I will share my journey in this post.

Resource Discovery in Terraform Providers

Terraform is powerful for its providers. You will find Terraform providers for all major cloud providers (Azure, AWS, Google etc.). The providers then allow us to define “resource” and “data source” in Terraform scripts. These resource and data source have arguments and attributes that one must know while creating terraform files. Luckily, they are documented nicely in Terraform site. However, it still requires us to jump back and forth to the documentation site and terraform file editor (i.e. VSCode).

Azure-Discovery

To make this experience easier, I wrote a crawler application that downloads the terraform providers (I am doing it for Azure, AWS and google for now) and discovers the attributes and arguments for each and every resource and data source. I also try to extract the documentation for every attributes and arguments from the terraform documentation site with a layman parsing (not 100% accurate but works for majority. Something I will improve soon).

GoogleAWS-discovery

This process generates JSON structure for each resource and data source, enriches them with the documentation and stores them in an Azure Blob Storage.

Building Infrastructure as code

Now that I have a structured data store with all resources and data sources for any terraform provider, I can leverage that building a user interface on top of it. To keep things a bit organized, I started with a concept of “project”.

workflow
The workflow

Project

I can start by creating a project (well, it can be a product too, but let’s not get to that debate). Project is merely a logical boundary here.

Blueprint

Within a project I can create Blueprint(s). Blueprint(s) are the entity that retains the elements of the infrastructure that we are aiming to create. For instance, a Blueprint targets to a Cloud provider (i.e. Azure). Then I can create the elements (resource and data sources) within the blueprint (i.e. Azure Web App, Cosmos DB etc.).

provider-configuration

Blueprints keeps the base structure of all the infrastructure elements. It allows defining variables (plain and simple terraform variables) so the actual values can vary in different environments (dev, test, pre-production, production etc.).

Once I am happy with the blueprint, I can download them as a zip – that contains the terraform scripts (main.tf and variable.tf). That’s it, we have our infrastructure as code in Terraform. I can execute them on a local development machine or check them in to source control – whatever I prefer.

storage_account

One can stop here and keep using the blueprint feature to generate Infrastructure as code. That’s what it is for. However, the next features are just to make the overall experience of running terraform a bit easier.

Environments

Next to blueprint, we can create as many environments we want. Again, just a logical entity to keep isolation of actual deployment for different environments.

Deployments

Deployment entity is the glue that ties a blueprint to a specific environment. For instance, I can define a blueprint for “order management” service (or micro-service maybe?), create an environment as “test” and then create a “deployment” for “order management” on “test”. This is where I can define constant values to the blueprint variable that are specific to the test environment.

Terraform State

Perhaps the most important aspect the deployment entity holds is the terraform state management. Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. This state is stored by default in a local file named “terraform.tfstate”, but it can also be stored remotely, which works better in a team environment. Defining the state properties (varies in different cloud providers) in deployment entity makes the remote state management easier – specifically in team environment. It will configure the remote state to the appropriate remote backend. For instance, when the blueprint cloud provider is set to Azure, it will configure Azure Storage account as terraform state remote backend, for AWS it will pick S3 automatically.

e2e

Terraform plan

Once we have deployment entity configured, we can directly from the user interface run “terraform plan”. The terraform plan command creates an execution plan. Unless explicitly disabled, it performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the blueprint. This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

Terraform apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan. Like “plan”, the “apply” command can also be issued directly from the user interface.

Terraform plan and apply both are issued in an isolated docker container and the output is captured and displayed back to the user interface. However, there’s a cost associated running docker containers on cloud, therefore, it’s disabled in the public site.

Final thoughts

It was fun to write a tool like this. I recommend you give it a go. Especially if you are stepping into Terraform. It can also be helpful for experienced Terraform developers – specifically with the on-screen documenation, type inferance and discovery features.

Some features, I have working progress:

  • Ability to define policy for each resources and data types
  • Save a Blueprint as custom module

Stay tuned!