Linkerd in Azure Kubernetes Service cluster

In this article I would document my journey on setting up Linkerd Service Mesh on Azure Kubernetes service.

Background

I have a tiny Kubernetes cluster. I run some workload there, some are useful, others are just try-out, fun stuffs. I have few services that need to talk to each other. I do not have a lot of traffic to be honest, but I sometimes curiously run Apache ab to simulate load and see how my services perform under stress. Until very recently I was using a messaging (basically a pub-sub) pattern to create reactive service-to-service communication. Which works great, but often comes with a latency. I can only imagine, if I were to run these service to service communication for a mission critical high-traffic performance-driven scenario (an online game for instance), this model won’t fly well. There comes the need for a service-to-service communication pattern in cluster.

What’s big deal? We can have REST calls between services, even can implement gRPC for that matter. The issue is things behaves different at scale. When many services talks to many others, nodes fail in between, network address of PODs changes, new PODs show up, some goes down, figuring out where the service sits becomes quite a challenging task.

Then Kubernetes comes to rescue, Kubernetes provides “service”, that gives us service discovery out of the box. Which is awesome. Not all issues disappeared though. Services in a cluster need fault-tolerances, traceability and most importantly, “observability”.  Circuit-breakers, retry-logics etc. implementing them for each service is again a challenge. This is exactly the Service Mesh addresses.

Service mesh

From thoughtworks radar:

Service mesh is an approach to operating a secure, fast and reliable microservices ecosystem. It has been an important steppingstone in making it easier to adopt microservices at scale. It offers discovery, security, tracing, monitoring and failure handling. It provides these cross-functional capabilities without the need for a shared asset such as an API gateway or baking libraries into each service. A typical implementation involves lightweight reverse-proxy processes, aka sidecars, deployed alongside each service process in a separate container. Sidecars intercept the inbound and outbound traffic of each service and provide cross-functional capabilities mentioned above.

Some of us might remember Aspect Oriented programming (AOP) – where we used to separate cross cutting concerns from our core-business-concerns. Service mesh is no different. They isolate (in a separate container) these networking and fault-tolerance concerns from the core-capabilities (also running in container).

Linkerd

There are quite several service mesh solutions out there – all suitable to run in Kubernetes. I have used earlier Envoy and Istio. They work great in Kubernetes as well as VM hosted clusters. However, I must admit, I developed a preference for Linkerd since I discovered it. Let’s briefly look at how Linkerd works. Imagine the following two services, Service A and Service B. Service A talks to Service B.

service-2-service

When Linkerd installed, it works like an interceptor between all the communication between services. Linkerd uses sidecar pattern to proxy the communication by updating the KubeProxy IP Table.

Linkerd-architecture.png

Linkerd implants two sidecar containers in our PODs. The init container configures the IP table so the incoming and outgoing TCP traffics flow through the Linkerd Proxy container. The proxy container is the data plane that does the actual interception and all the other fault-tolerance goodies.

Primary reason behind my Linkerd preferences are performance and simplicity. Ivan Sim has done performance benchmarking with Linkerd and Istio:

Both the Linkerd2-meshed setup and Istio-meshed setup experienced higher latency and lower throughput, when compared with the baseline setup. The latency incurred in the Istio-meshed setup was higher than that observed in the Linkerd2-meshed setup. The Linkerd2-meshed setup was able to handle higher HTTP and GRPC ping throughput than the Istio-meshed setup.

Cluster provision

Spinning up AKS is easy as pie these days. We can use Azure Resource Manager Template or Terraform for that. I have used Terraform to generate that.

resource "azurerm_resource_group" "cloudoven" {
name = "cloudoven"
location = "West Europe"
}
resource "azurerm_kubernetes_cluster" "cloudovenaks" {
name = "cloudovenaks"
location = "${azurerm_resource_group.cloudoven.location}"
resource_group_name = "${azurerm_resource_group.cloudoven.name}"
dns_prefix = "cloudovenaks"
agent_pool_profile {
name = "default"
count = 1
vm_size = "Standard_D1_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
agent_pool_profile {
name = "pool2"
count = 1
vm_size = "Standard_D2_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "98e758f8r-f734-034a-ac98-0404c500e010"
client_secret = "Jk==3djk(efd31kla934-=="
}
tags = {
Environment = "Production"
}
}
output "client_certificate" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config.0.client_certificate}"
}
output "kube_config" {
value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config_raw}"
}

view raw
Kuberentes-iac
hosted with ❤ by GitHub

Service deployment

This is going to take few minutes and then we have a cluster. We will use the canonical emojivoto app (“buoyantio/emojivoto-emoji-svc:v8”) to test our Linkerd installation. Here’s the Kubernetes manifest file for that.

apiVersion: v1
kind: Namespace
metadata:
name: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: emoji
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: voting
namespace: emojivoto
kind: ServiceAccount
apiVersion: v1
metadata:
name: web
namespace: emojivoto
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: emoji
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: emoji-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: emoji-svc
spec:
serviceAccountName: emoji
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-emoji-svc:v8
name: emoji-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: emoji-svc
namespace: emojivoto
spec:
selector:
app: emoji-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: voting
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: voting-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: voting-svc
spec:
serviceAccountName: voting
containers:
env:
name: GRPC_PORT
value: "8080"
image: buoyantio/emojivoto-voting-svc:v8
name: voting-svc
ports:
containerPort: 8080
name: grpc
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: voting-svc
namespace: emojivoto
spec:
selector:
app: voting-svc
clusterIP: None
ports:
name: grpc
port: 8080
targetPort: 8080
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: web
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: web-svc
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: web-svc
spec:
serviceAccountName: web
containers:
env:
name: WEB_PORT
value: "80"
name: EMOJISVC_HOST
value: emoji-svc.emojivoto:8080
name: VOTINGSVC_HOST
value: voting-svc.emojivoto:8080
name: INDEX_BUNDLE
value: dist/index_bundle.js
image: buoyantio/emojivoto-web:v8
name: web-svc
ports:
containerPort: 80
name: http
resources:
requests:
cpu: 100m
status: {}
apiVersion: v1
kind: Service
metadata:
name: web-svc
namespace: emojivoto
spec:
type: LoadBalancer
selector:
app: web-svc
ports:
name: http
port: 80
targetPort: 80
apiVersion: apps/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
name: vote-bot
namespace: emojivoto
spec:
replicas: 1
selector:
matchLabels:
app: vote-bot
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: vote-bot
spec:
containers:
command:
emojivoto-vote-bot
env:
name: WEB_HOST
value: web-svc.emojivoto:80
image: buoyantio/emojivoto-web:v8
name: vote-bot
resources:
requests:
cpu: 10m
status: {}

view raw
emoji-manifest.yml
hosted with ❤ by GitHub

With this IaC – we can run Terraform apply to provision our AKS cluster in Azure.

Azure Pipeline

Let’s create a pipeline for the service deployment. The easiest way to do that is to create a service connection to our AKS cluster. We go to the project settings in Azure DevOps project, pick Service connections and create a new service connection of type “Kubernetes connection”.

Azure DevOps connection

Installing Linkerd

We will create a pipeline that installs Linkerd into the AKS cluster. Azure Pipeline now offers “pipeline-as-code” – which is just an YAML file that describes the steps need to be performed when the pipeline is triggered. We will use the following pipeline-as-code:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl get'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: get
arguments: nodes
script: |
curl -sL https://run.linkerd.io/install | sh
export PATH=$PATH:$HOME/.linkerd2/bin
linkerd version
linkerd check –pre
linkerd install | kubectl apply -f –
linkerd check
displayName: 'Linkerd – Installation'

We can at this point trigger the pipeline to install Linkerd into the AKS cluster.

Linkerd installation (2)

Deployment of PODs and services

Let’s create another pipeline as code that deploys all the services and deployment resources to AKS using the following Kubernetes manifest file:

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml

In Azure Portal we can already see our services running:

Azure KS

Also in Kubernetes Dashboard:

Kub1

We have got our services running – but they are not really affected by Linkerd yet. We will add another step into the build pipeline to tell Linkerd to do its magic.

pool:
name: Hosted Ubuntu 1604
steps:
task: KubectlInstaller@0
displayName: 'Install Kubectl latest'
task: Kubernetes@1
displayName: 'kubectl apply'
inputs:
kubernetesServiceEndpoint: CloudOvenKubernetes
command: apply
useConfigurationFile: true
configuration: src/services/emojivoto/all.yml
script: 'src/services/emojivoto/all.yml | linkerd inject – | kubectl apply -f –'
displayName: 'Inject Linkerd'

Next thing, we trigger the pipeline and put some traffic into the service that we have just deployed. The emoji service is simulating some service to service invocation scenarios and now it’s time for us to open the Linkerd dashboard to inspect all the distributed traces and many other useful matrix to look at.

linkerd-censored

We can also see kind of an application map – in a graphical way to understand which service is calling who and what is request latencies etc.

linkerd-graph

Even fascinating, Linkerd provides some drill-down to the communications in Grafana Dashboard.

ezgif.com-gif-maker.gif

Conclusion

I have enjoyed a lot setting it up and see the outcome and wanted to share my experience with it. If you are looking into Service Mesh and read this post, I strongly encourage to give Linkerd a go, it’s awesome!

Thanks for reading.

Author: Moim Hossain

I develop software that solve problems! All opinions are my own!

One thought on “Linkerd in Azure Kubernetes Service cluster”

  1. That is really fascinating, You are a very skilled blogger. I’ve joined your feed and look forward to in search of extra of your excellent post. Additionally, I’ve shared your web site in my social networks!

    Like

Leave a Reply to Javier Czyz Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s