In this article I would document my journey on setting up Linkerd Service Mesh on Azure Kubernetes service.
Background
I have a tiny Kubernetes cluster. I run some workload there, some are useful, others are just try-out, fun stuffs. I have few services that need to talk to each other. I do not have a lot of traffic to be honest, but I sometimes curiously run Apache ab to simulate load and see how my services perform under stress. Until very recently I was using a messaging (basically a pub-sub) pattern to create reactive service-to-service communication. Which works great, but often comes with a latency. I can only imagine, if I were to run these service to service communication for a mission critical high-traffic performance-driven scenario (an online game for instance), this model won’t fly well. There comes the need for a service-to-service communication pattern in cluster.
What’s big deal? We can have REST calls between services, even can implement gRPC for that matter. The issue is things behaves different at scale. When many services talks to many others, nodes fail in between, network address of PODs changes, new PODs show up, some goes down, figuring out where the service sits becomes quite a challenging task.
Then Kubernetes comes to rescue, Kubernetes provides “service”, that gives us service discovery out of the box. Which is awesome. Not all issues disappeared though. Services in a cluster need fault-tolerances, traceability and most importantly, “observability”. Circuit-breakers, retry-logics etc. implementing them for each service is again a challenge. This is exactly the Service Mesh addresses.
Service mesh
From thoughtworks radar:
Service mesh is an approach to operating a secure, fast and reliable microservices ecosystem. It has been an important steppingstone in making it easier to adopt microservices at scale. It offers discovery, security, tracing, monitoring and failure handling. It provides these cross-functional capabilities without the need for a shared asset such as an API gateway or baking libraries into each service. A typical implementation involves lightweight reverse-proxy processes, aka sidecars, deployed alongside each service process in a separate container. Sidecars intercept the inbound and outbound traffic of each service and provide cross-functional capabilities mentioned above.
Some of us might remember Aspect Oriented programming (AOP) – where we used to separate cross cutting concerns from our core-business-concerns. Service mesh is no different. They isolate (in a separate container) these networking and fault-tolerance concerns from the core-capabilities (also running in container).
Linkerd
There are quite several service mesh solutions out there – all suitable to run in Kubernetes. I have used earlier Envoy and Istio. They work great in Kubernetes as well as VM hosted clusters. However, I must admit, I developed a preference for Linkerd since I discovered it. Let’s briefly look at how Linkerd works. Imagine the following two services, Service A and Service B. Service A talks to Service B.
When Linkerd installed, it works like an interceptor between all the communication between services. Linkerd uses sidecar pattern to proxy the communication by updating the KubeProxy IP Table.
Linkerd implants two sidecar containers in our PODs. The init container configures the IP table so the incoming and outgoing TCP traffics flow through the Linkerd Proxy container. The proxy container is the data plane that does the actual interception and all the other fault-tolerance goodies.
Primary reason behind my Linkerd preferences are performance and simplicity. Ivan Sim has done performance benchmarking with Linkerd and Istio:
Both the Linkerd2-meshed setup and Istio-meshed setup experienced higher latency and lower throughput, when compared with the baseline setup. The latency incurred in the Istio-meshed setup was higher than that observed in the Linkerd2-meshed setup. The Linkerd2-meshed setup was able to handle higher HTTP and GRPC ping throughput than the Istio-meshed setup.
Cluster provision
Spinning up AKS is easy as pie these days. We can use Azure Resource Manager Template or Terraform for that. I have used Terraform to generate that.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| resource "azurerm_resource_group" "cloudoven" { | |
| name = "cloudoven" | |
| location = "West Europe" | |
| } | |
| resource "azurerm_kubernetes_cluster" "cloudovenaks" { | |
| name = "cloudovenaks" | |
| location = "${azurerm_resource_group.cloudoven.location}" | |
| resource_group_name = "${azurerm_resource_group.cloudoven.name}" | |
| dns_prefix = "cloudovenaks" | |
| agent_pool_profile { | |
| name = "default" | |
| count = 1 | |
| vm_size = "Standard_D1_v2" | |
| os_type = "Linux" | |
| os_disk_size_gb = 30 | |
| } | |
| agent_pool_profile { | |
| name = "pool2" | |
| count = 1 | |
| vm_size = "Standard_D2_v2" | |
| os_type = "Linux" | |
| os_disk_size_gb = 30 | |
| } | |
| service_principal { | |
| client_id = "98e758f8r-f734-034a-ac98-0404c500e010" | |
| client_secret = "Jk==3djk(efd31kla934-==" | |
| } | |
| tags = { | |
| Environment = "Production" | |
| } | |
| } | |
| output "client_certificate" { | |
| value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config.0.client_certificate}" | |
| } | |
| output "kube_config" { | |
| value = "${azurerm_kubernetes_cluster.cloudovenaks.kube_config_raw}" | |
| } |
Service deployment
This is going to take few minutes and then we have a cluster. We will use the canonical emojivoto app (“buoyantio/emojivoto-emoji-svc:v8”) to test our Linkerd installation. Here’s the Kubernetes manifest file for that.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| — | |
| apiVersion: v1 | |
| kind: Namespace | |
| metadata: | |
| name: emojivoto | |
| — | |
| kind: ServiceAccount | |
| apiVersion: v1 | |
| metadata: | |
| name: emoji | |
| namespace: emojivoto | |
| — | |
| kind: ServiceAccount | |
| apiVersion: v1 | |
| metadata: | |
| name: voting | |
| namespace: emojivoto | |
| — | |
| kind: ServiceAccount | |
| apiVersion: v1 | |
| metadata: | |
| name: web | |
| namespace: emojivoto | |
| — | |
| apiVersion: apps/v1beta1 | |
| kind: Deployment | |
| metadata: | |
| creationTimestamp: null | |
| name: emoji | |
| namespace: emojivoto | |
| spec: | |
| replicas: 1 | |
| selector: | |
| matchLabels: | |
| app: emoji-svc | |
| strategy: {} | |
| template: | |
| metadata: | |
| creationTimestamp: null | |
| labels: | |
| app: emoji-svc | |
| spec: | |
| serviceAccountName: emoji | |
| containers: | |
| – env: | |
| – name: GRPC_PORT | |
| value: "8080" | |
| image: buoyantio/emojivoto-emoji-svc:v8 | |
| name: emoji-svc | |
| ports: | |
| – containerPort: 8080 | |
| name: grpc | |
| resources: | |
| requests: | |
| cpu: 100m | |
| status: {} | |
| — | |
| apiVersion: v1 | |
| kind: Service | |
| metadata: | |
| name: emoji-svc | |
| namespace: emojivoto | |
| spec: | |
| selector: | |
| app: emoji-svc | |
| clusterIP: None | |
| ports: | |
| – name: grpc | |
| port: 8080 | |
| targetPort: 8080 | |
| — | |
| apiVersion: apps/v1beta1 | |
| kind: Deployment | |
| metadata: | |
| creationTimestamp: null | |
| name: voting | |
| namespace: emojivoto | |
| spec: | |
| replicas: 1 | |
| selector: | |
| matchLabels: | |
| app: voting-svc | |
| strategy: {} | |
| template: | |
| metadata: | |
| creationTimestamp: null | |
| labels: | |
| app: voting-svc | |
| spec: | |
| serviceAccountName: voting | |
| containers: | |
| – env: | |
| – name: GRPC_PORT | |
| value: "8080" | |
| image: buoyantio/emojivoto-voting-svc:v8 | |
| name: voting-svc | |
| ports: | |
| – containerPort: 8080 | |
| name: grpc | |
| resources: | |
| requests: | |
| cpu: 100m | |
| status: {} | |
| — | |
| apiVersion: v1 | |
| kind: Service | |
| metadata: | |
| name: voting-svc | |
| namespace: emojivoto | |
| spec: | |
| selector: | |
| app: voting-svc | |
| clusterIP: None | |
| ports: | |
| – name: grpc | |
| port: 8080 | |
| targetPort: 8080 | |
| — | |
| apiVersion: apps/v1beta1 | |
| kind: Deployment | |
| metadata: | |
| creationTimestamp: null | |
| name: web | |
| namespace: emojivoto | |
| spec: | |
| replicas: 1 | |
| selector: | |
| matchLabels: | |
| app: web-svc | |
| strategy: {} | |
| template: | |
| metadata: | |
| creationTimestamp: null | |
| labels: | |
| app: web-svc | |
| spec: | |
| serviceAccountName: web | |
| containers: | |
| – env: | |
| – name: WEB_PORT | |
| value: "80" | |
| – name: EMOJISVC_HOST | |
| value: emoji-svc.emojivoto:8080 | |
| – name: VOTINGSVC_HOST | |
| value: voting-svc.emojivoto:8080 | |
| – name: INDEX_BUNDLE | |
| value: dist/index_bundle.js | |
| image: buoyantio/emojivoto-web:v8 | |
| name: web-svc | |
| ports: | |
| – containerPort: 80 | |
| name: http | |
| resources: | |
| requests: | |
| cpu: 100m | |
| status: {} | |
| — | |
| apiVersion: v1 | |
| kind: Service | |
| metadata: | |
| name: web-svc | |
| namespace: emojivoto | |
| spec: | |
| type: LoadBalancer | |
| selector: | |
| app: web-svc | |
| ports: | |
| – name: http | |
| port: 80 | |
| targetPort: 80 | |
| — | |
| apiVersion: apps/v1beta1 | |
| kind: Deployment | |
| metadata: | |
| creationTimestamp: null | |
| name: vote-bot | |
| namespace: emojivoto | |
| spec: | |
| replicas: 1 | |
| selector: | |
| matchLabels: | |
| app: vote-bot | |
| strategy: {} | |
| template: | |
| metadata: | |
| creationTimestamp: null | |
| labels: | |
| app: vote-bot | |
| spec: | |
| containers: | |
| – command: | |
| – emojivoto-vote-bot | |
| env: | |
| – name: WEB_HOST | |
| value: web-svc.emojivoto:80 | |
| image: buoyantio/emojivoto-web:v8 | |
| name: vote-bot | |
| resources: | |
| requests: | |
| cpu: 10m | |
| status: {} | |
| — |
With this IaC – we can run Terraform apply to provision our AKS cluster in Azure.
Azure Pipeline
Let’s create a pipeline for the service deployment. The easiest way to do that is to create a service connection to our AKS cluster. We go to the project settings in Azure DevOps project, pick Service connections and create a new service connection of type “Kubernetes connection”.
Installing Linkerd
We will create a pipeline that installs Linkerd into the AKS cluster. Azure Pipeline now offers “pipeline-as-code” – which is just an YAML file that describes the steps need to be performed when the pipeline is triggered. We will use the following pipeline-as-code:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pool: | |
| name: Hosted Ubuntu 1604 | |
| steps: | |
| – task: KubectlInstaller@0 | |
| displayName: 'Install Kubectl latest' | |
| – task: Kubernetes@1 | |
| displayName: 'kubectl get' | |
| inputs: | |
| kubernetesServiceEndpoint: CloudOvenKubernetes | |
| command: get | |
| arguments: nodes | |
| – script: | | |
| curl -sL https://run.linkerd.io/install | sh | |
| export PATH=$PATH:$HOME/.linkerd2/bin | |
| linkerd version | |
| linkerd check –pre | |
| linkerd install | kubectl apply -f – | |
| linkerd check | |
| displayName: 'Linkerd – Installation' | |
We can at this point trigger the pipeline to install Linkerd into the AKS cluster.
Deployment of PODs and services
Let’s create another pipeline as code that deploys all the services and deployment resources to AKS using the following Kubernetes manifest file:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pool: | |
| name: Hosted Ubuntu 1604 | |
| steps: | |
| – task: KubectlInstaller@0 | |
| displayName: 'Install Kubectl latest' | |
| – task: Kubernetes@1 | |
| displayName: 'kubectl apply' | |
| inputs: | |
| kubernetesServiceEndpoint: CloudOvenKubernetes | |
| command: apply | |
| useConfigurationFile: true | |
| configuration: src/services/emojivoto/all.yml | |
In Azure Portal we can already see our services running:
Also in Kubernetes Dashboard:
We have got our services running – but they are not really affected by Linkerd yet. We will add another step into the build pipeline to tell Linkerd to do its magic.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pool: | |
| name: Hosted Ubuntu 1604 | |
| steps: | |
| – task: KubectlInstaller@0 | |
| displayName: 'Install Kubectl latest' | |
| – task: Kubernetes@1 | |
| displayName: 'kubectl apply' | |
| inputs: | |
| kubernetesServiceEndpoint: CloudOvenKubernetes | |
| command: apply | |
| useConfigurationFile: true | |
| configuration: src/services/emojivoto/all.yml | |
| – script: 'src/services/emojivoto/all.yml | linkerd inject – | kubectl apply -f –' | |
| displayName: 'Inject Linkerd' | |
Next thing, we trigger the pipeline and put some traffic into the service that we have just deployed. The emoji service is simulating some service to service invocation scenarios and now it’s time for us to open the Linkerd dashboard to inspect all the distributed traces and many other useful matrix to look at.
We can also see kind of an application map – in a graphical way to understand which service is calling who and what is request latencies etc.
Even fascinating, Linkerd provides some drill-down to the communications in Grafana Dashboard.
Conclusion
I have enjoyed a lot setting it up and see the outcome and wanted to share my experience with it. If you are looking into Service Mesh and read this post, I strongly encourage to give Linkerd a go, it’s awesome!
Thanks for reading.
Reverse Engineering 
That is really fascinating, You are a very skilled blogger. I’ve joined your feed and look forward to in search of extra of your excellent post. Additionally, I’ve shared your web site in my social networks!
LikeLike