Restricting Unverified Kubernetes Content with Docker Content Trust

Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

Signed tags
Image source: Docker Content Trust

Through DCT, image publishers can sign their images and image consumers can ensure that the images they pull are signed. Publishers could be individuals or organizations manually signing their content or automated software supply chains signing content as part of their release process.

Azure Container Registry implements Docker’s content trust model, enabling pushing and pulling of signed images. Once Content Trust is enabled in Azure Container Registry, signing an image is extremely easy as below:

Signing an image from console

Problem statement

Now that we can have DCT enabled in Azure Container Registry (i.e. allows pushing signed images into the repository), we want to make sure Kubernetes Cluster would deny running any images that are not signed.

However, Azure Kubernetes Service doesn’t have the feature (as of the today while writing this article) to restrict only signed images to be executed in the cluster. This doesn’t mean we’re stuck until AKS releases the feature. We can implement the content trust restriction using a custom Admission controller. In this article I want to share how one can create their own custom Admission controller to achieve DCT in an AKS cluster.

What are Admission Controllers?

In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether.

Admission Controller Phases
Image source: Kubernetes Documentations

The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both.

In this article we will create a validation controller that would check if the docker image signature and will disallow Kubernetes to run any unsigned image for a selected Azure Container Registry. And we will create our Admission controller in .net core.

Writing a custom Admission Controller

Writing Admission Controller is fairly easy and straightforward – they’re just web APIs (Webhook) get invoked by API server while a resource is about to be created/updated/deleted etc. Webhook can respond to that request with an Allowed or Disallowed flag.

Kubernetes uses mutual TLS for all the communication with Admission controllers hence, we would need to create self-signed certificate for our Admission controller service.

Creating Self Signed Certificates

Following is a bash script that would generate a custom CA (certificate authority) and a pair of certificates for our web hook API.

#!/usr/bin/env bash

# Generate the CA cert and private key
openssl req -nodes -new -x509 -keyout ca.key -out ca.crt -subj "/CN=TailSpin CA"
# Generate the private key for the tailspin server
openssl genrsa -out tailspin-server-tls.key 2048
# Generate a Certificate Signing Request (CSR) for the private key, and sign it with the private key of the CA.

## The CN should be in this format: <service name>.<namespace>.svc
openssl req -new -key tailspin-server-tls.key -subj "/CN=tailspin-admission.tailspin.svc" \
    | openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -out tailspin-server-tls.crt

We will create a simple core web api project. It’s as simple as a File->New web API project. Except, we would configure Kestrel to use the TLS certificates (created above) while listening by modifying the Program.cs as below.

public static IHostBuilder CreateHostBuilder(string[] args) =>
      .ConfigureWebHostDefaults(webBuilder => {
         .UseKestrel(options => {                        
 options.ConfigureHttpsDefaults(connectionOptions => 
     {                                    connectionOptions.AllowAnyClientCertificate();
connectionOptions.OnAuthenticate = (context, options) => {                                        Console.WriteLine($"OnAuthenticate:: TLS Connection ID: {context.ConnectionId}");
   options.ListenAnyIP(443, async listenOptions => {
     var certificate = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_CERT);
     var privateKey = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_KEY);

      Console.WriteLine("Certificate and Key received...creating PFX..");
      var pfxCertificate = CertificateHelper.GetPfxCertificate(
      Console.WriteLine("Service is listening to TLS port");

Next, we will create a middle-ware/handler in Startup.cs to handle the Admission Validation requests.

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)

The implementation of the middleware looks as following:

public Func<HttpContext, Func<Task>, Task> GetAdminissionMiddleware()
            var acrName = Environment.GetEnvironmentVariable("RegistryFullName"); // "/subscriptions/XX/resourceGroups/YY/providers/Microsoft.ContainerRegistry/registries/ZZ";
            return async (context, next) =>
                if (context.Request.Path.HasValue && context.Request.Path.Value.Equals("/admission"))
                    using var streamReader = new StreamReader(context.Request.Body);
                    var body = await streamReader.ReadToEndAsync();
                    var payload = JsonConvert.DeserializeObject<AdmissionRequest>(body);

                    var allowed = true;
                    if (payload.Request.Operation.Equals("CREATE") || payload.Request.Operation.Equals("UPDATE"))
                        foreach(var container in payload.Request.Object.Spec.Containers)
                            allowed = (await RegistryHelper.VerifyAsync(acrName, container.Image)) && allowed;
                    await GenerateResponseAsync(context, payload, allowed);
                await next();

What we are doing here is, once we receive a request from API server about a POD to be created or updated, we intercept the request, validate if the image has Digital Signature in place (DCT) and reply to API server accordingly. Here’s the response to API server:

private static async Task GenerateResponseAsync(HttpContext context, AdmissionRequest payload, bool allowed)
            context.Response.Headers.Add("content-type", "application/json");
            await context
                .WriteAsync(JsonConvert.SerializeObject(new AdmissionReviewResponse
                    ApiVersion = payload.ApiVersion,
                    Kind = payload.Kind,
                    Response = new ResponsePayload
                        Uid = payload.Request.Uid,
                        Allowed = allowed

Now let’s see how we can verify the image has a signed tag in Azure Container Registry.

public class RegistryHelper
        private static HttpClient http = new HttpClient();

        public async static Task<bool> VerifyAsync(string acrName, string imageWithTag)
            var imageNameWithTag = imageWithTag.Split(":".ToCharArray());
            var credentials = SdkContext

            var azure = Azure
            var azureRegistry = await azure.ContainerRegistries.GetByIdAsync(acrName);
            var creds = await azureRegistry.GetCredentialsAsync();           

            var authenticationString = $"{creds.Username}:{creds.AccessKeys[AccessKeyType.Primary]}";
            var base64EncodedAuthenticationString = 
            http.DefaultRequestHeaders.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await http.GetAsync($"https://{azureRegistry.Name}{imageNameWithTag[0]}/_tags/{imageNameWithTag[1]}");
            var repository = JsonConvert.DeserializeObject<RepositoryTag>(await response.Content.ReadAsStringAsync());

            return repository.Tag.Signed;

This class uses Azure REST API for Azure Container Registry to check if the image tag was digitally signed. That’s all. We would create a docker image (tailspin-admission:latest) for this application and deploy it to Kubernetes with the following manifest:

apiVersion: apps/v1
kind: Deployment
  name: tailspin-admission
  namespace: tailspin
  replicas: 1
      app: tailspin-admission
        app: tailspin-admission
        "": linux
      - name: tailspin-admission
        image: ""
        - containerPort: 443
apiVersion: v1
kind: Service
  name: tailspin-admission
  namespace: tailspin
  type: LoadBalancer
    app: tailspin-admission
    - port: 443
      targetPort: 443

Now our Admission Controller is running, we need to register this as a validation web hook with the following manifest:

kind: ValidatingWebhookConfiguration
  name: tailspin-admission-controller
  - name: tailspin-admission.tailspin.svc
    admissionReviewVersions: ["v1", "v1beta1"]
    failurePolicy: Fail
        name: tailspin-admission
        namespace: tailspin
        path: "/admission"
      caBundle: ${CA_PEM_B64}
      - operations: [ "*" ]
        apiGroups: [""]
        apiVersions: ["*"]
        resources: ["*"]

Here the ${CA_PEM_B64} needs to be filled with the base64 of our CA certificate that we generated above. The following bash script can do that:

# Read the PEM-encoded CA certificate, base64 encode it, and replace the `${CA_PEM_B64}` placeholder in the YAML
# template with it. Then, create the Kubernetes resources.
ca_pem_b64="$(openssl base64 -A <"../certs/ca.crt")"
(sed -e 's@${CA_PEM_B64}@'"$ca_pem_b64"'@g' <"./manifests/webhook-deployment.template.yaml") > "./manifests/webhook-deployment.yaml"

We can now deploy this manifest (KubeCtl) to our AKS cluster. And we are done! The cluster will now prevent running any unsigned images coming from our Azure Container Registry.


A critical aspect for DCT feature on AKS is enabling a solution which can satisfy all requirements such as content moving repositories or across registries which may extend beyond the current scope of the Azure Container Registry content trust feature as seen today. And enabling DCT on each AKS node is not a feasible solution as many Kubernetes images are not signed.

A custom Admission controller allow you to avoid these limitations and complexity today still being able to enforce Content Trust specific to your own ACR and images only. This article shows a very quick and simple way to create a custom Admission Controller – in .net core and how easy it is to create your own security policy for your AKS cluster.

Hope you find it useful and interesting.

OpenSSL as Service

OpenSSL is awesome! Though, requires little manual work to remember all the commands, executing them in a machine that has OpenSSL installed. In this post, I’m about to build an HTTP API over OpenSSL, with the most commonly used commands (and the possibility to extend it further – as required). This will help folks who wants to run OpenSSL in a private network but wants to orchestrate it in their automation workflows.


Ever wanted to automate the TLS (also known as SSL) configuration process for your web application? You know, the sites that served via HTTPS and Chrome shows a green “secure” mark in address bar. Serving site over HTTP is insecure (even for static contents) and major browsers will mark those sites as not secure, Chrome already does that today.

Serving contents via HTTPS involves buying a digital certificate (aka SSL/TLS certificate) from certificate authorities (CA). The process seemed complicated (sometimes expensive too) by many average site owners or developers. Let’s encrypt addressed this hardship and made it painless. It’s an open certificate authority that provides free TLS certificates in an automated and elegant way.

However, free certificates might not be ideal for enterprise scenarios. Enterprise might have a requirement to buy certificate from a specific CA. In many cases, that process is manual and often complicated and slow. Typically, the workflow starts by generating a Certificate Signing request (also known as CSR) which requires generating asymmetric key pair (a public and private key pair). Which is then sent to CA to get a Digital Identity certificate. This doesn’t stop here. Once the certificate is provided by the CA, sometimes (Specially if you are in IIS, .net or Azure world) it’s needed to be converted to a PFX (Personal Information Exchange) file to deploy the certificate to the web server.

PFX (aka PKCS #12) is a file format defines an archive file format for storing many cryptography objects as a single file. It’s used to bundle a private key with it’s X.509 certificate or bundling all the members of a chain of trust. This file may be encrypted and signed. The internal storage containers (aka SafeBags), may also be encrypted and signed.

Generating CSR, converting a Digital Identity certificate to PFX format are often done manually. There are some online services that allows you generating CSRs – via an API or an UI. These are very useful and handy, but not the best fit for an enterprise. Because the private keys need to be shared with the online provider – to generate the CSR. Which leads people to use the vastly popular utility – OpenSSL in their local workstation – generating CSRs. In this article, this is exactly what I am trying to avoid. I wanted to have an API over OpenSSL – so that I can invoke it from my other automation workflow running in the Cloud.

Next, we will see how we can expose the OpenSSL over HTTP API in a Docker container, so we can run the container in our private enterprise network and orchestrate this in our certificate automation workflows.

The Solution Design

We will write a .net core web app, exposing the OpenSSL command via web API. Web API requests will fork OpenSSL process with the command and will return the outcome as web API response.

OpenSSL behind .net core Web API

We are using System.Diagnostics.Process to lunch OpenSSL in our code. This is assuming we will have OpenSSL executable present in our path. Which we will ensure soon with Docker.

        private static StringBuilder ExecuteOpenSsl(string command)
            var logs = new StringBuilder();
            var executableName = "openssl";
            var processInfo = new ProcessStartInfo(executableName)
                Arguments = command,
                UseShellExecute = false,
                RedirectStandardError = true,
                RedirectStandardOutput = true,
                CreateNoWindow = true

            var process = Process.Start(processInfo);
            while (!process.StandardOutput.EndOfStream)
            return logs;

This is simply kicking off OpenSSL executable with a command and capturing the output (or errors). We can now use this in our Web API controller.

    /// <summary>
    /// The Open SSL API
    /// </summary>
    public class OpenSslController : Controller
        /// <summary>
        /// Creates a new CSR
        /// </summary>
        /// Payload info
        /// The CSR with private key
        public async Task Csr([FromBody] CsrRequestPayload payload)
            var response = await CertificateManager.GenerateCSRAsync(payload);
            return new JsonResult(response);

This snippet only shows one example, where we are receiving a CSR generation request and using the OpenSSL to generate, returning the CSR details (in a base64 encoded string format) as API response.

Other commands are following the same model, so skipping them here.

Building Docker Image

Above snippet assumes that we have OpenSSL installed in the machine and the executable’s path is registered in our system’s path. We will turn that assumption to a fact by installing OpenSSL in our Docker image.

FROM microsoft/aspnetcore:2.0 AS base

RUN apt-get update -y
RUN apt-get install openssl

Here we are using aspnetcore:2.0 as our base image (which is a Linux distribution) and installing OpenSSL right after.

Let’s Run it!

I have built the docker image and published it to Docker Hub. All we need is to run it:


The default port of the web API is 80, though in this example we will run it on 8080. Let’s open a browser pointing to:


Voila! We have our API’s. Here’s the Swagger UI for the web API.


And we can test our CSR generation API via Postman:


The complete code for this web app with Docker file can be found in this GitHub Repository. The Docker image is in Docker Hub.

Thanks for reading.

Secure Azure Web sites with Web Application Gateway wtih end-to-end SSL connections

The Problem

In order to met higher compliance demands and often as security best practices, we want to put an Azure web site behind an Web Application Firewall (aka WAF). The WAF provides known malicious security attack vectors mitigation’s defined in OWASP top 10 security vulnerabilities. Azure Application Gateway is a layer 7 load balancer that provides WAF out of the box. However, restricting a Web App access with Application Gateway is not trivial.
To achieve the best isolation and hence protection, we can provision Azure Application Service Environment (aka ASE) and put all the web apps inside the virtual network of the ASE. The is by far the most secure way to lock down a web application and other Azure resources from internet access. But ASE deployment has some other consequences, it is costly, and also, because the web apps are totally isolated and sitting in a private VNET, dev-team needs to adopt a unusual deployment pipeline to continuously deploy changes into the web apps. Which is not an ideal solution for many scenarios.
However, there’s an intermediate solution architecture that provides WAF without getting into the complexities that AES brings into the solution architecture, allowing sort of best of both worlds. The architecture looks following:

The idea is to provision an Application Gateway inside a virtual network and configure it as a reverse proxy to the Azure web app. This means, the web app should never receive traffics directly, but only through the gateway. The Gateway needs to configure with the custom domain and SSL certificates. Once a request receives, the gateway then off-load the SSL and create another SSL to the back-end web apps configured into a back-end pool. For a development purpose, the back-end apps can use the Azure wildcard certificates (* but for production scenarios, it’s recommended to use a custom certificate. To make sure, no direct traffic gets through the azure web apps, we also need to white-list the gateway IP address into the web apps. This will block every requests except the ones coming through the gateway.

How to do that?

I have prepared an Azure Resource Manager template into this Github repo, that will provision the following:

  • Virtual network (Application Gateway needs a Virtual network).
  • Subnet for the Application Gateway into the virtual network.
  • Public IP address for the Application Gateway.
  • An Application Gateway that pre-configured to protect any Azure Web site.

How to provision?

Before you run the scripts you need the following:
  • Azure subscription
  • Azure web site to guard with WAF
  • SSL certificate to configure the Front-End listeners. (This is the Gateway Certificate which will be approached by the end-users (browsers basically) of your apps). Typically a Personal Information Exchange (aka pfx) file.
  • The password of the pfx file.
  • SSL certificate that used to protect the Azure web sites, typically a *.cer file. This can be the * for development purpose.
You need to fill out the parameters.json file with the appropriate values, some examples are given below:
        "vnetName": {
            "value": "myvnet"
        "appGatewayName": {
            "value": "mygateway"
        "azureWebsiteFqdn": {
            "value": ""
        "frontendCertificateData": {
            "value": ""
        "frontendCertificatePassword": {
            "value": ""
        "backendCertificateData": {
            "value": ""
Here, frontendCertificateData needs to be Base64 encoded content of your pfx file.
Once you have the pre-requisites, go to powershell and run:
    $> ./deploy.ps1 `
        -subscriptionId "" `
        -resourceGroupName ""
This will provision the Application Gatway in your resource group.

Important !

The final piece of work that you need to do, is to whitelist the IP address of the Application Gatway into your Azure Web App. This is to make sure, nobody can manage a direct access to your Azure web app, unless they come through the gateway only.


Contribution is always appreciated.