Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.
Through DCT, image publishers can sign their images and image consumers can ensure that the images they pull are signed. Publishers could be individuals or organizations manually signing their content or automated software supply chains signing content as part of their release process.
Azure Container Registry implements Docker’s content trust model, enabling pushing and pulling of signed images. Once Content Trust is enabled in Azure Container Registry, signing an image is extremely easy as below:
Problem statement
Now that we can have DCT enabled in Azure Container Registry (i.e. allows pushing signed images into the repository), we want to make sure Kubernetes Cluster would deny running any images that are not signed.
However, Azure Kubernetes Service doesn’t have the feature (as of the today while writing this article) to restrict only signed images to be executed in the cluster. This doesn’t mean we’re stuck until AKS releases the feature. We can implement the content trust restriction using a custom Admission controller. In this article I want to share how one can create their own custom Admission controller to achieve DCT in an AKS cluster.
What are Admission Controllers?
In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether.
The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both.
In this article we will create a validation controller that would check if the docker image signature and will disallow Kubernetes to run any unsigned image for a selected Azure Container Registry. And we will create our Admission controller in .net core.
Writing a custom Admission Controller
Writing Admission Controller is fairly easy and straightforward – they’re just web APIs (Webhook) get invoked by API server while a resource is about to be created/updated/deleted etc. Webhook can respond to that request with an Allowed or Disallowed flag.
Kubernetes uses mutual TLS for all the communication with Admission controllers hence, we would need to create self-signed certificate for our Admission controller service.
Creating Self Signed Certificates
Following is a bash script that would generate a custom CA (certificate authority) and a pair of certificates for our web hook API.
#!/usr/bin/env bash
# Generate the CA cert and private key
openssl req -nodes -new -x509 -keyout ca.key -out ca.crt -subj "/CN=TailSpin CA"
# Generate the private key for the tailspin server
openssl genrsa -out tailspin-server-tls.key 2048
# Generate a Certificate Signing Request (CSR) for the private key, and sign it with the private key of the CA.
## NOTE
## The CN should be in this format: <service name>.<namespace>.svc
openssl req -new -key tailspin-server-tls.key -subj "/CN=tailspin-admission.tailspin.svc" \
| openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -out tailspin-server-tls.crt
We will create a simple asp.net core web api project. It’s as simple as a File->New asp.net web API project. Except, we would configure Kestrel to use the TLS certificates (created above) while listening by modifying the Program.cs as below.
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder => {
webBuilder
.UseStartup<Startup>()
.UseKestrel(options => {
options.ConfigureHttpsDefaults(connectionOptions =>
{ connectionOptions.AllowAnyClientCertificate();
connectionOptions.OnAuthenticate = (context, options) => { Console.WriteLine($"OnAuthenticate:: TLS Connection ID: {context.ConnectionId}");
};
});
options.ListenAnyIP(443, async listenOptions => {
var certificate = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_CERT);
var privateKey = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_KEY);
Console.WriteLine("Certificate and Key received...creating PFX..");
var pfxCertificate = CertificateHelper.GetPfxCertificate(
certificate,
privateKey);
listenOptions.UseHttps(pfxCertificate);
Console.WriteLine("Service is listening to TLS port");
});
});
});
Next, we will create a middle-ware/handler in Startup.cs to handle the Admission Validation requests.
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.Use(GetAdminissionMiddleware());
The implementation of the middleware looks as following:
public Func<HttpContext, Func<Task>, Task> GetAdminissionMiddleware()
{
var acrName = Environment.GetEnvironmentVariable("RegistryFullName"); // "/subscriptions/XX/resourceGroups/YY/providers/Microsoft.ContainerRegistry/registries/ZZ";
return async (context, next) =>
{
if (context.Request.Path.HasValue && context.Request.Path.Value.Equals("/admission"))
{
using var streamReader = new StreamReader(context.Request.Body);
var body = await streamReader.ReadToEndAsync();
var payload = JsonConvert.DeserializeObject<AdmissionRequest>(body);
var allowed = true;
if (payload.Request.Operation.Equals("CREATE") || payload.Request.Operation.Equals("UPDATE"))
{
foreach(var container in payload.Request.Object.Spec.Containers)
{
allowed = (await RegistryHelper.VerifyAsync(acrName, container.Image)) && allowed;
}
}
await GenerateResponseAsync(context, payload, allowed);
}
await next();
};
}
What we are doing here is, once we receive a request from API server about a POD to be created or updated, we intercept the request, validate if the image has Digital Signature in place (DCT) and reply to API server accordingly. Here’s the response to API server:
private static async Task GenerateResponseAsync(HttpContext context, AdmissionRequest payload, bool allowed)
{
context.Response.Headers.Add("content-type", "application/json");
await context
.Response
.WriteAsync(JsonConvert.SerializeObject(new AdmissionReviewResponse
{
ApiVersion = payload.ApiVersion,
Kind = payload.Kind,
Response = new ResponsePayload
{
Uid = payload.Request.Uid,
Allowed = allowed
}
}));
}
Now let’s see how we can verify the image has a signed tag in Azure Container Registry.
public class RegistryHelper
{
private static HttpClient http = new HttpClient();
public async static Task<bool> VerifyAsync(string acrName, string imageWithTag)
{
var imageNameWithTag = imageWithTag.Split(":".ToCharArray());
var credentials = SdkContext
.AzureCredentialsFactory
.FromServicePrincipal(Environment.GetEnvironmentVariable("ADClientID"),
Environment.GetEnvironmentVariable("ADClientSecret"),
Environment.GetEnvironmentVariable("ADTenantID"),
AzureEnvironment.AzureGlobalCloud);
var azure = Azure
.Configure()
.WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
.Authenticate(credentials)
.WithSubscription(Environment.GetEnvironmentVariable("ADSubscriptionID"));
var azureRegistry = await azure.ContainerRegistries.GetByIdAsync(acrName);
var creds = await azureRegistry.GetCredentialsAsync();
var authenticationString = $"{creds.Username}:{creds.AccessKeys[AccessKeyType.Primary]}";
var base64EncodedAuthenticationString =
Convert.ToBase64String(ASCIIEncoding.UTF8.GetBytes(authenticationString));
http.DefaultRequestHeaders.Add("Authorization", "Basic " + base64EncodedAuthenticationString);
var response = await http.GetAsync($"https://{azureRegistry.Name}.azurecr.io/acr/v1/{imageNameWithTag[0]}/_tags/{imageNameWithTag[1]}");
var repository = JsonConvert.DeserializeObject<RepositoryTag>(await response.Content.ReadAsStringAsync());
return repository.Tag.Signed;
}
}
This class uses Azure REST API for Azure Container Registry to check if the image tag was digitally signed. That’s all. We would create a docker image (tailspin-admission:latest) for this application and deploy it to Kubernetes with the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tailspin-admission
namespace: tailspin
spec:
replicas: 1
selector:
matchLabels:
app: tailspin-admission
template:
metadata:
labels:
app: tailspin-admission
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: tailspin-admission
image: "acr.io/tailspin-admission:latest"
ports:
- containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
name: tailspin-admission
namespace: tailspin
spec:
type: LoadBalancer
selector:
app: tailspin-admission
ports:
- port: 443
targetPort: 443
Now our Admission Controller is running, we need to register this as a validation web hook with the following manifest:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: tailspin-admission-controller
webhooks:
- name: tailspin-admission.tailspin.svc
admissionReviewVersions: ["v1", "v1beta1"]
failurePolicy: Fail
clientConfig:
service:
name: tailspin-admission
namespace: tailspin
path: "/admission"
caBundle: ${CA_PEM_B64}
rules:
- operations: [ "*" ]
apiGroups: [""]
apiVersions: ["*"]
resources: ["*"]
Here the ${CA_PEM_B64} needs to be filled with the base64 of our CA certificate that we generated above. The following bash script can do that:
# Read the PEM-encoded CA certificate, base64 encode it, and replace the `${CA_PEM_B64}` placeholder in the YAML
# template with it. Then, create the Kubernetes resources.
ca_pem_b64="$(openssl base64 -A <"../certs/ca.crt")"
(sed -e 's@${CA_PEM_B64}@'"$ca_pem_b64"'@g' <"./manifests/webhook-deployment.template.yaml") > "./manifests/webhook-deployment.yaml"
We can now deploy this manifest (KubeCtl) to our AKS cluster. And we are done! The cluster will now prevent running any unsigned images coming from our Azure Container Registry.
Summary
A critical aspect for DCT feature on AKS is enabling a solution which can satisfy all requirements such as content moving repositories or across registries which may extend beyond the current scope of the Azure Container Registry content trust feature as seen today. And enabling DCT on each AKS node is not a feasible solution as many Kubernetes images are not signed.
A custom Admission controller allow you to avoid these limitations and complexity today still being able to enforce Content Trust specific to your own ACR and images only. This article shows a very quick and simple way to create a custom Admission Controller – in .net core and how easy it is to create your own security policy for your AKS cluster.
Hope you find it useful and interesting.
A service principal has a secret, often referred as Client Secret. This can be analogous to the password of a user principal. The Service Principal ID (often known as Application ID or Client ID) and Client Secret together can authenticate an application to Azure AD for a secure resource access. In our earlier example, we needed to keep this client secret (the only secret) in our configuration file, to gain access to the Key vault. Client secrets have expiration period that up to the application developers to renew to keep things more secure. In a large solution this can easily turn into a difficult job to keep all the service principal secrets renewed with short expiration time.
