Restricting Unverified Kubernetes Content with Docker Content Trust

Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

Signed tags
Image source: Docker Content Trust


Through DCT, image publishers can sign their images and image consumers can ensure that the images they pull are signed. Publishers could be individuals or organizations manually signing their content or automated software supply chains signing content as part of their release process.


Azure Container Registry implements Docker’s content trust model, enabling pushing and pulling of signed images. Once Content Trust is enabled in Azure Container Registry, signing an image is extremely easy as below:

Signing an image from console

Problem statement

Now that we can have DCT enabled in Azure Container Registry (i.e. allows pushing signed images into the repository), we want to make sure Kubernetes Cluster would deny running any images that are not signed.

However, Azure Kubernetes Service doesn’t have the feature (as of the today while writing this article) to restrict only signed images to be executed in the cluster. This doesn’t mean we’re stuck until AKS releases the feature. We can implement the content trust restriction using a custom Admission controller. In this article I want to share how one can create their own custom Admission controller to achieve DCT in an AKS cluster.

What are Admission Controllers?

In a nutshell, Kubernetes admission controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercept (authenticated) API requests and may change the request object or deny the request altogether.

Admission Controller Phases
Image source: Kubernetes Documentations

The admission control process has two phases: the mutating phase is executed first, followed by the validating phase. Consequently, admission controllers can act as mutating or validating controllers or as a combination of both.

In this article we will create a validation controller that would check if the docker image signature and will disallow Kubernetes to run any unsigned image for a selected Azure Container Registry. And we will create our Admission controller in .net core.

Writing a custom Admission Controller

Writing Admission Controller is fairly easy and straightforward – they’re just web APIs (Webhook) get invoked by API server while a resource is about to be created/updated/deleted etc. Webhook can respond to that request with an Allowed or Disallowed flag.

Kubernetes uses mutual TLS for all the communication with Admission controllers hence, we would need to create self-signed certificate for our Admission controller service.

Creating Self Signed Certificates

Following is a bash script that would generate a custom CA (certificate authority) and a pair of certificates for our web hook API.

#!/usr/bin/env bash

# Generate the CA cert and private key
openssl req -nodes -new -x509 -keyout ca.key -out ca.crt -subj "/CN=TailSpin CA"
# Generate the private key for the tailspin server
openssl genrsa -out tailspin-server-tls.key 2048
# Generate a Certificate Signing Request (CSR) for the private key, and sign it with the private key of the CA.

## NOTE
## The CN should be in this format: <service name>.<namespace>.svc
openssl req -new -key tailspin-server-tls.key -subj "/CN=tailspin-admission.tailspin.svc" \
    | openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -out tailspin-server-tls.crt

We will create a simple asp.net core web api project. It’s as simple as a File->New asp.net web API project. Except, we would configure Kestrel to use the TLS certificates (created above) while listening by modifying the Program.cs as below.

public static IHostBuilder CreateHostBuilder(string[] args) =>
  Host.CreateDefaultBuilder(args)
      .ConfigureWebHostDefaults(webBuilder => {
        webBuilder
         .UseStartup<Startup>()
         .UseKestrel(options => {                        
 options.ConfigureHttpsDefaults(connectionOptions => 
     {                                    connectionOptions.AllowAnyClientCertificate();
connectionOptions.OnAuthenticate = (context, options) => {                                        Console.WriteLine($"OnAuthenticate:: TLS Connection ID: {context.ConnectionId}");
     };
   });
   options.ListenAnyIP(443, async listenOptions => {
     var certificate = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_CERT);
     var privateKey = await ResourceReader.GetEmbeddedStreamAsync(ResourceReader.Certificates.TLS_KEY);

      Console.WriteLine("Certificate and Key received...creating PFX..");
      var pfxCertificate = CertificateHelper.GetPfxCertificate(
                certificate,
                privateKey);
      listenOptions.UseHttps(pfxCertificate);
      Console.WriteLine("Service is listening to TLS port");
      });
   });
});

Next, we will create a middle-ware/handler in Startup.cs to handle the Admission Validation requests.

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.Use(GetAdminissionMiddleware());

The implementation of the middleware looks as following:

public Func<HttpContext, Func<Task>, Task> GetAdminissionMiddleware()
        {            
            var acrName = Environment.GetEnvironmentVariable("RegistryFullName"); // "/subscriptions/XX/resourceGroups/YY/providers/Microsoft.ContainerRegistry/registries/ZZ";
            return async (context, next) =>
            {
                if (context.Request.Path.HasValue && context.Request.Path.Value.Equals("/admission"))
                {
                    using var streamReader = new StreamReader(context.Request.Body);
                    var body = await streamReader.ReadToEndAsync();
                    var payload = JsonConvert.DeserializeObject<AdmissionRequest>(body);

                    var allowed = true;
                    if (payload.Request.Operation.Equals("CREATE") || payload.Request.Operation.Equals("UPDATE"))
                    {                        
                        foreach(var container in payload.Request.Object.Spec.Containers)
                        {
                            allowed = (await RegistryHelper.VerifyAsync(acrName, container.Image)) && allowed;
                        }
                    }
                    await GenerateResponseAsync(context, payload, allowed);
                }
                await next();
            };
        }

What we are doing here is, once we receive a request from API server about a POD to be created or updated, we intercept the request, validate if the image has Digital Signature in place (DCT) and reply to API server accordingly. Here’s the response to API server:

private static async Task GenerateResponseAsync(HttpContext context, AdmissionRequest payload, bool allowed)
        {
            context.Response.Headers.Add("content-type", "application/json");
            await context
                .Response
                .WriteAsync(JsonConvert.SerializeObject(new AdmissionReviewResponse
                {
                    ApiVersion = payload.ApiVersion,
                    Kind = payload.Kind,
                    Response = new ResponsePayload
                    {
                        Uid = payload.Request.Uid,
                        Allowed = allowed
                    }
                }));
        }

Now let’s see how we can verify the image has a signed tag in Azure Container Registry.

public class RegistryHelper
    {
        private static HttpClient http = new HttpClient();

        public async static Task<bool> VerifyAsync(string acrName, string imageWithTag)
        {
            var imageNameWithTag = imageWithTag.Split(":".ToCharArray());
            var credentials = SdkContext
                .AzureCredentialsFactory
                .FromServicePrincipal(Environment.GetEnvironmentVariable("ADClientID"),
                Environment.GetEnvironmentVariable("ADClientSecret"),
                Environment.GetEnvironmentVariable("ADTenantID"),
                AzureEnvironment.AzureGlobalCloud);

            var azure = Azure
                .Configure()
                .WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic)
                .Authenticate(credentials)
                .WithSubscription(Environment.GetEnvironmentVariable("ADSubscriptionID"));
           
            var azureRegistry = await azure.ContainerRegistries.GetByIdAsync(acrName);
            var creds = await azureRegistry.GetCredentialsAsync();           

            var authenticationString = $"{creds.Username}:{creds.AccessKeys[AccessKeyType.Primary]}";
            var base64EncodedAuthenticationString = 
                Convert.ToBase64String(ASCIIEncoding.UTF8.GetBytes(authenticationString));
            http.DefaultRequestHeaders.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await http.GetAsync($"https://{azureRegistry.Name}.azurecr.io/acr/v1/{imageNameWithTag[0]}/_tags/{imageNameWithTag[1]}");
            var repository = JsonConvert.DeserializeObject<RepositoryTag>(await response.Content.ReadAsStringAsync());

            return repository.Tag.Signed;
        }
    }

This class uses Azure REST API for Azure Container Registry to check if the image tag was digitally signed. That’s all. We would create a docker image (tailspin-admission:latest) for this application and deploy it to Kubernetes with the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tailspin-admission
  namespace: tailspin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tailspin-admission
  template:
    metadata:
      labels:
        app: tailspin-admission
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: tailspin-admission
        image: "acr.io/tailspin-admission:latest"
        ports:
        - containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: tailspin-admission
  namespace: tailspin
spec:
  type: LoadBalancer
  selector:
    app: tailspin-admission
  ports:
    - port: 443
      targetPort: 443

Now our Admission Controller is running, we need to register this as a validation web hook with the following manifest:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: tailspin-admission-controller
webhooks:
  - name: tailspin-admission.tailspin.svc
    admissionReviewVersions: ["v1", "v1beta1"]
    failurePolicy: Fail
    clientConfig:
      service:
        name: tailspin-admission
        namespace: tailspin
        path: "/admission"
      caBundle: ${CA_PEM_B64}
    rules:
      - operations: [ "*" ]
        apiGroups: [""]
        apiVersions: ["*"]
        resources: ["*"]

Here the ${CA_PEM_B64} needs to be filled with the base64 of our CA certificate that we generated above. The following bash script can do that:

# Read the PEM-encoded CA certificate, base64 encode it, and replace the `${CA_PEM_B64}` placeholder in the YAML
# template with it. Then, create the Kubernetes resources.
ca_pem_b64="$(openssl base64 -A <"../certs/ca.crt")"
(sed -e 's@${CA_PEM_B64}@'"$ca_pem_b64"'@g' <"./manifests/webhook-deployment.template.yaml") > "./manifests/webhook-deployment.yaml"

We can now deploy this manifest (KubeCtl) to our AKS cluster. And we are done! The cluster will now prevent running any unsigned images coming from our Azure Container Registry.

Summary

A critical aspect for DCT feature on AKS is enabling a solution which can satisfy all requirements such as content moving repositories or across registries which may extend beyond the current scope of the Azure Container Registry content trust feature as seen today. And enabling DCT on each AKS node is not a feasible solution as many Kubernetes images are not signed.

A custom Admission controller allow you to avoid these limitations and complexity today still being able to enforce Content Trust specific to your own ACR and images only. This article shows a very quick and simple way to create a custom Admission Controller – in .net core and how easy it is to create your own security policy for your AKS cluster.

Hope you find it useful and interesting.

Access Control management via REST API – Azure Data Lake Gen 2

Background

A while ago, I have built an web-based self-service portal that facilitated multiple teams in the organisation, setting up their Access Control (ACLs) for corresponding data lake folders.

The portal application was targeting Azure Data Lake Gen 1. Recently I wanted to achieve the same but on Azure Data Lake Gen 2. At the time of writing this post, there’s no official NuGet package for ACL management targeting Data Lake Gen 2. One must rely on REST API only.

Read about known issues and limitations of Azure Data Lake Storage Gen 2

Further more, the REST API documentations do not provide example snippets like many other Azure resources. Therefore, it takes time to demystify the REST APIs to manipulate ACLs. Good new is, I have done that for you and will share a straight-forward C# class that wraps the details and issues correct REST API calls to a Data Lake Store Gen 2.

About Azure Data Lake Store Gen 2

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics. Data Lake Storage Gen2 is significantly different from it’s earlier version known as Azure Data Lake Storage Gen1, Gen2 is entirely built on Azure Blob storage.

Data Lake Storage Gen2 is the result of converging the capabilities of two existing Azure storage services, Azure Blob storage and Azure Data Lake Storage Gen1. Gen1 Features such as file system semantics, directory, and file level security and scale are combined with low-cost, tiered storage, high availability/disaster recovery capabilities from Azure Blob storage.

Let’s get started!

Create a Service Principal

First we would need a service principal. We will use this principal to authenticate to Azure Active Directory (using OAuth 2.0 protocol) in order to authorize our REST calls. We will use Azure CLI to do that.

az ad sp create-for-rbac --name ServicePrincipalName
Add required permissions

Now you need to grant permission for your application to access Azure Storage.

  • Click on the application Settings
  • Click on Required permissions
  • Click on Add
  • Click Select API
  • Filter on Azure Storage
  • Click on Azure Storage
  • Click Select
  • Click the checkbox next to Access Azure Storage
  • Click Select
  • Click Done

App

Now we have Client ID, Client Secret and Tenant ID (take it from the Properties tab of Azure Active Directory – listed as Directory ID).

Access Token from Azure Active Directory

Let’s write some C# code to get an Access Token from Azure Active Directory:

Creating ADLS Gen 2 REST client

Once we have the token provider, we can jump in implementing the REST client for Azure Data Lake.

Data Lake  ACLs and POSIX permissions

The security model for Data Lake Gen2 supports ACL and POSIX permissions along with some extra granularity specific to Data Lake Storage Gen2. Settings may be configured through Storage Explorer or through frameworks like Hive and Spark. We will do that via REST API in this post.

There are two kinds of access control lists (ACLs), Access ACLs and Default ACLs.

  • Access ACLs: These control access to an object. Files and folders both have Access ACLs.
  • Default ACLs: A “template” of ACLs associated with a folder that determine the Access ACLs for any child items that are created under that folder. Files do not have Default ACLs.

Here’s the table of allowed grant types:

acl1

While we define ACLs we need to use a short form of these grant types. Microsoft Document explained these short form in below table:

posix

However, in our code we would also simplify the POSIX ACL notations by using some supporting classes as below. That way REST client consumers do not need to spend time building the short form of their aimed grant criteria’s.

Now we can create methods to perform different REST calls, let’s start by creating a file system.

Here we are retrieving a Access Token and then issuing a REST call to Azure Data Lake Storage Gen 2 API to create a new file system. Next, we will create a folder and file in it and then set some Access Control to them.

Let’s create the folder:

And creating file in it. Now, file creation (ingestion in Data Lake) is not that straight forward, at least, one can’t do that by a single call. We would have to first create an empty file, then we can write some content in it. We can also append content to an existing file. Finally, we would require to flush the buffer so the new content gets persisted.

Let’s do that, first we will see how to create an empty file:

The above snippet will create an empty file, now we will read all content from a local file (from PC) and write them into the empty file in Azure Data Lake that we just created.

Right! Now time to set Access control to the directory or files inside a directory. Here’s the method that we will use to do that.

The entire File system REST API class can be found here. Here’s an example how we can use this methods from a console application.

Conclusion

Until, there’s an Official Client Package released, if you’re into Azure Data Lake Store Gen 2 and wondering how to accomplish these REST calls – I hope this post helped you to move further!

Thanks for reading.

 

Inter-process communication on Windows Containers

Background

Legacy monolith applications that are built to run on single beefy server can take advantage of containers to simplify the deployment model and also potentially opens possibility to re-architect piece by piece without triggering a complete rewrite. I ran into a scenario where I am considering wrap up a large monolith (with many threads in it) into multiple containers and introduce some mode of execution. Therefore, each container instance runs a specific mode of operations and leads to a micro-service-based architecture in future. Splitting into containers is rather easier, but then I needed to introduce an IPC mechanism to enable communication between these container instances. In this post, I will write some IPC options that I have exercised in these scenarios.

The application is written in .net framework; therefore, I couldn’t use .net core and Linux machines. I have only investigated windows containers. I have tried following technologies for IPC and did some bench marking on latency.

Environment and Hardware specs

Most of these IPC technologies (e.g. TCP, gRPC, Web Sockets) also allow remote invocations, but I have only tried on single machine- as that’s what I wanted to investigate. I have run these benchmarks on windows 10 client machine with following configuration:

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362

Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores

[Host]: .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.8.3815.0

Windows Container: Quick refresh

Windows Server containers provide application isolation through process and namespace isolation technology. That is often referred to as process-isolated containers. A Windows Server container shares a kernel with the container host and all containers running on the host. These process-isolated containers don’t provide a hostile security boundary and shouldn’t be used to isolate untrusted code. Because of the shared kernel space, these containers require the same kernel version and configuration.

However, windows containers also provide a different type of isolation – called Hyper-V isolation. Hyper-V isolation expands on the isolation provided by Windows Server containers by running each container in a highly optimized virtual machine.

HyperV conainers
Windows Container Hyper-V isolation

In this configuration, the container host doesn’t share its kernel with other containers on the same host. These containers are designed for hostile multi-tenant hosting with the same security assurances of a virtual machine. Since these containers don’t share the kernel with the host or other containers on the host, they can run kernels with different versions and configurations (within supported versions). For example, all Windows containers on Windows 10 use Hyper-V isolation to utilize the Windows Server kernel version and configuration.

Running a container on Windows with or without Hyper-V isolation is a runtime decision. We can initially create the container with Hyper-V isolation, and then later at runtime choose to run it as a Windows Server container instead.

I have run the IPC stack for each technology in three different setups.

  • Bare metal (running on my windows 10 client)
  • Two containers (server and client) running in Hyper-V isolation (–isolation=hyperv)
  • Two containers (server and client) running in Process isolation (–isolation=process)

Measure/benchmark

IPC – by its nature is a non-deterministic operation. Hence, I wanted to measure and focus on latencies in my investigation instead of throughputs. I created some IPC handshake applications that exchanges approximately 1 KB of message from client to server. I ran it in different frequencies (>10000 times) and measured the percentiles.

And of course, I am running Docker for Windows with following version:

docker-version

WCF TCP/IP Channel

TCP channel is probably the most commonly used binding in WCF applications. Here I have the simple WCF server and client that sends some bytes over the wire. TCP is a connection-based, stream-oriented delivery service with end-to-end error detection and correction. Connection-based means that a communication session between hosts is established before exchanging data. A host is any device on a TCP/IP network identified by a logical IP address.

The sample hosts a TCP server and waits for clients to connect. Once the client is connected, client sends 1KB bytes to the server for a n number of times.

WCF server

WCF Client

Running it on bare-metal:

TCP bare-metal

Now running it inside two containers (server and client) in Hyper-V isolation:

TCP-HyperV

We can see that it adds latency compare to Bare-metal. Let’s run it on process isolation mode:

TCP-Process

That improved a lot, almost as bare-metal.

gRPC

Like many RPC systems, gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. By default, gRPC uses protocol buffers as the Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages. It is possible to use other alternatives if desired.

I have written a similar sample project as I did for TCP channel above but this time both the server and client uses gRPC for messaging.

Lets run the same exercise with gRPC.

gRPC

What I see is, the process isolation is pretty darn good compare to Hyper-V isolation.

Web Sockets

Web Sockets (over HTTPS) gives a easy programming model for network communication.  The WebSocket API is an advanced technology that makes it possible to open a two-way interactive communication session between the user’s browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

I didn’t write or programmed web socket API directly though, I have used the SignalR self-hosting to do that.

In this exercise I created the same messaging with web sockets.

Web-socket

Unix Domain Sockets

Like I have mentioned above, Linux and .net core was not my option for this exercise. However, I couldn’t resist give it a shot running the same messaging over an Unix-domain-socket on Linux kernel.

Unix domain socket or IPC socket is a data communications endpoint for exchanging data between processes executing on the same host operating system. Valid socket types in the UNIX domain are: SOCK_STREAM (compare to TCP), for a stream-oriented socket; SOCK_DGRAM (compare to UDP), for a datagram-oriented socket that preserves message boundaries (as on most UNIX implementations, UNIX domain datagram sockets are always reliable and don’t reorder datagrams); and SOCK_SEQPACKET (compare to SCTP), for a sequenced-packet socket that is connection-oriented, preserves message boundaries, and delivers messages in the order that they were sent. The Unix domain socket facility is a standard component of POSIX operating systems.

Here’s what I get when running the similar messaging that leverages Unix domain sockets:

UDS

That’s blazing fast! Sadly I couldn’t use it for my purpose.

Summary

Putting all the numbers into a chart, I get this:

Graph

Disclaimer 1: Bench marking is difficult – it has so many moving factors to get everything right. I wouldn’t put any conclusive statement on it, like certain IPC technique is faster than other. But the source codes are included and you can run it on your environment and make your one judgement.

Disclaimer 2: Another interesting technology I needed to try out was windows named-pipes. The source code is in the same repository, but I couldn’t get it to work while sharing between containers. I will update the post once I have some progress there.

All remarks/questions are always welcome, Thanks for reading.

CloudOven – Terraform at ease!

TL;DR:

  • URL: CloudOven 

  • Use Google account or sign-up 
  • Google Chrome please! (I’ve not tested on other browsers yet)

e2e

Background

In recent years I have spent fair amount of time in design and implementation of Infrastructure as code in larger enterprise context. Terraform seemed to be a tool of choice when it comes to preserve the uniformity in Infrastructure as code targeting multiple cloud providers. It is rapidly becoming a de facto choice for creating and managing cloud infrastructures by writing declarative definitions. It’s popular because the syntax of its files is quite readable and because it supports several cloud providers while making no attempt to provide an artificial abstraction across those providers. The active community will add support for the latest features from most cloud providers.

However, rolling out Terraform in many enterprises has its own barrier to face. Albeit the syntax (HCL) is neat, but not every developers or Infrastructure operators in organizations finds it easy. There’s a learning curve and often many of us lose momentum discovering the learning effort. I believe if we could make the initial ramp-up easier more people would play with it.

That’s one of my motivation for this post, following is the other one.

Blazor meets Terraform

Lately I was learning Blazor – the new client-side technology from Microsoft. Like many others, I find one effective way learning a new technology by creating/building solution to a problem. I have decided to build a user interface that will help creating terraform scripts easier. I will share my journey in this post.

Resource Discovery in Terraform Providers

Terraform is powerful for its providers. You will find Terraform providers for all major cloud providers (Azure, AWS, Google etc.). The providers then allow us to define “resource” and “data source” in Terraform scripts. These resource and data source have arguments and attributes that one must know while creating terraform files. Luckily, they are documented nicely in Terraform site. However, it still requires us to jump back and forth to the documentation site and terraform file editor (i.e. VSCode).

Azure-Discovery

To make this experience easier, I wrote a crawler application that downloads the terraform providers (I am doing it for Azure, AWS and google for now) and discovers the attributes and arguments for each and every resource and data source. I also try to extract the documentation for every attributes and arguments from the terraform documentation site with a layman parsing (not 100% accurate but works for majority. Something I will improve soon).

GoogleAWS-discovery

This process generates JSON structure for each resource and data source, enriches them with the documentation and stores them in an Azure Blob Storage.

Building Infrastructure as code

Now that I have a structured data store with all resources and data sources for any terraform provider, I can leverage that building a user interface on top of it. To keep things a bit organized, I started with a concept of “project”.

workflow
The workflow

Project

I can start by creating a project (well, it can be a product too, but let’s not get to that debate). Project is merely a logical boundary here.

Blueprint

Within a project I can create Blueprint(s). Blueprint(s) are the entity that retains the elements of the infrastructure that we are aiming to create. For instance, a Blueprint targets to a Cloud provider (i.e. Azure). Then I can create the elements (resource and data sources) within the blueprint (i.e. Azure Web App, Cosmos DB etc.).

provider-configuration

Blueprints keeps the base structure of all the infrastructure elements. It allows defining variables (plain and simple terraform variables) so the actual values can vary in different environments (dev, test, pre-production, production etc.).

Once I am happy with the blueprint, I can download them as a zip – that contains the terraform scripts (main.tf and variable.tf). That’s it, we have our infrastructure as code in Terraform. I can execute them on a local development machine or check them in to source control – whatever I prefer.

storage_account

One can stop here and keep using the blueprint feature to generate Infrastructure as code. That’s what it is for. However, the next features are just to make the overall experience of running terraform a bit easier.

Environments

Next to blueprint, we can create as many environments we want. Again, just a logical entity to keep isolation of actual deployment for different environments.

Deployments

Deployment entity is the glue that ties a blueprint to a specific environment. For instance, I can define a blueprint for “order management” service (or micro-service maybe?), create an environment as “test” and then create a “deployment” for “order management” on “test”. This is where I can define constant values to the blueprint variable that are specific to the test environment.

Terraform State

Perhaps the most important aspect the deployment entity holds is the terraform state management. Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. This state is stored by default in a local file named “terraform.tfstate”, but it can also be stored remotely, which works better in a team environment. Defining the state properties (varies in different cloud providers) in deployment entity makes the remote state management easier – specifically in team environment. It will configure the remote state to the appropriate remote backend. For instance, when the blueprint cloud provider is set to Azure, it will configure Azure Storage account as terraform state remote backend, for AWS it will pick S3 automatically.

e2e

Terraform plan

Once we have deployment entity configured, we can directly from the user interface run “terraform plan”. The terraform plan command creates an execution plan. Unless explicitly disabled, it performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the blueprint. This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

Terraform apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan. Like “plan”, the “apply” command can also be issued directly from the user interface.

Terraform plan and apply both are issued in an isolated docker container and the output is captured and displayed back to the user interface. However, there’s a cost associated running docker containers on cloud, therefore, it’s disabled in the public site.

Final thoughts

It was fun to write a tool like this. I recommend you give it a go. Especially if you are stepping into Terraform. It can also be helpful for experienced Terraform developers – specifically with the on-screen documenation, type inferance and discovery features.

Some features, I have working progress:

  • Ability to define policy for each resources and data types
  • Save a Blueprint as custom module

Stay tuned!

 

Continuously deploy Blazor SPA to Azure Storage static web site

Lately I am learning ASP.net Blazor – the relatively new UI framework from Microsoft. Blazor is just awesome – the ability to write c# code both in server and client side is extremely productive for .net developers. From Blazor documentations:

Blazor lets you build interactive web UIs using C# instead of JavaScript. Blazor apps are composed of reusable web UI components implemented using C#, HTML, and CSS. Both client and server code is written in C#, allowing you to share code and libraries.

I wanted to write a simple SPA (Single Page Application) and run it as server-less. Azure Storage offers hosting static web sites for quite a while now. Which seems like a very nice option to run a Blazor SPA which executes into the user’s browser (within the same Sandbox as JavaScript does). It also a cheap way to run a Single Page application in Cloud.

I am using GitHub as my source repository for free (in a private repository). Today wanted to create a pipeline that will continuously deploy my Blazor app to the storage account. Azure Pipelines seems to have pretty nice integration with GitHub and it’s has a free tier as well . If either our GitHub repository or pipeline is private, Azure Pipeline still provide a free tier. In this tier, one can run one free parallel job that can run up to 60 minutes each time until we’ve used 1800 minutes per month. That’s pretty darn good for my use case.

I also wanted to build the project many times in my local machine (while developing) in the same way it gets built in the pipeline. Being a docker fan myself, that’s quite no-brainer. Let’s get started.

Pre-requisite

I have performed few steps before I ran after the pipeline – that are beyond the scope of this post.

  • I have created an Azure Subscription
  • Provisioned resource groups and storage account
  • I have created a Service Principal and granted Contributor role to the storage account

Publishing in Docker

I have created a docker file that will build the app, run unit tests and if all goes well, it will publish the app in a folder. All of these are standard dotnet commands.
Once, we have the application published in a folder, I have taken the content of that folder to a Azure CLI docker base image (where CLI is pre-installed) and thrown away the rest of the intermediate containers.

Here’s our docker file:

The docker file expects few arguments (basically the service principal ID, the password of the service principal and the Azure AD tenant ID – these are required for Azure CLI to sign-in to my Azure subscription). Here’s how we can build this image now:

Azure Pipeline as code

We have now the container, time to run it every time a commit has been made to the GitHub repository. Azure Pipeline has a yaml format to define pipeline-as-code – which is another neat feature of Azure Pipelines.

Let’s see how the pipeline-as-code looks like:

I have committed this to the same repository into the root folder.

Creating the pipeline

We need to login to Azure DevOps and create a project (if there’s none). From the build option we can create a new build definition.

ado

The steps to create build definition is very straightforward. It allows us to directly point to a GitHub repository that we want to build.
Almost there. We need to supply the service principal ID, password, tenant ID and storage account names to this pipeline – because both our docker file and the pipeline-as-code expected them as dependencies. However, we can’t just put their values and commit them to GitHub. They should be kept secret.

Azure Pipeline Secret variables

Azure Pipeline allows us to define secret variable for a pipeline. We need to open the build definition in “edit” mode and then go to the top-right ellipses button as below:

ado1

Now we can define the values of these secret and keep them hidden (there’s a lock icon there).

ad02

That’s all what we need. Ready to deploy the code to Azure storage via this pipeline. If we now go an make a change in our repository it will trigger the pipeline and sure enough it will build-test-publish-deploy to Azure storage as a Static SPA.

Thanks for reading!

OpenSSL as Service

OpenSSL is awesome! Though, requires little manual work to remember all the commands, executing them in a machine that has OpenSSL installed. In this post, I’m about to build an HTTP API over OpenSSL, with the most commonly used commands (and the possibility to extend it further – as required). This will help folks who wants to run OpenSSL in a private network but wants to orchestrate it in their automation workflows.

Background

Ever wanted to automate the TLS (also known as SSL) configuration process for your web application? You know, the sites that served via HTTPS and Chrome shows a green “secure” mark in address bar. Serving site over HTTP is insecure (even for static contents) and major browsers will mark those sites as not secure, Chrome already does that today.

Serving contents via HTTPS involves buying a digital certificate (aka SSL/TLS certificate) from certificate authorities (CA). The process seemed complicated (sometimes expensive too) by many average site owners or developers. Let’s encrypt addressed this hardship and made it painless. It’s an open certificate authority that provides free TLS certificates in an automated and elegant way.

However, free certificates might not be ideal for enterprise scenarios. Enterprise might have a requirement to buy certificate from a specific CA. In many cases, that process is manual and often complicated and slow. Typically, the workflow starts by generating a Certificate Signing request (also known as CSR) which requires generating asymmetric key pair (a public and private key pair). Which is then sent to CA to get a Digital Identity certificate. This doesn’t stop here. Once the certificate is provided by the CA, sometimes (Specially if you are in IIS, .net or Azure world) it’s needed to be converted to a PFX (Personal Information Exchange) file to deploy the certificate to the web server.

PFX (aka PKCS #12) is a file format defines an archive file format for storing many cryptography objects as a single file. It’s used to bundle a private key with it’s X.509 certificate or bundling all the members of a chain of trust. This file may be encrypted and signed. The internal storage containers (aka SafeBags), may also be encrypted and signed.

Generating CSR, converting a Digital Identity certificate to PFX format are often done manually. There are some online services that allows you generating CSRs – via an API or an UI. These are very useful and handy, but not the best fit for an enterprise. Because the private keys need to be shared with the online provider – to generate the CSR. Which leads people to use the vastly popular utility – OpenSSL in their local workstation – generating CSRs. In this article, this is exactly what I am trying to avoid. I wanted to have an API over OpenSSL – so that I can invoke it from my other automation workflow running in the Cloud.

Next, we will see how we can expose the OpenSSL over HTTP API in a Docker container, so we can run the container in our private enterprise network and orchestrate this in our certificate automation workflows.

The Solution Design

We will write a .net core web app, exposing the OpenSSL command via web API. Web API requests will fork OpenSSL process with the command and will return the outcome as web API response.

OpenSSL behind .net core Web API

We are using System.Diagnostics.Process to lunch OpenSSL in our code. This is assuming we will have OpenSSL executable present in our path. Which we will ensure soon with Docker.

        private static StringBuilder ExecuteOpenSsl(string command)
        {
            var logs = new StringBuilder();
            var executableName = "openssl";
            var processInfo = new ProcessStartInfo(executableName)
            {
                Arguments = command,
                UseShellExecute = false,
                RedirectStandardError = true,
                RedirectStandardOutput = true,
                CreateNoWindow = true
            };

            var process = Process.Start(processInfo);
            while (!process.StandardOutput.EndOfStream)
            {
                logs.AppendLine(process.StandardOutput.ReadLine());
            }
            logs.AppendLine(process.StandardError.ReadToEnd());
            return logs;
        }

This is simply kicking off OpenSSL executable with a command and capturing the output (or errors). We can now use this in our Web API controller.

    /// <summary>
    /// The Open SSL API
    /// </summary>
    [Produces("application/json")]
    [Route("api/OpenSsl")]
    public class OpenSslController : Controller
    {
        /// <summary>
        /// Creates a new CSR
        /// </summary>
        /// Payload info
        /// The CSR with private key
        [HttpPost("CSR")]
        public async Task Csr([FromBody] CsrRequestPayload payload)
        {
            var response = await CertificateManager.GenerateCSRAsync(payload);
            return new JsonResult(response);
        }

This snippet only shows one example, where we are receiving a CSR generation request and using the OpenSSL to generate, returning the CSR details (in a base64 encoded string format) as API response.

Other commands are following the same model, so skipping them here.

Building Docker Image

Above snippet assumes that we have OpenSSL installed in the machine and the executable’s path is registered in our system’s path. We will turn that assumption to a fact by installing OpenSSL in our Docker image.

FROM microsoft/aspnetcore:2.0 AS base

RUN apt-get update -y
RUN apt-get install openssl

Here we are using aspnetcore:2.0 as our base image (which is a Linux distribution) and installing OpenSSL right after.

Let’s Run it!

I have built the docker image and published it to Docker Hub. All we need is to run it:

Untitled-1

The default port of the web API is 80, though in this example we will run it on 8080. Let’s open a browser pointing to:

http:localhost:8080/ 

Voila! We have our API’s. Here’s the Swagger UI for the web API.

swagger

And we can test our CSR generation API via Postman:

Postman

The complete code for this web app with Docker file can be found in this GitHub Repository. The Docker image is in Docker Hub.

Thanks for reading.

Resilient Azure Data Lake Analytics (ADLA) Jobs with Azure Functions

Azure Data Lake Analytics is an on-demand analytics job service that allows writing queries to transform data and grab insights efficiently. The analytics service can handle jobs of any scale instantly by setting the dial for how much power you need.

JObs

In many organizations, these jobs could play a crucial role and reliability of these job executions could be business critical. Lately I have encountered a scenario where a particular USQL job has failed with following error message:

Usql – Job failed due to internal system error – NM_CANNOT_LAUNCH_JM

A bit of research on Google revealed, it’s a system error, which doesn’t leave a lot of diagnostic clue to reason out. Retrying this job manually (by button clicking on portal) yielded success! Which makes it a bit unpredictable and uncertain. However, uncertainty like this is sort of norm while developing Software for Cloud. We all read/heard about Chaos Monkeys of Netflix.

What is resiliency?

Resiliency is the capability to handle partial failures while continuing to execute and not crash. In modern application architectures — whether it be micro services running in containers on-premises or applications running in the cloud — failures are going to occur. For example, applications that communicate over networks (like services talking to a database or an API) are subject to transient failures. These temporary faults cause lesser amounts of downtime due to timeouts, overloaded resources, networking hiccups, and other problems that come and go and are hard to reproduce. These failures are usually self-correcting. (Source)
Today I will present an approach that mitigated this abrupt job failure.

The Solution Design

Basically, I wanted to have a job progress watcher, waiting to see a failed job and then resubmit that job as a retry-logic. Also, don’t want to retry more than once, which has potential to repeat a forever-failure loop. I can have my watcher running at a frequency – like every 5 minutes or so.

Azure Functions

Azure Functions continuously impressing me for its lightweight built and consumption-based pricing model. Functions can run with different triggers, among them time schedule trigger- that perfectly fits my purpose.

Prerequisites

The function app needs to retrieve failed ADLA jobs and resubmit them as needed. This can be achieved with the Microsoft.Azure.Management.DataLake.Analytics, Version=3.0.0.0 NuGet package. We will also require Microsoft.Rest. ClientRuntime.Azure.Authentication, Version=2.0.0.0 NuGet package for Access Token retrievals.

Configuration

We need a Service Principal to be able to interact with ADLA instance on Azure. Managed Service Identity (written about it before) can also be used to make it secret less. However, in this example I will use Service Principal to keep it easier to understand. Once we have our Service Principal, we need to configure them in Function Application Settings.

Hacking the function

[FunctionName("FN_ADLA_Job_Retry")]

public static void Run([TimerTrigger("0 0 */2 * * *")]TimerInfo myTimer, TraceWriter log)

{

var accountName = GetEnvironmentVariable("ADLA_NAME");

var tenantId = GetEnvironmentVariable("TENANT_ID");

var clientId = GetEnvironmentVariable("SERVICE_PRINCIPAL_ID");

var clientSecret = GetEnvironmentVariable("SERVICE_PRINCIPAL_SECRET");

 

ProcessFailedJobsAsync(tenantId, clientId, clientSecret, accountName).Wait();

}

That’s our Azure Function scheduled to be run every 2 hours. Once we get a trigger, we retrieve the AD tenant ID, Service Principal ID, secret and the account name of target ADLA.

Next thing we do, write a method that will give us a ADLA REST client – authenticated with Azure AD, ready to make a call to ADLA account.

private static async Task GetAdlaClientAsync(

string clientId, string clientSecret, string tenantId)

{

var creds = new ClientCredential(clientId, clientSecret);

var clientCreds = await ApplicationTokenProvider

.LoginSilentAsync(tenantId, creds);

 

var adlsClient = new DataLakeAnalyticsJobManagementClient(clientCreds);

return adlsClient;

}

The DataLakeAnalyticsJobManagementClient class comes from Microsoft.Azure.Management.DataLake.Analytics, Version=3.0.0.0 NuGet package that we have already installed into our project.

Next, we will write a method that will get us all the failed jobs,

private static async Task<Microsoft.Rest.Azure.IPage>

GetFailedJobsAsync(string accountName, DataLakeAnalyticsJobManagementClient client)

{

// We are ignoring the data pages that has older jobs

// If that's important to you, use CancellationToken to retrieve those pages

return await client.Job

.ListAsync(accountName,

new ODataQuery(job => job.Result == JobResult.Failed));

}

We have now the capability to retrieve failed jobs, great! Now we should write the real logic that will check for failed jobs that never been retried and resubmit them.

private const string RetryJobPrefix = "RETRY-";

public static async Task ProcessFailedJobsAsync(

string tenantId, string clientId, string clientSecret, string accountName)

{

var client = await GetAdlaClientAsync(clientId, clientSecret, tenantId);

 

var failedJobs = await GetFailedJobsAsync(accountName, client);

 

foreach (var failedJob in failedJobs)

{

// If it's a retry attempt we will not kick this off again.

if (failedJob.Name.StartsWith(RetryJobPrefix)) continue;

 

// we will retry this with a name prefixed with a RETRY

var retryJobName = $"{RetryJobPrefix}{failedJob.Name}";

 

// Before we kick this off again, let's check if we already have retried this before..

if (!(await HasRetriedBeforeAsync(accountName, client, retryJobName)))

{

var jobDetails = await client.Job.GetAsync(accountName, failedJob.JobId.Value);

var newJobID = Guid.NewGuid();

 

var properties = new USqlJobProperties(jobDetails.Properties.Script);

var parameters = new JobInformation(

retryJobName,

JobType.USql, properties,

priority: failedJob.Priority,

degreeOfParallelism: failedJob.DegreeOfParallelism,

jobId: newJobID);

 

// resubmit this job now

await client.Job.CreateAsync(accountName, newJobID, parameters);

}

}

}

private async static Task HasRetriedBeforeAsync(string accountName,

DataLakeAnalyticsJobManagementClient client, string name)

{

var jobs = await client.Job

.ListAsync(accountName,

new ODataQuery(job => job.Name == name));

 

return jobs.Any();

}

This is it all!

Final thoughts!

We can’t avoid failures, but we can respond in ways that will keep our system up or at least minimize downtime. In this example, when one Job fails unpredictably, its effects can cause the system to fail.

We should build our own mitigation against these uncertain factors – with automation.

Deploying Azure web job written in .net core

Lately I have written a .net core web job and wanted to publish it via CD (continuous deployment) from Visual Studio Online. Soon I figured, Azure Web Job SDK doesn’t support (yet) .net core. The work I expected will take 10 mins took about an hour.

If you are also figuring out this, this blog post is what you are looking for.

I will describe the steps and provide a PowerShell script that does the deployment via Kudu API. Kudu is the Source Control management for Azure app services, which has a Zip API that allows us to deploy zipped folder into an Azure app service.

Here are the steps you need to follow. You can start by creating a simple .net core console application. Add a Power Shell file into the project that will do the deployment in your Visual Studio online release pipeline. The Power Shell script will do the following:

  • Publish the project (using dotnet publish)
  • Make a zip out of the artifacts
  • Deploy the zip into the Azure web app

Publishing the project

We will use dotnet publish command to publish our project.

$resourceGroupName = "my-regource-group"
$webAppName = "my-web-job"
$projectName = "WebJob"
$outputRoot = "webjobpublish"
$ouputFolderPath = "webjobpublish\App_Data\Jobs\Continuous\my-web-job"
$zipName = "publishwebjob.zip"

$projectFolder = Join-Path `
    -Path "$((get-item $PSScriptRoot ).FullName)" `
    -ChildPath $projectName
$outputFolder = Join-Path `
    -Path "$((get-item $PSScriptRoot ).FullName)" `
    -ChildPath $ouputFolderPath
$outputFolderTopDir = Join-Path `
    -Path "$((get-item $PSScriptRoot ).FullName)" `
    -ChildPath $outputRoot
$zipPath = Join-Path `
    -Path "$((get-item $PSScriptRoot ).FullName)" `
    -ChildPath $zipName

if (Test-Path $outputFolder)
  { Remove-Item $outputFolder -Recurse -Force; }
if (Test-path $zipName) {Remove-item $zipPath -Force}
$fullProjectPath = "$projectFolder\$projectName.csproj"

dotnet publish "$fullProjectPath"
     --configuration release --output $outputFolder

Create a compressed artifact folder

We will use System.IO.Compression.Filesystem assembly to create the zip file.

Add-Type -assembly "System.IO.Compression.Filesystem"
[IO.Compression.Zipfile]::CreateFromDirectory(
        $outputFolderTopDir, $zipPath)

Upload the zip into Azure web app

Next step is to upload the zip file into the Azure web app. This is where we first need to fetch the credentials for the Azure web app and then use the Kudu API to upload the content. Here’s the script:

function Get-PublishingProfileCredentials
         ($resourceGroupName, $webAppName) {

    $resourceType = "Microsoft.Web/sites/config"
    $resourceName = "$webAppName/publishingcredentials"

    $publishingCredentials = Invoke-AzureRmResourceAction `
                 -ResourceGroupName $resourceGroupName `
                 -ResourceType $resourceType `
                 -ResourceName $resourceName `
                 -Action list `
                 -ApiVersion 2015-08-01 `
                 -Force
    return $publishingCredentials
}

function Get-KuduApiAuthorisationHeaderValue
         ($resourceGroupName, $webAppName) {

    $publishingCredentials =
      Get-PublishingProfileCredentials $resourceGroupName $webAppName

    return ("Basic {0}" -f `
        [Convert]::ToBase64String( `
        [Text.Encoding]::ASCII.GetBytes(("{0}:{1}"
           -f $publishingCredentials.Properties.PublishingUserName, `
        $publishingCredentials.Properties.PublishingPassword))))
}

$kuduHeader = Get-KuduApiAuthorisationHeaderValue `
    -resourceGroupName $resourceGroupName `
    -webAppName $webAppName

$Headers = @{
    Authorization = $kuduHeader
}

# use kudu deploy from zip file
Invoke-WebRequest `
    -Uri https://$webAppName.scm.azurewebsites.net/api/zipdeploy `
    -Headers $Headers `
    -InFile $zipPath `
    -ContentType "multipart/form-data" `
    -Method Post

# Clean up the artifacts now
if (Test-Path $outputFolder)
      { Remove-Item $outputFolder -Recurse -Force; }
if (Test-path $zipName) {Remove-item $zipPath -Force}

PowerShell task in Visual Studio Online

Now we can leverage the Azure PowerShell task in Visual Studio Release pipeline and invoke the script to deploy the web job.

That’s it!

Thanks for reading, and have a nice day!

CQRS and ES on Azure Table Storage

Lately I was playing with Event Sourcing and command query responsibility segregation (aka CQRS) pattern on Azure Table storage. Thought of creating a lightweight library that facilitates writing such applications. I ended up with a Nuget package to do this. here is the GitHub Repository.

A lightweight CQRS supporting library with Event Store based on Azure Table Storage.

Quick start guide

Install

Install the SuperNova.Storage Nuget package into the project.

Install-Package SuperNova.Storage -Version 1.0.0

The dependencies of the package are:

  • .NETCoreApp 2.0
  • Microsoft.Azure.DocumentDB.Core (>= 1.7.1)
  • Microsoft.Extensions.Logging.Debug (>= 2.0.0)
  • SuperNova.Shared (>= 1.0.0)
  • WindowsAzure.Storage (>= 8.5.0)

Implemention guide

Write Side – Event Sourcing

Once the package is installed, we can start sourcing events in an application. For example, let’s start with a canonical example of UserController in a Web API project.

We can use the dependency injection to make EventStore avilable in our controller.

Here’s an example where we register an instance of Event Store with DI framework in our Startup.cs

// Config object encapsulates the table storage connection string
services.AddSingleton(new EventStore( ... provide config ));

Now the controller:

[Produces("application/json")]
[Route("users")]
public class UsersController : Controller
{
public UsersController(IEventStore eventStore)
{
this.eventStore = eventStore; // Here capture the event store handle
}

... other methods skipped here
}

Aggregate

Implementing event sourcing becomes way much handier, when it’s fostered with Domain Driven Design (aka DDD). We are going to assume that we are familiar with DDD concepts (especially Aggregate Roots).

An aggregate is our consistency boundary (read as transactional boundary) in Event Sourcing. (Technically, Aggregate ID’s are our partition keys on Event Store table – therefore, we can only apply an atomic operation on a single aggregate root level.)

Let’s create an Aggregate for our User domain entity:

using SuperNova.Shared.Messaging.Events.Users;
using SuperNova.Shared.Supports;

public class UserAggregate : AggregateRoot
{
private string _userName;
private string _emailAddress;
private Guid _userId;
private bool _blocked;

Once we have the aggregate class written, we should come up with the events that are relevant to this aggregate. We can use Event storming to come up with the relevant events.

Here are the events that we will use for our example scenario:

public class UserAggregate : AggregateRoot
{

... skipped other codes

#region Apply events
private void Apply(UserRegistered e)
{
this._userId = e.AggregateId;
this._userName = e.UserName;
this._emailAddress = e.Email;
}

private void Apply(UserBlocked e)
{
this._blocked = true;
}

private void Apply(UserNameChanged e)
{
this._userName = e.NewName;
}
#endregion

... skipped other codes
}

Now that we have our business events defined, we will define our commands for the aggregate:

public class UserAggregate : AggregateRoot
{
#region Accept commands
public void RegisterNew(string userName, string emailAddress)
{
Ensure.ArgumentNotNullOrWhiteSpace(userName, nameof(userName));
Ensure.ArgumentNotNullOrWhiteSpace(emailAddress, nameof(emailAddress));

ApplyChange(new UserRegistered
{
AggregateId = Guid.NewGuid(),
Email = emailAddress,
UserName = userName
});
}

public void BlockUser(Guid userId)
{
ApplyChange(new UserBlocked
{
AggregateId = userId
});
}

public void RenameUser(Guid userId, string name)
{
Ensure.ArgumentNotNullOrWhiteSpace(name, nameof(name));

ApplyChange(new UserNameChanged
{
AggregateId = userId,
NewName = name
});
}
#endregion


... skipped other codes
}

So far so good!

Now we will modify the web api controller to send the correct command to the aggregate.

public class UserPayload 
{
public string UserName { get; set; }
public string Email { get; set; }
}

// POST: User
[HttpPost]
public async Task Post(Guid projectId, [FromBody]UserPayload user)
{
Ensure.ArgumentNotNull(user, nameof(user));

var userId = Guid.NewGuid();

await eventStore.ExecuteNewAsync(
Tenant, "user_event_stream", userId, async () => {

var aggregate = new UserAggregate();

aggregate.RegisterNew(user.UserName, user.Email);

return await Task.FromResult(aggregate);
});

return new JsonResult(new { id = userId });
}

And another API to modify existing users into the system:

//PUT: User
[HttpPut("{userId}")]
public async Task Put(Guid projectId, Guid userId, [FromBody]string name)
{
Ensure.ArgumentNotNullOrWhiteSpace(name, nameof(name));

await eventStore.ExecuteEditAsync(
Tenant, "user_event_stream", userId,
async (aggregate) =>
{
aggregate.RenameUser(userId, name);

await Task.CompletedTask;
}).ConfigureAwait(false);

return new JsonResult(new { id = userId });
}

That’s it! We have our WRITE side completed. The event store is now contains the events for user event stream.

EventStore

Read Side – Materialized Views

We can consume the events in a seperate console worker process and generate the materialized views for READ side.

The readers (the console application – Azure Web Worker for instance) are like feed processor and have their own lease collection that makes them fault tolerant and resilient. If crashes, it catches up form the last event version that was materialized successfully. It’s doing a polling – instead of a message broker (Service Bus for instance) on purpose, to speed up and avoid latencies during event propagation. Scalabilities are ensured by means of dedicating lease per tenants and event streams – which provides pretty high scalability.

How to listen for events?

In a worker application (typically a console application) we will listen for events:

private static async Task Run()
{
var eventConsumer = new EventStreamConsumer(
... skipped for simplicity
"user-event-stream",
"user-event-stream-lease");

await eventConsumer.RunAndBlock((evts) =>
{
foreach (var @evt in evts)
{
if (evt is UserRegistered userAddedEvent)
{
readModel.AddUserAsync(new UserDto
{
UserId = userAddedEvent.AggregateId,
Name = userAddedEvent.UserName,
Email = userAddedEvent.Email
}, evt.Version);
}

else if (evt is UserNameChanged userChangedEvent)
{
readModel.UpdateUserAsync(new UserDto
{
UserId = userChangedEvent.AggregateId,
Name = userChangedEvent.NewName
}, evt.Version);
}
}

}, CancellationToken.None);
}

static void Main(string[] args)
{
Run().Wait();
}

Now we have a document collection (we are using Cosmos Document DB in this example for materialization but it could be any database essentially) that is being updated as we store events in event stream.

Conclusion

The library is very light weight and havily influenced by Greg’s event store model and aggreagate model. Feel free to use/contribute.

Thank you!

Prompt for Save changes in MMC 3.0 Application (C#)

Microsoft Management Console 3.0 is a managed platform to write and host application inside the standard Windows configuration console. It provides a simple, consistent and integrated management user interface and administrative console. In one of the product I am currently working with uses this MMC SDK. We used it to develop the configuration panel of administrative purposes.

That’s the quick background description of this post, however, this post is not meant for a guy who never worked with MMC SDK. I am assuming you already know all the basics about it.

In our application it has quite a number of ScopeNodes and each of them has associated details pages (in MMC terminology they are View, in our case most of them are FormView ). All of them have application data rendered in a WinForm’s UserControl. We allow user to modify those configuration data in place.
But the problem begins when user move to a different scope node and then close the MMC console window. Now the changes the user made earlier are not saved. As you already know that MMC console is an MDI application and the scope nodes are completely isolated from each others. Therefore, you can’t prompt user to save or discard the pending changes.

I googled a lot to get a solution for this, but ended up with a myriad of frustrations. Many people also faced the same problem and later they implemented with a pop-up dialog, so that they can handle the full lifecycle of saving functionalities. But that causes a lot of development works when you have too many scope nodes. You need to define 2 forms for each of them. One for the read-only display, another is a pop-up to edit the data. For my product, it was not viable really. In fact, it’s even looks nasty to get a pop-up for each node configuration.
Anyway, I finally resolved my problem by myself. It’s not a very good way to settle this issue, but it works superb for my purpose. So here is what I did to fix this problem. I have a class that will intercept the Windows messages and will take action while user is trying to close the main Console.



/// Subclassing the main window handle's WndProc method to intercept
/// the close event
internal class SubClassHWND : NativeWindow
{
private const int WM_CLOSE = 0x10;
private MySnapIn _snapIn; // MySnapIn class is derived from SnapIn (MMC SDK)
private List _childNativeWindows;

// Constructs a new instance of this class
internal SubClassHWND(MySnapIn snapIn)
{
this._snapIn = snapIn;
this._childNativeWindows = new List();
}

// Starts the hook process
internal void StartHook()
{
// get the handle
var handle = Process.GetCurrentProcess().MainWindowHandle;

if (handle != null && handle.ToInt32() > 0)
{
// assign it now
this.AssignHandle(handle);
// get the childrens
foreach (var childHandle in GetChildWindows(handle))
{
var childSubClass = new SubClassHWND(this._snapIn);

// assign this
childSubClass.AssignHandle(childHandle);

// keep the instance alive
_childNativeWindows.Add(childSubClass);
}
}
}

// The overriden windows procedure
protected override void WndProc(ref Message m)
{
if (_snapIn != null && m.Msg == WM_CLOSE)
{ // if we have a valid snapin instance
if (!_snapIn.CanCloseSnapIn(this.Handle))
{ // if we can close
return; // don't close this then
}
}
// delegate the message to the chain
base.WndProc(ref m);
}

// Requests the handle to close the window
internal static void RequestClose(IntPtr hwnd)
{
SendMessage(hwnd.ToInt32(), WM_CLOSE, 0, 0);
}

// Send a Windows Message
[DllImport("user32.dll")]
public static extern int SendMessage(int hWnd,
int Msg,
int wParam,
int lParam);


[DllImport("user32")]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool EnumChildWindows(IntPtr window, EnumWindowProc callback, IntPtr i);

public static List GetChildWindows(IntPtr parent)
{
List result = new List();
GCHandle listHandle = GCHandle.Alloc(result);
try
{
EnumWindowProc childProc = new EnumWindowProc(EnumWindow);
EnumChildWindows(parent, childProc, GCHandle.ToIntPtr(listHandle));
}
finally
{
if (listHandle.IsAllocated)
listHandle.Free();
}
return result;
}

private static bool EnumWindow(IntPtr handle, IntPtr pointer)
{
GCHandle gch = GCHandle.FromIntPtr(pointer);
List list = gch.Target as List;
if (list == null)
{
throw new InvalidCastException("GCHandle Target could not be cast as List");
}
list.Add(handle);
// You can modify this to check to see if you want to cancel the operation, then return a null here
return true;
}

public delegate bool EnumWindowProc(IntPtr hWnd, IntPtr parameter);
}

This class has the “WndProc” – windows message pump (or dispatcher method) method that will receive all the messages that are sent to the MMC console main window. Also this class will set message hook to all the child windows hosted inside the MMC MDI window.

Now we will only invoke the StartHook method from the SnapIn to active this interception hook.



// The subclassed SnapIn for my application
public class MySnapIn : SnapIn
{
internal SubClassHWND SubClassHWND
{
get;
private set;
}

protected MySnapIn()
{
// create the subclassing support
SubClassHWND = new SubClassHWND(this);
}

// Start the hook now
protected override void OnInitialize()
{
SubClassHWND.StartHook()
}

Now we have the option to do something before closing the application, like prompting with a Yes, No and Cancel dialog like NotePad does for a dirty file.



// Determins if the snapin can be closed now or not
internal bool CanCloseSnapIn(IntPtr requestWindow)
{
if (IsDirty)
{ // found a node dirty, ask user if we can
// close this dialog or not
this.BeginInvoke(new System.Action(() =>
{
using (var dlg = new SnapInCloseWarningDialog())
{
var dlgRes = Console.ShowDialog(dlg);

switch (dlgRes)
{
case DialogResult.Yes:
SaveDirtyData(); // save them here
IsDirty = false; // set to false, so next
// time the method
// will not prevent
// closing the application
SubClassHWND.RequestClose(requestWindow);
break;
case DialogResult.No:
IsDirty = false;
SubClassHWND.RequestClose(requestWindow);
break;
case DialogResult.Cancel: break;// Do nothing
}
}
}));
return false;
}
return true;
}


One small problem remains though. The dispatcher method gets the WM_CLOSE in a thread that can’t display a Window due to the fact that the current thread is not really a GUI thread. So we have to do a tricky solution there. We need to display the prompt by using a delegate (using BeginInvoke) and discard the current WM_CLOSE message that we intercepted already.
Later when a choice has been made (user selected yes, no or cancel), if they selected ‘Yes’ then we have to close the application after saving the data. If ‘no’ selected we will have to close the SnapIn as well. Only for ‘Cancel’ we don’t have to do anything. So only thing is critical is how we can close this window again. Here is how we can do that:

Notice that SnapIn’s CanCloseSnapIn method does have a parameter which is the pointer (an instance of IntPtr in this case) of the window handle that has been closed by the user. This has been done on purpose. This will offer the possiblity to send a WM_CLOSE again to that same window. So even if user closes the MDI child it will only close the child window only after save- which is just perfect!

Hope this will help somebody struggling with the same gotcha.