CloudOven – Terraform at ease!

TL;DR:

  • URL: CloudOven 

  • Use Google account or sign-up 
  • Google Chrome please! (I’ve not tested on other browsers yet)

e2e

Background

In recent years I have spent fair amount of time in design and implementation of Infrastructure as code in larger enterprise context. Terraform seemed to be a tool of choice when it comes to preserve the uniformity in Infrastructure as code targeting multiple cloud providers. It is rapidly becoming a de facto choice for creating and managing cloud infrastructures by writing declarative definitions. It’s popular because the syntax of its files is quite readable and because it supports several cloud providers while making no attempt to provide an artificial abstraction across those providers. The active community will add support for the latest features from most cloud providers.

However, rolling out Terraform in many enterprises has its own barrier to face. Albeit the syntax (HCL) is neat, but not every developers or Infrastructure operators in organizations finds it easy. There’s a learning curve and often many of us lose momentum discovering the learning effort. I believe if we could make the initial ramp-up easier more people would play with it.

That’s one of my motivation for this post, following is the other one.

Blazor meets Terraform

Lately I was learning Blazor – the new client-side technology from Microsoft. Like many others, I find one effective way learning a new technology by creating/building solution to a problem. I have decided to build a user interface that will help creating terraform scripts easier. I will share my journey in this post.

Resource Discovery in Terraform Providers

Terraform is powerful for its providers. You will find Terraform providers for all major cloud providers (Azure, AWS, Google etc.). The providers then allow us to define “resource” and “data source” in Terraform scripts. These resource and data source have arguments and attributes that one must know while creating terraform files. Luckily, they are documented nicely in Terraform site. However, it still requires us to jump back and forth to the documentation site and terraform file editor (i.e. VSCode).

Azure-Discovery

To make this experience easier, I wrote a crawler application that downloads the terraform providers (I am doing it for Azure, AWS and google for now) and discovers the attributes and arguments for each and every resource and data source. I also try to extract the documentation for every attributes and arguments from the terraform documentation site with a layman parsing (not 100% accurate but works for majority. Something I will improve soon).

GoogleAWS-discovery

This process generates JSON structure for each resource and data source, enriches them with the documentation and stores them in an Azure Blob Storage.

Building Infrastructure as code

Now that I have a structured data store with all resources and data sources for any terraform provider, I can leverage that building a user interface on top of it. To keep things a bit organized, I started with a concept of “project”.

workflow
The workflow

Project

I can start by creating a project (well, it can be a product too, but let’s not get to that debate). Project is merely a logical boundary here.

Blueprint

Within a project I can create Blueprint(s). Blueprint(s) are the entity that retains the elements of the infrastructure that we are aiming to create. For instance, a Blueprint targets to a Cloud provider (i.e. Azure). Then I can create the elements (resource and data sources) within the blueprint (i.e. Azure Web App, Cosmos DB etc.).

provider-configuration

Blueprints keeps the base structure of all the infrastructure elements. It allows defining variables (plain and simple terraform variables) so the actual values can vary in different environments (dev, test, pre-production, production etc.).

Once I am happy with the blueprint, I can download them as a zip – that contains the terraform scripts (main.tf and variable.tf). That’s it, we have our infrastructure as code in Terraform. I can execute them on a local development machine or check them in to source control – whatever I prefer.

storage_account

One can stop here and keep using the blueprint feature to generate Infrastructure as code. That’s what it is for. However, the next features are just to make the overall experience of running terraform a bit easier.

Environments

Next to blueprint, we can create as many environments we want. Again, just a logical entity to keep isolation of actual deployment for different environments.

Deployments

Deployment entity is the glue that ties a blueprint to a specific environment. For instance, I can define a blueprint for “order management” service (or micro-service maybe?), create an environment as “test” and then create a “deployment” for “order management” on “test”. This is where I can define constant values to the blueprint variable that are specific to the test environment.

Terraform State

Perhaps the most important aspect the deployment entity holds is the terraform state management. Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. This state is stored by default in a local file named “terraform.tfstate”, but it can also be stored remotely, which works better in a team environment. Defining the state properties (varies in different cloud providers) in deployment entity makes the remote state management easier – specifically in team environment. It will configure the remote state to the appropriate remote backend. For instance, when the blueprint cloud provider is set to Azure, it will configure Azure Storage account as terraform state remote backend, for AWS it will pick S3 automatically.

e2e

Terraform plan

Once we have deployment entity configured, we can directly from the user interface run “terraform plan”. The terraform plan command creates an execution plan. Unless explicitly disabled, it performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the blueprint. This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

Terraform apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan. Like “plan”, the “apply” command can also be issued directly from the user interface.

Terraform plan and apply both are issued in an isolated docker container and the output is captured and displayed back to the user interface. However, there’s a cost associated running docker containers on cloud, therefore, it’s disabled in the public site.

Final thoughts

It was fun to write a tool like this. I recommend you give it a go. Especially if you are stepping into Terraform. It can also be helpful for experienced Terraform developers – specifically with the on-screen documenation, type inferance and discovery features.

Some features, I have working progress:

  • Ability to define policy for each resources and data types
  • Save a Blueprint as custom module

Stay tuned!

 

Continuously deploy Blazor SPA to Azure Storage static web site

Lately I am learning ASP.net Blazor – the relatively new UI framework from Microsoft. Blazor is just awesome – the ability to write c# code both in server and client side is extremely productive for .net developers. From Blazor documentations:

Blazor lets you build interactive web UIs using C# instead of JavaScript. Blazor apps are composed of reusable web UI components implemented using C#, HTML, and CSS. Both client and server code is written in C#, allowing you to share code and libraries.

I wanted to write a simple SPA (Single Page Application) and run it as server-less. Azure Storage offers hosting static web sites for quite a while now. Which seems like a very nice option to run a Blazor SPA which executes into the user’s browser (within the same Sandbox as JavaScript does). It also a cheap way to run a Single Page application in Cloud.

I am using GitHub as my source repository for free (in a private repository). Today wanted to create a pipeline that will continuously deploy my Blazor app to the storage account. Azure Pipelines seems to have pretty nice integration with GitHub and it’s has a free tier as well . If either our GitHub repository or pipeline is private, Azure Pipeline still provide a free tier. In this tier, one can run one free parallel job that can run up to 60 minutes each time until we’ve used 1800 minutes per month. That’s pretty darn good for my use case.

I also wanted to build the project many times in my local machine (while developing) in the same way it gets built in the pipeline. Being a docker fan myself, that’s quite no-brainer. Let’s get started.

Pre-requisite

I have performed few steps before I ran after the pipeline – that are beyond the scope of this post.

  • I have created an Azure Subscription
  • Provisioned resource groups and storage account
  • I have created a Service Principal and granted Contributor role to the storage account

Publishing in Docker

I have created a docker file that will build the app, run unit tests and if all goes well, it will publish the app in a folder. All of these are standard dotnet commands.
Once, we have the application published in a folder, I have taken the content of that folder to a Azure CLI docker base image (where CLI is pre-installed) and thrown away the rest of the intermediate containers.

Here’s our docker file:

The docker file expects few arguments (basically the service principal ID, the password of the service principal and the Azure AD tenant ID – these are required for Azure CLI to sign-in to my Azure subscription). Here’s how we can build this image now:

Azure Pipeline as code

We have now the container, time to run it every time a commit has been made to the GitHub repository. Azure Pipeline has a yaml format to define pipeline-as-code – which is another neat feature of Azure Pipelines.

Let’s see how the pipeline-as-code looks like:

I have committed this to the same repository into the root folder.

Creating the pipeline

We need to login to Azure DevOps and create a project (if there’s none). From the build option we can create a new build definition.

ado

The steps to create build definition is very straightforward. It allows us to directly point to a GitHub repository that we want to build.
Almost there. We need to supply the service principal ID, password, tenant ID and storage account names to this pipeline – because both our docker file and the pipeline-as-code expected them as dependencies. However, we can’t just put their values and commit them to GitHub. They should be kept secret.

Azure Pipeline Secret variables

Azure Pipeline allows us to define secret variable for a pipeline. We need to open the build definition in “edit” mode and then go to the top-right ellipses button as below:

ado1

Now we can define the values of these secret and keep them hidden (there’s a lock icon there).

ad02

That’s all what we need. Ready to deploy the code to Azure storage via this pipeline. If we now go an make a change in our repository it will trigger the pipeline and sure enough it will build-test-publish-deploy to Azure storage as a Static SPA.

Thanks for reading!