Enforce stateless containers on Azure

Background

In the constantly changing realm of containerization and cloud computing, the demand for dynamic and scalable solutions has taken center stage. I recently faced a distinctive challenge from a client who was determined to attain stateless containers throughout their Azure infrastructure. While Azure Kubernetes Services (AKS) provided a built-in policy to restrict stateful containers, the same level of control was not readily available for Azure Container Apps (ACA) and Azure Container Instances (ACI). In this blog post, I’ll share my journey in helping the customer by crafting custom policies for ACA and ACI to ensure a consistent, stateless container environment.

Stateless Containers on Azure Kubernetes Services (AKS):

The first challenge was to enforce statelessness on Azure Kubernetes Services (AKS). Fortunately, AKS offers built-in policies that can be leveraged to control various aspects of the cluster. I opted to use Azure Policy, a service in Azure that you can use to create, assign, and manage policies. There is a built-in policy in Azure that can be used for this challenge. The policy named [Preview]: Kubernetes clusters should restrict creation of given resource type. This is a policy that prevents deploying a certain Kubernetes resource that we can provide via parameter. Essentially, assign this policy with a resource kind set to “PersistentVolumeClaim“. Which triggers when a container gets deployed with PersistentVolumeClaim.

In orde for it to work, The AKS must enable the Azure Policy Add-On which deploys the Gatekeeper (from Open Policy Agent) within the cluster to sync Azure Policies and evaluate violations Just-In-Time using the Admission controller hooks.

This policy effectively denies any AKS cluster that attempts to use storage profiles other than Managed Disks, ensuring a stateless container environment.

Stateless Containers on Azure Container Apps (ACA):

Moving on to Azure Container Apps (ACA), the challenge was more intricate as there wasn’t a built-in policy for restricting stateful containers. In this scenario, I opted for Azure Custom Policy Configuration, a feature that extends Azure Policy to evaluate settings.

Here’s the policy rule for that:

{
  "if": {
    "allOf": [
      {
        "field": "type",
        "equals": "Microsoft.App/containerApps"
      },
      {
        "count": {
          "field": "Microsoft.App/containerApps/template.volumes[*]",
          "where": {
            "field": "Microsoft.App/containerApps/template.volumes[*].storageType",
            "equals": "AzureFile"
          }
        },
        "greaterOrEquals": 1
      }
    ]
  },
  "then": {
    "effect": "[parameters('effect')]"
  }
}

With this policy.json we can now deploy the policy definition to the subscription or management group.

jq .properties.parameters < policy.json > azurepolicy.parameters.json
jq .properties.policyRule < policy.json > azurepolicy.rules.json

subscriptionId=$(az account show --query id --output tsv)

az policy definition create --name StatelessContainerApps \
  --display-name "[Custom] Container Apps should be stateless" \
  --description "Container App should not configure persistent volume mount using Azure FileShare so they can be stateless." \
  --metadata version="0.1.0" category="Container Apps" preview=true \
  --mode Indexed \
  --params "@azurepolicy.parameters.json" \
  --rules "@azurepolicy.rules.json" \
  --subscription $subscriptionId

Once the policy definition is deployed we can assign it to any scope (subscription or management group) to start enforcing it. This policy denies any ACA application that contains containers with volume mounts, ensuring a stateless configuration.

Stateless Containers on Azure Container Instances (ACI):

For Azure Container Instances (ACI), a similar challenge presented itself – the absence of a built-in policy for enforcing statelessness. Therefore, following the same route as we did above for Azure Container Apps.

I developed a custom policy definition that scans the ACI instances for containers with persistent storage. Here’s a simplified example:

{
  "if": {
    "allOf": [
      {
        "field": "type",
        "equals": "Microsoft.ContainerInstance/containerGroups"
      },
      {
        "count": {
          "field": "Microsoft.ContainerInstance/containerGroups/volumes[*]",
          "where": {
            "field": "Microsoft.ContainerInstance/containerGroups/volumes[*].name",
            "equals": "azurefile"
          }
        },
        "greaterOrEquals": 1
      }
    ]
  },
  "then": {
    "effect": "[parameters('effect')]"
  }
}

This policy denies any ACI container group that contains containers with volume mounts, ensuring the desired stateless configuration.

Conclusion:

With these policies we can now audit (or deny) stateful containers deployed either on AKS, ACA or ACIs and monitor the compliancy in single place.

In the dynamic world of container orchestration, achieving statelessness is crucial for scalability and efficiency. While Azure provides built-in policies for some services, custom policies can bridge the gaps. In this case, crafting policies for AKS, ACA, and ACI ensured a consistent and stateless container environment for the customer. As container technologies continue to evolve, the ability to create tailored solutions becomes increasingly valuable.