TL; DR: This article and accompanying source codes helps you setting up an automated secret or certificate rotations for Azure service principals and Azure DevOps service connections. Jump into the technical part here.
Background
If you are deploying applications on Azure from Azure DevOps service connections (or GitHub actions) you are most likely using a Service Principal (via AAD app registration). These are identities that are created and used to non-interactively communicate Azure from external systems. These are unattended processes, many of the Identity protection capabilities (i.e., Multi-Factor Authentication) are not easy to apply on them. Hence, you need to take extra measures to minimize the risk of possible compromises. Here are few obvious measures that you should consider for these service accounts (service principals).
- Create service principals for each specific tasks. For instance, do NOT use single service principals that has access to non-production and production environments simultaneously.
- Grant minimum Azure Roles that the service principal needs (the least privilege principle).
Managed identities
Besides these, you can further strengthen and enhance your security posture by leveraging Managed Identities on Azure.
Managed identities eliminate the need for developers to manage credentials. Managed identities provide an identity for applications to use when connecting to resources that support Azure Active Directory (Azure AD) authentication. However, one can only leverage Managed Identity if the process is running on a compute hosted on Azure that supports Managed Identity (i.e., Azure VM, App services etc.). However, majority of the Azure Pipelines are running on Microsoft Hosted agents where you do not have Managed Identity. Therefore, you need to use service principals with secrets and there come the vulnerabilities too. It’s important to acknowledge that, you can create self-hosted agents on Azure DevOps (or GitHub runners) which can be hosted on Azure VM with Managed Identities. However, that has cost implications and not an attractive alternative to many businesses.
The question then is what we can do to improve the security of service principal secrets.
Automatic credential rotations
Azure AD service principals offer two flavors of authentications, password based, and certificate based. If you can’t adopt self-hosted agents and Managed Identities, you should at least make sure the passwords or certificates are frequently rotated. More frequently you rotate, more difficult it becomes for an attacker to cause any damage with a compromised credential.
Frequently (let’s say daily) rotating these secrets is a toil that you must automate, specially when you have large number of service principals. Rest of the article will give you some insights (with example codes) how to achieve just that.
Rotate service principal credentials.
First thing first, we would need service that can create new secret (or certificate) and delete older ones. As this process would need high privileged Azure AD roles (to create or delete secrets) we will not create another service principal for that. That would not serve our objective, i.e., trying to secure service principal by introducing even more powerful service principal that has secrets.
Azure Function with Managed Identity
We will use Managed identity for that and host it in an Azure Function. Considering cost and simplicity Azure Functions stands out to be the most viable option for this. Upon creating the function on Azure with Managed identity we need to grant AAD roles to the identity – so that it can (only from within the function) read and write Azure AD Application information (i.e., Service Principals). The following Azure CLI (bash file) commands show how to do that.
functionName="Az_FUNC_NAME"
msiObjectId=$(az resource list -n $functionName --query [*].identity.principalId --out tsv)
graphAppID=$(az ad sp list --display-name "Microsoft Graph" --query [0].objectId --out tsv)
appReadWriteOwnedByRoleId=$(az ad sp list --display-name "Microsoft Graph" --query "[0].appRoles[?value=='Application.ReadWrite.OwnedBy' && contains(allowedMemberTypes, 'Application')].id" --output tsv)
appReadWriteAllRoleId=$(az ad sp list --display-name "Microsoft Graph" --query "[0].appRoles[?value=='Application.ReadWrite.All' && contains(allowedMemberTypes, 'Application')].id" --output tsv)
uri=https://graph.microsoft.com/v1.0/servicePrincipals/$msiObjectId/appRoleAssignments
body="{'principalId':'$msiObjectId','resourceId':'$graphAppID','appRoleId':'$appReadWriteOwnedByRoleId'}"
az rest --method post --uri $uri --body $body --headers "Content-Type=application/json"
body="{'principalId':'$msiObjectId','resourceId':'$graphAppID','appRoleId':'$appReadWriteAllRoleId'}"
az rest --method post --uri $uri --body $body --headers "Content-Type=application/json"
Rotate secrets
Once the function has granted rights to read/write secrets for AAD applications we can leverage the Microsoft Graph APIs to create new secrets for an application (or Service Principal). I am using C# on .net 5 for these examples, but you can do this in PowerShell or any other language too.
var newPassCred = await graph.Applications[application.Id]
.AddPassword(new PasswordCredential
{
DisplayName = $"Auto Generated: {now}",
Hint = $"Auto Generated: {now}",
StartDateTime = now,
EndDateTime = now.AddDays(payload.LifeTimeInDays)
})
.Request().PostAsync();
Also, we can delete older credentials the same way:
await graph
.Applications[application.Id]
.RemovePassword(oldPassCred.KeyId.Value)
.Request().PostAsync();
Rotate certificates
Rotating certificates would require us to write a bit more code in contrast to the secret rotation above. You can bring your own certificate for this -which will reduce the code little bit. But I would go with a Self-Signed Certificate that the Azure function will generate on the fly and attach that to the service principals’ credentials.
Self-Signed certificate
Generating self-signed certificate in .net 5 is super convenient with the new Cryptography APIs. The following method will do that job for us.
public static X509Certificate2 CreateSelfSignedCertificateAsync(
string CA = "MoHA Corp Inc.",
int validForDays = 30)
{
using RSA rootRSA = RSA.Create(4096);
using RSA certRSA = RSA.Create(2048);
var certAuthorityReq = new CertificateRequest(
$"CN={CA}",
rootRSA,
HashAlgorithmName.SHA256,
RSASignaturePadding.Pkcs1);
certAuthorityReq.CertificateExtensions.Add(
new X509BasicConstraintsExtension(true, false, 0, true));
certAuthorityReq.CertificateExtensions.Add(
new X509SubjectKeyIdentifierExtension(certAuthorityReq.PublicKey, false));
using var rootCertificate = certAuthorityReq
.CreateSelfSigned(
DateTimeOffset.UtcNow.AddDays(-30),
DateTimeOffset.UtcNow.AddDays(365));
var csr = new CertificateRequest(
$"CN={CA}",
certRSA,
HashAlgorithmName.SHA256,
RSASignaturePadding.Pkcs1);
csr.CertificateExtensions.Add(
new X509BasicConstraintsExtension(false, false, 0, false));
csr.CertificateExtensions.Add(
new X509KeyUsageExtension(
X509KeyUsageFlags.DigitalSignature
| X509KeyUsageFlags.NonRepudiation
| X509KeyUsageFlags.CrlSign
| X509KeyUsageFlags.DataEncipherment
| X509KeyUsageFlags.KeyAgreement
| X509KeyUsageFlags.KeyCertSign
| X509KeyUsageFlags.KeyEncipherment,
false));
csr.CertificateExtensions.Add(
new X509EnhancedKeyUsageExtension(new OidCollection { new Oid("1.3.6.1.5.5.7.3.8") }, true));
csr.CertificateExtensions.Add(
new X509SubjectKeyIdentifierExtension(csr.PublicKey, false));
using var certificate = csr.Create(
rootCertificate,
DateTimeOffset.UtcNow.AddDays(-1),
DateTimeOffset.UtcNow.AddDays(validForDays),
new byte[] { 1, 2, 3, 4 });
return certificate.CopyWithPrivateKey(certRSA);
}
With that dynamically generated certificate we can go back to our Graph API to configure the certificate as key credentials for AAD applications (or service principals).
var selfSignedCertificate =
CertificateUtils.CreateSelfSignedCertificateAsync(validForDays: payload.LifeTimeInDays);
var certificateCredentail = new KeyCredential
{
StartDateTime = now,
EndDateTime = now.AddDays(payload.LifeTimeInDays),
Type = "AsymmetricX509Cert",
Usage = "Verify",
Key = CertificateUtils.GetPfxAsBytes(selfSignedCertificate)
};
var app = new Application
{
KeyCredentials = new List<KeyCredential> { certificateCredentail }
};
await graph.Applications[application.Id].Request().UpdateAsync(app);
Unlike secret rotation, we do not need to delete the old certificate as we are overwriting the older ones immediately.
Azure DevOps service connection
We have now a function that can be scheduled at night (or every six hours) to rotate service principals secrets and certificates, however, we need to update the Azure DevOps to use the newly created credentials too. That bit is easier to accomplish. We will use the Azure DevOps Rest API to do that.
public async Task<VstsServiceEndpoint> GetServiceEndpointsAsync(Guid projectId, Guid endpointId)
{
var requestPath = $"{OrgName}/{projectId}/_apis/serviceendpoint/endpoints/{endpointId}?api-version=6.0-preview.4";
return await GetRestAsync<VstsServiceEndpoint>(requestPath);
}
public async Task<VstsServiceEndpoint> UpdateServiceEndpointsAsync(Guid projectId, Guid endpointId, VstsServiceEndpoint endpoint)
{
var requestPath = $"{OrgName}/{projectId}/_apis/serviceendpoint/endpoints/{endpointId}?api-version=6.0-preview.4";
return await PutRestAsync<VstsServiceEndpoint, VstsServiceEndpoint>(requestPath, endpoint);
}
Update secrets in Azure DevOps
These two methods do plain REST calls to Azure DevOps – one reads a Service Connection (aka Service endpoint) and other updates. With that, we can update a newly generate password credentials for Azure AD applications (or service principals) as follows:
var endpoint = await azdo.GetServiceEndpointsAsync(payload.ProjectId, endpointId);
endpoint.Authorization.Parameters.Serviceprincipalkey = newPassCred.SecretText;
await azdo.UpdateServiceEndpointsAsync(payload.ProjectId, endpoint.Id, endpoint);
Update certificates in Azure DevOps
Certificate updates needs little extra work from us. Earlier we issued self-signed certificates and configured that to Azure AD apps/Service Principals. We would need to generate the PEM format from that certificate first then update it via Azure DevOps REST APIs.
public static string GeneratePEMWithPrivateKeyAsString(X509Certificate2 certificate)
{
var sb = new StringBuilder();
AsymmetricAlgorithm key = certificate.GetRSAPrivateKey();
sb.AppendLine(new string(PemEncoding.Write("PRIVATE KEY", key.ExportPkcs8PrivateKey())));
sb.AppendLine();
sb.AppendLine(new string(PemEncoding.Write("CERTIFICATE", certificate.GetRawCertData())));
return sb.ToString();
}
Then update the service connection:
endpoint.Authorization.Parameters
.ServicePrincipalCertificate = CertificateUtils.GeneratePEMWithPrivateKeyAsString(selfSignedCertificate);
await azdo.UpdateServiceEndpointsAsync(payload.ProjectId, endpoint.Id, endpoint);
Conclusion
That’s all for today. You can find the complete solution in this GitHub repository. This is under MIT license, you are free to use, modify the code anyway you want. Of course, I would appreciate if you acknowledged if this code helped you. That surely motivates and makes my day!
Thanks for reading!
Reverse Engineering 