Rotating Temporal CA Certificates

Certificates must be rotated, eventually...

By: Chris - Backend Lead Engineer DelightRoom

The Crisis

One crisp autumn day, we started receiving warning emails like the one below from Temporal Cloud.

Looking into it, the certificate we were currently using had been created about a year ago and was nearing its expiration date.

For reference, you can check the exact expiration date of a certificate using the openssl terminal command.

$ openssl x509 -in ca.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            58:78:12:b0:c9:aa:15:dd:11:bd:f2:e2:48:b5:27:dc:3b:8b:e0:58
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = DelightRoom
        Validity
            Not Before: Nov  8 18:23:15 2022 GMT
            Not After : Nov  8 18:23:15 2023 GMT
        Subject: O = DelightRoom
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:b4:b9:5f:8f:6e:de:b1:99:f5:10:0b:8e:d5:ac:
...

Since the expiration is 16:23:15 on November 8, 2023, GMT, it essentially expires in the early hours of November 9th in Korean time. If we don't rotate it immediately, critical services could stop working!

Wait! ✋️ Suddenly you keep mentioning Temporal, Cloud, and so on... What are those?

Temporal is a platform and underlying technology that enables durable execution of application code. Temporal Cloud is the cloud offering of this open-source tech. I'll cover this in more detail in a separate post. For the purposes of this article, you just need to know that certificate management is necessary because the external Temporal Cloud Server and our internal application, Temporal Worker, communicate using mTLS.

Research

We could have just regenerated and updated the certificates using the exact same method we used when we first adopted Temporal about a year ago, but I did a quick bit of research to make this rotation process more efficient and systematic.

The following official Temporal documentation explains the general considerations and how-tos for managing certificates when using Temporal Cloud.

How to manage certificates in Temporal Cloud | Temporal Documentation

On the blog of Keith Tenzer, a Solution Architect at Temporal, I found a post specifically tailored for those looking to do a CA certificate rotation.

Temporal Cloud Certificate Rotation

Using the second post as a reference, since it seemed a bit friendlier and less complex, let's go ahead and rotate the certificate.

Rotating the Certificate

CA certificate rotation generally consists of the following steps.

1. Generate a new CA certificate

2. Upload the CA certificate to the server

3. Generate a Leaf certificate for the client to use

4. Update the client to use the new Leaf certificate

5. Verify it works

6. Delete the old CA certificate that is nearing expiration

Generating a CA Certificate

First, we need to renew the CA certificate that gets uploaded to the Temporal server (in our case, the Temporal Cloud server). Conveniently, if you're using Cloud, you'll start getting notification emails 10 days before expiration.

tcld

There are several ways to generate a certificate. First, I tried tcld (https://github.com/temporalio/tcld), the CLI tool for Temporal Cloud.

$ tcld gen ca --org DelightRoom -d 1y --ca-cert ca.pem --ca-key ca.key

You can easily generate a CA certificate by running the command above in your desired location.

Unfortunately, we couldn't use the CA generated this way this time. This was because appending a CA certificate with the same subject (“O=DelightRoom”) as the existing certificate to the bundle wasn't allowed. Of course, we could have uploaded it by creating it under a different org, but I wanted to keep the org the same. As an alternative, I tried changing another attribute like the common name, but tcld didn't support that yet.

certstrap

So, I pivoted to an alternative: certstrap (https://github.com/square/certstrap).

$ certstrap init --common-name DelightRoomCertAuth \\
  --organization DelightRoom --expires "3 years"

Created out/DelightRoomCertAuth.key
Created out/DelightRoomCertAuth.crt
Created out/DelightRoomCertAuth.crl

The command above generates a CA certificate with the CN DelightRoomCertAuth. We set the expiration to 3 years. By default, it gets created in the out folder like you see above.

Uploading the CA Certificate

tcld also provides a feature to upload the CA certificate to be used in a namespace. Therefore, we'll make use of this. One thing to be careful about is that Temporal clients like Workers and API servers attached to the existing CA certificate will still be active, so you must only append the new one without deleting the old one.

The following command appends the new certificate to the CA certificate list used by the namespace called primary-prod.xAxPa on Temporal Cloud.

$ tcld n ca add --namespace primary-prod.xAxPa -f out/DelightRoomCertAuth.crt

Generating a Leaf Certificate for the Client

Now let's generate a leaf certificate that will be authenticated via that CA, so it can be used by the client. We'll use certstrap for this as well. We'll continue in the same directory as before. Then the out folder acts almost like a staging area, utilizing the crt and key inside it right away.

First, we make a certificate request. Let's use the namespace name as the Common name.

$ certstrap request-cert --common-name primary-prod.xAxPa

Created out/primary-prod.xAxPa.key
Created out/primary-prod.xAxPa.csr

After that, we proceed with signing using the new CA's key.

$ certstrap sign primary-prod.xAxPa --CA DelightRoomCertAuth

Created out/primary-prod.xAxPa.crt from out/primary-prod.xAxPa.csr signed by out/DelightRoomCertAuth.key

Now, the Temporal client can just utilize the generated primary-prod.xAxPa.key and primary-prod.xAxPa.crt!

Dropping this straight into production is a bit nerve-wracking, so we can test it out beforehand to see if it works properly. Using the Temporal CLI tool (https://github.com/temporalio/cli), you can test it relatively easily as shown below. If there's an issue, you'll get a certificate error.

$ temporal workflow list --address primary-prod.xAxPa.tmprl.cloud:7233 --namespace primary-prod.xAxPa --tls-cert-path out/primary-prod.xAxPa.crt --tls-key-path out/primary-prod.xAxPa.key

Updating the Client-Side Certificate

Now we need to rotate the certificates used by the clients to the newly generated leaf certificate. This process can vary widely depending on how you're actually applying configurations and secrets to your deployment environment. Since we inject secrets via Doppler → External Secret → K8s Secret, we were able to update it relatively painlessly.

Since issues can also arise during this process, I highly recommend deploying by gradually scaling up replicas using the new certificate rather than outright killing the existing clients immediately.

After the update, monitor until all clients using the old certificate have shut down.

Removing the Old CA Certificate

Once you've confirmed that no clients are using the old certificate, you can now remove the old, soon-to-expire CA certificate from the namespace. You can remove it directly via Temporal Cloud's Web UI, or you can conveniently use a terminal command like the following.

$ tcld n ca remove --namespace primary-prod.xAxPa -f old_ca.pem

This requires the old CA certificate, so make sure to have it ready beforehand if you don't already have it locally.

Follow-Up

For smaller companies like ours that don't use a separate certificate management infrastructure, there is the burden of manually auditing and updating the authentication system as seen above. And this process harbors potential risks like human error. For instance, what if a certificate expiration warning email was sent to the admin, but they missed it or it went to spam? We could easily suffer a sudden outage of a critical service.

To solve this problem, we plan to proceed with the following follow-up tasks.

• Build an internal expiration risk check and warning alert system for the CA and leaf certificates currently in use.

• Automate rotation to periodically and automatically update the certificates at an interval shorter than the expiration period.

If management becomes too difficult down the road—due to the number of certificates getting too large, for example—adopting a dedicated certificate management system, even if it costs money, could be an option.

Wrapping Up

In this post, we've looked at how to rotate CA certificates when using Temporal Cloud. Whether you're using Temporal Cloud or hosting it yourself, you'll probably run into the need to rotate certificates periodically. I hope this post was at least somewhat helpful to Temporal users facing this situation.