Automating Kubernetes TLS with cert-manager and Managed CAs

Most engineers operating Kubernetes clusters intuitively assume that all traffic within and to their cluster is encrypted by default. This assumption, unfortunately, is often incorrect. While communication between your kubectl client and the Kubernetes API server typically uses TLS, and the API server's connection to etcd is usually secured depending on provisioning, traffic between your pods is plaintext by default. Similarly, Ingress traffic from the internet to your services only becomes encrypted if you explicitly configure TLS.

Kubernetes' design philosophy provides powerful primitives but intentionally leaves implementation details like comprehensive certificate management to the user. This approach, while flexible, introduces a notorious pain point: manual certificate provisioning, expiration, and rotation. Forgetting to rotate certificates can lead to significant outages.

This is where cert-manager steps in. It's a Kubernetes controller that streamlines certificate management by watching for Certificate resources, automatically requesting certificates from configured issuers (like Let's Encrypt or an internal Certificate Authority), storing them in Kubernetes Secrets, and rotating them before they expire. With cert-manager, you declare your desired certificate state, and it ensures that state is maintained, freeing your applications and engineers from the operational burden of TLS.

How cert-manager Automates TLS

cert-manager extends the Kubernetes API with Custom Resources (CRs) to define its operations. The primary resources you'll interact with are:

• Issuer and ClusterIssuer: These define the source of your certificates. An Issuer is namespace-scoped, suitable for application-specific CAs, while a ClusterIssuer is cluster-wide, ideal for shared infrastructure like Let's Encrypt.
• Certificate: This resource declares your intent to obtain a certificate, specifying details like the domain names, the desired Issuer/ClusterIssuer, and the Kubernetes Secret where the certificate and private key should be stored.
• CertificateRequest: Automatically created by cert-manager in response to a Certificate resource, this represents an individual certificate signing request sent to the issuer.

When a Certificate is created, cert-manager generates a CertificateRequest with a Certificate Signing Request (CSR), passes it to the configured issuer, handles any necessary validation challenges, receives the signed certificate, and stores it in the specified Secret. It then continuously monitors the certificate's expiry, automatically initiating renewal well before it becomes invalid (typically when two-thirds of its validity period has passed).

Encrypting External Traffic with Let's Encrypt (ACME)

For public-facing services, Let's Encrypt is a popular choice for obtaining free, trusted TLS certificates. cert-manager integrates with Let's Encrypt via the ACME (Automatic Certificate Management Environment) protocol. To prove domain ownership, ACME offers two primary challenge types:

1. HTTP-01: cert-manager creates a temporary HTTP endpoint (http://<your-domain>/.well-known/acme-challenge/<token>). Let's Encrypt validates ownership by requesting this URL and matching the expected token. This method requires your cluster to be internet-reachable on port 80 and is suitable for single domains.
2. DNS-01: cert-manager creates a temporary DNS TXT record (_acme-challenge.<your-domain>). Let's Encrypt checks for this record. This method is crucial for private clusters that are not publicly exposed via HTTP, and it's the only way to obtain wildcard certificates (e.g., *.example.com). The trade-off is that it requires cert-manager to have API access to your DNS provider to automate record creation.

When using a ClusterIssuer configured for ACME, you can secure an Ingress resource simply by adding an annotation like cert-manager.io/cluster-issuer: letsencrypt-production and defining the TLS section with the desired hosts and secretName. cert-manager then handles the entire issuance process.

Securing Internal Service-to-Service Traffic with an Internal CA

For traffic between services within your Kubernetes cluster, using a public CA like Let's Encrypt is often impractical or undesirable. Instead, you can establish an internal Certificate Authority (CA) using cert-manager.

The process typically involves:

1. Self-signed Issuer: First, you create a ClusterIssuer or Issuer of type Self-signed. This issuer generates its own root certificate and private key, establishing your internal CA.
2. CA Issuer: Next, you create a ClusterIssuer or Issuer of type CA, which references the Secret containing the self-signed root certificate and private key. This CA issuer is then used to sign certificates for your internal services.

Once your internal CA is set up, internal applications can request certificates via Certificate resources, specifying the internal CA issuer. The issued certificates, stored in Secrets, can then be mounted into pods. This enables mutual TLS (mTLS) between internal services, providing strong encryption and authentication without needing public domain validation or external trust chains. Applications simply read the certificates from the Secret, and cert-manager ensures they remain valid.

Practical Takeaways

cert-manager effectively bridges the gap between Kubernetes' declarative model and the complexities of certificate management. By automating the full lifecycle—issuance, storage, and rotation—it eliminates a significant source of operational overhead and potential outages due to expired certificates. Whether you're securing public-facing Ingress endpoints with Let's Encrypt or establishing robust service-to-service encryption with an internal CA, cert-manager provides a reliable, scalable solution.

FAQ

Q: Why isn't Kubernetes traffic encrypted by default, especially pod-to-pod communication?

A: Kubernetes is designed as a platform that provides primitives, leaving specific security implementations like comprehensive network encryption to the user. This allows for flexibility in choosing security solutions (e.g., service mesh, network policies, or cert-manager for TLS) that best fit a particular environment's requirements rather than imposing a one-size-fits-all approach.

Q: What's the main difference between HTTP-01 and DNS-01 ACME challenges, and when should I use each?

A: HTTP-01 challenges require your cluster to be reachable from the internet on port 80, as cert-manager creates a temporary HTTP endpoint for validation. It's simpler to set up for single domains. DNS-01 challenges, conversely, involve creating a temporary DNS TXT record and do not require inbound HTTP access, making them suitable for private clusters or when obtaining wildcard certificates (*.example.com). DNS-01 requires API integration with your DNS provider.

Q: How does cert-manager prevent certificate expiration outages?

A: cert-manager continuously monitors the validity of all certificates it manages. When a certificate approaches its expiry date (by default, when two-thirds of its validity period has passed), cert-manager automatically initiates a renewal process. It obtains a new certificate from the configured issuer and updates the associated Kubernetes Secret, thereby replacing the old certificate before it expires and preventing service disruption.