Kubernetes and Certificate Management

Cert Manager and OpenSSL

Cert-Manager

Cert-Manager is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry.

In my experience, I have used and configured Cert-Manager with fairly short lived certs that rotate every 30 to 60 days. After that period of time, and within the window that you configure Cert-Manager to reissue the cert, the Certs automatically rotate and then you are good to go. Cert-manager is a powerful and helpful tool that is automating that process.

In simple language and terms, Cert-Manager says, "hey I need a cert for this domain”, then goes out to your Certificate Authority, for example Letsencrypt, and Letsencrypt verifies that you own that that domain. After the verification is complete, (Certificate Request, Certificate Signing Request or CSR, Challenge, and Order) a signed Certificate is sent back, and Cert-Manager then stores that in a Secret. Then, Cert-Manager is also responsible for periodically checking what the expiration date is.

Ways Letsencrypt Verifies You Are Who You Say You Are
  • DNS challenge - "Hey, I expect this DNS record to exist, and when I hit this DNS record, I expect it to give me this value back.
    • So for example say you are using a CA (Certificate Authority) like Letsencrypt, what A DNS challenge does is once you have created a DNS record, like a subdomain (for example foo.example.com), Letsencrypt will go and do a DNS resolve on that. If it gets back what it expects to see, in simple terms, it says: "okay, you are who you say you are, I'm going to issue this cert to you." Cert-Manager is doing this automatically.

    If the cert contains some variation of CA:TRUE, it makes sense to distribute it, if it does not, then there is no benefit to distributing it. The chain of trust itself can be explained in terms of the certificate structure.

    If we start from your server's cert (i.e. the cert you would typically use for apache):
    • Your server cert is generated by a certificate authority, based on a CSR you provide. That CSR contains your public key, and some other information the CA may or may not be interested in using.
    • The CA receives your CSR, and (generally), will create a signed certificate using an intermediate CA. Your certificate will have two public keys: one for the subject (which is the public key extracted from your CSR), and one for the issuing party, which is the CA's intermediate certificate.
    • The CA's intermediate certificate is itself a signed certificate (with some special options), where the subject key will be the intermediate CA's public key (which is the same public key as is found in the Issuer Public Key of your server certificate). The signing (or issuing) key will be the CA's root certificate (alternatively, it will be another intermediate).
    • The CA's root certificate is (generally) a self-signed certificate. In practice, this means that the Issuer and Subject keys are the same public key.

    Chain of Trust

    The chain of trust this builds can be summarized as (I am starting from the "bottom"):
    • The root CA delegates day-to-day activities to an intermediate CA and signs a certificate representing this fact (i.e setting CA:TRUE, KeyUsage: keyCertSign).
    • In other words, the root CA 'trusts' the intermediate to issue certificates on its behalf (hopefully after completing a series of mandatory validation steps).

    Note that (root) CAs are generally self-signed - a CA is trusted generally because it abides by a set of processes and procedures that are felt to ensure it does not issue certificates to people who should not have them. Which is why I can't go out and get myself a certificate saying I am google. This is therefore a convention of sorts (albeit one backed by formal mechanisms), and if you start your own CA, distributing its root (and intermediate) certificates achieves exactly the same thing as distributing the certs of other CAs does: it makes certificates issued by that CA be accepted as valid (i.e. trustworthy) by the system.

    OpenSSL

    In more practical terms, the trust store (for openSSL) is essentially a bunch of files with a specific naming convention. (See https://www.openssl.org/docs). When a certificate is verified its root CA must be "trusted" by OpenSSL. This typically means that the CA certificate must be placed in a directory or file and the relevant program configured to read it. That directory is /etc/ssl/certs. (There are others, such as /etc/pki on RH-likes. https://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu#719047 provides some further overview of how one adds a certificate to the store, as does https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate).

    The short version is that update-ca-certificates automates the process, and is the commonly used tool. The slightly longer version is that certificates need to be stored by hash (e.g. in files named based on the output of openssl x509 -hash -in cert.pem), so that openSSL can efficiently validate chains. (See https://www.openssl.org/docs/man1.0.2/apps/c_rehash.html for some additional background).

    The server certificate, in this discussion, is defined as trusted or not based on its chain of trust. If you think about how curl https://google.com works, it is a bit easier to understand:

    • curl opens a TLS connection to google, which returns a certificate.
    • curl looks at that 'end-user' certificate - i.e. the server's certificate, and looks specifically for the issuer.
    • If the issuer is 'known' in the trust store, the remainder of the trust chain is validated (i.e. the server certificate issuer's certificate is checked, and if it has an issuer other than itself, that issuer is checked, etc). Only if the full trust chain can be validated is the certificate considered valid.
    • However, it would be impractical to distribute trust chains if end-user certificates needed to be included.
    • Which is sort of the point of a chain of trust: you generally need to have the issuer certificates around, but you do not need the end certificates to be distributed, because they tell you who the issuer is.
    • And you can verify they aren't lying, because you can check the issuers public key (and the chain of trust that establishes it is the proper key), and validate whether the end-user (i.e. server) certificate was really signed by the private counterpart of the issuer's public key.
    • So, as a general practice, should not distribute the end-user certs, but only the chain of trust - or, in simpler terms: distribute all the certs you generate where the BasicConstraints (legitimately) say something at least CA:TRUE. Do not distribute anything that does not meet that condition, because it is a waste of your time and disk space - everything that does not say CA:TRUE can be validated against something that does.

    Written: June 14, 2024
    BACK HOME