📒 ✍🏾 🎒 Let's encrypt SSL certificates with cert-manager in Kubernetes 🍢 👁‍🗨 👩‍💻

In this article I will talk about how to automate the ordering and renewal of certificates from Let's Encrypt (and not only) for Ingress in Kubernetes using the add-on cert-manager. But I'll start with a brief introduction to the essence of the problem.

A bit of educational program

The HTTP protocol, developed in the early 90s of the last century, entered our everyday life so tightly that it’s hard to imagine at least a day without using it. However, by itself, it does not even provide a minimum level of security when exchanging information between a user and a web server. HTTPS (“S” - secure) comes to the rescue: using the packaging of the transmitted data in SSL / TLS, this protocol has proven itself in protecting information from interception and is actively promoted by the industry.

For example, Google has adhered to 2014“HTTPS everywhere” position and even lowers the priority of sites without it in the search results. This “propaganda” does not bypass ordinary consumers either: modern browsers warn their users about the presence and correctness of SSL certificates for visited sites.

The cost of a certificate for a personal site starts from tens of dollars. Not always buying it is justified and appropriate. Fortunately, since the end of 2015, a free alternative in the form of Let's Encrypt (LE) certificates is available . This nonprofit project was created by Mozilla enthusiasts in order to cover most of the websites with encryption.

Certificate Authority issues domain-validated certificates(the simplest among those available on the market) with a validity period of 90 days, and for many years it has been possible to issue the so-called wildcard certificate for several subdomains.

To obtain the certificate, the site uses the algorithms described in the Automated Certificate Management Environment (ACME) protocol , created specifically for Let's Encrypt. When using it, confirmation of domain ownership is carried out by requests through the placement of a specific HTTP code (called HTTP-01 ) or the installation of DNS records (DNS-01) - more about them will be given below.

Certification manager

Cert-manager is a special project for Kubernetes, which is a set of CustomResourceDefinitions (hence the limitation on the minimum supported version of K8s - v1.12) for CA configuration (certification authorities) and direct ordering of certificates. Installing CRD in a cluster is trivial and comes down to using one YAML file:

kubectl create ns cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

( There is also the possibility of installing using Helm.)

To initiate the ordering procedure, the cluster must have the resources of certification authorities (CAs) declared: Issueror ClusterIssuer, - which are used to sign the CSR (certificate issuance requests). The difference between the first resource and the second is in scope:

Issuer can be used within the same namespace,
ClusterIssuer is a global cluster object.

Practice with cert-manager

No. 1. Self Signed Certificate

Let's start with the simplest case - ordering a self-signed certificate. This option is quite common, for example, for dynamic test environments for developers or in the case of using an external balancer that terminates SSL traffic.

The resource Issuerwill look like this:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

And in order to issue a certificate, it is necessary to describe the resource Certificate, where it is indicated how to issue it (see the section issuerRefbelow) and where the private key (field secretName) is located. After that, Ingress will need to refer to this key (see section tlsc spec):

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: selfsigned
  commonName: "yet-another.website"
  dnsNames:
  - "yet-another.website"
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: app
spec:
  tls:
  - hosts:
    - "yet-another.website"
    secretName: tls-secret
  rules:
  - host: "yet-another.website"
    http:
      paths:
      - path: /
        backend:
          serviceName: app
          servicePort: 8080

A few seconds after adding these resources to the cluster, the certificate will be issued. You can see the corresponding report in the output of the command:

kubectl -n app describe  certificate selfsigned-crt
...
  Normal  GeneratedKey  5s    cert-manager  Generated a new private key
  Normal  Requested     5s    cert-manager  Created new CertificateRequest resource "selfsigned-crt-4198958557"
  Normal  Issued        5s    cert-manager  Certificate issued successfully

If you look at the secret resource itself, then it contains:

private key tls.key,
root certificate ca.crt,
Our self-signed certificate tls.crt.

The contents of these files can be seen using the utility openssl, for example, like this:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:01:59 2020 GMT
notAfter=May 10 21:01:59 2020 GMT
issuer=O = cert-manager, CN = yet-another.website

It is worth noting that in the general case, the certificate issued using this will not be trusted by theIssuer connected clients . The reason is simple: it does not have a CA (see the note on the project website ) . To avoid this, you need to specify in the path to the secret file where it is contained . This can be a corporate CA organization - to sign the certificates issued for Ingress with a key that is already used for the needs of other server services / information systems.Certificateca.crt

No. 2. Let's Encrypt Certificate with HTTP Validation

To issue LE certificates, as mentioned earlier, two types of domain ownership confirmation are available : HTTP-01 and DNS-01.

The first approach (HTTP-01) is to launch a small web server with a separate deployment, which will send to the Internet via the link <YOUR_DOMAIN > /. Well-known / acme-challenge / <TOKEN> some data requested from the certification server. Therefore, this method implies the availability of Ingress from outside on port 80 and the resolution of the DNS record of the domain into public IP addresses.

The second option to verify the issued certificate (DNS-01) comes fromfrom having an API to the DNS server hosting the domain records. Issuer, using the indicated tokens, creates TXT records on the domain, which the ACME server then receives during confirmation. Among the officially supported DNS providers are CloudFlare, AWS Route53, Google CloudDNS and others, including its own implementation ( acme-dns ).

Note : Let's Encrypt has fairly strict limits on requests to ACME servers. In order not to get into a long ban, it is recommended to use the letsencrypt-staging certificate type for debugging (the difference is only in the ACME server).

So, we describe the resources:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory 
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
       ingress:
         class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: letsencrypt
  commonName: yet-another.website
  dnsNames:
  - yet-another.website

Note that the address of the staging server is specified as servery acme(c Issuer) . Replace it with a battle will be possible later. Applying this configuration, we trace the entire order path:

Creation Certificatespawned a new resource CertificateRequest:

kubectl -n app describe certificate le-crt
...
Created new CertificateRequest resource "le-crt-1127528680"

In his description - a mark on the creation Order:

kubectl -n app describe certificaterequests le-crt-1127528680
…
Created Order resource app/le-crt-1127528680-1805948596

It Orderdescribes with what parameters the test passes and what its current status is. Such verification is carried out by the resource Challenge:

kubectl -n app describe order le-crt-1127528680-1805948596
…
Created Challenge resource "le-crt-1127528680-1805948596-1231544594" for domain "yet-another.website"

Finally, the details of this resource contain information about the status of the scan itself:

kubectl -n app describe challenges le-crt-1127528680-1805948596-1231544594
...
  Reason:      Successfully authorized domain                                                                                                                                                                      
...
  Normal  Started         2m45s  cert-manager  Challenge scheduled for processing
  Normal  Presented       2m45s  cert-manager  Presented challenge using http-01 challenge mechanism
  Normal  DomainVerified  2m22s  cert-manager  Domain "yet-another.website" verified with "http-01" validation

If all the conditions have been met (i.e. the domain is accessible from the outside, there is no ban from the LE ...) - in less than a minute, the certificate will be issued. If successful, a describe certificate le-crtrecord will appear in the output Certificate issued successfully.

Now you can safely change the server address to combat ( https://acme-v02.api.letsencrypt.org/directory) and re-order the real certificates already signed not Fake LE Intermediate X1, but Let's Encrypt Authority X3.

To do this, you first need to delete the resource Certificate: otherwise, no order procedures are activated, because the certificate already exists and it is relevant. Removing a secret will lead to its immediate return with a message to describe certificate:

  Normal  PrivateKeyLost  44s                   cert-manager  Lost private key for CertificateRequest "le-crt-613810377", deleting old resource

It remains to apply the “combat” manifesto for Issuerthe one already described above Certificate(it has not changed):

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
       ingress:
         class: nginx

After receiving the message Certificate issued successfully, describecheck it:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:11:48 2020 GMT
notAfter=May 10 21:11:48 2020 GMT
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

Number 3. Wildcard LE with DNS validation

We will complicate the task by writing out a certificate immediately to all subdomains of the site and using this time the DNS check (via CloudFlare).

First, get the token in the CloudFlare control panel for working through the API:

Profile → API Tokens → Create Token.
Set permissions as follows:
- Permissions:
  - Zone - DNS - Edit
  - Zone - Zone - Read
- Zone Resources:
  - Include - All Zones
Copy the token received after saving (for example:) y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M.

Create a Secret in which this token will be stored, and refer to it in Issuer:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
type: Opaque
stringData:
  api-token: y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory 
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - dns01:
        cloudflare:
          email: my-cloudflare-acc@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-crt
spec:
  secretName: tls-secret
  issuerRef:
    kind: Issuer
    name: letsencrypt
  commonName: yet-another.website
  dnsNames:
  - "yet-another.website"
  - "*.yet-another.website"

(Do not forget to use staging if you are experimenting!)

Let’s go through the process of confirming domain ownership:

kubectl -n app describe challenges.acme.cert-manager.io le-crt-613810377-1285319347-3806582233
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for dns-01 challenge propagation: DNS record for "yet-another.website" not yet propagated
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    54s   cert-manager  Challenge scheduled for processing
  Normal  Presented  53s   cert-manager  Presented challenge using dns-01 challenge mechanism

A TXT record will appear in the panel:

... and after a while the status will change to:

Domain "yet-another.website" verified with "dns-01" validation

Make sure that the certificate is valid for any subdomain:

kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -text |grep DNS:
          DNS:*.yet-another.website, DNS:yet-another.website

Validation by DNS, as a rule, does not happen quickly, since most DNS providers have a data update period that shows how much time elapses from the moment the DNS record is changed to the actual update of all the provider’s DNS servers. However, the ACME standard also provides a combination of two verification options, which can be used to expedite the receipt of a certificate for the main domain. In this case, the description Issuerwill be as follows:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - selector:
        dnsNames:
        - "*.yet-another.website"
      dns01:
        cloudflare:
          email: my-cloudflare-acc@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token            
    - selector:
        dnsNames:
        - "yet-another.website"
      http01:
        ingress:
          class: nginx

If you apply this configuration, two resources will be created Challenge:

kubectl -n app describe orders le-crt-613810377-1285319347
…
  Normal  Created  3m29s  cert-manager  Created Challenge resource "le-crt-613810377-1285319347-3996324737" for domain "yet-another.website"                 
  Normal  Created  3m29s  cert-manager  Created Challenge resource "le-crt-613810377-1285319347-1443470517" for domain "yet-another.website"

Number 4. Using Ingress Special Annotations

In addition to the direct path to creating certificates in cert-manager, it is possible to use a component called ingress-shim and not explicitly create resources Certificate. The idea is that with the help of special Ingress annotations, the certificate will be automatically ordered using the one indicated in them Issuer. The result is roughly the following Ingress resource:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - "yet-another.website"
    secretName: tls-secret
  rules:
  - host: "yet-another.website"
    http:
      paths:
      - path: /
        backend:
          serviceName: app
          servicePort: 8080

For correct operation, only the presence of Issuer is enough here, that is, creating one less entity.

In addition, there is an obsolete annotation kube-lego - kubernetes.io/tls-acme: "true", - which requires a Issuerdefault task when installing cert-manager using Helm parameters (or in the manager container launch options).

We at the company do not use these options and cannot advise them due to the opacity of the approaches used to order SSL certificates (and at the same time to various problems that arise ), but still decided to mention in the article for a more complete picture.

Instead of a conclusion

Through simple manipulations with CRD, we learned how to write auto-renewable, self-signed, and free SSL certificates from the Let's Encrypt project for domain sites launched as part of Ingresss in Kubernetes clusters.

The article provides examples of solving the most common problems in our practice. However, the cert-manager functions are not limited to the features described above. On the utility website you can find examples of working with other services - for example, a bundle with Vault or the use of external issuing centers (issuers).

Let's encrypt SSL certificates with cert-manager in Kubernetes