The Kubernetes Series - SSL/TLS Certificates

by Rik Nieu

(Photo by Markus Spiske on Unsplash)

In the previous post we had a brief look at the 3 ways we can authenticate users to our cluster. In this post we look at SSL/TLS certificates in particular.

If you use Kubeadm to create your cluster, this should all be handled for you automatically. Kubeadm will also handle certificate renewals for you if you don't have a complex use-case. But let's look at some of the details so that we have a better understanding as to whats going on behind the scenes.

SSL/TLS Certificates

As mentioned in the previous post, TLS certificates encrypts the data sent and received, and confirms sender and receiver identities in a more secure way. Since SLL/TLS certificates use asymmetric encryption, it implies there are two parts needed to securely send data. These parts are public and private keys.

Public keys and Private keys

Exactly how TLS encryption works is well beyond the scope of this post, so we will only look at it from a genera, vague and high-level point of view.

The asymmetrical encryption process consists of multiple steps and parts. The use of signed certificates for verified and encrypted communication is also called Public Key Infrastructure(PKI). When sending information from Sender to Receiver via PKI, the general parts involved are the Private key, Public key and a CA-signed certificate.

The process goes more or less like this: a Private and Public key-pair is created with complex mathematical calculations by the Sender. The Public key will be used by others to encrypt data to sent to you, data only you will be able to decrypt with your Private key.

Identifying information like a domain name or email is then encrypted with the Private key - this is called symmetrical encryption. Next, the signed/encrypted data is packaged with the Public key into an object called a Certificate Signing Request(CSR). This is the start of the asymmetrical encryption process, where two keys are used to encode and decode messages.

The CSR is then sent to a Certificate Authority(CA), which is a third-party server both Sender and Receiver have defined as trusted. The CA signs the CSR with its own Private key, and it gets passed along to the Receiver. This new CA-signed certificate is called the Identity Certificate, the message the Sender will send to the Receiver.

The Receiver will then get this package, and verify that it is indeed from the Sender by checking if it is signed by the trusted CA. Once verified as trusted, the Receiver can send data back to the Sender and encrypt it with the Sender Public key it received in the Senders' signed certificate. When it receives the message back, the Sender will then be able to decrypt the message from the Receiver(after also checking in with the CA and getting the Receivers Public key) with its Private key.

Private keys can usually be identified by having "-key" somewhere in its name or as a .key extension. Public keys(certificates) have the extension .crt or .pem, with no "-key" in the filename.

Client Certificates and Server Certificates

The CA-signed certificate used to send information from your machine to the server is called the Client Certificate. The signed certificate used to send information from the server to you machine is called the Server Certificate.

Generating CA, Client and Server Certificates

Let's look at the different entities on a Kubernetes cluster that need Client Certificates and the ones that need Server Certificates. All of these, by the way, also needs to be signed by our CA. With a private cluster, we will be hosting our own CA on the Master node - using public CAs for private networks would be prohibitively expensive.

Entities requiring Server Certificates

  • ETCD service
  • kube-apiserver
  • kubelets on Worker nodes

Entities requiring Client Certificates

  • Kubectl users
  • kube-scheduler(for communicating with kube-apiserver)
  • kube-controller-manager(for communicating with kube-apiserver)
  • kube-proxy(for communicating with kube-apiserver)
  • kube-apiserver(for communicating with kubelets and ETCD server)

CA Certificates

We will need to generate all these certificates, but let's start with the CA certificates, since all others will need to be signed by them. We'll use openssl to generate certificates.

openssl genrsa -out kube-ca.key 2048

That, as you can see from the naming convention, is our Private Key. Now let's create the Public key / Certificate Signing Request combo;

openssl req -new -key kube-ca.key -subj "/CN=KUBE-CA" -out kube-ca.csr

Note how we gave it a identifiable name with Certificate Name(CN) and then symmetrically encoded it with our Private key generated in the previous step. Now, even though this is the CA CSR, we need to sign the certificate as well. The CA is the only self-signed certificate we will use - all others will be signed by the CA.

openssl x509 -req -in kube-ca.csr -signkey kube-ca,key -out kube-ca.crt

Now the CSR is signed(in this case self-signed) with the CA Private key. Note the difference between the certificate request and signed certificate extensions;

  • request –> .csr
  • signed –> .crt

A copy of the kube-ca.crt will be needed on every Node, in order for the Clients to be able to verify the legitimacy of certs sent from Servers, and for Servers to verify the identities of certs sent from Clients.

Signing Client certificates with CA certificates

Needless to say, your entire cluster access control determined by your CA authority, that's why your CA certs need to preferably held on a secure server where dodgy busy-bodies don't have access to them. They are essentially the keys to your castle. By default your CA files are stored on the Master node.

Kubernetes has a built-in certificate signing api called the Certificates API, via the Control Plane Service kube-controller-manager .

If a new user wanted kubectl access to your cluster, they would need to generate a Private key, and send you as administrator a signing request. It would then need to be signed with your cluster CA private key and certificate and to sent back as a signed certificate to the user.

Let's generate a kubectl user Client Cert and sign it with the CA.

openssl genrsa -out user.key 2048
openssl req -new -key user.key -subj "/CN=KUBECTL-USR/O=system:masters" -out user.csr

If you wanted to you could manually sign the certificate as an admin user by logging into the Master node and executing the following command;

openssl x509 -req -in user.csr -CA kube-ca.crt -CAkey kube-ca.key -out user.crt

(Note two things - One; the CN=KUBECTL gives the certificate its name, and Two; the O=system:masters specify the group roles to the user - more on that later.)

Now let's sign the certificate with the Certificates API instead. You would create a Kubernetes CertificateSigningRequest object with something similar to the following definitions file(note the groups and usages permissions);

apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: name-surname
spec:
  groups:
  - system:authenticated
  usages:
  - digital signature
  - key encipherment
  - server auth
  request:
     LS0t... (base64 string of .csr file)

The request key is required to be in a base64 format. Generate and copy/paste it from the users' certificate signing request(user.csr above);

cat user.csr | base64

Administrators can view pending requests submitted to the Certificates API with;

kubectl get csr

They can then approve/reject them with;

kubectl certificate approve name-surname

Now we can view the signed certificate with;

kubectl get csr name-surname -o yaml

The signed certificate is the encoded base64 value under they key certificate. You can then decode the base64 value and share with the user, so that they can save it as a signed .crt key to be used to access the cluster.

Now, the user can sign-in to the cluster with a signed certificate as authentication. Let's do that via command-line;

kubectl get pods --server=test-server:6443 --client-key=user.key --client-certificate=user.crt --certificate-authority=ca.crt

This works, but it's a schlep to have to add all those flags to  queries the whole time. We can instead save this information as a config file in our home directory under the folder ~/.kube/config.

Accessing your cluster - the KubeConfig File

The KubeConfig file contains 3 main parts;

  • users(--client-key, --client-certificate, --certificate-authority)
  • clusters (--server)
  • context(user@test-server)

The context part specifies which clusters you have access to, the users specifies the right and roles your user accounts have, and context bridges your users with your clusters.

Lets look at the KubeConfig definition file;

apiVersion: v1
kind: Config
current-context: user@test-server <-- The default context
clusters:
- name: test-server
  cluster:
    certificate-authority: ca.crt
    server: https://test-server:6443
- name: second-server
  cluster:
    certificate-authority: ca.crt
    server: https://second-server:6443    
contexts:
- name: user@test-server
  context:
    cluster: test-server
    user: user
  namespace: business
- name: second-user@second-server
  context:
    cluster: second-server
    user: second-user 
    namespace: HR
users:
- name: user
  user:
    client-certificate: user.crt
    client-key: user.key
- name: second-user
  user:
    client-certificate: second-user.crt
    client-key: second-user.key

We don't have to create this file like a Kubernetes object, it's just a config file used by kubectl on your local machine. You can view the content of this config file with;

kubectl config view

Now let's use kubectl to switch context;

kubectl config use-context second-user@second-server

If you view you config file again, you'll see your file has been updated too.

Multiple KubeConfig files

You might have multiple KubeConfig files and you can set the active context and namespace from a different config file like so;

kubectl config --kubeconfig=~/.kube/other-kube-config use-context finance

Server Certificates

Server Certificates are generated in much the same way as Client Certificates, except for the kube-apiserver service. It needs multiple identifying names to be added to its certificate, due to legacy naming conventions and such. To do this we can use a .cnf dictionary file to list all it's different possible names.

Let's create a kube-api-server.cnf dictionary file;

[req]
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 10.96.0.0
IP.2 = 172.17.0.80

And now let's reference it in our CSR command it;

openssl genrsa -out kube-api-server.key 2048
openssl req -new -key kube-api-server.key -subj "/CN=KUBE-API-SERVER" -out kube-api-server.csr -config kube-api-server.cnf
openssl x509 -req -in kube-api-server.csr -CA kube-ca.crt -CAkey kube-ca.key -out kube-api-server.crt

Ok, now what do we do with these certs?

For creating the kube-apiserver Service, we specify them in our service YAML definition files or with command-line labels on creation;

  • --client-ca-file = {kube-ca.pem} <– the CA authority certificate to verify client coms
  • --tls-cert-file = {/var/lib/kubernetes/kube-api-server.crt} <– Server Cert
  • --tls-private-key-file= {/var/lib/kubernetes/kube-api-server.key} <– Server Key
  • --etcd-cafile = {kube-ca.crt} <– the CA authority certificate to verify etcd coms
  • --etcd-certfile = {kube-api-server-etcd-client.crt} <– the Client certificate to verify etcd coms
  • --etcd-keyfile{kube-api-server-etcd-client.key} <– the Client key to encode etcd coms

Kubelets Certs

Kubelet Server Certs gets named after the nodes they run on, but we can also add more names with a .cnf dictionary.

openssl req -new -key kubet-server.key -subj "/CN=NODE01" -out kubet-server.csr -config kubet-server.cnf

Kubelet Client Cert names start with the keyword system:node to indicate to the kube-apiserver which node is trying to authenticate on it. This is required to give it the right permissions/roles, in this case a system component role. Let look at that below

openssl req -new -key kubet-client.key -subj "/CN=SYSTEM:NODE:NODE01" -out kubet-client.csr -config kubet-client.cnf

After their keys and certs have been generated you specify it in the kubelet-config.yaml file like so;

authentication:
  x509:
    clientCAFile: "/var/lib/kubernetes/kube-ca.crt"
tlsCertFile: "/var/lib/kubelet/node01.crt"
tlsPrivateKeyFile: "/var/lib/kubelet/node01.key"

Inspecting Certificates

You can view the location of certificates used for a particular service by viewing its manifest files, in the case of a cluster generated with kubeadm, or viewing the service running on your master node.

Let's take the default case for the kube-apiserver on a cluster create with kudeadm.

We fist view the kube-apiserver.yaml file to find the server certificate under the flag tls-cert-file;

cat /etc/kubernetes/manifests/kube-apiserver.yaml

This gives us the location of the server cert as /etc/kubernetes/pki/apiserver.crt. Now we can view the details of the certificate with openssl, like so;

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

Conclusion

Phew, that was a tough one! This post was by no means comprehensive but you should have a better understanding on how PKI works and how you can use it to access your cluster securely.

The next post throttles down the intensity again, and we'll have a look at User and Cluster Roles in order to control who can do what on our cluster.