The Kubernetes Series - Static Pods & Multiple Schedulers

by Rik Nieu

(Photo by Vlad Shapochnikov on Unsplash)

We looked at one edge-case for how the kube-schedulers is not used to deploy Pods by looking at DaemonSets  - now let's look at two other instances.

Static Pods

It's technically possible to have only one Worker Node in your cluster, with no supporting Master Node to orchestrate scheduling, networking or anything else, for that matter. These Pods are called Static Pods.

Creating Static Pods on a Node

Static Pods are created with YAML file definition too, but since there's no Master Node with an api to create and provision your Pod, you will have to add the definition to the Worker Node itself. You do this by placing the definition file in the following directory on your Static Node.

/etc/kubernetes/manifests

You can change this default manifest path if you like by editing the kubelet service pod-manifest-path property. Or you can add a config property to the kubelet service, pointing to a config YAML file. Then, inside this config YAML file, you can add the custom manifest path under the property staticPodPath.

Remember, you can inspect a kubelet service with the following command;

ps -aux | grep kubelet

The scheduling functions previously provided by the Master Node via the kube-scheduler service will now be taken over and approximated by the kubelet service on your Worker Node. If the Pod crashes, the kubelet will automatically restart it by loading the definitions from the manifests folder on the node. In a similar way, any changes to the manifest files will force the Pods to be recreated, and removing files will also delete any Pods created with them.

And since this is a single node with no scheduling managed by a Master Node, you can only add Pod definitions to the node. No ReplicationSets or Deployments or DaemonSets - those all rely on a Master Node with a kube-scheduler, remember? Only Static Pods.

View Static Pods on a Node

Since there is no Master node and thus no kube-API-server to handle kubectl commands, you have to view the running pods on the Static Node the same way you would view singular Docker instances from the command line;

Docker ps

Static Pods On The Same Cluster As A Master Node

You might be wondering if it's possible to create a Static Pod on a node in a cluster where there is indeed a Master Node running, and yes, you can. Kubectl will even allow you to view and describe these Pods, but it will not allow you to update, replace or delete them. You will have to do that yourself by editing the manifest YAML files.

The kube-scheduler service also pretends that these Pods don't exist, as the kubelet on the Node in question will take over their management instead.

Master Node - a Static Pod?

You can create your own Master Nodes from a Static Pod by installing the kubelet service on the Pod, and then putting YAML definition files pointing to Docker images for all the Master Static Plane components in the pod manifests folder; eg the API-serve, scheduler controller and etcd store.

If any of there services were to crash the kubelet will automatically recreate them and keep your stuff going. This is more-or-less how the kubeadm creates a cluster for you, by the way - all the Control Place Components are thus created as separate Pods, not as services.

Multiple Schedulers

You might run into a use-case where you would need to define a custom scheduler for a particular application, but still have the default kube-scheduler managing other applications on you cluster. We will not being going into too much detail about why you would want a custom scheduler here, by safe to say that you can indeed do it, and below are some instruction to get you started, at least.

You can create a customer scheduler by running a new instance of the scheduler binary, and giving it a custom name with the --scheduler-name=name flag. The default scheduler is created by not passing a name, creating the default scheduler with the name default-scheduler.

You can also create it with kubeadm, or with a custom definition in the manifest folder, as per above.

With the manifest file case, you can duplicate the scheduler YAML file and under the command dictionary list, add a --schedule-name property with the name of your new scheduler. If you have one Master Node, set the --leader-elect property to false, or if you do have multiple Master Nodes, lock the scheduler with the property --lock-object-name.

Conclusion

With this post we learned that you can create Pods that work without having a Master Node or kube-scheduler to manage them, in the form of Static Pods. We also learned that you can have multiple schedulers running on the same cluster, although we did not get into too much detail as to why or exactly how you would do this.

The next post will deal with basic solutions for logging and monitoring.