Choosing a Bare Metal Kubernetes Distribution
Table of Contents
A common complaint regarding Kubernetes is that it is overwhelmingly complex.
In my experience, the perceived complexity is a result of needing to understand the whole system at a broad level before starting to work with it, rather than inherent complexity of any individual component.
When setting up your first cluster you are immediately faced with several critical decisions that might not be easy to answer for a newcomer: Which distribution of Kubernetes? What host OS? How will you manage persistent storage? What is a CNI? What about virtual IPs? What is GitOps? Etc…
When I decided to transition from Docker to Kubernetes in my homelab, I started with the most obvious question first: which distribution of Kubernetes to run.
Here is a brief overview of popular choices for bare metal k8s clusters, and some notes regarding my decision making process.
Popular distributions #
Talos #
Talos is A Minimalist, fully API-managed Linux distro that is designed from the ground up solely for running k8s. This was my first choice initially, and I did install and test it out.
| Pros | Cons |
|---|---|
| Minimal | Compatibility issues |
| Security by design | Less community adoption |
| No SSH | No SSH |
| Upstream k8s | |
| Immutable host | |
| Manages k8s and host |
I ultimately abandoned Talos mainly due to incompatibility with my preferred storage platform, Longhorn. If you do not need in-cluster persistent storage, or are planning on deploying rook, Talos is a fantastic choice.
K3s #
K3s is an edge focused, stripped back k8s distribution by SUSE, deployed on top of an existing Linux distro.
K3s appears to be the most popular choice among homelabs based on my unscientific perusal of public git repositories (see k8s at home search). K3s is very well documented and there is a great community of users behind it.
| Pros | Cons |
|---|---|
| Minimal | Need to manage host separately |
| Greater community adoption | |
| Well documented | |
| system-upgrade-controller |
This is what I ultimately settled on.
Typhoon #
Typhoon is a minimal k8s distro that is deployed on either Flatcar Linux or Fedora CoreOS.
| Pros | Cons |
|---|---|
| Minimal | Less community adoption |
| Upstream k8s | No in-place upgrades |
| Immutable host | |
| Manages k8s and host |
I strongly considered using this, but for my use case of a single small cluster, lacking the ability to perform in-place Kubernetes upgrades was a deal breaker.
I can deal with unforeseen consequences of upgrades given that I am the sole user of my cluster, and would much rather have the ability to upgrade and risk breakage than deal with re-provisioning the entire cluster each time.
Others #
Some alternative choices that I am aware of but have not looked into in detail:
I’m sure there are more options that I haven’t listed here, but when deploying something with as many moving parts as Kubernetes, it seems prudent to stick with popular choices.
My choice? #
All the options listed have their pros and cons, but after briefly playing with and abandoning Talos, I ultimately went with the safe option and deployed k3s.
Compatibility with my preferred storage provider, Longhorn was a critical factor, but the excellent and easy to use system-upgrade-controller is the killer feature for me. Having a well understood cluster life cycle with support for automatic upgrades is simply awesome, and goes a long way towards minimizing administrative overhead.
While not having to manage the host OS would be nice, there are fantastic immutable distro options here: Fedora CoreOS and Flatcar Linux, that get you most of the way to Talos, without sacrificing flexibility and/or capability.