So, with all cloud technologies these days. It is clear the core layers are moving faster than the supporting elements can keep up. For instance, Rancher or Portainer. Now, this is not about ether one of them but instead how to setup core infrastructure to meet the needs of upper layers. That is, how to pick a specific version of Kubernetes to install.

Pick Servers

For me, Debian 11 is the ones I am using and I also install on Proxmox running Ceph as a storage layer.

I followed this cloud image setup blog I wrote a while ago. Once done I would make a "containerd" or Docker template to make it easiest to install both master and nodes for Kubernetes. I used this article to summarize this below based on my cloud image above.

Setup "overlay" and "br_netfilter"

#add needed modules
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

#Manually add the above.
sudo modprobe overlay
sudo modprobe br_netfilter

#Add needed iptables info
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

#apply settings from above.
sudo sysctl --system

So, this is critical for proper container networking it applies to both "containerd" and Docker configurations.

Install "containerd" or Docker

So, first one must choose the container layer that Kubernetes is going to use. Newer Kubernetes recommends using Containerd as opposed to Docker, but use whaterver is required for overall success. That is if only worried about server utilizing Kubernetes then use Containerd as it is the latest recommended container layer. Otherwise, use Docker. Both work well so this may be a personal choice. Remember containerd is a sublayer of Docker so it is kind of a mute point. Just remember, Docker is heavier than containerd. So, more mean more vectors of attack and layers of fat that may not be needed. So, for a light as possible system just install containerd. See this article for more info.

Option 1: "containerd" Setup

Remember, this is a lighter container layer. Both are valid just this option is a slimlined kubernetes only way.

sudo apt  update
sudo apt -y install containerd

Now, it is time to setup containerd configuration.

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1

add the following, on line below [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

SystemdCgroup = true

sudo vi +/"\[plugins\.\"io\.containerd\.grpc\.v1\.cri\"\.containerd\.runtimes\.runc\.options\]" /etc/containerd/config.toml

Ok, it will highlight the line. Now, press "i" to go into insert mode now go to the end of the highlighted line. Then, hit <enter> then tab over twice. Finally, put SystemdCgroup = true in this new line.

Overall, the area should look like below:

       [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          privileged_without_host_devices = false
          base_runtime_spec = ""
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
                SystemdCgroup = true
    [plugins."io.containerd.grpc.v1.cri".cni]

To save, press ":". Now at the " : " type wq!

Now it is time to apply the changes. Like below:

sudo systemctl restart containerd
sudo systemctl enable containerd

Finally, containerd is setup.

Option 2: Docker Setup

So, for full docker install here is how to do it. Especially step 1 in that article.

Decide on GUI

That is, Rancher or Protainer. This is not going to be decided here. Instead I am going to focus on core layers to help with all Kubernetes. That is, the very core layers to make a useful kubernetes cluster.

Currently, for Rancher this is the results of running the helm install on latest Kubernetes

debian@k8-m-01:~$ helm install rancher rancher-latest/rancher   --namespace cattle-system   --set hostname=rancher.dmz.alshowto.com   --set bootstrapPassword=admin
Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.25.0-0 which is incompatible with Kubernetes v1.25.3

So, it is obvious this will not install on Kubernetes at this version.

Install Helm on Debian Master Node or Somewhere

Yes, for now I am going to install Helm on the master node. Also, It may be wise to not install Helm on the master node. Instead, create an installer or use wsl client and install helm there. This will prevent unneeded items on master nodes and keep them focused on Kubernetes only. For example, for security that is only have things in master related to master things. Adding add potential vectors to attack the cluster.

Put in Needed apt Settings

Below, is from helm install page.

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

sudo apt-get install apt-transport-https --yes

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

sudo apt-get update

sudo apt-get install helm

How To Tell Kubernetes Version

Simple, run this command

kubectl version

It will tell you the version of both the client aka kubectl and server aka kubernetes master cluster version simular to this:

debian@k8-m-01:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

Below, this will print all available versions of Kubernetes.

curl -s https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages | grep Version | awk '{print $2}'

Install a Specific Version of Kubernetes

Below, command will install specific version of Kubernetes. Remember, replace <version> with the version required to be installed.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update -q && \
  sudo apt-get install -qy kubelet=<version> kubectl=<version> kubeadm=<version>

Once this is done it is time to setup the master and node servers. First, setup a master. Also, it is important to setup at least one master and then add another master and all needed node servers.

Setup Hosts File

Oh, it is important to setup the hosts to make sure all is working correctly so modify /etc/hosts file. Note, in my cloud configuration it is recommended to actually modify the file: /etc/cloud/templates/hosts.debian.tmpl Also, make sure to add all servers to all other server both masters and nodes need to know where each is located. Remember. the above is for cloud servers if modifying a non cloud server it may be just fine to modify file /etc/hosts directly. Below, is an example:

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 k8-m-01 k8-m-01
127.0.0.1 localhost

192.168.2.200 k8-m-01
192.168.2.202 k8-n-01
192.168.2.203 k8-n-02
192.168.2.204 k8-n-03

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Setup First Master

So, the next thing to do is setup the very first master kubernetes server. For that, run the line below on server to be a master.

sudo kubeadm init --control-plane-endpoint=k8s-master

To Interact with Cluster

Once kubeadm returns it is necessary to setup kubectl via the lines below. With this, it will allow direct interaction with cluster via kubectl as current user. Note: this works with non root user just fine and is recommended to be run as non root.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Check Status of Nodes and Cluster

kubectl get nodes
kubectl cluster-info dump