Intro
Well, there is quite a bit to setting up a Kubernetes (K8) cluster. Remember this, this is a full K8 cluster setup with all needed steps to get K8 running on Debian 11.
Found this article: https://www.linuxtechi.com/install-kubernetes-cluster-on-debian/
This includes the following:

Components
2 master nodes and 2 worker nodes. All will have Docker installed and all will be running in ProxMox
Also, I will be setting up a HAProxy server using Debian 11 as a load balancer between the master nodes. Note, this will become the address used to access the master nodes. This VIP will balance and failover between any master nodes configured to work in this cluster. At least two masters are recommended in an HA setup. Also, it would be possible to add this configuration to pfSense and avoid the added HAProxy server. However, for this demo I am leaving pfSense out of the picture. In the end, I will include a new series of posts that will outline my entire HA Proxmox setup that is running the k8 cluster.
Pre Step 1: Define IP’s and DNS Names
So, best to plan out all names domain info and IP’s needed for this configuration.
FQDN | IP | Type | Description |
---|---|---|---|
k8master.cld.alshowto.com | 192.168.50.200 | Load Balancer | Main Load Balancer for Master Node access |
k8-m-01.cld.alshowto.com | 192.168.50.202 | Master Node 1 | First master node in cluster |
k8-m-02.cld.alshowto.com | 192.168.50.203 | Master Node 2 | Second master node in cluster |
k8-m-03.cld.alshowto.com | 192.168.50.204 | Master Node 3 | Third master node in cluster |
k8-n-01.cld.alshowto.com | 192.168.50.205 | Worker Node | First worker node in cluster |
k8-n-02.cld.alshowto.com | 192.168.50.206 | Worker Node | Second worker node in cluster |
k8-n-03.cld.alshowto.com | 192.168.50.207 | Worker Node | Third worker node in cluster |
Also, it is important to add all of the names into DNS and make sure each is pingable from each other. At least, set the names and ip in the /etc/hosts file on each server that will interact with cluster.
Step 1: Install VMs on ProxMox

1-1: Install pfSense
First and Most important, install pfSense to serve the DMZ/CLD router. So, I have many sub-domains and and have associated vlans to each of the sub domains
192.168.0.0/23 | NA | non-vlan LAN subnet. Original subnet that I use for general catch all. | pfSense LAN |
192.168.2.0/24 | 2 | DMZ vlan this is where my general web server are running | pfSense DMZ |
192.168.5.0/24 | 5 | This is my Ceph cluster_network only used for my main ceph partition. | |
192.168.6.0/24 | 6 | This is my Ceph public_network only used for my main ceph partition. This allows access to ceph admin gui. | |
192.168.50.0/24 | 50 | CLD network for kubernetes cluster. | pfSense CLD |
NA | 999 | WAN VLAN 10g Port 2 untags and tags this VLAN | All WAN access via ISP DHCP |

As shown above, this is my current VLAN and subnet running on two switches. A 10g switch and a 1g switch. I am not going to go into all the setup of VLAN as this post would never end if I did so. Only point, it is important to segment out what works for you in your environment. Having DMZ and CLD separate from LAN protects each from the others. Overall, this adds layers of isolation. So, look for a post related to this setup in the future. See, here is the start of that post.
Step 2: Make HAProxy VM
Simple, just create a Debian 11 container or VM and install HAProxy on it. Then run the following apt on it.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install haproxy
Then configure the HAProxy config at /etc/haproxy/haproxy.cfg
sudo nano /etc/haproxy/haproxy.cfg
Below, is an example of the config file. Note: this does not modify global or defaults this the two … elements. Therefore, the top four lines are only for reference within the default config file. Overall, adjustments may have to be made.
<kube-balancer-ip> = Load balancer ip or it can also simply be a *
<kube-master0-ip> = Master 0 ip for master node 0.
<kube-master1-ip> = Master 1 ip for master node 1.
global
....
defaults
....
frontend kubernetes
bind <kube-balancer-ip>:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
frontend http_front
mode http
bind <kube-balancer-ip>:80
default_backend http_back
frontend https_front
mode http
bind <kube-balancer-ip>:443
default_backend https_back
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master-0 <kube-master0-ip>:6443 check fall 3 rise 2
server k8s-master-1 <kube-master1-ip>:6443 check fall 3 rise 2
backend http_back
mode http
server k8s-master-0 <kube-master0-ip>:32059 check fall 3 rise 2
server k8s-master-0 <kube-master1-ip>:32059 check fall 3 rise 2
backend https_back
mode http
server k8s-master-0 <kube-master0-ip>:32423 check fall 3 rise 2
server k8s-master-0 <kube-master1-ip>:32423 check fall 3 rise 2
Step 2: Remove swap
Note, If using Debain 11 cloud images no swap so this can be skipped.
nano /etc/fstab
# swap was on /dev/sda6 during installation # UUID=18b43798-486f-499d-9edf-2c551b34b5a1 none swap sw 0 0
Comment the following line:
# swap was on /dev/sda6 during installation # UUID=18b43798-486f-499d-9edf-2c551b34b5a1 none swap sw 0 0
Step 3: Install Container Layer
No matter the below layers this should be applied to allow Kubernetes to run correctly with respect to iptables.
# Setup Kernel Parameters
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system
Done, so next is to pick Containerd or full Docker install. Remember, Containerd is in both but Containerd only is a smaller footprint and thus less attack vectors exist. So, the decision is yours and what requirements that are needed for your use case. Personally, I would install less and then if needed you can add full Docker if it is found it is needed.
Containerd Version – Note: this is currently broken
Simple Containerd install, this is a lighter that Docker containerd only layer to run Kubernetes without the full Docker layer in place. Thus a smaller attack vector exists as it does not contain the full Docker stack on server running Kubernetes.
sudo apt update
sudo apt -y install containerd
Now, it is time to setup containerd config to work correctly with Kubernetes and use SystemdCgroup as required for Kubernetes.
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i '/SystemdCgroup = true/d' /etc/containerd/config.toml
new_string="\[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options\]\n\ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup = true"
sudo sed -i "s/\[plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runc.options\]/$new_string/" /etc/containerd/config.toml
Finally, restart and confirm containerd is using config and containerd is enabled.
sudo catsystemctl restart containerd
sudo systemctl enable containerd
Docker Version
See here and follow 1-a and 1-b.
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker
docker run hello-world
So, setting up Docker to use SystemdCgroup here is one way to configure that option in /etc/docker/daemon.json
.
sudo nano /etc/docker/daemon.json
Fix SystemdCgroup in Docker
Now, if this is a new file as it should be type in the below code.
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
Restart docker
sudo systemctl restart docker
Now really fix SystemCgroup!!! in Containerd layer
However, it has never worked for me so then I do the steps in this section to really fix SystemCgroup.
Create default Containerd config with SystemdCgroup = true
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i '/SystemdCgroup = true/d' /etc/containerd/config.toml
sudo sed -i "s/SystemdCgroup = false/SystemdCgroup = true/" /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd
Step 4: Install K8 on VM
Install needed packages.
KEYRING=/usr/share/keyrings/kubernetes.gpg
#Get Key add to keyring
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee "$KEYRING" >/dev/null
#add associated packages to apt
echo 'deb [signed-by=/usr/share/keyrings/kubernetes.gpg] http://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kurb.list >/dev/null
sudo apt update
sudo apt install kubelet kubeadm kubectl -y
sudo apt-mark hold kubelet kubeadm kubectl
Step 5: Make First Master Node
ClusterConfiguration
So, now it is time to create the first node. First, create a cluster create yaml file like below called kubeadmcf.yaml.
cd ~
nano kubeadmcf.yaml
Note:
<LB_FQDN> = fully qualified domain name for Load Balancer. DNS or hosts; actually both should be setup to include this info.
<LB_IP> = Load Balancer IP.
<6443> = optional port for Load Balancer related port for control plain if left off then it will default to 6443.
<podSubnet> = pod network subnet cidr like 10.50.0.0/16
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
certSANs:
- "<LB_FQDN>"
- "<LB_IP>"
controlPlaneEndpoint: "<LB_FQDN>:<6443>"
networking:
podSubnet: "<podSubnet>" # --pod-network-cidr
Mine looks like this:
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
certSANs:
- "k8master.cld.alshowto.com"
- "192.168.50.200"
controlPlaneEndpoint: "k8master.cld.alshowto.com:6443"
networking:
podSubnet: "10.50.0.0/16" # --pod-network-cidr
So with this config file run the following command
sudo kubeadm init --config=kubeadmcf.yaml
Then once cluster is created and if using a non root user then do this.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Finally, do the same for root so the last commands can be ran.
sudo mkdir -p /root/.kube
sudo cp /etc/kubernetes/admin.conf /root/.kube/config
sudo chown root:root /root/.kube/config
Configure Calico as CNI
Now, it is time to create the cluster network interface. I have used Calico because it supports BGP so I can configure my opnSense router that is routing my cloud network allow for proper routing to the pods directly.
Create a file called calico-installation.yaml and past the following
cd ~
nano calico-installation.yaml
# This section includes base Calico installation configuration.
# For more information, see: https://docs.projectcalico.org/v3.19/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Configures Calico networking.
calicoNetwork:
bgp: Enabled
# Note: The ipPools section cannot be modified post-install.
ipPools:
- blockSize: 26
cidr: 10.50.0.0/16
encapsulation: IPIP
natOutgoing: Enabled
nodeSelector: all()
Now it is time to run that config into kubectl
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl apply -f calico-installation.yaml
Now create calicoctl.cfg
nano calicoctl.cfg
Copy the config text below into newly created file
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/root/.kube/config"
Next install calicoctl direct from github.com
cd ~
wget https://github.com/projectcalico/calicoctl/releases/download/v3.14.0/calicoctl 1> /dev/null 2> /dev/null
sudo chmod +x calicoctl
sudo mv calicoctl /usr/local/bin/
sudo mkdir /etc/calico
sudo cp calicoctl.cfg /etc/calico/
Finally, it is time to finalize the config of Calico create a file called calico-bgpconfiguration.yaml
nano calico-bgpconfiguration.yaml
then copy below config text into newly created file.
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: true
asNumber: 65000
---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: my-global-peer
spec:
peerIP: 192.168.50.200
asNumber: 65000
Save the file and execute with calicoctl
sudo calicoctl apply -f calico-bgpconfiguration.yaml
Common Errors
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: E1007 05:47:14.357349 847 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-10-07T05:47:14Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
The above, this error occurs when SystemdCgroup = false.
At last time to finalize kubectl
kubectl create configmap haproxy-kubernetes-ingress
sudo touch /etc/default/kubelet
echo "KUBELET_EXTRA_ARGS=--node-ip=192.168.50.200" | sudo tee /etc/default/kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet
Step 6: Install K8 Worker Nodes
kubeadm join 192.168.1.139:6443 --token 4bp4wb.kpfgj1x7zulq5sy1 \
--discovery-token-ca-cert-hash sha256:eeb7acbd33ae54874c0481af94a6d3b8abcd84c61a7552d5e19099ec8498ece8
-control-plane
kubeadm join 192.168.1.139:6443 --token 4bp4wb.kpfgj1x7zulq5sy1 \
--discovery-token-ca-cert-hash \sha256:eeb7acbd33ae54874c0481af94a6d3b8abcd84c61a7552d5e19099ec8498ece8
Other Resources
I found this example helpful and modified some of my post info based on info in here especially useful was info around HAProxy setup for master nodes. https://medium.com/@sven_50828/setting-up-a-high-availability-kubernetes-cluster-with-multiple-masters-31eec45701a2
As far as external load balancing of workload data this article is very helpful. Note, the difference here is working with actual load balancing within kubernetes and not just load balancing the access to the master controllers. This expands on the knowledge of HAProxy and introduces the concepts needed to reduce the complexity of getting ingress access to kubernetes. I will expand on this with a separate post once I have more time to process all of the needed logic to also implement into pfSense (OpnSense) with the end goal of adding some of the GBP peering into the pfSense layer since I use that as my router anyway.