So, I thought I had a article related to this when I was answering this post on Server fault. However, I could not find one so I wrote this one to put into Serverfault. Hopefully I will have my HA Kubernetes figured out to also post into the serverfault article as it may be a more efficient way to LoadBalance (LB) in Kubernetes. Anyway, this is kind of the first step anyway and this will work for anything including standalone web servers. Especially, if I can get my HA Kubernetes working as it utilizes Calico BPG and pfSense FFR plugin to get Pods overlayed into my pfSense routed subnet. One note, this is more of a journey then a final complete destination article I am only providing some working and work-in-progress options to help understand some very complex topics in both pfSense and Kubernetes. Hopefully, by the end of this journey it is a truly workable solution for your needs. Or at least, provided you the tools needed to complete your own journey into these abstract but powerful tools.

Domain

First, have to have this in order to serve anything. Now, it is possible to get away with some really free options like DuckDNS or NoIp and that may be good enough for testing purposes. Overall, it is usually better to have an actual Domain Host such as GoDaddy or Epik to provide a real domain name. I know Epik even allows some domains to be purchased for a lifetime. Well, they used to have forever and here. Sad, I was going to do that for my domain. Oh well, just another change I had to make to this article. So, it looks like Epik made that change 6 days ago so around December 14, 2022 they stopped offering forever domains. So, looks like forever may be possible from here. I will try it out later. Anyway, for professional looking websites and web apps it is recommended to get a solid domain from a reputable registrar.

DNS

Critical step, It does not matter if configuring for only internal or external URL’s what does matter is that there is a DNS record that gets you to the pfSense router and provides a unique path for HAProxy to route through to the associated backend server. See with proper setup it does not even matter that the web server is in Kubernetes or is bare-metal or VM or whatever is need. I have this example already about routing the pfSense’s own web through pfSense HAProxy and yes if https is going to be used it is critical to make sure port 443 is not used by pfSense’s own web interface. So, that article is needed to make this all work seamless behind pfSense.

Pick a DNS Provider

I know it is possible to create your own DNS server with BIND and have all kinds of fun setting it up and getting it all to work and then after about a week you finally have a working DNS server that you then can continue to do the real work you were going to do before you decided to setup your DNS. So, see even the last sentence got to be to crazy and that was just talking about it. That reminds me, Keep It Simple Stupid (KISS). I know this personally because I have many awesome experiences setting up BIND. In the end, Pick a existing provider you can revisit that later if you are so inclined! Unless, you already have a very workable BIND solution then go to town and use it. After all, DNS is DNS. Oh, use DNSSEC to protect your domain against DNS attacks. After all, security is an onion and you have to layer it to really protect your assets. Again, this is why I use an external DNS provider today and Cloudflare has free DNS options so look into it if you do not have a workable solution for true DNS routing. Yes, most domain hosting providers also have DNS options, but Cloudflare is great to setup Let’s Encrypt see my article on that here.

Internal DNS aka Split DNS

Again, KISS use pfSense’s own DNS services to set this up here is an example. That reminds me, many websites act different behind HAProxy so make sure to research how to configure them for HAProxy or even if it is your own site how it will act behind HAProxy. Again, I would configure site standalone and internally. Then, once working then migrate to Kubernetes. Then finally, stand up externally facing. Also, internal DNS does not have to be externally facing. In fact, most cases it should not be the same. Setting up in HAProxy it is not difficult to move from internal to external facing when ready. I know, somethings can only be run in Kubernetes. So, in that case know how to test it without adding to many layers all at once. Remember, as the implementer you want to know how each layer interacts and impacts the flow of the application though the system.

HAProxy Config

I have included my working example of this alshowto.com website configured to run behind HAProxy.

# Automaticaly generated, dont edit manually.
# Generated on: 2022-12-17 00:37
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

frontend wan-https-access
	bind			165.23.32.190:443 name 165.23.32.190:443   ssl crt-list /var/etc/haproxy/wan-https-access.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			matches-alshowto.com	var(txn.txnhost) -m str -i alshowto.com
	acl			match-wp.alshowto.com	var(txn.txnhost) -m str -i wp.alshowto.com
	acl			match-next.alshowto.com	var(txn.txnhost) -m str -i next.alshowto.com
	acl			match-wm-alshowto	var(txn.txnhost) -m str -i webmail.alshowto.com
	acl			aclcrt_wan-https-access	var(txn.txnhost) -m reg -i ^([^\.]*)\.alshowto\.com(:([0-9]){1,5})?$
	acl			aclcrt_wan-https-access	var(txn.txnhost) -m reg -i ^alshowto\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	http-request set-header X-Forwarded-Proto https  if   aclcrt_wan-https-access
	http-request redirect code 301 location https://alshowto.com%[capture.req.uri]  if  match-wp.alshowto.com aclcrt_wan-https-access
	use_backend WordPress-Main-Page_ipvANY  if  matches-alshowto.com aclcrt_wan-https-access
	use_backend next-Main-Page-copy_ipvANY  if  match-next.alshowto.com aclcrt_wan-https-access
	use_backend wm-alshowto_ipvANY  if  match-wm-alshowto aclcrt_wan-https-access
	use_backend WordPress-Main-Page_ipvANY  if   aclcrt_wan-https-access

frontend internal-lan-front
	bind			192.168.2.1:443 name 192.168.2.1:443   ssl crt-list /var/etc/haproxy/internal-lan-front.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			matches-dmz-rtr	var(txn.txnhost) -m str -i rtr.dmz.alshowto.com
	acl			aclcrt_internal-lan-front	var(txn.txnhost) -m reg -i ^([^\.]*)\.dmz\.alshowto\.com(:([0-9]){1,5})?$
	acl			aclcrt_internal-lan-front	var(txn.txnhost) -m reg -i ^dmz\.alshowto\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend dmz-rtr-gui-backend_ipvANY  if  matches-dmz-rtr aclcrt_internal-lan-front
	use_backend dmz-rtr-gui-backend_ipvANY  if   aclcrt_internal-lan-front

backend WordPress-Main-Page_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		100000
	timeout server		100000
	retries			3
	option			httpchk GET / 
	server			mainwordsrv 192.168.2.11:80 id 101 check inter 1000  

backend next-Main-Page-copy_ipvANY
	mode			http
	id			105
	log			global
	timeout connect		100000
	timeout server		100000
	retries			3
	option			httpchk GET / 
	server			mainnextsrv 192.168.2.5:80 id 101 check inter 1000  

backend wm-alshowto_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET / 
	server			wm-alshowto-srv 192.168.2.39:80 id 106 check inter 1000  

backend dmz-rtr-gui-backend_ipvANY
	mode			http
	id			103
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET / 
	server			dmz-rtr-gui 192.168.2.1:8080 id 104 check inter 1000

Network Configuration

Put lightly, this is a complex setup! However, once I figured this out I was able to easily spin up many pfSense servers. Each pfSense serves a specific purpose. LAN, DMZ and CLD are the main VLAN based subnets running though a 1g fiber WAN. Critical to me, was the ability to serve WAN as a VLAN. So that is what I did below is my “edge” switch.

Setup

Here is my managed “edge” 10g ports

For my situation I also have vlan 999 and this is a special vlan for it is where my WAN connects. That is, all pfSense routers have a NIC pointing to VLAN 999 and then a NIC pointing to associated VLAN for example VLAN 2 for DMZ or VLAN 50 for CLD.

So, port 1 goes to a tp-link managed 24 port POE switch which only support 1g SFP so that is why it is yellow. Port 2 goes to Fiber router. Port 3 and 4 are LAG’ed to ProxMox pve4 vm server. Port 5 and 6 are LAG’ed to ProxMox pve6 vm server. Port 7 and 8 are LAG’ed to ProxMox pve7 vm server. Also note, Port 2 is 10g so the fiber router provided by my ISP is capable of 10g speeds and so is my WAN. Currently, I am paying for a 1g connection and may go up to a 2g if I see to much traffic on my WAN. However, currently it is not even close to a problem.

DMZ

Very critical, I also have a cld subnet but for most a dmz subnet should be good enough. Again, this helps with the security onion concept where there are layers of security protecting the infrastructure. Managed switches help me distribute my workloads according to the vlan that they are running within and add yet another layer of separation between the subnets. For example, my subnets fall within the 196.168.0.0/16 layer. Most are /24 with the exception of my main LAN which is a /23 subnet. So, that main LAN falls between 192.168.0.0 and 192.168.1.255. I will call that vlan 1 for simplicity. Also, most managed switches have vlan 1 so real vlan is usually between vlan 2 and 4096. Technically, there are some reserved vlans but for most general users that is not a concern. Still I would stay way from vlan 1 as every switch I have ever seen setup internal vlan to 1 so it is best to not use vlan 1. My case, vlan 2 is my DMZ so 192.168.2.0/24 is the subnet that I use for DMZ. CLD resides on vlan 50 so following the pattern it is at 192.168.50.0/24. Oh, that is also why I have 0 and 1 vlans combined into LAN subnet since this works well for all unmanaged switches and routers and overall dumb devices in general as they fall on 192.168.0.0/23 subnet.