So, I thought I had a article related to this when I was answering this post on Server fault. However, I could not find one so I wrote this one to put into Serverfault. Hopefully I will have my HA Kubernetes figured out to also post into the serverfault article as it may be a more efficient way to LoadBalance (LB) in Kubernetes. Anyway, this is kind of the first step anyway and this will work for anything including standalone web servers. Especially, if I can get my HA Kubernetes working as it utilizes Calico BPG and pfSense FFR plugin to get Pods overlayed into my pfSense routed subnet. One note, this is more of a journey then a final complete destination article I am only providing some working and work-in-progress options to help understand some very complex topics in both pfSense and Kubernetes. Hopefully, by the end of this journey it is a truly workable solution for your needs. Or at least, provided you the tools needed to complete your own journey into these abstract but powerful tools.
Domain
First, have to have this in order to serve anything. Now, it is possible to get away with some really free options like DuckDNS or NoIp and that may be good enough for testing purposes. Overall, it is usually better to have an actual Domain Host such as GoDaddy or Epik to provide a real domain name. I know Epik even allows some domains to be purchased for a lifetime. Well, they used to have forever and here. Sad, I was going to do that for my domain. Oh well, just another change I had to make to this article. So, it looks like Epik made that change 6 days ago so around December 14, 2022 they stopped offering forever domains. So, looks like forever may be possible from here. I will try it out later. Anyway, for professional looking websites and web apps it is recommended to get a solid domain from a reputable registrar.
DNS
Critical step, It does not matter if configuring for only internal or external URL's what does matter is that there is a DNS record that gets you to the pfSense router and provides a unique path for HAProxy to route through to the associated backend server. See with proper setup it does not even matter that the web server is in Kubernetes or is bare-metal or VM or whatever is need. I have this example already about routing the pfSense's own web through pfSense HAProxy and yes if https is going to be used it is critical to make sure port 443 is not used by pfSense's own web interface. So, that article is needed to make this all work seamless behind pfSense.
Pick a DNS Provider
I know it is possible to create your own DNS server with BIND and have all kinds of fun setting it up and getting it all to work and then after about a week you finally have a working DNS server that you then can continue to do the real work you were going to do before you decided to setup your DNS. So, see even the last sentence got to be to crazy and that was just talking about it. That reminds me, Keep It Simple Stupid (KISS). I know this personally because I have many awesome experiences setting up BIND. In the end, Pick a existing provider you can revisit that later if you are so inclined! Unless, you already have a very workable BIND solution then go to town and use it. After all, DNS is DNS. Oh, use DNSSEC to protect your domain against DNS attacks. After all, security is an onion and you have to layer it to really protect your assets. Again, this is why I use an external DNS provider today and Cloudflare has free DNS options so look into it if you do not have a workable solution for true DNS routing. Yes, most domain hosting providers also have DNS options, but Cloudflare is great to setup Let's Encrypt see my article on that here.
Internal DNS aka Split DNS
Again, KISS use pfSense's own DNS services to set this up here is an example. That reminds me, many websites act different behind HAProxy so make sure to research how to configure them for HAProxy or even if it is your own site how it will act behind HAProxy. Again, I would configure site standalone and internally. Then, once working then migrate to Kubernetes. Then finally, stand up externally facing. Also, internal DNS does not have to be externally facing. In fact, most cases it should not be the same. Setting up in HAProxy it is not difficult to move from internal to external facing when ready. I know, somethings can only be run in Kubernetes. So, in that case know how to test it without adding to many layers all at once. Remember, as the implementer you want to know how each layer interacts and impacts the flow of the application though the system.
HAProxy Config
I have included my working example of this alshowto.com website configured to run behind HAProxy.
# Automaticaly generated, dont edit manually.
# Generated on: 2022-12-17 00:37
global
maxconn 1000
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbproc 1
nbthread 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
frontend wan-https-access
bind 165.23.32.190:443 name 165.23.32.190:443 ssl crt-list /var/etc/haproxy/wan-https-access.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl matches-alshowto.com var(txn.txnhost) -m str -i alshowto.com
acl match-wp.alshowto.com var(txn.txnhost) -m str -i wp.alshowto.com
acl match-next.alshowto.com var(txn.txnhost) -m str -i next.alshowto.com
acl match-wm-alshowto var(txn.txnhost) -m str -i webmail.alshowto.com
acl aclcrt_wan-https-access var(txn.txnhost) -m reg -i ^([^\.]*)\.alshowto\.com(:([0-9]){1,5})?$
acl aclcrt_wan-https-access var(txn.txnhost) -m reg -i ^alshowto\.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
http-request set-header X-Forwarded-Proto https if aclcrt_wan-https-access
http-request redirect code 301 location https://alshowto.com%[capture.req.uri] if match-wp.alshowto.com aclcrt_wan-https-access
use_backend WordPress-Main-Page_ipvANY if matches-alshowto.com aclcrt_wan-https-access
use_backend next-Main-Page-copy_ipvANY if match-next.alshowto.com aclcrt_wan-https-access
use_backend wm-alshowto_ipvANY if match-wm-alshowto aclcrt_wan-https-access
use_backend WordPress-Main-Page_ipvANY if aclcrt_wan-https-access
frontend internal-lan-front
bind 192.168.2.1:443 name 192.168.2.1:443 ssl crt-list /var/etc/haproxy/internal-lan-front.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl matches-dmz-rtr var(txn.txnhost) -m str -i rtr.dmz.alshowto.com
acl aclcrt_internal-lan-front var(txn.txnhost) -m reg -i ^([^\.]*)\.dmz\.alshowto\.com(:([0-9]){1,5})?$
acl aclcrt_internal-lan-front var(txn.txnhost) -m reg -i ^dmz\.alshowto\.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend dmz-rtr-gui-backend_ipvANY if matches-dmz-rtr aclcrt_internal-lan-front
use_backend dmz-rtr-gui-backend_ipvANY if aclcrt_internal-lan-front
backend WordPress-Main-Page_ipvANY
mode http
id 100
log global
timeout connect 100000
timeout server 100000
retries 3
option httpchk GET /
server mainwordsrv 192.168.2.11:80 id 101 check inter 1000
backend next-Main-Page-copy_ipvANY
mode http
id 105
log global
timeout connect 100000
timeout server 100000
retries 3
option httpchk GET /
server mainnextsrv 192.168.2.5:80 id 101 check inter 1000
backend wm-alshowto_ipvANY
mode http
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server wm-alshowto-srv 192.168.2.39:80 id 106 check inter 1000
backend dmz-rtr-gui-backend_ipvANY
mode http
id 103
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server dmz-rtr-gui 192.168.2.1:8080 id 104 check inter 1000Network Configuration
Put lightly, this is a complex setup! However, once I figured this out I was able to easily spin up many pfSense servers. Each pfSense serves a specific purpose. LAN, DMZ and CLD are the main VLAN based subnets running though a 1g fiber WAN. Critical to me, was the ability to serve WAN as a VLAN. So that is what I did below is my "edge" switch.
Setup
Here is my managed "edge" 10g ports
For my situation I also have vlan 999 and this is a special vlan for it is where my WAN connects. That is, all pfSense routers have a NIC pointing to VLAN 999 and then a NIC pointing to associated VLAN for example VLAN 2 for DMZ or VLAN 50 for CLD.
So, port 1 goes to a tp-link managed 24 port POE switch which only support 1g SFP so that is why it is yellow. Port 2 goes to Fiber router. Port 3 and 4 are LAG'ed to ProxMox pve4 vm server. Port 5 and 6 are LAG'ed to ProxMox pve6 vm server. Port 7 and 8 are LAG'ed to ProxMox pve7 vm server. Also note, Port 2 is 10g so the fiber router provided by my ISP is capable of 10g speeds and so is my WAN. Currently, I am paying for a 1g connection and may go up to a 2g if I see to much traffic on my WAN. However, currently it is not even close to a problem.
DMZ
Very critical, I also have a cld subnet but for most a dmz subnet should be good enough. Again, this helps with the security onion concept where there are layers of security protecting the infrastructure. Managed switches help me distribute my workloads according to the vlan that they are running within and add yet another layer of separation between the subnets. For example, my subnets fall within the 196.168.0.0/16 layer. Most are /24 with the exception of my main LAN which is a /23 subnet. So, that main LAN falls between 192.168.0.0 and 192.168.1.255. I will call that vlan 1 for simplicity. Also, most managed switches have vlan 1 so real vlan is usually between vlan 2 and 4096. Technically, there are some reserved vlans but for most general users that is not a concern. Still I would stay way from vlan 1 as every switch I have ever seen setup internal vlan to 1 so it is best to not use vlan 1. My case, vlan 2 is my DMZ so 192.168.2.0/24 is the subnet that I use for DMZ. CLD resides on vlan 50 so following the pattern it is at 192.168.50.0/24. Oh, that is also why I have 0 and 1 vlans combined into LAN subnet since this works well for all unmanaged switches and routers and overall dumb devices in general as they fall on 192.168.0.0/23 subnet.