Overview

What is HAProxy? Well, put simply it allows forward redirection of web traffic to another server. Ok, why do this? Here are the top reasons:

1. Load balancing
2. URL parsing to other servers via url
3. Obfuscation of Web Server
4. Separation of Traffic via protocols such as TCP port 80, 443, 25, etc..
5. Protection of web server
6. Offload of SSL certs

Setup HAProxy

Simple, pick a server. For me, I chose a public VPS as it has a good static IPv4 IP.

First, Run this command to make sure system is current as possible. Note, adjust as you see fit.

sudo apt-get update
sudo apt-get upgrade

Next, Run apt install of HAProxy

sudo apt install haproxy

Now, Confugure HAProxy

nano /etc/haproxy/haproxy.cfg

Below, this is my current config for my site

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 50000

        # Default SSL material locations
        lua-load /etc/haproxy/haproxy-acme-validation-plugin-0.1.1/acme-http01-webroot.lua
        ca-base /etc/ssl/certs
        crt-base /etc/haproxy/ssl

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
        option forwardfor
        option http-server-close

# Configuration for HTTP site
#frontend http-in
#       bind 142.202.205.204:80
#       acl url_acme_http01 path_beg /.well-known/acme-challenge/
#       http-request use-service lua.acme-http01 if METH_GET url_acme_http01
#       redirect scheme https code 301 if !{ ssl_fc }
#
#frontend https
#       bind 142.202.205.204:443 ssl crt alshowto.com.pem
#       mode http
#       default_backend web-https

frontend myfrontend
        mode http
        bind 142.202.205.204:80
        bind 142.202.205.204:443 ssl crt alshowto.com.pem
        #handle ssl certs
        acl url_acme_http01 path_beg /.well-known/acme-challenge/
        # Redirect HTTP to HTTPS
        http-request redirect scheme https unless { ssl_fc }
        default_backend web-https

backend default
        mode http
        balance roundrobin  # Load Balancing algorithm
        option httpchk
        option forwardfor
        server WEB1 192.168.0.234:80 weight 1 maxconn 512 check

backend web-https
        mode http
        balance roundrobin
        option httpchk
        option forwardfor
        option http-server-close
        http-request set-header X-Forwarded-Proto https if { ssl_fc }
        server WEB1 192.168.0.234:80 weight 1 maxconn 512 check

Configure Let's Encrypt

Above, it is possible to see that it is configured to acl on /.well-known/acme-challenge/

frontend myfrontend
        mode http
...
acl url_acme_http01 path_beg /.well-known/acme-challenge/
...

Install certbot

First, install certbot

sudo apt install certbot

Get the haproxy acme validation acme-http01-webroot.lua from github

Next, get the lua file needed from github that actuall does the inline support for Let's Encrypt

sudo cd /etc/haproxy
sudo wget https://github.com/janeczku/haproxy-acme-validation-plugin/archive/refs/tags/0.1.1.tar.gz
sudo tar -xf 0.1.1.tar.gz
sudo cd haproxy-acme-validation-plugin-0.1.1/
sudo ls

Now, it is visible that the lua file is in this above folder.

Create a shell script to execute install

Now, Create a bash script to create ssl certs for supported domains.

sudo nano /etc/haproxy/haproxy-acme-validation-plugin-0.1.1/cert-create-haproxy.sh

#!/bin/bash

# Path to the letsencrypt-auto tool
LE_TOOL=/usr/bin/letsencrypt

# Directory where the acme client puts the generated certs
LE_OUTPUT=/etc/letsencrypt/live

# Concat the requested domains
DOMAINS=""
EMAIL="name@somedomain.com"
for DOM in "$@"
do
 if [[ "$DOM" == *@* ]]; then
   EMAIL= "$DOM"
 else
   DOMAINS+=" -d $DOM"
 fi
done

# Create or renew certificate for the domain(s) supplied for this tool
$LE_TOOL --agree-tos --renew-by-default certonly $DOMAINS --text --webroot --webroot-path /var/lib/haproxy --email $EMAIL

mkdir /etc/haproxy/ssl
# Cat the certificate chain and the private key together for haproxy
cat $LE_OUTPUT/$1/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${1}.pem

# Reload the haproxy daemon to activate the cert
systemctl reload haproxy

Now, make cert-create-haproxy.sh executable

sudo chmod +x cert-create-haproxy.sh

Finally, here is how to run this script.

./cert-create-haproxy.sh alshowto.com www.alshowto.com apauna@alshowto.com

Finally, I would always run this shell script manually the first time to accept the yes as shown below

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: yes

Create cron job

So, now that the initial run is done it is a good time to create a cron job to run this thing automagically

See below, this is an example that I have to run this job at 2 am the 1st of every month. Also note, the alshowto.com.log in /var/log/ folder is overwritten every time it is run as to save space if you have a delete on the file then you could change the > to >> to have it append to the file rather then overwrite.

0 2 1 * * bash /etc/haproxy/haproxy-acme-validation-plugin-0.1.1/cert-create-haproxy.sh alshowto.com www.alshowto.com apauna@alshowto.com > /var/log/alshowto.com-certbot.log

Diagnostics

service haproxy restart

Here are the typical errors encountered.

Job for haproxy.service failed because the control process exited with error code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.

What to do

Run systemctl

systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-09-11 21:24:28 UTC; 11s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
    Process: 4824 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=1/FAILURE)
        CPU: 8ms

Sep 11 21:24:28 vps.server.com systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Sep 11 21:24:28 vps.server.com systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 11 21:24:28 vps.server.com systemd[1]: Failed to start HAProxy Load Balancer.
Sep 11 21:24:28 vps.server.com systemd[1]: haproxy.service: Scheduled restart job, restart counter is at 5.
Sep 11 21:24:28 vps.server.com systemd[1]: Stopped HAProxy Load Balancer.
Sep 11 21:24:28 vps.server.com systemd[1]: haproxy.service: Start request repeated too quickly.
Sep 11 21:24:28 vps.server.com systemd[1]: haproxy.service: Failed with result 'exit-code'.
Sep 11 21:24:28 vps.server.com systemd[1]: Failed to start HAProxy Load Balancer.

run tail on haproxy.log

tail /var/log/haproxy.log
Sep 11 21:24:27 vps haproxy[4818]: [ALERT] 253/212427 (4818) : Fatal errors found in configuration.
Sep 11 21:24:27 vps haproxy[4820]: [ALERT] 253/212427 (4820) : parsing [/etc/haproxy/haproxy.cfg:48] : 'bind 142.202.205.204:443' : Didn't find any certificate for bundle '/etc/ssl/private/alshowto.com/haproxy.pem'.
Sep 11 21:24:27 vps haproxy[4820]: [ALERT] 253/212427 (4820) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Sep 11 21:24:27 vps haproxy[4820]: [ALERT] 253/212427 (4820) : Fatal errors found in configuration.
Sep 11 21:24:27 vps haproxy[4822]: [ALERT] 253/212427 (4822) : parsing [/etc/haproxy/haproxy.cfg:48] : 'bind 142.202.205.204:443' : Didn't find any certificate for bundle '/etc/ssl/private/alshowto.com/haproxy.pem'.
Sep 11 21:24:27 vps haproxy[4822]: [ALERT] 253/212427 (4822) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Sep 11 21:24:27 vps haproxy[4822]: [ALERT] 253/212427 (4822) : Fatal errors found in configuration.
Sep 11 21:24:28 vps haproxy[4824]: [ALERT] 253/212428 (4824) : parsing [/etc/haproxy/haproxy.cfg:48] : 'bind 142.202.205.204:443' : Didn't find any certificate for bundle '/etc/ssl/private/alshowto.com/haproxy.pem'.
Sep 11 21:24:28 vps haproxy[4824]: [ALERT] 253/212428 (4824) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Sep 11 21:24:28 vps haproxy[4824]: [ALERT] 253/212428 (4824) : Fatal errors found in configuration.

Certbot failures

So, look at the log file /var/log/letsencrypt/letsencrypt.log

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: yes
Account registered.
Requesting a certificate for alshowto.com and www.alshowto.com
Performing the following challenges:
http-01 challenge for alshowto.com
http-01 challenge for www.alshowto.com
Using the webroot path /var/lib/haproxy for all unmatched domains.
Waiting for verification...
Challenge failed for domain alshowto.com
Challenge failed for domain www.alshowto.com
http-01 challenge for alshowto.com
http-01 challenge for www.alshowto.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: alshowto.com
   Type:   unauthorized
   Detail: 142.202.205.204: Invalid response from
   http://alshowto.com/.well-known/acme-challenge/XeR-nglNMNtpiCBbixW1_4vdtwqoI6PwBiWNU3XUet8:
   404

   Domain: www.alshowto.com
   Type:   unauthorized
   Detail: 142.202.205.204: Invalid response from
   http://www.alshowto.com/.well-known/acme-challenge/phEc4p738GI9w2VaLFHsEGRD3_9nDKdHXYXapCFXXdg:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
./cert-create-haproxy.sh: line 20: /etc/haproxy/ssl/alshowto.com.pem: No such file or directory
root@vps:/etc/haproxy/haproxy-acme-validation-plugin-0.1.1# service haproxy restart
root@vps:/etc/haproxy/haproxy-acme-validation-plugin-0.1.1# ./cert-create-haproxy.sh alshowto.com www.alshowto.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

GitLab

So, I also have the GitLab project.