Using HAProxy to Pass the K8s API on CentOS8

So from last time I have a Kubernetes cluster built on top of RancherOS with a HAProxy loadbalancer on CentOS 8 sitting in front of it. Lately, I’ve been working on tightening down my home firewall rules and had restricted traffic to the loadbalancer to only ports 80 and 443. That didn’t have any effect on the different web services I was running but the next time I pulled out kubectl I couldn’t get a connection. Turns out, duh, you also need to pass the Kubernetes API port 6443 through the load balancer if you want to connect.

Seems simple enough to fix, add the two entries below to /etc/haproxy/haproxy.cfg and reboot the service.

# ----------
# /etc/haproxy/haproxy.cfg
# ----------

...

frontend rancher_k8s_api
    bind *:6443
    mode tcp
    default_backend rancher_k8s_api_backend

...

backend rancher_k8s_api_backend
    mode tcp
    option tcp-check
    server rancher-1 rancher-1.example.com:6443 check
    server rancher-2 rancher-2.example.com:6443 check
    server rancher-3 rancher-3.example.com:6443 check

$ sudo systemctl restart haproxy

Except of course it isn’t! Looking at sudo journalctl -xe it turns out SELinux was blocking HAProxy from binding to port 6443. Fortunately, it did provide some options for how to resolve the problem. Since the Kubernetes API is effectively HTTP traffic I reconfigured SELinux to tag 6443 appropriately and restarted HAProxy again.

$ sudo semanage port -a -t http_port_t -p tcp 6443
$ sudo systemctl restart haproxy

The last thing to do is login to Rancher real quick and pull down a new copy of my Kubeconfig that points to the load balancer instead of at one of the control plane nodes. With that we’re back up and running!