Issues encountered when setting up VMware Tanzu

None of the hosts connected to this vCenter are licensed for Workload Management

If you get issue, open vCenter in private or incognito mode in your browser. Seems to be a known issue.

“HTTP communication could not be completed with status 404” when deploying Supervisor cluster

You might get 404 error, but those can be ignored. Seems to be a known issue.

Asymmetric routing with VMware Tanzu and HAproxy 3-leg setup

When I installed HAproxy I chose the “Frontend Network” deployment options in my install, which gave me 3 nic for network, “Management”, “Workload”, and “Frontend”. After deploying the namespace in workload management, the deployment just kept working in a “Creating” status and did not seem to have any progress.

After troubleshooting it, it seemed like the control plane node for the namespace that was deployed did not have access to the load balancer vip addresses, and therefor could not access the api it uses. Also I noticed when I logged in on ssh to the control plane, my ssh connection stopped responding after minutes. I also saw that my pfSense that I use as firewall started dropping packets for the connections on the interface.

This all seems to be due to how routing rules have been set up on the HAproxy node, Supervisor Controller nodes, and the Control Plane node. The HAproxy and Supervisor Controllers have IP routing rules that routes all traffic on the correct interface directly to the gateway. No direct local traffic. This seems to be correct to avoid the issue of asymmetric routing. However the Control Plane nodes only have one nic, and no IP routing rules have been spesified, so this sens traffic to the local network directly to the needed server.

My pfSense clearly did not like this asymmetric routing, so when it started receiving packets that it did not know about in it’s connection state table, it started dropping it, which again created issues.

The only “good” way I found to solve this was to create spesific firewall rules on the pfSense to avoid connection state tracking on this traffic as mentioned on this page under “Manual Fix”. However it probably would’ve been better if the Control Plane node also had IP routing rules forcing all traffic to go to the gateway, but that would be something for VMware to fix.

References

• (1): https://www.virtuallyghetto.com/2020/11/quick-tip-rebooting-vcsa-causes-vsphere-with-tanzu-to-show-esxi-hosts-not-licensed-for-workload-management.html “Quick Tip – Rebooting VCSA causes vSphere with Tanzu to show ESXi hosts not licensed for Workload Management”
• (2): https://www.virtuallyghetto.com/2020/11/why-am-i-seeing-http-communication-status-404-error-when-configuring-vsphere-with-tanzu-how-to-fix.html “Why am I seeing HTTP communication status 404 error when configuring vSphere with Tanzu & how to fix?”
• (3): https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html “Troubleshooting Asymmetric Routing”