HCX is more just one component, but the main one is called HCX Manager. It is deployed as first and it is the one you can login to using https://FQDN_OR_IP:9443. Web UI is always a first step in troubleshooting because you can quickly check or restart services. And the most important, start SSH service to get to the console.
> ccli
Welcome to HCX Central CLI
Few simple commands that you can ran are:
> list
To list the connected service appliances:
> go 0
to select a specific appliance
> hc -d
to run a detailed healtcheck on the selected appliance, like this one…it is in a pretty bad shape:
> ssh
to connect to a selected appliance (no username and password required) to check networking, routing etc but also to view the logs. For Interconnect appliance (HCX-WAN-IX) /var/log/vmware/hbrsrv.log and /var/log/vmware/mobilityagent.log are the most valuable in troubleshooting.
To leave ccli just type > exit.
On HCX Manager the best destination for log analysis is: /common/logs/admin/app.log , /common/logs/admin/job.log and /common/logs/admin/web.log.
The most common issues that may occur during setup are mostly networking ones around interconnect between sites, Management Network, Uplink Network and vMotion Network.
And HCX Plugin in vCenter will show the following: tunnel status down
We can go through a very long list checking open ports running > ping, > netcat -vz : https://ports.vmware.com/home/VMware-HCX
We can take a shortcut as well (not sure if this is supported method but I believe we are good to go if we only want to edit something) and check HCX Mongo DB:
> mongo hybridity
> show collections
will list all the tables in the database. The table that is worth checking is the following (from what I checked it works on HCX Cloud connector/on-prem site where you can RUN DIAGNOSTICS on service mesh ):
> db.ServiceMeshDiags.find().pretty()
Look for entries:
"message" : "Diagnostics completed. There are 7 failed probes.",
"status" : "FAILED",
------------------------------
"status" : "FAILURE",
"error" : {
"output" : "",
"message" : "Failed to reach destination"
}
}
],
"status" : "ERROR",
"message" : "HCX-NET-EXT is unable to reach HCX-NET-EXT-PEER on the ports 4500. Please ensure firewall is not blocking the ports or routing is correctly configured."
------------------------------
{
"type" : "REACHABILITY_HTTPS_CONNECT",
"source" : "x.x.x.x",
"destination" : "x.x.x.x",
"sourcePort" : 0,
"destPort" : 443,
"destType" : "HCX-WAN-IX",
"protocol" : "TCP",
"status" : "FAILURE",
"error" : {
"output" : "",
"message" : "Failed to connect to target"
},
status" : "ERROR",
"message" : "HCX is unable to reach HCX-WAN-IX on the ports 443. Please ensure firewall is not blocking the ports or routing is correctly configured."
-----------------------------
"type" : "REACHABILITY_TCP_CONNECT",
"source" : "x.x.x.x",
"destination" : "x.x.x.x",
"sourcePort" : 0,
"destPort" : 8000,
"destType" : "Deployment_HostSystem",
"protocol" : "TCP",
"status" : "FAILURE",
"error" : {
"output" : "dial tcp x.x.x.x:8000: connect: no route to host",
"message" : ""
},
This table is a real time-saver!