vSAN unlike vMotion does not have dedicated TCP/IP stack. This means it uses default gateway.
In clusters where vSAN uses a single L2 domain it is not a problem. In cases where there are multiple L2 domains within a cluster (stretched cluster, dedicated L2 domain per site, clusters that span racks in a Leaf and Spine topology) we need to define static routes to reach other L2 domains.
It is important to know that when you enter a dedicated gateway address for the vSAN network (Override default gateway for this adapter) it does not override the routing table on the ESXi host:
ESXi attempts to route all traffic through the default gateway of the default TCP/IP stack (Management) instead.
So far, the only option to route traffic via dedicated gateway for vSAN is to create a static route using this command:
esxcli network ip route ipv4 add --gateway IPv4_address_of_router --network IPv4_address
Other useful commands:
esxcli network ip route ipv4 list
esxcli network ip route ipv4 remove -n network_ip/mask -g gateway_ip
esxcfg-route -l
Let’s not forget about the most important one after every change in the network: vmkping:
If you have Jumbo Frames configured in your environment, run vmkping with -d (disable fragmentation) and -s (size).
