Having a small vSAN cluster might be a challenge during maintenance work. It might be even a bigger challenge when we face a host failure, especially we do not have +1 host to migrate data.
Imagine a situation when you have a 2-node vSAN cluster (esx-05 and esx-06) and one host (esx-05) is down. Our VMs will work because they will be reading and writing to the other copy of VMDK on esx-06. VMs that run on esx-05 will be restarted by HA on esx-06.
The question is will we be able to create NEW VMs?
With default SPBM: vSAN Default Storage Policy when we require 2 copies of the VMDK we would not have sufficient Fault Domains to create a vSAN object.
That is why with 2-Node clusters we should consider enabling one of the Advanced Policy Rules: Force Data Provisioning. This policy will create an object on the vSAN datastore event if it does not have enough Fault Domains available = just one host when there are 2 required (+ Witness).
With Force Provisioning enabled, we can create a new VM (VM5), even if host (esx-05) is not accessible. Both Witness node and the second host (esx-06) are available and form a vSAN cluster together. Creating VMs is possible because vSAN creates objects with FTT=0 policy. It means that temporarily we have just one copy do the data.
As soon as the second host (esx-05) is up, we can either manually immediately Repair Object (apply actions to be compliant to the SPBM) or we can wait till vSAN timer kicks in (default setting is 60min).
Here we can see that all of our objects are healthy because vSAN object health has a green status.
This screen shows that objects are complaint to their original policy of having two copies of the data.
Another question is will it work for Erasure Coding eg. RAID-5?
If Force Provisioning is enabled, we can for example create a VM when we have just 3 nodes out of 4 available. But objects will have again only FTT=0 policy.
Storage Policy Compliance is Noncompliant in this case, but VM runs fine on just one copy (but we cannot afford any other failures).
When our host is back, objects will be re-written to match Storage Policies = to be compliant to the policies.
It is worth to mention that there are some exceptions where Force Provisioning will not help us. When a VMDK has FTT=1 mirror policy and we have 3 out of 4 hosts available, we will not be able to change the policy to FTT=1 RAID-5, event when it has Force Provisioning setting enabled.
Will we be able to change the policy from FTT=1 RAID-5 to FTT=1 mirror when we have 3 out of 4 hosts running?
Yes we will.
If you want to play with vSAN Storage Polices, you can use VMware Hands On Labs like I did. There is a vSAN lab where can work on 8 ESXis and test our scenarios without touching our production evnironments.