While browsing the familiar vCenter UI right after an upgrade from 6.7U3 to 7.0U1 I noticed some small but nice changes I wanted to share.
Enable Capacity Reserve for vSAN datastore
Cluster -> Configure-> vSAN Services: this part just got a new option: Enable Capacity Reserve. To be able to see the benefits of this improvement, lets see how it was in pre-7.0 era.
In my 4-node vSAN cluster where hosts have around 17,49 TB capacity each, the datastore capacity is 69,86 TB. When actually written capacity (used by VM) reached 49,47 TB which was more than 70% of the cluster space, it was still ‘all green’ for the cluster in CAPACITY USAGE view. Don’t look at dedupe savings, this is just a test environment.
The only indication that the datastore was getting full and that we could potentially run into rebuild issues if a host failed was vSAN Skyline Health check “What if the most consumed host fails”.
In this situation having datastore which was 70% full already and 47 TB of VM data, loosing 17,49 TB would mean datastore size around 52 TB. 47/52 means datastore full in 90%. But I was still able to create new VMs, this limit was soft.
For vSAN 6.7 to stay on the safe side we had to use fixed 25-30% of slack (free) space regardless of the cluster size. 25-30% guaranteed there would be enough space for rebuilds and cluster operations in case of a node failure but in many cases it was too much and not everyone was an expert in monitoring the cluster size. On the other hand, not everyone could afford to constantly monitor the space making sure no one creates an automation task to provision hundreds of VMs over the weekend that could make vSAN datastore full.
Capacity Reserve is a feature that helps to control the vSAN capacity and is specific to your cluster size. It will also protect the cluster from provisioning new VMs, so datastore will not get full.
In my case, I still have 4-node vSAN Cluster with capacity 69,86 TB. When I enable Capacity Reserve feature, my VMs use only 9,19 TB of vSAN datastore. I don’t get to set the capacity reservations, the exact values are calculated and set for me. When I select Operations reserve, it is up to me to select also Host rebuild reserve (the first one has to be selected to be able to select the second).
Looking at capacity Overview of the vSAN datastore, I see that the system has reserved 16,54 TB for the host rebuild (23.68 %) and 1.78 TB for operations (2.54%).
When I create more VMs and vSAN tasks, my capacity utilisation reaches 45.45 TB (the utilisation is now almost the same as in the first scenario). I can also see the Operations reserve increased to 7.08 TB (10%). It is because I created more VMs and more tasks so more cluster resources would be needed for operations.
You can see Capacity Overview changed colour to yellow, showing I am reaching my space limit. Actually “free” space on disks is now 815 GB. The remaining capacity is secured in case of a host failure or rebuilds.
You can also see an alert in vCenter: vSAN Health alarm “Cluster disk space utilization” triggered although cluster still has free capacity. Isn’t it little bit like HA Admission Control for a storage?
Even though there was still some space on vSAN datastore, it prevented me from creating new VMs.
We also get Disk space vSAN Health alert “This cluster level disk usage is above 34 854 GB, which is 80% of the host rebuild tolerance threshold”.
And in addition to that, also ‘What if the most consumed host fails’ is triggered independently of the other alerts.
It seems the feature is very effective and finally we can put a hard stop and prevent vSAN from getting full.