What to do when you have SSD drives in your ESXi hosts and no spinning disks? Use the new Virtual Flash Read Cache (vFRC)! VMware’s VSAN is off the table without spinning disks, so this is the best way to take advantage of the host’s resources. I’m using this in a VDI environment, but it can be used for any VM with read intensive applications and databases. vSphere 5.5 and above is required for this feature. The web GUI is the only way to configure vFRC.
vFRC utilizes flash memory to form two optional read caches:
1. 1. Virtual Flash Host Swap Cache- Used when the ESXi host runs out of physical memory. This feature takes the place of the older SSD Host Cache introduced in vSphere 5.0
2. 2. Virtual Flash Read Cache- Used for virtual machines with high Read workloads. Write-through settings are automatic and not configurable. Writes go directly to the remote storage, bypassing the cache.
Each ESXi host in the VDI cluster has a local 400GB SSD serving as the vFRC resource. When assigned to the vFRC, the disk is formatted in VFFS (Virtual Flash File System).
When a VM is assigned an amount of vFRC, it is absolute reservation and will be used. The cache is warmed through VMware algorithms and the VM will hit the cache first if the needed data is available.
vFRC data is not synchronized throughout the cluster. It is volatile data.
Independent ESXi Host Configuration:
1. Each host has been assigned 4GB for the Host Swap Cache, with the remainder of the SSD assigned to the Virtual Flash Resource Capacity. The capacity of this resource is used for each singular VM’s assigned vFRC amount.
vCenter HA/DRS Considerations:
1. HA clusters are affected directly by vFRC. When a VM is assigned a vFRC resource, it is dependent upon that resource for boot processes. Each host in the HA cluster must have enough vFRC resources to accommodate an n+1 or higher configuration. Even if HA is set to ignore resource limitations (CPU, RAM), vFRC cannot be ignored and HA will fail for certain VMs if vFRC resources run out or a vFRC cache is not configured on the destination host.
2. DRS clusters are affected directly by vFRC. If vFRC is configured for a VM, vCenter will treat this setting as a soft host-affinity rule. DRS will not move a VM unless it is absolutely necessary.
The existing data in a VM’s vFRC cache can be migrated with the VM or not. If the data is moved, the migration will take longer depending on the size of the cache.
4. Cache Warming
vFRC caches will need to be warmed up if the VM is migrated without its cache data on the previous host. If the VM is migrated with its cache, the data remains warm.
In the existing VDI Cluster, HA and DRS are enabled, while SDRS is illegal and is disabled. The sizing of the independent VM caches is based on the ability to survive one host failure in the cluster.
Virtual Machine Requirements:
1. – Each VM must be at virtual hardware level 10 (vmx-10) to be configured with vFRC. Currently, these VMs will be managed only by the VMware Web GUI.
2. – Each VM has to be assigned a vFRC size independently. In a VDI implementation, enable vFRC on the parent VM and choose the cache size. The clones of the parent will receive the same settings upon creation.
3. – Clone destruction will free the vFRC reservation on the host.
– vFRC settings for the individual machine can be found in the hardware settings. Expanding the disk and clicking the Advanced section of Virtual Flash Read Cache reveals the block size options.
vFRC Sizing Decisions:
Reference: Nick Marshall’s Blog
Static Limits for VDI Pools with an expected limit of 1000 VMs:
1. Amount of VMs per ESXi host: 512 MB
vFRC Cache Capacity per host: 360GB
The original cache limit per VM is set to 512MB. In the case of 512 VMs running on one host, the maximum vFRC reservations would be 256GB. In the current VDI cluster, with no hardware additions and the assumption that the VDI project will include 1000 users; this scenario protects the cluster from one host failure.
When a VM is assigned an amount of vFRC it is an absolute reservation and the space is carved out in the ESXi host’s vFRC Capacity Pool immediately. There is no over-allocation process.
vFRC Block Size Decision:
The block size determines the effectiveness of the cache (hit and miss ratio) and the hardware memory resources of the host. The smaller the block size, the more blocks there are to manage. The management of vFRC falls on the host and its hardware resources.
It is recommended to match the block size of the OS and the workloads. The block size for the 32-bit version of Windows 7, which is the parent of the VDI clones, is 4KB. After running a test on the block sizes being used by a test VM, there was enough evidence to configure the block size of the cache at 4KB.
In the graph above, the frequency of the block size and the actual block size is separated by a comma. The highlighted stat shows that the 4K block size was used 29,593 times in a 20 minute period.
- This stat may change with other production users. Consider, it will take individual changes of each VM to change the block size of the vFRC it is assigned.
Monitoring can be done in two place: The vCenter web client and through a SSH session.
The web client offers there two stats, at the VM level:
The vFRC View was customized and is not standard. The Virtual Disk category contains two vFRC stats:
- I/O per second
- Latency – Keep in mind, the latency is in MICROseconds, not the standard milliseconds. In the above, the latency is 0.70 milliseconds, well below to 10ms limit for performance obstruction.
Establishing a SSH session and running some commands yield the following, probably more important stats:
For the test VM, AdminTest-1, the cache hit percentage was 22%, with a total of 1.4M I/O’s in the time the testing took place, which was about 15-20 minutes.