July | 2013 | rhyshammond.com

I’ve seen this issue a couple times in my travels, in the past we would have to dig down in the CLI but in the latest version of vSphere, there is a new tool built right into the vSphere Client.

Firstly, consider the following

Run VM consolidation outside business hours if the VM is considered critical production.
Ensure you have a recent backup of the VM before proceeding as consolidation can fail.
The remove snapshot process can take a long time to complete if the snapshots are large.
If the consolidation process is stopped before completing, it may result in data corruption.
The virtual machine performance may be degraded while the snapshot consolidation process.

To check if virtual machine consolidation is required:

Select a vCenter Server host or a cluster and click the Virtual Machines tab.
Right-click the menu bar for any virtual machine column and click Needs Consolidation. The Needs Consolidation column appears.
- A Yes status indicates that the snapshot files for the virtual machine should be consolidated and that the virtual machine’s Tasks and Events tab shows a configuration problem.
- A No status indicates that the files are OK.
Note: A Configuration Issue warning is also displayed in the Summary tab of the virtual machine indicating that virtual machine disk consolidation is needed.

To consolidate snapshots:

Right-click the virtual machine and click Snapshot > Consolidate.
Check the Need Consolidation column to verify that the task succeeded. To check the Consolidation progress see Commands to monitor snapshot deletion in ESX 2.5/3.x/4.x and ESXi 3.x/4.x/5.x (1007566)If the task succeeded, the Configuration Issues message clears and the Needs Consolidation value is No.

Note: The consolidate option allows you to consolidate the delta disks into the base disk when there are no snapshots shown in the Snapshot Manager. It does not fix the underlying issue that may have caused this to occur initially.

Consolidating snapshots may be required when:

The snapshot commit fails due to locked files. For details on how to release the lock on locked files, see Investigating virtual machine file locks on ESXi/ESX (10051).
There is a temporary loss of communication between the vCenter Server and the ESXi/ESX host during snapshot commit. To refresh the connectivity, restart the ESXi/ESX host management agents. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).

Why is uploading files to VMFS so slow?

Sourced from; http://cormachogan.com/2013/07/18/why-is-uploading-files-to-vmfs-so-slow/#more-1892

This is something which has come up numerous times, and behavior which many of you have observed. There seems to be some issue with uploading files to a VMFS datastore. In fact, in one example, we had someone report that it took 10 minutes to upload a Windows 7 ISO to an iSCSI datastore and less than 1 minute to upload the same ISO to an NFS datastore. Both datastores were very healthy and fast, and both had to run VMs on them. There have been variations of this behavior reported before. This post will try to explain why.The issue with copying files to VMFS is slower than NFS due to the metadata operational overheads. Copying files piece-by-piece results in a lot of metadata updates as the file grows. One possible solution is to make a large file, copy it and truncate it to size at the end. This is why we always recommend using vmkfstools to move virtual machines between datastores at the command line. As a file grows in little chunks, VMFS will have to continually invoke metadata operations to allocate new resources on the VMFS. Much of this would require the disk to be locked to ensure data integrity. NFS doesn’t need to do that as it has a very different file locking mechanism.

One way of verifying this is to look at the VAAI statistics for VMFS datastores backed by VAAI capable storage arrays, or SCSI reservation requests/reservations per second for the non-VAAI datastores. A simple example is to gzip a very large VMDK on a VMFS volume which sorts of does the same things (lots of very small reads and writes). What you will observe is a very large number of locks occurring. Even with VAAI, we observed via esxtop a very large number of ATS and ZERO requests taking place on the volume:

ATS ATSF ZERO ZERO_F MBZERO/s
42052 3 61370 0 5.72

From this, we can surmise that the gzip operation locks the resource/block, zeroes it out, and then copies the new contents in. The same would hold through for a file upload operation to VMFS either via the UI, or a CLI copy command.

Hopefully, this sheds some light on why these types of operations are slower on VMFS than they are on NFS.