Why is uploading files to VMFS so slow?

Sourced from; http://cormachogan.com/2013/07/18/why-is-uploading-files-to-vmfs-so-slow/#more-1892

This is something which has come up numerous times, and behavior which many of you have observed. There seems to be some issue with uploading files to a VMFS datastore. In fact, in one example, we had someone report that it took 10 minutes to upload a Windows 7 ISO to an iSCSI datastore and less than 1 minute to upload the same ISO to an NFS datastore.  Both datastores were very healthy and fast, and both had to run VMs on them.  There have been variations of this behavior reported before. This post will try to explain why.The issue with copying files to VMFS is slower than NFS due to the metadata operational overheads. Copying files piece-by-piece results in a lot of metadata updates as the file grows.  One possible solution is to make a large file, copy it and truncate it to size at the end. This is why we always recommend using vmkfstools to move virtual machines between datastores at the command line. As a file grows in little chunks, VMFS will have to continually invoke metadata operations to allocate new resources on the VMFS. Much of this would require the disk to be locked to ensure data integrity. NFS doesn’t need to do that as it has a very different file locking mechanism.

One way of verifying this is to look at the VAAI statistics for VMFS datastores backed by VAAI capable storage arrays, or SCSI reservation requests/reservations per second for the non-VAAI datastores. A simple example is to gzip a very large VMDK on a VMFS volume which sorts of does the same things (lots of very small reads and writes). What you will observe is a very large number of locks occurring. Even with VAAI, we observed via esxtop a very large number of ATS and ZERO requests taking place on the volume:

ATS       ATSF    ZERO     ZERO_F    MBZERO/s
42052         3    61370               0     5.72

From this, we can surmise that the gzip operation locks the resource/block, zeroes it out, and then copies the new contents in. The same would hold through for a file upload operation to VMFS either via the UI, or a CLI copy command.

Hopefully, this sheds some light on why these types of operations are slower on VMFS than they are on NFS.