You will need to ensure you have VMware Tools installed and running before you can use this utility. As of right now, customers can get a hold of this utility by filing an SR with VMware Support and referencing PR 1385761. The workaround that has been developed is a tiny standalone command-line utility called vmware-resolutionSet which runs within the Mac OS X Guest and allows you to configure a custom display resolution. Given this is a non-trivial fix, VMware Engineering has been working hard on a providing a workaround which would still allow users to set a custom resolution from within the GuestOS. The reason for this behavior is that Apple has changed the way in which it remembers previously used modes and would automatically fall back to this versus retaining the custom mode using the Display Preferences.

For customers who are running Mac OS X 10.9 (Mavericks) or newer in a Virtual Machine, you may have noticed that you can no longer set a custom display resolution beyond the default 1024x768 in either VMware Fusion and vSphere, regardless of the amount of video memory that has been allocated.