In some cases, it may be necessary to share the same virtual disk (vmdk or RMD) between 2 (or more) virtual machines (VMs) on VMWare ESXi. This configuration is most commonly used in clustering scenarios. You can share the vmdk file located on the shared storage, or on a local datastore on the ESXi host. If you want to use the shared disk on different ESXi hosts, you can use only the shared VMFS (datastore), or SAN/iSCSI/vSAN storage. A shared virtual disk is a VMDK file that two or more virtual machines can read and write at the same time.
VMware Multi-Writer Mode for Shared VMDK
VMware VMFS is a clustered file system shared between multiple ESXi hosts. By default VMFS, NFS datastores, vSAN are not allowing multiple virtual machines to access the same .vmdk file simultaneously (locks are used). This prevent VMs from accessing other virtual machines disks, and avoid data corruption or loss. However, there is a VMware technology that allows sharing VMFS-backed disks between several VMs. This feature is called Multi-writer VMDK. The Multi-Writer option ensures that a guest host clustering application (such as Oracle RAC or Microsoft MSCS) doesn’t cause data loss when accessing from two or more virtual machines.
Hint. Multi-writer is also used for VMWare Fault Tolerance when primary and standby virtual machines access a shared vmdk file simultaneously.
Virtual disks, which must be available simultaneously in several VMWare virtual machines, use the Multi-writer technology (available starting from VMware ESXI 5.5 and newer). In most cases, this shared disk operation mode is used in cluster solutions Oracle RAC and Microsoft MSCS (Microsoft Cluster Service) clusters.
The main restrictions of shared VMWare disks in Multi-Writer mode:
- You won’t be able to perform the online migration of running VMs with shared disk to another host (vMotion), or datastore (Storage vMotion). You can migrate between ESXi hosts only powered off virtual. Hint. vMotion is only supported for Oracle RAC cluster with up to 8 ESXi hosts;
- If you try to write data to such a vmdk from the guest operating system (for example, creating a folder), then this data (folder) will be visible only on the host that created it. Those, the Multi-Writer VMDK technology is needed just for clustering, and it’s impossible to use it as a shared disk with automatic file synchronization;
- You cannot expand such a VMDK disk online (if you try to increase the size of the vmdk, you get the error: The disk extend operation failed: The virtual disk requires a feature not supported by this program. Hot-extend is currently supported only for VMFS flat virtual disks without snapshots opened in persistent mode);
- You can’t suspend a VM with shared disk;
- Snapshot creation and Changed Block Tracking technology are not supported (only for independent-persistent disk starting from vSphere 5.1 update 2);
- Virtual disks in multi-writer mode must not be connected to the virtual NVMe controller.
Sharing a VMDK Disk Between Two Virtual Machine on ESXi Hosts
Suppose the shared external storage (connected to each ESXi host using iSCSI or Fiber Channel SAN) has already been presented to all VMWare ESXi 6.5 hosts. These hosts running the two VMs (node1 and node2) you want to add a shared virtual disk. On both virtual machines, you need to add a new SCSI controller.
In the vSphere Client inventory select the first virtual machine (Node 1), and select Actions > Edit Settings.
To add a new virtual device select Add New Device > SCSI controller in the dropdown list, and press the Add button.
Select LSI Logic SAS as a type of SCSI controller. Be sure to create a new SCSI controller, don’t use a default SCSI 0 controller.
Now you need to choose the SCSI Bus sharing mode:
- Virtual — if you want to share a virtual disk between VMs on the same ESXi host;
- Physical — used when you need to share vmdk file between VMs on different ESXi hosts (if you have multiple ESXi hosts in VMWare vSphere cluster then you must select this bus sharing mode).
Hint. None mode is used only for not clustering Multi-writer drives.
Press OK. Next, you need to add a new virtual disk on the first VM (Add New Device > Hard Disk > Add) with the following disk settings:
- VM Storage Policy: optional;
- Location: you need to select a shared datastore name that will store the vmdk file;
- Disk Provisioning: select Thick provision eager zeroed (shared virtual disk cannot be zeroed thick or thin provisioned. You can use other disk provisioned types if your shared vmdk file is located on the vSAN 6.7 Patch 01 datastore). Note. You can convert thick provision lazy zeroed disk to thin;
- Sharing: Multi-writer;
- Virtual Device Node: select the SCSI controller created earlier;
- Disk mode: Independent – Persistent – in this mode, you can’t create snapshots for a virtual drive you want to share.
Hint. If you enabled VMWare Fault Tolerance protection for a virtual machine, the multi-write VMDK mode is enabled automatically for virtual disks.
In the same way, you need to add a new SCSI controller to another virtual machine (Node 2). Then you need to add a new disk (Existing Hard Disk) in the settings of the second virtual machine.
Select the VMFS Datastore, which stores the shared vmdk file that you created earlier for the first VM. Select the name of the first virtual machine, and in the middle pane select the desired vmdk file.
Also, you need to select the Sharing mode — Multi-Writer for this disk.
If you want to use more than one shared disk on VMware, keep in mind that each of them doesn’t require the addition SCSI controller. You can connect up to 16 different drives for a single SCSI controller.
Make sure you use the same SCSI device address for a shared vmdk disk on both virtual machines. Therefore, if you added SCSI Controller 1 when creating a SCSI controller, you should select it. For example, if you have a new address SCSI(1:0) on VM (node1) for a new disk (this means the first disk on the second SCSI controller), you should use the same address SCSI(1:0) for the shared disk on the second VM (node2). It is important!
Note. In ESXi versions, before ESXi vSphere 6.0 update 1, the MultiWriter mode for vmdk is also supported, but you cannot enable it from the vSphere Web Client interface. You can add a multi-writer flag by shutting down the VM and manually editing the vmx file of the virtual machine, adding the following line at the end of the file:SCSI1:0.sharing = “multi-writer”
You can also add this parameter in the VM properties: Options > General > Configuration Parameters: Scsi1:0:sharing “multi-writer”.
After changing the virtual machine’s settings, connect to the console of the guest OS virtual machine. In this case, the VMs running Windows Server so you can remotely connect to them using RDP (how to enable RDP remotely). Run the Computer Management console (compmgmt.msc), and expand Storage > Disk Management section. Right-click and select Rescan Disk.
The system detects the new disk and offers to initialize it. Select the desired partition table (MBR or GPT), create a new partition, and format it. Similar operations must be performed on the second VM. After these settings, both VMs are using a shared disk. Now you can proceed to set up a cluster solution based on Oracle RAC and Microsoft Cluster Services (MSCS).
Note. Keep in mind that using multi-writer VMDKs for non-clustered scenarios is not supported. If the shared VMDK disk is online on both ESXi hosts, you won’t see changes made to the disk by one virtual machine in the other virtual machine. This will happen until you bring the disk offline, and then bring it online.
When trying to migrate a VM with a connected vmdk disk in MultiWriter mode, an error will appear:
Virtual machine is configured to use a device that prevents the operation: Device ‘SCSI controller X’ is an SCSI controller engaged in bus-sharing.
To resolve this issue turn off the VM, and perform cold vMotion.
In a similar way, you can provide simultaneous access from different VMs to the same raw LUN via Raw Device Mapping (RDM) mode. In this case, the RDM disk is added to node1 as usual, and when adding a disk to node2 instead of selecting the LUN number, you need to select the mapping VMDK file for the RDM disk on the VMFS Datastore.
Starting from vSphere 6.7 Update 1 you can share a single virtual disk in multi-writer mode on more than 8 hosts simultaneously. This behavior is supported for VMFS5/VMFS6 datastores as well after you enabled the /VMFS3/GBLAllowMW config option.