Virtual Disk(s) Locked (code 1935)

Situation

A VMware backup plan terminates with the following warning: Virtual disk(s) locked. Cannot back up VM(s): some virtual disks are locked with the list of VMs that failed to back up.

Cause

To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

• VMNAME.vswp
• DISKNAME-flat.vmdk
• DISKNAME-ITERATION-delta.vmdk
• VMNAME.vmx
• VMNAME.vmxf
• vmware.log

Solution

Initial Quick Test

Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host helps the lock.

Identify the Locked File

1. Power on the virtual machine, this process should fail and display an error message. Make note of this error.
2. Connect to the ESXi host the virtual machine is on with an SSH session.
3. Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or SEsparse file for VMFS, or the .UUID.lck file for vSAN.

vmfsfilelockinfo takes these parameters:

• File to be tested
• Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

For example:

vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u administrator@vsphere.local

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.10 with user administrator@vsphere.local
Password: xXxXxXxXxXx
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.178
Searching on Host 192.168.1.179
Searching on Host 192.168.1.180
MAC Address : xx:xx:xx:xx:xx:xx

Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

Total time taken : 0.27 seconds.

4. To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:

For example:

lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

Cartel | World name | Type | fd | Description
36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk

This shows that the file is locked by a virtual machine having Cartel ID 36202.

5. Now display the list of active Cartel IDs by with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
Display Name: Alternate_VM27
Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
………

The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.

6. If no processes are shown, the following script can search for VMs with the locked vmdk mounted.

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/"  '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Remove the Lock

1. Power off the virtual machine holding the lock.
2. Reboot the ESXi host that holds the lock.