Windows Server 2019 vs. VMware Snapshots with quiescing: an elegant solution to the problem



Hello, Habr!

Windows Server 2019 virtual machines with EFI emulation on VMware have a problem with Application-Aware snapshots. It looks like this: a snapshot is done, reaches 100%, hangs for 5 minutes, and then falls into the error Failed to quiesce the virtual machine.
 


The investigation showed that the cause of this error was a conflict between the Windows Server 2019 VSS service and the VMware Snapshot Provider, which is responsible for application quiescing. This thing prepares the virtual machine for snapshots: it stops applications and write operations so that after recovery from snapshots, all data is consistent.

Microsoft has claimed and claims that Application-Aware Snapshots are perfectly supported by the OS. Indeed, such a problem does not occur in Hyper-V. VMware modestly invited its users to simply turn off quiescing (so what, that snapshot will not be consistent at the application level). Or refuse system disks with GPT-marking and EFI emulation (in 2020!). 

Today I’ll show you how to get around this problem beautifully and get consistent application-aware snapshots of Windows Server 2019 virtual machines, and at the same time I’ll remind you of the Microsoft-supported way to install Windows Server on a virtual machine.

Who's guilty?


Start over. When creating a virtual machine on VMware ESXi in VM Options, the default emulation is BIOS emulation. And, while there are no special requirements for security or the size of the system disk, such a scenario is quite appropriate. But as soon as we need to configure Secure Boot or the number of vCPUs exceeds 128, we will need to create a VM with UEFI emulation. If only because there are:

  • Secure Boot - protection against booting from extraneous media.
  • Encryption support (requires TPM support, but in ESXi 6.7 the same trouble).
  • Support for drives over 2TB (doubtful, but it happens).
  • Support for a large number of processors.

If you install Windows on VMware VMs, as we used to do (create VMs, palm off ISO, boot with handles, next, next ...), then when GPT-partitioning the system disk, the Recovery section appears next to the EFI partition and the partition with the system.

We recall the materiel:

Recovery partition - a protected partition on the system hard drive, which is used to restore the factory settings of the system in case of problems. When you install Windows Server on a GPT disk, Recovery writes the Windows Recovery Environment (WinRE). Computer manufacturers, as a rule, supplement this section with their custom settings and drivers tailored for a specific hardware.

Stop! 

But we are not working with hardware, but with a virtual machine. No rollbacks to any factory settings are required. In the end, if you want to return the VM to its original state, we simply redeploy it. Therefore, the Recovery partition on the VM does not make sense and is not particularly needed.

The Recovery section is protected from modification and accidental deletion: it "does not freeze." As a result, a conflict arises between the VSS VMware Snapshot Provider and Microsoft Windows VSS: the VMware provider is trying to snapshot all the partitions of the disk, and Windows Server does not allow it to do this. This good news is reported to us by the logs:



What to do?


It is treated simply. You can install Windows on a VM using the Convert Windows Image cmdlet . Actually, just such an installation script is recommended and supported by Microsoft itself (this script is on the Windows Server 2016 RTM image in the directory with the Nano Server image, but this is a different and no less interesting story;)).

The script will automatically create a template with the updated sysprepped Windows Server, the necessary drivers, and even roles. This approach is interesting because the installation and configuration of the OS can be automated, since the script supports unattend.xml. In addition, this option provides freedom for engineering: from pre-launching a VM on Hyper-V with installation through Invoke-Command to creating a custom service through editing the registry. 

At the output, we get a thick VHD-disk, which you can slip VMware , start the VM, put VMware Tools and convert to a template. As a result, the VM will have GPT markup inside, suitable for running on an EFI VM and without Recovery section by design. As a bonus: in my example, I will immediately create a paravirtual adapter to provide performance. And most importantly - for such a virtual machine, application-aware snapshot can be done without problems.

Now everything is detailed in steps.

Driver extraction


  1. Download the image from the repository .
  2. Mount the image.

  3. Run cmd.exe and go to the root of the mounted image
  4. Run the installer with the keys:
    setup64.exe /A /P <  >
  5. In the second step of the wizard, we repeat the path for unpacking:

  6. The drivers are here: VMWToolsExtract \ VMware \ VMware Tools \ VMware \ Drivers

Windows Server 2019


  1. . , .
  2. : Servicing Stack Update. How to get this update , MS update catalog:

  3. Windows Server 2019 Cumulative Update Servicing Stack Update.



  4. CAB- :



  1. PowerShell .

  2. Install-Module -Name Convert-WindowsImage
  3. - Convert-WindowsImage:
    $ConvertWindowsImageParam = @{
    	#    Windows Server 2019
    	SourcePath = "C:\work\en_windows_server_2019.iso"
    	#   
    	VHDFormat = "VHD"
    	#   
    	SizeBytes = 40GB
    	#   - GPT
    	DiskLayout = "UEFI"
    	BCDinVHD = "VirtualMachine"
    	#   
    	DiskType = "Fixed"
    	#  - Windows Server 2019 Standard (Desktop Expirience)
    	<#
    	 :
    	1 - Windows Server 2019 Standard (Core)
    	2 - Windows Server 2019 Standard (Desktop Expirience)
    	3 - Windows Server 2019 Datacenter (Core)
    	4 - Windows Server 2019 Datacenter (Desktop Expirience)
    	#>
    	Edition = @("2")
    	#  Remote Desktop
    	RemoteDesktopEnable = $True
    	#   
    	Feature = @("TelnetClient","WindowsServerBackup","NetFx3")
    	#   (     CAB)
    	#       :  SSU,  
    	Package = @("C:\work\Windows10.0-KB4523204-x64.cab","C:\work\Windows10.0-KB4534321-x64_PSFX.cab")
    	#     
    	Driver = @("C:\work\VMWToolsExtract\VMware\VMware Tools\VMware\Drivers")
    	}
  4. Windows Server 2019 :
    Convert-WindowsImage @ConvertWindowsImageParam





Installation takes up to 30 minutes.

In this example, the output is a fully prepared OS in the Sysprepped state, but nothing prevents us from using Unattend.xml for customization (Customization specifications are a little puzzled when you see the sysprep template and the VM does not expand).

Read more about the keys of this wonderful kmlet here and in Get-Help of the kmlet itself.

After that, you can load the disk onto ESXi, create a VM and install VMware Tools. And deploy virtual machines without worrying about quiescing snapshots issues.

But what if there is already a β€œdefective” template?


It is enough to simply delete the Recovery partition via diskpart.exe and move the EFI and system partitions for the beauty of the markup (but you can not do this). Is it supported? On a virtual machine - definitely!

  1. We start cmd.exe.
  2. We start diskpart.
  3. Get the list of drives:
    list disk
  4. Select a disk:
    select disk n

    where n is the disk number.
  5. We list the sections:
    list partition
  6. We look at the Recovery (n) partition number and select it:
    select partition n
  7. Remove protection from the section:
    gpt attributes=0x8000000000000000
  8. Delete:
    delete partition
  9. Profit!





VM is ready for Quescing Snapshots!

Instead of a conclusion


This approach is just one of the options for deploying Windows Server and solving the KB60395 problem . Of course, in this case we can make thin VDH and then use the quemu-img utility to convert the virtual disk to the OVF archive, which ESXi understands very well. Such a scenario is quite appropriate, for example, in the case when the virtual disk needs to be transferred over the network. Understanding the history of the issue and the true nature of the problem always makes it possible to find a solution suitable for a particular infrastructure. Technical implementation is a field for interesting experiments and fantasies, which is unlimited.

All Articles