Nutanix AHV offered one more troubleshooting helping hand feature called ‘Nutanix AHV Memory dump log generator’ for Windows VM. In case of Windows hang / hung / crash / not responding / unresponsive Windows VM state due to problem in the netkvm driver, high network load may increase and/or any other issue, can generate Windows VM memory dump, error log for hang / hung / crashed / not responding Windows VM in real time in Nutanix AHV.
Windows VM hosted on Nutanix AHV can provide the real time memory dump / error log that is very helpful to get the actual issue / error of Windows VM’s crashing / hanging state.
Prerequisites of Windows VM
To generate / collect Nutanix AHV hosted Window VM Memory dump / error log prerequisites are:
Step 1: Install latest Nutanix VirtIO driver because older than 1.1.4 have an issue in vioscsi driver that prevents memory dump creation. You may see progress stuck at 0% while OS tries to write dump to disk. Please upgrade Nutanix VirtIO to version 1.1.4 or newer. Download Nutanix Virito 1.1.4 Tool
Read also: How to do Nutanix Guest Tool NGT Installations in Windows and Linux VM
OR
If VirtIO cannot be upgraded, following workaround can be used:
- Add new IDE disk to affected VM. Size needs to be determined based on the need, but 15-20 Gb should be more than enough for kernel dumps.
- Format it and assign drive letter.
- Configure ‘Dedicated Dump File’ and ‘Dump File’ options to use newly added disk as described in https://blogs.msdn.microsoft.com/ntdebugging/2010/04/02/how-to-use-the-dedicateddumpfile-registry-value-to-overcome-space-limitations-on-the-system-drive-when-capturing-a-system-memory-dump/
Step 2: Enable memory dump setting in windows VM
You must be logged on as an administrator or a member of the Administrators group to complete this procedure. If your computer is connected to a network, network policy settings may prevent you from completing this procedure.
To enable memory dump setting, follow these steps:
- In Control Panel, select System and Security > System.
- Select Advanced system settings, and then select the Advanced tab.
- In the Startup and Recovery area, select Settings.
- Make sure that Kernel memory dump or Complete memory dump is selected under Writing Debugging Information.
- Restart the computer.
Note: You can change the dump file path by edit the Dump file field. In other words, you can change the path from %SystemRoot%\Memory.dmp to point to a local drive that has enough disk space, such as E:\Memory.dmp.
Read also: Windows 10 High CPU Consumption Issue On Nutanix AHV
Step 3: Use Non-Maskable Interrupt NMI
Starting from Windows 8 and Windows Server 2012 OS later are configured to crash on Non-Maskable Interrupt ( NMI ) by default, so no additional configuration required.
On some computers, you cannot use keyboard to generate a crash dump file. For example, Hewlett-Packard (HP) BladeSystem servers from the Hewlett-Packard Development Company are managed through a browser-based graphical user interface (GUI). A keyboard is not attached to the HP BladeSystem server.
In these cases, you must generate a complete crash dump file or a kernel crash dump file by using the Non-Maskable Interrupt (NMI) switch that causes an NMI on the system processor.
Read Also: Top 5 Microsoft Unknown Services Possibly You Don’t Know
To do this, follow these steps:
Follow the steps in this section carefully. Serious problems might occur if you modify the registry incorrectly. Before you modify it, back up the registry for restoration in case problems occur.
- In Registry Editor, locate the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl
- Right-click CrashControl, point to New, and then click DWORD Value.
- Type NMICrashDump, and then press Enter.
- Right-click NMICrashDump, and then select Modify.
- In the Value data box, type 1, and then select OK.
- Restart the computer.
- Hardware vendors, such as HP, IBM, and Dell, may provide an Automatic System Recovery (ASR) feature. You should disable this feature during troubleshooting. For example, if the HP and Compaq ASR feature is enabled in the BIOS, disable this feature while you troubleshoot to generate a complete Memory.dmp file. For the exact steps, contact your hardware vendor.
- Enable the NMI switch in the BIOS or by using the Integrated Lights Out (iLO) Web interface. Note For the exact steps, see the BIOS reference manual or contact your hardware vendor.
- Test this method on the server by using the NMI switch to generate a dump file. You will see a STOP 0x00000080 hardware malfunction.
Source Microsoft KB: https://docs.microsoft.com/en-us/windows/client-management/generate-kernel-or-complete-crash-dump
Nutanix AHV – Generate Memory Dump
Lets explore the way to generate / collect / get unresponsive , hang , hung, crashed Windows VM Memory dump / error log hosted on Nutanix AHV hypervisor
The only way to generate a memory dump from hung VM is to inject NMI interrupt into a Windows guest and triggering Windows to bugcheck.
Note: Starting from Windows 8 and Windows Server 2012 OS later are configured to crash on NMI by default, so no additional configuration required.
To send NMI from the Nutanix AHV host to the Windows VM, run the following commands:
Step 1: On any Nutanix CVM run the following command and note VM UUID:
nutanix@cvm$ acli vm.list
Step 2: On the Nutanix AHV host (where the VM is running), run the following command to send the NMI.
[root@AHV ~]# virsh inject-nmi <VM_UUID>
Step 3: VM will bugcheck (crash / BSOD) and reboot once memory dump is written.
Step 4: Collect Memory Dunp error log – By default, memory dump is saved in following location inside VM:
%SYSTEMDRIVE%\Windows\memory.dmp
Example: C:\Windows\memory.dmp
Read Also: Microsoft Integrating Linux Kernel Inside Windows 10
Windows VM – Memory Dump Log Analysis
After collecting the Windows VM logs from Nutanix AHV if found following logs:
Child-SP RetAddr Call Site
00 fffff802`3d2a3c08 fffff802`3bc371c2 nt!KeBugCheckEx
01 fffff802`3d2a3c10 fffff802`3be2de4d hal!HalBugCheckSystem+0x7e
02 fffff802`3d2a3c50 fffff802`3bc37fa1 nt!WheaReportHwError+0x22d
03 fffff802`3d2a3cb0 fffff802`3be51c20 hal!HalHandleNMI+0xfe
04 fffff802`3d2a3ce0 fffff802`3bdc77c2 nt!KiProcessNMI+0x150
05 fffff802`3d2a3d30 fffff802`3bdc7636 nt!KxNmiInterrupt+0x82
06 fffff802`3d2a3e70 fffff802`3bcfcd6b nt!KiNmiInterrupt+0x176
02 (Inline Function) --------`-------- netkvm!CNdisSpinLock::Lock+0xa 03 (Inline Function) --------`-------- netkvm!CLockedContext<CNdisSpinLock>::{ctor}+0xa 04 (Inline Function) --------`-------- netkvm!CParaNdisTX::DoWithTXLock+0xa 05 ffffd001`b4bb9d20 fffff800`a89d1b7c netkvm!CParaNdisTX::NBLMappingDone+0x25 06 ffffd001`b4bb9d50 fffff802`3bc06a37 netkvm!CNBL::RegisterMappedNB+0x6c 07 ffffd001`b4bb9d80 fffff800`a7254e82 hal!HalBuildScatterGatherListV2+0x207
Issue Found: Windows guests running on AHV may crash ( bugcheck / BSOD ) or hang due to problem in the netkvm driver. High network load may increase chances of encountering this issue.
VirtIO NetKVM driver issue Solution
The issue is resolved in Nutanix VirtIO 1.1.0 and newer versions.
Download latest Nutanix AHV Virito Driver 1.1.4 for Windows
Please check How to Ugrading Nutanix VirtIO for Windows
Conclusion
Nutanix AHV is the market leading cloud enabled hypervisor that is evolving as per customer requirement as Nutanix embedded the Windows VM memory dump / Kernel error log generating feature to easily collect the windows hang / crash / unreponsive state error logs to troubleshoot the issue.
Thanks to being with your favorite HyperHCI Tech Blog to stay tuned with latest Technology.!