none
Using a Memory Dump to troubleshoot FSLogix issues RRS feed

  • General discussion

  • Using a Memory Dump to troubleshoot FSLogix issues

    Getting a memory dump is often used to troubleshoot issues with FSLogix when a machine is hung/frozen.

     

    Sometimes when a problem is caused by software, it doesn't manifest as a crash/blue screen. The machine can grind to a halt where users may not be able to logon and standard methods of creating a memory dump (also known as a crash dump) do not work.

    Memory dumps are very useful for diagnosing the cause of a software defect. But as mentioned above, not all scenarios result in a crash (which if Windows is configured correctly; see below "Windows memory dump configuration") will result in a memory dump file.

     

    With VMware you can use the following steps to get a memory dump without having a crash occur organically or through manipulation:

    1. If the machine gets in a state where it doesn't respond to mouse or keyboard (or if it does respond it is extremely slow) and even sometimes when it is not 'hung'.
    2. Suspend the VM from the management interface (vSphere or other). Note: this is not a pause. In some interfaces this may be a 'save' command.
    3. Download the VMSS and probably also the VMEM file (see the Note: at the end of this step) from the datastore to a local disk. If for some reason the .vmss will not download via the VMware console, try changing the extension to .txt and then try downloading. The VMSS file should be equivalent in size to the RAM allocated to the server.
      1. Note: If there is a .vmem file with the same date and time as the .vmss, then retrieve both. This will typically happen with the latest VMware hypervisor versions.
    4. VMware has a tool, vmss2core.exe which will extract the memory dump from the VMSS file. We {FSLogix) can do this step if necessary.
      1. Note: If there is the .vmem file as noted above in step 3, then it will also need to be a parameter on the command line when calling the vmss2core.exe. See article #1 below,
    5. The developers can analyze this dump file with windbg just as they can any other memory dump.

     

    Additional information on gathering crash dumps from Citrix:

    #1 - https://support.citrix.com/article/CTX125086

    #2 - https://support.citrix.com/article/CTX127871

    #3 - https://support.citrix.com/article/CTX123642

     

    Windows memory dump configuration

    When Windows does 'crash' it can automatically generate a memory (crash) dump file. This article from Microsoft explains how to configure this: https://support.microsoft.com/en-us/help/927069/how-to-generate-a-complete-crash-dump-file-or-a-kernel-crash-dump-file-by-using-an-nmi-on-a-windows-based-system. View the steps under the section "More Information".

    • For FSLogix troubleshooting please use one of the following:
    1. a Kernel dump or
    2. a Complete memory dump

    Monday, June 24, 2019 11:03 PM
    Owner

All replies

  • How to Get a Memory dump from machines running on Hyper-V

    Machines which hang (become unresponsive) but do not blue screen can have their memory extracted for analysis by this method


    Sometimes machines having problems do not blue screen and thus do not create a memory dump. Memory dumps are very useful to understand/analyze what is happening with driver-level code.

    When the machines are running in a virtual environment there are additional options. The machine can be paused or suspended in the hypervisor even when the machine itself is completely non-responding. The memory state of the machine is saved to a file and this file can be converted to a standard memory dump via tools from the various hypervisor vendors.

    For Hyper-V the details on how to do this are below:


    Monday, July 15, 2019 5:00 PM
    Owner
  • How to Get a Memory Dump from hung physical Windows Machine

    Getting a memory dump is often the best way to diagnose code problems


    Windows memory dump configuration

    When Windows does 'crash' it can automatically generate a memory (crash) dump file. This article from Microsoft explains how to configure this: https://support.microsoft.com/en-us/help/927069/how-to-generate-a-complete-crash-dump-file-or-a-kernel-crash-dump-file-by-using-an-nmi-on-a-windows-based-system. View the steps under the section "More Information".

    • For our purposes here at FSLogix a "Complete" memory dump will be of the most use as we typically have User Mode information which we need to look at. And that info is contained in a complete memory dump. If the information needed is only related to our driver, then a "Kernel" memory dump should suffice.

    Systems may sometimes experience a problem where the system 'locks up' but no Blue Screen is triggered. Having the 'crash dump', we can debug the exact state of the system when the 'lock up' occurs and determine the cause and then the resolution.

    For more information please see:

    https://kb.vmware.com/s/article/1007819

    https://blogs.technet.microsoft.com/yongrhee/2015/04/05/coming-soon-how-to-generate-a-kernel-or-a-complete-memory-dump-file-in-windows-server-2012-and-windows-server-2012-r2/



    Monday, July 15, 2019 5:03 PM
    Owner
  • How to get a Memory Dump in a hosted environments with Driver (Crasher.sys)

    Instructions on how to use the driver that will intentionally cause the machine to crash so the memory dump is saved by the system.


     

    1. To make sure we get the information we need, in System Properties > Start Up and Recovery, the "write debugging information" dropbox field should say 'Complete..'. Sometimes if only the driver is involved in the issue a Kernel dump will suffice. For non-persistent environments the "dumpfile:" field needs to be directed to a persistent storage location that the system has access to.

      If you do not have a persistent C drive, please set it up with an extra drive attached to the system and store the dump file there. Make sure you create the folder you specify on the extra drive or that it exists already. The picture below shows where the dump file will go, type it in and select OK.
      Note: the location where the dump file (for a Complete level dump) is stored must have at least as much space as there is RAM on the system. If the system has 24GB of RAM then the location the dump file will be stored must have slightly more than 24GB of free space available.
        
              
    2. If you have a persistent C drive you can leave it at the default location below.



    3. To install, download thisfile, extract the files and run the config.bat file. Note: it might just say config if you are not showing file types.

    4. If you want to change the time that it takes to crash the system, the default is 5 seconds, you can edit the config.bat file in notepad and change the time. If you already ran the .bat file before you change this you will just run the .bat again after your changes are made and saved.



    5. Then to activate the driver run (local on the machine) from an admin level command line: sc start crasher. This command can also be run remotely if done as an account which has rights on the machine having the issue by: sc \\<machinename> start crasher.



    6. In 5 seconds it will crash the machine. 



    7. A blue screen will occur and the memory dump file will be created.

    8. After the system restarts, get the memory dump file from the location specified earlier.



    Monday, July 15, 2019 5:10 PM
    Owner
  • How to enable/Configure Application Crash Dumps


    1. Create Registry Entry

    Crash dumps are created automatically by Windows if the following registry key is present:

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
    1. Create the registry key LocalDumps if it is not present already.
    2. Add the DWORD value DumpType and set to 2 to get a Full dump which gives more information to work with.
      1. For additional information on application dumps please see: https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx

    Reboot to apply this setting.

    2. Reproduce the issue

    1. Reproduce the problem (i.e. make the application crash).
    2. Locate the crash dump file in %LOCALAPPDATA%\CrashDumps. This is in the user's profile at \<user_name>\AppData\Local\CrashDumps. Please check each user's profile that may have have the application crash.
      1. Note that if the crashing application runs under the System account, that resolves to C:\Windows\System32\config\systemprofile\AppData\Local\CrashDumps.
    3. Attach the Application crash dump file to your helpdesk ticket.
    Monday, July 15, 2019 5:13 PM
    Owner