none
ChangeTracker.sys seems to leak memory RRS feed

  • Question

  • Several servers we manage suddenly (since approx. 4-6 weeks) seem to crash / freeze after a few days because the memory is full, through poolmon we found out that the cause seems to be the ChangeTracker driver (tag: MSpy). It appears to be taking up memory without releasing memory.


    We are still trying to figure out the exact process causing this and if there is any pattern in this behaviour.

    I cannot imagine us being the only one having this issue.


    Good to know that this issue only occured on Server 2019 Standard so far.


    Thursday, September 10, 2020 11:01 AM

All replies

  • Try R.clicking my computer/properties/advance/virtual memory tab/change/in properties/set all storage to:Let system manage/click set 2X/close out,restart pc.Also,start task mgr,proceses,locate the the service that has most pc usage,right click,go to open file location,check its defaults.

    Also try these microsoft links for Win Server memory.

    Server Hardware Performance Considerations Microsoft Docs

    Turbo-Charging Windows NT Microsoft Docs (4)

    Hope it helps,otherwise open Event Viewer/Windows/System events/locate the time of event-properties/see what the OS says....


    Tuesday, September 15, 2020 4:06 AM
  • It's probably even better to know that the issue also occurs on Server 2016 Datacenter (v. 1607, Build 14393.3930).

    MSpy shows currently 31.9 GB in poolmon.

    That's a terminal server with 47 GB RAM, so it's not frozen yet, but the users there already reported some discomfort.

    Friday, October 2, 2020 2:02 PM
  • We have exactly the same situation, multiple servers, Windows 2019 server standard, Windows 2016 server standard and Windows 2012 R2 server standard. Mostly RDS servers in our case. I do not have a solution to the problem.

    Thursday, October 8, 2020 9:19 AM
  • We are having the same issue. ChangeTracker.sys existing in C:\Windows\System32\drivers and the program files of our backup software. This does not existing in another other server that does not have the backup software.

    Do your servers have backup software installed?

    I am trying to figure out if this is a Windows issue or if we should be contacting our backup vendor.

    Thanks!

    Thursday, October 8, 2020 4:37 PM
  • If you're using SolarWinds Backup and Recovery / MSP Backup, contact your backup vendor. The Backup Accelerator (which uses and likely installed that ChangeTracker.sys kernel driver from the Win7 DDK) function of Backup Manager (at least versions 20.6-20.9) had a memory leak which should be resolved in version 20.10 released three days ago.
    Thursday, October 8, 2020 6:20 PM
  • Thank you for confirming this! I contacted Solarwinds and they provided the 20.10 download and installed on the servers that were experiencing the issue often to make sure it resolves it.
    Thursday, October 8, 2020 8:38 PM
  • Please confirm this is a fix.  Solarwinds is making us jump through hoops to obtain that version.
    Friday, October 9, 2020 12:50 AM
  • As of just a short while ago, 20.10 is in General Availability via the primary download link as well.

    https://www.solarwindsmsp.com/products/backup-recovery/backup-manager

    Friday, October 9, 2020 2:00 PM
  • The bad news, at least one of my endpoints seems to still be exhibiting symptoms of a memory leak on 20.10, but it has not yet been rebooted since the upgrade yesterday. Everything looked great at the time OF the upgrade...
    Friday, October 9, 2020 4:41 PM
  • Confirmed 20.10 did not fix this. I reach out to Solarwinds support. They told me to run the command below which I did on the servers having the issue frequently. I assume this will prevent the hangups because the exe and driver causing the issue gets turned off, in other words a short term fix.

    net stop "Backup Service Controller"

    cd C:\Windows\System32\drivers

    ren ChangeTracker.sys ChangeTracker.sys.old

    cd "c:\Program files\Backup Manager"

    ren BackupUP.exe BackupUP.exe.old

    net start "Backup Service Controller"

    Saturday, October 10, 2020 12:56 PM
  • Here is the latest response from them. It looks like they will disable the change tracker in a future release. I told them I had the same issue with 20.10 and they did not specify which version of 20.10 will fix the issue.

    "Thank you for your response, we have no specific version of 20.10 as of yet as our DevTeam currently looking into it and it should automatically update all your servers for the new version but after disabling the backup accelerator (changeTracker) probably resolved the issue"

    Monday, October 19, 2020 1:57 PM
  • I'm a Solarwinds MSP Backup user as well and I have the same problem on Windows 10 laptop after it goes to sleep and wakes up a few times. Memory goes up to 95% and BackupFP.exe process can't be terminated. System performance get really bad. Eventually the system has a BoD and when I check WhoCrashed it's related to the ChangeTrack.sys. 

    Unfortunately on this machine I'm still running version 20.6.

    Friday, October 23, 2020 3:40 AM
  • I've been running an internal test build (still 20.10 because it's still October 2020) they provided me for the past week and have not yet seen the memory leak on any devices upon which I've installed it. A new build of 20.10 has just gone into General Availability today (20.10.0.20292, up from the last 20.10 GA build of 20.10.0.20279); I recommend reinstalling/upgrading to the new version/build just made available from the primary download site.
    Friday, October 23, 2020 6:46 PM