locked
child process limits in service context and conhost.exe RRS feed

  • Question

  • I've found that, when our server is running as a windows service, the number of child processes that it can create is limited to about 106.  The odd thing is, CreateProcess() call comes back with a pid, but then the process just fails to materialize.  Meanwhile if I run our server as a console application, this limit goes away or is several times higher.

    Also, If I set the DETACHED_PROCESS flag, this limit goes away, or is at least several times higher.  However, (see my other post) I get failures if I set DETACHED_PROCESS and call CreateProcesssWithLogonW().

    This page (http://blogs.msdn.com/b/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx) hints that deaktop heap exhaustion could be a factor when running "too many services", is it possible that running "too many processes in a server context" might also be a problem?

    Thanks, john


    John Lilley Chief Architect RedPoint Global Inc.

    Thursday, February 28, 2013 11:49 PM

Answers

  • Yes we did find a solution.  The problem is definitely the windows desktop heap.  Note that this *only* applies to programs that are running as services, because the default desktop heap size for services is so much smaller than that for applications.  In our case, we were able to launch about 100 child processes before running out of resources without the change.  With the change, this number can be increased considerably.

    WARNING: this affects the desktop heap of *all* services!  Do not make it larger than necessary or you will push the system to consume more resource and you may bump up against problems in the total available desktop heap size.

    This is the answer we've given to our end users on our knowledgebase:

    If you find that you cannot open more than about 100 total projects, even on a very large RAM server, you may have run into a limit of the Windows "desktop heap size".

    The problem is that service sessions under windows (where the services run) have less of this "desktop heap" space available for creating windows, as described here:

    http://blogs.msdn.com/b/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx

    The short version is:

    • Services get smaller desktop heaps than interactive sessions. 
    • Desktop heap size limits the number of windows
    • Each sub-server creates one or more “windows” even if we can’t see them.

    Solution:

    0) Backup your registry before making any changes!

    1) Run regedit.exe as administrator 

    2) Edit the registry value:
         HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

    3) You will see a string like:
    %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16

    The critical bit is:
    SharedSection=1024,20480,768

    The second number (20480) is the size for interactive sessions.  The third number (768) is the size of non-interactive (services) sessions.   Note how the third number is 26x smaller than the second. Experimentally, we found that changing this to:

    SharedSection=1024,20480,2048

    Increased the project limit from 106 to 270, almost perfectly scaling with the heap size.  Pick a value that reflects the maximum number of projects that you expect to be opened simultaneously by all users on the system.  Do not make this value larger than necessary, and no larger than 8192, as each service in your system will consume more of a precious resource.


    John Lilley Chief Architect RedPoint Global Inc.

    • Marked as answer by john.e.lilley Saturday, February 14, 2015 4:20 AM
    Thursday, July 4, 2013 1:47 PM

All replies

  • Interesting problem, John. 

    What operating system are you using? Is it x86?

    Desktop heap exhaustion is definitely a possibility but there are other things to look at as well, such as GDI resources and USER objects.

    Mark Russinovich has a blog series on Pushing the Limits of Windows that is worth a serious study, including how to use Process Explorer to see which of these resource limitations may be playing a role in your particular scenario.

    Keep posting your findings. I'm very interested in the outcome.

    Friday, March 1, 2013 12:54 AM
  • Hi John,

    Did you solve your problem?
    If so, would you mind sharing the solution with us, it will be very great and helpful for other members of community who stuck in similar case.

    Have a nice day!
    Regrads,


    Elegentin Xie
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, March 6, 2013 6:29 AM
  • Hello All,

    I am having a very similar issue.  I am running a WCF service which requires client processors to be run in the background.  I can spawn exactly 86 child processes before process.Start returns false.  No exception or error message, simply stops spawning our clients.  This is consistent behavior on my desktop.

    I tried this on a co-workers computer (laptop and it stopped at 83 processes consistently.

    Both boxes are running 64bit Windows 7.

    Also strange is that as I stop the conhost.exe programs associated with my clients (a single conhost.exe gets spawned for each client) I can start more processors.  When I stop the conhost.exe the associated client remains in tact.  So if I stop one conhost.exe (86 get spawned with my 86 clients) I can spawn the 87th client, leaving 86 conhost.exe and 87 clients.  It is almost as if windows will not allow more than 86 conhost.exe running at a single time.  Expected behavior?  Way to stop spawning conhost.exe (although from other forums it seems that this console host is needed for security from shatter attacks)?

    Any thoughts, suggestions?  

    Thank you!

    Thursday, July 4, 2013 12:33 AM
  • Yes we did find a solution.  The problem is definitely the windows desktop heap.  Note that this *only* applies to programs that are running as services, because the default desktop heap size for services is so much smaller than that for applications.  In our case, we were able to launch about 100 child processes before running out of resources without the change.  With the change, this number can be increased considerably.

    WARNING: this affects the desktop heap of *all* services!  Do not make it larger than necessary or you will push the system to consume more resource and you may bump up against problems in the total available desktop heap size.

    This is the answer we've given to our end users on our knowledgebase:

    If you find that you cannot open more than about 100 total projects, even on a very large RAM server, you may have run into a limit of the Windows "desktop heap size".

    The problem is that service sessions under windows (where the services run) have less of this "desktop heap" space available for creating windows, as described here:

    http://blogs.msdn.com/b/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx

    The short version is:

    • Services get smaller desktop heaps than interactive sessions. 
    • Desktop heap size limits the number of windows
    • Each sub-server creates one or more “windows” even if we can’t see them.

    Solution:

    0) Backup your registry before making any changes!

    1) Run regedit.exe as administrator 

    2) Edit the registry value:
         HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

    3) You will see a string like:
    %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16

    The critical bit is:
    SharedSection=1024,20480,768

    The second number (20480) is the size for interactive sessions.  The third number (768) is the size of non-interactive (services) sessions.   Note how the third number is 26x smaller than the second. Experimentally, we found that changing this to:

    SharedSection=1024,20480,2048

    Increased the project limit from 106 to 270, almost perfectly scaling with the heap size.  Pick a value that reflects the maximum number of projects that you expect to be opened simultaneously by all users on the system.  Do not make this value larger than necessary, and no larger than 8192, as each service in your system will consume more of a precious resource.


    John Lilley Chief Architect RedPoint Global Inc.

    • Marked as answer by john.e.lilley Saturday, February 14, 2015 4:20 AM
    Thursday, July 4, 2013 1:47 PM
  • We also found that eliminating the conhost.exe (via the proves flag -- I think it is the DETACHED flag) increased the number of possible projects/executables.  However, this had undesirable side-effects, mostly that the child process cannot run as different user credentials.


    John Lilley Chief Architect RedPoint Global Inc.

    Thursday, July 4, 2013 1:49 PM