none
Long delay for ManagedEventWatcher __InstanceCreationEvent query as number of user sessions increases RRS feed

  • Question

  • We have a Windows service that monitors for process start events and sends notifications to client applications.
    We have discovered that the delay between when a process starts and when our EventArrivedEventHandler is called gets excessively long when the number of user sessions on the Windows server gets to about 80.
    The delay gets worse as the number of user sessions gets higher.
    The delays are not consistent. Even with 100 sessions some observed delays are short but most are too long and the maximum observed delay grows with the number of sessions.

    Here is one example of the delay we are seeing.
    A client application wrote its first log record to its log file at 11:05:34.076. Our EventArrivedEventHandler did not get notified of the process start event for the client application until 18 seconds later (at 11:05:52.188 ).
    We need the delay to be less than 5 seconds to be tolerable and would like the delay to be less than 3 seconds if possible.

    Is there something we can do to reduce the delay? Below are the details of our use of WMI.

    We are using an instance of class WqlEventQuery to represent a WMI event query in WQL format.
    We are constructing an instance of ManagementEventWatcher to consume events asynchronously.
    Below is how we are instantiating and running the query. Variable m_PollingIntervalInMilliseconds is set to 1000 by default.
                    WqlEventQuery query = new WqlEventQuery("__InstanceCreationEvent", new TimeSpan(0, 0, 0, 0, m_PollingIntervalInMilliseconds), "TargetInstance isa \"Win32_Process\"");
                    m_ManagementEventWatcher = new ManagementEventWatcher(query);
                    m_ManagementEventWatcher.EventArrived += new EventArrivedEventHandler(managementEventWatcher_EventArrived);
                    m_ManagementEventWatcher.Start();

    Our Windows service is not the only user of WMI services on the server. I do not know if there is contention with other users of WMI services or if there is something about the way we are consuming WMI services that is inefficient.


    • Edited by RossAtWFMC Monday, January 26, 2015 8:44 PM
    Monday, January 26, 2015 8:42 PM

Answers

  • Hello RossAtWFMC,

    It seems that the services are working with a complex environment, and currently, we do not have such an environment which could reproduce this issue you described. Anyway, I would like to share whatever I found and some suggestions about this issue:

    >> called gets excessively long when the number of user sessions on the Windows server gets to about 80.  The delay gets worse as the number of user sessions gets higher.

    This seems to show that the issue is related with the number of user sessions, it may be that when with lots of user sessions, there are something additional delay the event to be fired. As you mentions, there are other services on that server machine, if possible, you could make a test to run your WIM service only to see if it is still delayed.

    >> Is there something we can do to reduce the delay?

    I suggest that you could check this blog below which provide a way to debug with the .NET course code:

    http://blogs.msdn.com/b/dotnet/archive/2014/02/24/a-new-look-for-net-reference-source.aspx

    So that you could know which method inside costs most time.

    From your provided code, it is not very clear if you use multi threads in your service, if not and your event handler is short, you could have a try with it, and there is a discussion about this topic:

    https://social.msdn.microsoft.com/Forums/en-US/13f30e33-7f61-498e-a91a-ef982a63453c/event-handling-in-multithreaded-apps?forum=netfxbcl

    Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, January 27, 2015 6:40 AM
    Moderator
  • The problem was due to single-threading of the event handler. The long delay was an accumulation of delays as the events were queued for the handler. The solution was to change the event handler to spawn a new thread to finish handling the event.
    • Marked as answer by RossAtWFMC Friday, April 3, 2015 7:09 PM
    Friday, April 3, 2015 7:09 PM

All replies

  • Hello RossAtWFMC,

    It seems that the services are working with a complex environment, and currently, we do not have such an environment which could reproduce this issue you described. Anyway, I would like to share whatever I found and some suggestions about this issue:

    >> called gets excessively long when the number of user sessions on the Windows server gets to about 80.  The delay gets worse as the number of user sessions gets higher.

    This seems to show that the issue is related with the number of user sessions, it may be that when with lots of user sessions, there are something additional delay the event to be fired. As you mentions, there are other services on that server machine, if possible, you could make a test to run your WIM service only to see if it is still delayed.

    >> Is there something we can do to reduce the delay?

    I suggest that you could check this blog below which provide a way to debug with the .NET course code:

    http://blogs.msdn.com/b/dotnet/archive/2014/02/24/a-new-look-for-net-reference-source.aspx

    So that you could know which method inside costs most time.

    From your provided code, it is not very clear if you use multi threads in your service, if not and your event handler is short, you could have a try with it, and there is a discussion about this topic:

    https://social.msdn.microsoft.com/Forums/en-US/13f30e33-7f61-498e-a91a-ef982a63453c/event-handling-in-multithreaded-apps?forum=netfxbcl

    Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, January 27, 2015 6:40 AM
    Moderator
  • We are investigating the possiblity that single-threading is the problem. Our detailed logs show that, during the periods when the number of concurrent user sessions is high, our handler of process start events from WMI often takes about 1 second to complete and sometimes takes almost 2 seconds. Our logs also show that the handler is only handling one event-at-a-time. We have changed our event handler to be multithreaded but have not yet tested it with a high number of concurrent sessions. I intend to post again if the problem is solved by this change.
    Monday, February 2, 2015 7:08 PM
  • Hello RossAtWFMC,

    Any update? I have marked my reply as answer since I think it is helpful, if you think it provides no help, please unmark it.

    Thank you for your understanding and support.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, February 9, 2015 9:02 AM
    Moderator
  • No update yet. Your answer is a good answer and I'm okay with it being marked as such. The testing team has not been able to test the new module. When they do get around to testing it, and if test results indicate success, I will provide a specific answer and try to tie it in to your answer.
    Monday, February 9, 2015 1:47 PM
  • The problem was due to single-threading of the event handler. The long delay was an accumulation of delays as the events were queued for the handler. The solution was to change the event handler to spawn a new thread to finish handling the event.
    • Marked as answer by RossAtWFMC Friday, April 3, 2015 7:09 PM
    Friday, April 3, 2015 7:09 PM