locked
LimitedConcurrencyLevelTaskScheduler is not working well. RRS feed

  • Question

  • MSDN provides an example

    how to use LimitedConcurrencyLevelTaskScheduler.

    I have an application and I have been struggling with it for many days. Here is the code:

     LimitedConcurrencyLevelTaskScheduler myScheduler = new LimitedConcurrencyLevelTaskScheduler(portcount);
     List<Task> tasks = new List<Task>();
    
     for (int j = 0; j < dialCollection.Length; j++)
     {
          for (int i = 0; i < threadCount; i++)
          {
               var myFactory = new TaskFactory(myScheduler);
               var captureI = i + 1 + j * threadCount;
               var captureJ = dialCollection[j];
               Task t = myFactory.StartNew(() =>
               {
                  if (cToken.IsCancellationRequested)
                          return;
                   MakeTest(captureJ, captureI);// long running process.
               }, cToken);
                   tasks.Add(t);
           }
      }
      Task.WaitAll(tasks.ToArray());

    You see that there are two for loops. The inner for loop iterates with i; the outer for loop iterates with j. My problem is all tasks within the inner loop are running synchronous but failed in the outer loop scope. The application is a telephony application with the maximum port limit. I want to make call simultaneously.

    Let me give you an example. Say dialCollection.Length = 2 and threadCount = 1. So there are 2 tasks totally. However I only saw one task running, after it finishes then the second task came up.

    The reason I still can not figure it out. But when I look at the code on MSDN example.

    We notice that it used linkedlist.

    // The list of tasks to be executed  
    private readonly LinkedList<Task> _tasks = new LinkedList<Task>(); // protected by lock(_tasks) 
    
    I am not sure if it is the reason. Any advice? Can we use other collections? Please don't use Console.WriteLine or Thread.Sleep to simulate the long running process as it can not find the problem.



    Thursday, September 18, 2014 1:33 PM

Answers

  • The sample code you posted from them doesn't really seem to relate to what you're trying to do.  They create multiple threads but that doesn't really change the fact that those threads might be serialized.  This is no different than what your task code is doing.

    Based upon your disassembly of what they're doing the blocking could come from any # of places in their code but it is all based upon their internal implementation.  Most likely they need to block at certain points so this isn't surprising.  What you should probably do is contact the vendor and explain to them that you're seeing this particular call block while another thread is running the same code.  Most likely they can tell you what needs to happen on your side to unblock that call.  It could be something as simple releasing some resource or perhaps setting some flag.  I notice in the code that the UseIpcElements option causes a different set of code to execute.  It might be that you just need to set this option (however that is done) so the alternate (non-blocking) code would execute.  The vendor should be able to help you figure this out.

    • Marked as answer by ardmore Friday, September 19, 2014 6:02 PM
    Friday, September 19, 2014 6:00 PM

All replies

  • What is the maxDegreeOfParallelism which you use in your example? Is it higher than 2?

    Maybe your long running process is using some shared resources which are blocking each other.

    Thursday, September 18, 2014 2:01 PM
  • A big number.

    LimitedConcurrencyLevelTaskScheduler lcts = new LimitedConcurrencyLevelTaskScheduler(120);

    Thursday, September 18, 2014 2:06 PM
  • I'm not seeing this behavior.  The only change I made to your code was to move the declaration of the factory outside the loops as there is no reason to recreate it each time. Beyond that all the tasks were scheduled to run and the scheduler would run 2 tasks until it finished.  At any one time (except the last case) 2 tasks were being run in parallel.

    Michael Taylor
    http://blogs.msmvps.com/p3net

    Thursday, September 18, 2014 2:07 PM
  • Yes, I moved it outside the lops but the behavior persisted.

    var myFactory = new TaskFactory(myScheduler);
     for (int j = 0; j < dialCollection.Length; j++)
     {
          for (int i = 0; i < threadCount; i++)
          {
    I doubt that should I use BlockingCollection rather than LinkedList?

    Thursday, September 18, 2014 2:34 PM
  • The underlying collection used by the scheduler isn't relevant.  A linked list is fine.  Again, I'm not seeing any issues.  I'm wondering if the blocking behavior is inside your MakeTest method rather than any of the surrounding code.  Can you temporarily replace the functionality of MakeTest with something simple and confirm that multiple tasks are being created?  If they are then the issue is with your MakeTest method serializing itself.
    Thursday, September 18, 2014 2:55 PM
  • If I replaced the method of MakeTest with simple statement such as Thread.Sleep(2000) or Console.WriteLine(i,j or something), the multiple tasks were being created.

    If I insisted the original MakeTest method, the issue is inside the inner loop the multiple tasks were being created and running fine but blocking the outside loop tasks.

    The reason why I mentioned BlockingCollection is I posted a relevant question on stackoverflow, a guy named "Servy" said that I shouldn't be using either a LinkedList or a List.

    Thursday, September 18, 2014 3:11 PM
  • The sample makes significant use of locked code, but your posted code does not.

    Rudedog  =8^D


    Mark the best replies as answers. "Fooling computers since 1971."

    http://thesharpercoder.com/

    Thursday, September 18, 2014 3:29 PM
  • @Rudedog2, I used to add lock but it didn't work as well.

    for (int j = 0; j < dialCollection.Length; j++) { for (int i = 0; i < threadCount; i++) { lock (thisLock) {

    var captureI = i + 1 + j * threadCount;
                    var captureJ = dialCollection[j];
                    Task t = myFactory.StartNew(() =>


    Thursday, September 18, 2014 3:42 PM
  • So the locking issue is with your MakeTest method and not the scheduler.  I'm wondering if this is just confusion as to the expected behavior.  Given the above code the for loop will not block at all.  It'll schedule all the tasks and then block at the WaitAll call.  Can you confirm that you're seeing this with your MakeTest call?  If so then the scheduling is working correctly.  Now what you can expect to see is that at any particular moment in time 2 of the tasks should be running (ignoring the startup time of a task when it is scheduled).  When all the tasks complete your code will continue on.

    You didn't post the code for the scheduler but if you're using the original version from the MSDN article then it should work.  If you have switched the scheduler to use a BlockingCollection then you're going to see the problem you mentioned?  Did you switch it away from LinkedList?  If so then switch it back and see if your problem goes away.

    Thursday, September 18, 2014 3:48 PM
  • Yes, I am using the original version scheduler code from MSDN. Using Blockingcollection just got the same result. Inside my MakeTest I use a third party product, after I commented some code, sometimes it only has one task running. Perhaps the product could not support .net task?

    By the way, I have some wpf code in the method but I commented it out, the same result.

    Is any way to replace LimitedConcurrencyLevelTaskScheduler with an old pattern? I mean that I want to try the legacy .net framework skills such as thread only, I want to remove all Task.StartNew or Task.Run etc; Just use the lower level thread coding. Feel sadly. But please give me any links for scheduling threads.


    • Edited by ardmore Thursday, September 18, 2014 6:15 PM add
    Thursday, September 18, 2014 6:07 PM
  • You could but that isn't your problem.  There is no special logic needed for tasks.  All a task really is a thread pool thread that is managed by the framework rather than you writing the exact same code.  It doesn't really change how your code is going to behave. 

    If the code you're running is not multi-thread capable then no amount of threading/task will make it multi-threaded.  That is sort of what it sounds like is happening in your case.  You should probably follow up with the third party vendor to determine if they do allow you to use their component across multiple threads at the same time.  It is possible they do allow multi-threading but they serialize all requests (like a message pump).  In that case you cannot do anything to make them run in parallel.  Key indicators of this is sync objects like a mutex or semaphore or the lock keyword.

    Thursday, September 18, 2014 6:30 PM
  • I contacted the vendor, the product can make up to 50 calls or more at the same time. It depends on the license.

    I got their sample code, they just use threads and QUEUE CLASS.

    Each thread is for one call. After each call, the finally block it has

     public void DecrementCallCounters()
      {
            Interlocked.Decrement(ref m_ActiveCallCount);
      }
    I don't know whether the scheduler code from MSDN clean the list when the task down.

    Thursday, September 18, 2014 9:14 PM
  • The scheduler will properly handle the tasks as you observed.  I would recommend that you step through the MakeTest code and see if it is blocking inside the call.  This might be a little bit hard to do given the async nature so you might start by putting a debug statement (Debug.WriteLine) before the call to the vendor's API (inside MakeTest).  Put another debug statement after it.  Then run your app and see if you get 2 messages quickly.  If so then the scheduler is making the appropriate calls.

    Another issue could be that the thread is being scheduled on the current thread rather than a background thread.  Sometimes a task can be completed faster sync than async so the TPL has the ability to try to run a task on the current thread.  Try modifying your scheduler's TryExecuteTaskInline to return false.  This will prevent it from treating it as sync. 

    Thursday, September 18, 2014 9:45 PM
  • If keep the original scheduler as MSDN and add two debug statements around the call code.(print out timestamps) Let two threads running, the debug output were:

    Begin 20140919091714
    Begin 20140919091714
    //...
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    End 20140919091714
    End 20140919091714

    Secondly, I modified the code as

    protected sealed override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
            {
                 return false;
            }

    The debug output were:

    Begin 20140919092650
    Begin 20140919092650
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    // blah blah
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    End 20140919092652
    14/09/19 - 09:26:52.064 0014 (no thread name)     dial result for the dialing string xxxxxxxxxxxx at line 2
    WeakClientObject::TelephonyLink_OnExecuteClientCommand called
    End 20140919092652
    And in the second case, still only one line ring...



    • Edited by ardmore Friday, September 19, 2014 1:54 PM add
    Friday, September 19, 2014 1:35 PM
  • So the debug statements confirm that the scheduler is working correctly I believe.  2 tasks are scheduled and the calls are made together as you'd expect.  In the first case they both complete around the same time.  If there were more tasks scheduled then they would have started automatically.

    In the second scenario I don't understand why you have 2 begin outputs but 4 actual calls.  In both cases how many tasks should be scheduled altogether? 

    I should have mentioned this before (my apologies) but VS has a window for tasks.  Set a breakpoint inside your MakeTest method.  Then run the debugger.  When the breakpoint gets hit use Debug\Window\Tasks to pull up the list of running tasks.  In theory you should see most (if not all) the tasks scheduled (from the for loops) with 2 set to active.  This is the expected behavior. Keep pressing F5 and every time the debugger breaks in you should see 2 tasks active and the rest scheduled. Eventually you'll run out of tasks scheduled. If you are seeing this behavior then the scheduling is working fine and the problem is with the code you're calling. 

    Friday, September 19, 2014 3:20 PM
  • Thanks for your nice patience. I really appreciate it.

    I had two tasks ,the test result was:

    tasks

    Friday, September 19, 2014 3:47 PM
  • Aha, notice the blocked status?  That means the task was blocked waiting on something.  This commonly occurs when you're using the lock statement or waiting on a wait handle (like a semaphore).  My gut instinct is that either in your MakeTest call or in the lower level call(s) you're making someone is blocking.  Hence you are seeing the serialization process.  Since you have a BP on Dial it cannot be that call.  Looking further back in your method I would take a guess that it is the GetChannel() method call that is potentially blocking.    It could also be the cr.Add() call but I don't see what cr is so I don't know.  Try setting breakpoints on these lines and step through the code again to try to identify which call is causing the second task to block.  That is where your problem resides.
    Friday, September 19, 2014 3:58 PM
  • It is GetChannel() method. cr is an irreverent collection. I set a breakpoint at GetChannel line, then hit F5 twice got the below images.

    Then

    I decoding the dll and found it called a method such as:

    internal byte[] ExecuteCommandHelperAsync(byte[] buffer, string invoker)
    {
    	ElementsException ex = null;
    	byte[] array = null;
    	Session session = this.m_Session;
    	try
    	{
    		if (ClientObject.UseIpcElements)
    		{
    			base.WaitForConnection();
    			MRaw mRaw = new MRaw();
    			mRaw.RawPacket = buffer;
    			MRaw mRaw2 = (MRaw)this.m_Session.SendMessage(mRaw);
    			byte[] result = mRaw2.RawPacket;
    			return result;
    		}
    		AsyncWait asyncWait = new AsyncWait();
    		try
    		{
    			base.WaitForConnection();
    			VPacket vPacket = new VPacket(this.Guid, buffer.Length + 1024, "TelephonyServer.Async.Invoke", 1);
    			vPacket.BinaryWriter.Write(asyncWait.UniqueId);
    			vPacket.Serialize(buffer);
    			vPacket.DoneWriting();
    			array = this.TelephonyServerLink.ExecuteServerCommand(vPacket.Buffer);
    			VPacket.FromByteArray(array, "TelephonyClient.Async.Reply");
    			asyncWait.WaitForReply();
    			if (asyncWait.Exception != null)
    			{
    				throw asyncWait.Exception;
    			}
    			if (asyncWait.Reply == null)
    			{
    				throw new ElementsException("Async Reply was Null!");
    			}
    			byte[] result = asyncWait.Reply;
    			return result;
    		}
    		finally
    		{
    			asyncWait.Dispose();
    		}

    I contacted the vendor, their sample code likes:

    Thread threadPlaceCalls = new Thread(PlaceCalls);
     public void PlaceCalls()
     {
        while (true)
         {
                 // When true this means that we have ports available to dial out on.
                 if (m_ActiveCallCount < MaximumPorts)
    	     {
    		  // blah blah
    		    Thread oThread = new Thread(MakeTest());// get channel in this method
    		 // Increment the call count
                        IncrementCallCounters();
    		    oThread.Start();
    	      }
    	       else
    		{
    		 // This waits less time to see if a channel resource is available to use
                     bool isSignalled = AliveEvent.WaitOne(100);// ManualResetEvent
    
                        // If the reset event is signalled then we go ahead and break.
                        if (isSignalled)
                        {
                            break;
                        }
    	    }
    You see, they use thread, is there any way to combine their logic with mine?

    Friday, September 19, 2014 5:32 PM
  • The sample code you posted from them doesn't really seem to relate to what you're trying to do.  They create multiple threads but that doesn't really change the fact that those threads might be serialized.  This is no different than what your task code is doing.

    Based upon your disassembly of what they're doing the blocking could come from any # of places in their code but it is all based upon their internal implementation.  Most likely they need to block at certain points so this isn't surprising.  What you should probably do is contact the vendor and explain to them that you're seeing this particular call block while another thread is running the same code.  Most likely they can tell you what needs to happen on your side to unblock that call.  It could be something as simple releasing some resource or perhaps setting some flag.  I notice in the code that the UseIpcElements option causes a different set of code to execute.  It might be that you just need to set this option (however that is done) so the alternate (non-blocking) code would execute.  The vendor should be able to help you figure this out.

    • Marked as answer by ardmore Friday, September 19, 2014 6:02 PM
    Friday, September 19, 2014 6:00 PM