none
Question- lock in multithreading RRS feed

  • Question

  • A windows service had been running in the production for so long and suddenly we observed a sharp hike in the memory going up to 2 GB.

    While analyzing the memory dump of the service it was found that among 90% of the threads created were waiting for a lock to get released.

    The lock is held by a thread which had been live for more than an hour and it's stack Trace shows

    System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)
    System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].set_Item(System.__Canon, System.__Canon)

    Server.Utilities.AppServer.Register()

    The Register method locks a particular dictionary object looks something like this,

    public void Register()
            {
                lock(this.notificationCallbackDictionary)
                {
                    Dictionary<string, CNotificationMessageHeader> notificationDictionary = null;
                    if (false == this.notificationCallbackDictionary.TryGetValue(this.msgHeader.LocationId, out notificationDictionary))
                    {
                        notificationDictionary = new Dictionary<string, CNotificationMessageHeader>(StringComparer.InvariantCultureIgnoreCase);
                        this.notificationCallbackDictionary.Add(this.msgHeader.LocationId, notificationDictionary);
                    }

                    notificationDictionary[this.msgHeader.TerminalId] = msgHeader;
                }

            }

    from thread stack trace 'notificationDictionary[this.msgHeader.TerminalId] = msgHeader'  should have been the code that got called.

    This problem got resolved after performing a restart on the service, but in two months this had been the second time the same issue is noticed.

    I wonder why the thread would hang on an Insert() method for so long because only if it gets completed, the lock on the dictionary object will get released paving way for other threads to obtain a lock.

    Any thoughts??


    deva_8907

    Tuesday, September 3, 2013 8:28 AM

Answers

  • The low level transport layer is using TCP.   The recieved data is serialized at the server and then deserialized at the client when received.  I'm no sure but I think the received data was incomplete (corrupted) so the deserialize function failed.  The data the serialize function at the server take a VS object and turns it into a byte[] which includes the object type.  The client then has to convert the byte[] back into the object type.  I suspect this is what is occuring during the dictionary Insert method.  If the data is is incomplete (the byte array doesn't have enough bytes) the code will wait until the number proper number of bytes gets recieved.

    I suspect something happened to the TCP connection.  I would start by performing in a cmd.exe window the command "netstat -a" which will give you the status of the connection.  Run this command at both the client and server while the connection is running and then again after it hangs.  Look for the connection between client and server that contains the port number you are using.  Also check the task manager for available memory at client and server.  Sometimes there are issues with TCP having a memory leak.  There are a lot of reasons a TCP connection can handg up and I don't want to start guessing without more information which one may be happening in your case.  the netstat command and memory will give more information to help determine the cause of the problem.


    jdweng

    • Marked as answer by deva_8907 Tuesday, September 3, 2013 10:48 AM
    Tuesday, September 3, 2013 9:55 AM

All replies

  • The low level transport layer is using TCP.   The recieved data is serialized at the server and then deserialized at the client when received.  I'm no sure but I think the received data was incomplete (corrupted) so the deserialize function failed.  The data the serialize function at the server take a VS object and turns it into a byte[] which includes the object type.  The client then has to convert the byte[] back into the object type.  I suspect this is what is occuring during the dictionary Insert method.  If the data is is incomplete (the byte array doesn't have enough bytes) the code will wait until the number proper number of bytes gets recieved.

    I suspect something happened to the TCP connection.  I would start by performing in a cmd.exe window the command "netstat -a" which will give you the status of the connection.  Run this command at both the client and server while the connection is running and then again after it hangs.  Look for the connection between client and server that contains the port number you are using.  Also check the task manager for available memory at client and server.  Sometimes there are issues with TCP having a memory leak.  There are a lot of reasons a TCP connection can handg up and I don't want to start guessing without more information which one may be happening in your case.  the netstat command and memory will give more information to help determine the cause of the problem.


    jdweng

    • Marked as answer by deva_8907 Tuesday, September 3, 2013 10:48 AM
    Tuesday, September 3, 2013 9:55 AM
  • ok correct me if am wrong, you're saying the insert method might have got failed because the object that it's trying to update or insert has got the corrupted information because of a possible TCP related network issue.

    deva_8907

    Tuesday, September 3, 2013 10:50 AM