none
System.ServiceModel.CommunicationException---> System.Net.WebException: The underlying connection was closed RRS feed

  • Question

  • Hi All,

     

    I am getting following error while communication with CRM Dynamics via BizTalk 2006 R2. This error is very very intermittent. Out of 100 records only 20-25 fail. On certain days no records fail and on some days all records fail.

    The message load is very minimal and constant throughout.

     

    Any pointers how to start troubleshooting:

     

    System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://hero.com/CRMIntegrationOnline/CRMService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

       at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

       --- End of inner exception stack trace ---

       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResul

     

    Regards,

    Vikkingss

    Monday, January 2, 2012 11:20 AM

Answers

  • Hi Lars.......

    I have almost drilled down the reasons why it is happening in my case. I am using request response send ports in almost all the communication with CRM sharepoint and SAP. In my case 95% messages get processed without fail. on 5 % problematic messages I have observed that they fail because they are not able to perform their operations in end systems within stipulated time and could not return response. This is highly possible when the traffic load is high. but spiking is for few seconds only(luckily), so message at peak load becomes the prey and remaining get processed.

    Solution is to fine tune the end systems CRM, Sharepoint or SAP services, nothing much we can do from BizTalk end other than keeping the performance parapeters tuned.

    I have implemented a work around to it by increasing the retry count and retry interval, so that those unlucky messages get processed again after retry interval and if they are lucky they will get processed this time if load has not spiked again.

    I need to look for a solution for long term.

    Also to mention there could be plenty of causes which might be resulting this issue..so be open for a silly cause to a big one :):)

    Also Thanks Tord for your directions...It was a great help.

     

    Regards,

    vikingssss

    • Marked as answer by vikingss Friday, January 6, 2012 9:34 AM
    Friday, January 6, 2012 9:30 AM

All replies

  • And at CRM Side I am getting below error:

     

    ERROR WCFServices - An exception has occurred in GetUserDescription called with parameters: DomainName\Username. Details: Exception message: Failed to project selection.

    Exception details: System.Exception: Failed to project selection. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: An item with the same key has already been added.

       at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)

       at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

       at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)

     

     

    Any pointers please ...

    Monday, January 2, 2012 11:31 AM
  • Hello Vikingss!

    The way I read this you have a timeout towards the url hero.com ... What adapter are you using, and are you able to resume the messages ok?

    Try to change the timeout for the send port.

    updated:

    I didnt see your second post. It looks like it might be some problem with the web service, are the fields unique in the database where the information is stored, what are the "receiver getting" as a unique ID? Have you tried to "change" this either from your application or in the WS.

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

     


    • Edited by Tord G. Nordahl Monday, January 2, 2012 11:51 AM didnt see the last error message
    Monday, January 2, 2012 11:49 AM
  • Hi Tord,

    I have set the time outs value to max possible at BizTalk side(CLient). I am able to resume the instances but they are again failing with "No Subscribers Found" error.

     

    Thanks

    Vikingss 

    Monday, January 2, 2012 12:07 PM
  • Okay, without knowing how your application works, or how it send the information I have my usual suspect (and of course outside of biztalk).

    As i mentioned, you have some issues when you try to send data to the webservice, the webservice throws an exception stating that it cannot add the "row" because a key is allready in use.

    To example this, lets say you have a biztalk solution sending over information regarding a name and birthdate, a random number.

    The ws receives this and tries to update a database, however the random numer in this database is unique, so as long as the random number is unique and does not apear in the database it will add all the three columns, however, if the random number already exsists in the database it will throw an exception stating something like "An item with the same key has already been added."

    So I would take a look at the WS and see what it does.

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Monday, January 2, 2012 12:23 PM
  • Thanks Tord for your response.

    Interestingly I don't see any insert statements in the code, so don't think violation of unique key is happening. Only look up operations are being done. Today I noticed a different log at the CRM side but causing the same error:

    An exception has occurred in GetUserRolesDescription called with parameters: DomainName\Username. Details: Exception message: Failed to project selection.
    Exception details: System.Exception: Failed to project selection. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: No value nodes found for Type ABC.CRM.Common.CrmSdk.new_accountteam

    Best Regards,

    vikingsss

    Tuesday, January 3, 2012 7:24 AM
  • Could there be some load issues towards the CRM system? I've seen this a couple of times (without remembering the error messages) we had to tune down the application in order to make it work. When the load was too great the web services kneel.

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Tuesday, January 3, 2012 7:29 AM
  • Interestingly I don't see any insert statements in the code, so don't think violation of unique key is happening. 

    The unique key violation happens when the code tries to insert a object in a list/collection but the key is already added. Check for existence before adding the object

    "System.ArgumentException: An item with the same key has already been added"

    Check the method "GetUserDescription"

    Could be due to some cache updating?

    rgds /Peter

    Tuesday, January 3, 2012 10:33 AM
  • Tord, Actually you are correct. I think It is happening because of high message load I guess, since in the CRM code we are populating value in a list and for same user let us say the last thread has not completed and a new instance is trying to put the same value in the same list, this error is bound to come.

     

    I will fine tune from BizTalk end to handle the high message load but is there any way to fine tune from WCF end?

     

    One of my friends has suggested to use locks in the code but that will only solve "System.ArgumentException: An item with the same key has already been added" issue but not the "System.ArgumentException: No value nodes found for Type ABC.CRM.Common.CrmSdk.new_accountteam"

     

     

     

    Thanks & Regards,

    vikingsss


    • Edited by vikingss Tuesday, January 3, 2012 1:04 PM
    Tuesday, January 3, 2012 1:01 PM
  • I've had a smiliar issue, the solution was to scale down the application, however this was just a temporary solution, we had to remodule the entire application in order to get the desired functonality. Often the issue lies outside BizTalk (like this scenario). 

    So my tips:

    Redesign the WS at the CRM side
    Redesign the entire solution to work without the WS

    tempoary:

    If it the threads you can simply set a threshold for usable threads for the designated host, so that it will stop creating new threads when a certain threshold is reached. Give me a call if you want more information regarding this (you can find my number on my blog) Its a lot easier to explain tuning over the phone then on a forum!

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Tuesday, January 3, 2012 1:31 PM
  • Hi Tord,

    I reduced the Maximum number of messaging engine threads per CPU to 2 for the host but it does not look to help.

    I will now try to modify the throttling settings if that helps.

     

    Regards,

    Vikingss

    Wednesday, January 4, 2012 12:10 PM
  • And to add on I am getting the same error while accessing sharepoint sites too from BizTalk but it is happening highly intermittent once or twice a week, remaining smooth. I am using WCF-WSHTTP adapter for CRM and Sharepoint site as well.

    System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://a.b.c.d:81/_layouts/super/MyIntegrationService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
       --- End of inner exception stack trace ---
       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
       at

    So I am really confused, If Problem exist at BizTalk or CRM or Sharepoint side :(:(

    Thursday, January 5, 2012 6:15 AM
  • Hmm, you seem to have some issues. However these can be located in BizTalk or at sharepoint and the CRM site. Have you tried to run MessageBoxViewer (it might find something wrong somewhere)

    Can you also set up PAL to run for a week (or two) to see if what issues you might have.

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Thursday, January 5, 2012 11:16 AM
  • Hi Vikingss,

    I am troubleshooting the exact same set of problems at a client with BizTalk calling a WCF service hosted in SharePoint. First the timeout's, and then the System.ServiceModel.CommunicationException.

    Have tried throttling up or down, w/o any impact on the problem.

    Will update this thread if I get any wiser on what is happening.

    /Lars

    Thursday, January 5, 2012 8:33 PM
  • Hello Lars, did you try to run PAL to see if you have some bottlenecks somwhere?

    Best regards

    Tord Glad Nordahl
    Bouvet ASA, Norway
    http://www.BizTalkAdmin.com |@tordeman

    Please indicate ”Mark as Answer” if this post has answered the question.

    Thursday, January 5, 2012 8:35 PM
  • Yeah ... nothing there. I am playing with host throttling and the <ConnectionManagement> settings atm to see if it makes a diff.

    I really think though that this should be solved on the client side though ... As we don't have the luxury of design time deciding the setup of how the production env. will look in the future. Like - what happens if it runs on 20 cores instead of 10, or a couple servers are added.

    But ... still looking at the issue. Will update if I get any wiser ... :-)

    Thursday, January 5, 2012 8:47 PM
  • Hi Lars.......

    I have almost drilled down the reasons why it is happening in my case. I am using request response send ports in almost all the communication with CRM sharepoint and SAP. In my case 95% messages get processed without fail. on 5 % problematic messages I have observed that they fail because they are not able to perform their operations in end systems within stipulated time and could not return response. This is highly possible when the traffic load is high. but spiking is for few seconds only(luckily), so message at peak load becomes the prey and remaining get processed.

    Solution is to fine tune the end systems CRM, Sharepoint or SAP services, nothing much we can do from BizTalk end other than keeping the performance parapeters tuned.

    I have implemented a work around to it by increasing the retry count and retry interval, so that those unlucky messages get processed again after retry interval and if they are lucky they will get processed this time if load has not spiked again.

    I need to look for a solution for long term.

    Also to mention there could be plenty of causes which might be resulting this issue..so be open for a silly cause to a big one :):)

    Also Thanks Tord for your directions...It was a great help.

     

    Regards,

    vikingssss

    • Marked as answer by vikingss Friday, January 6, 2012 9:34 AM
    Friday, January 6, 2012 9:30 AM
  • Okay ... just a quick update regarding what happened on my "version" of this issue.

    I must have been having a bad day, because I had put terminate shapes inside most of my ExceptionHandler shapes ... and some of them are in a parallel shape.  so what happened was that the orchestration was shut down while one branch was waiting for a response back on its WCF service call, because the faulty branch received a soap fault back.  After a while - none of the calls to the branch that didnt fail came through, and all calls to that service gave me the CommunicationExceptions.  

    Must be something "hanging" on the service side (A SharePoint hosted WCF service) or on the BizTalk side.

    In any case ... once I removed the "Terminate" shapes, and allowed the other branch to receive a reply back, I haven't had this problem.  I went back and reproduced it, so pretty sure this IS the issue for me. But what is hanging and giving me these errors I still don't know....

     

     

    Thursday, January 12, 2012 10:47 AM