none
WCF with streaming mode timeout after two calls when client and service are not on same machine RRS feed

  • Question

  • Hi exports,

    I struggled this issue for almost a week, very frustrating. 

    I have weird timeout issue for my WCF application. On the service side, it is downloading file using stream mode. The method has a message contract type object as its parameter and output parameter. The object contains a Stream type member defined as MessageBodyMember. When my client app call the WCF method, the first and second time calls are ok, even with very big file stream. But then the client app call the third time same method same parameter(same small file) then it is timeout with below exception. The thing which i don't understand is that same code if I run the client app and service app on same machine(client app is running on same machine where the service app hosted on its IIS), then there is no issue. The issue happens when the client and service app are cross the network. Which i don't understand at all.

    Any ides?

    Many thanks!

    Lan

    -------------------------------- exception ------------------------------

    The request channel timed out attempting to send after 00:00:30. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

    -------------------------------- exception ------------------------------

    My binding setting on service side

    <binding name="BasicHttpBinding_IDocWS"  
    maxBufferPoolSize="2048000" maxBufferSize="2048000" 
    maxReceivedMessageSize="2147483647" messageEncoding="Mtom" 
    transferMode="Streamed" sendTimeout="00:10:00" 
    receiveTimeout="00:10:00" openTimeout="00:10:00" 
    closeTimeout="00:10:00" >
    
    <readerQuotas maxDepth="2000000" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
    
    <security mode="None">
       <transport clientCredentialType="None"></transport>
    </security> 
    
    </binding>


    ZL @ Hypertec

    Thursday, August 17, 2017 3:54 PM

Answers

  • Hi Edward,

    Finally I figured it out. :) I have to close the Stream object at client side. It looks like WCF only allow two Stream object opened at one client app. But the timeout exception really doesn't mean anything and confuse developer. Also I don't know why I can't keep more than two Stream object open at one client app, is there a way to set ? Another thing I don't understand is that if the client app and service app are at same machine then there is no issue at all.

    Not sure how Stream object can be closed or disposed at service side, sometimes the Stream object can be re-read at the client side multiple times. I don't know how the two Stream object at client side and service side communicate with each other. Let's say I close the client side Stream, is the service side Stream will be closed automatically?

    Many thanks !

    Lan


    ZL @ Hypertec

    • Marked as answer by Zhang Lan Wednesday, August 23, 2017 2:37 PM
    Wednesday, August 23, 2017 2:37 PM
  • Hi ZL,

    Since simple operation works correctly, it seems it is not related with the windows configuration or service configuration.

    Per to your Stream method code, I failed to find anything wrong which will cause this issue.

    Could you share us a simple solution which contains service and client? I will try to reproduce your issue at my side for further trouble shooting.

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Zhang Lan Wednesday, August 23, 2017 2:37 PM
    Wednesday, August 23, 2017 1:38 AM

All replies

  • BTW, i have disabled the Windows Firwall and antivirus protection, so it is not firewall or antivirus issue.

    ZL @ Hypertec

    Thursday, August 17, 2017 4:11 PM
  • Also, if the parameter object of the method doesn't contain Stream member, then there is no issue. It looks like the server is blocking the client because of the stream object is not released? But I already tried to read all byte data from the stream at client app side and even i tried close the web service client, but still same issue. But I don't know how to release the stream object on server side.

    ZL @ Hypertec

    Thursday, August 17, 2017 4:16 PM
  • Hi ZL,

    >>The request channel timed out attempting to send after 00:00:30

    Did you set any timeout on client app.config?

    According your binding setting name, it seems you are using BasicHttpBinding, right? As my understanding, there is no default timeout value is 30 seconds for BasicHttpBinding. ChannelInitializationTimeout default value is 30 seconds, but it applies to TCP or named pipes.

    Could you try to set the same binding on client side like your service side?

    For checking the depth exception, I suggest you enable Configuring Tracing on both client and service side.

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Friday, August 18, 2017 1:47 AM
  • Hi Edward,

    Thanks for your response! In my binding setting, there is a "sendTimeout="00:10:00" and I have same setting on client side. I believe the timeout setting function well because if I set it to "sendTimeout="00:00:30" at client side then it is timeout at 30 seconds. Now it is timeout at 10 minutes which is exactly the same setting in my binding setting in both service and client side. Also I think usually the web method shouldn't take that long time, in my first and 2nd attempt with same parameter, it just take even less than one second. I will try the tracing on server side. But I don't understand that why it is ok if the client and service app are located at same machine.

    Lan


    ZL @ Hypertec

    Friday, August 18, 2017 6:07 PM
  • Hi Edward,

    I enabled the trace in service side, but I didn't find anything useful or something like exception message.

    Any idea?

    Lan


    ZL @ Hypertec

    Friday, August 18, 2017 9:01 PM
  • Hi Edward,

    I enabled the trace on client app side, see below exception I received. But in general it is almost same like what I see from my client app. According to the exception message, the client is timeout at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream().

    Any idea?

    Thanks!

    Lan

    Unhandled exception.  Exception details: System.TimeoutException: The request channel timed out attempting to send after 00:00:59. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---&gt; System.TimeoutException: The HTTP request to 'http://aaa.com/DocWS.svc' has exceeded the allotted timeout of 00:00:59. The time allotted to this operation may have been a portion of a longer timeout. ---&gt; System.Net.WebException: The operation has timed out
       at System.Net.HttpWebRequest.GetRequestStream(TransportContext&amp; context)
       at System.Net.HttpWebRequest.GetRequestStream()
       at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()



    ZL @ Hypertec


    • Edited by Zhang Lan Friday, August 18, 2017 9:28 PM
    Friday, August 18, 2017 9:27 PM
  • Hi Edward,

    I add some logs in web method of service app side, it shows that the service app didn't receive the request data at 3rd call. It looks like the IIS server is not willing to accept the 3rd request. As I described in my previous email, the 1st and 2nd call are working fine. But other methods that don't return a Stream object don't have this issue at all. So I guess it is something that service app only support 2 stream operation?

    Lan


    ZL @ Hypertec

    Friday, August 18, 2017 9:58 PM
  • I tried close the wcf client object after every time I call the method and regenerate the wcf client object before call the web method, but still same result.

    Any idea?

    Thanks!

    Lan


    ZL @ Hypertec

    Friday, August 18, 2017 10:58 PM
  • I paste the behaviors setting in my service side web.config file just in case.

    Lan

    <behaviors>
          <serviceBehaviors>
            <behavior>
              <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="11000" maxConcurrentSessions="10000" />
              <dataContractSerializer ignoreExtensionDataObject="true" maxItemsInObjectGraph="2147483647" />
              <!-- To avoid disclosing metadata information, set the values below to false before deployment -->
              <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true"/>
              <!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
              <serviceDebug includeExceptionDetailInFaults="true"/>
              <serviceTimeouts transactionTimeout="00:10:00" />
            </behavior>
          </serviceBehaviors>
        </behaviors>


    ZL @ Hypertec

    Friday, August 18, 2017 11:01 PM
  • Hi ZL,

    Could you share us the complete configuration for service and client side? I am confused about this error message which indicate sendTimeout is 00:00:59.

    If you create a new service, will you reproduce this error?

    If you only change the new service configuration to like your current service, will you reproduce this error?

    If it could be reproduced, could you share us the detailed steps? We will try to make a test with the exact steps.

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, August 21, 2017 6:05 AM
  • Hi Edward,

    Thanks again for your response!

    Please see attached complete web.config and app.config of service app and client app. I removed some sensitive data such like DB connection string.

    For error message of sendTimeout 00:00:59, that is because I set it to 59 seconds in both client and server side on purpose. I want to see if the setting affect the behavior. My original setting was 00:10:00 which is 10 minutes. In my testing cases, the data are very small, 59 seconds should be far more enough. As I mentioned, I can get the data within less than 1 sec in the 1st call and 2nd call, but then it is timeout at 59 seconds in the 3rd call.

    For creating a new service, if I create a new service client each time I call the method at same client app, it is same error. But weird thing is that if I launch the second client app instance(2nd client app process) then I can continue the 1st and 2nd call and of course it is timeout at 3rd call. If I launch another client app instance again then it is like the first and second client app instance. See below my code snip. It looks like each client app is stuck at 3rd call.

    foreach (string di_id in att_list)
    {
        DocsWSClient ws1 = new DocsWSClient();
        Stream doc_sr = Stream.Null;
        doc_sr = ws1.DownloadAttachmentById(di_id);
        byte[] bytes = FileUtil.ReadAllByte(doc_sr);
        MessageBox.Show("From " + ws1.Endpoint.Address.Uri.AbsoluteUri + ":  file length: " + bytes.Length.ToString());
    
        ws1.Close();
    }

    Server side web.config

    <?xml version="1.0"?>
    <configuration>
    
      <configSections>
        <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" >
        </sectionGroup>
      </configSections>
      <appSettings>
        <add key="aspnet:UseTaskFriendlySynchronizationContext" value="true" />
        <add key="AppGuid" value="C9E7514A-0E12-4799-A9DF-2B7195025BBE"/>
        <add key="keyTester" value="tester123"/>
      </appSettings>
    
      <connectionStrings>
      </connectionStrings>
    
      <system.web>
        <compilation debug="true" targetFramework="4.5" />
        <httpRuntime targetFramework="4.5" maxRequestLength="8192000" executionTimeout="600" />
      </system.web>
      <system.serviceModel>
        <diagnostics performanceCounters="All" />
        <bindings>
          <basicHttpBinding>
            
            <binding name="BasicHttpBinding_IDocsWS" maxBufferPoolSize="2048000" maxBufferSize="2048000" maxReceivedMessageSize="2147483647" messageEncoding="Mtom" transferMode="Streamed" sendTimeout="00:00:59" receiveTimeout="00:00:59" openTimeout="00:00:59" closeTimeout="00:00:59" >
              <readerQuotas maxDepth="2000000" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
              <security mode="None">
                <transport clientCredentialType="None"></transport>
              </security> 
            </binding>
          </basicHttpBinding>
        </bindings>
        
        <services>
          <service name="HEBIS.PilotDocksWS" >
            <endpoint address="" binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IDocsWS" contract="Doc.IDocsWS" name="BasicHttpBinding_IDocsWS">
            </endpoint>
          </service>
        </services>
    
        <client>
        </client>
    
        <behaviors>
          <serviceBehaviors>
            <behavior>
              <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="11000" maxConcurrentSessions="10000" />
              <dataContractSerializer ignoreExtensionDataObject="true" maxItemsInObjectGraph="2147483647" />
              <!-- To avoid disclosing metadata information, set the values below to false before deployment -->
              <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true"/>
              <!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
              <serviceDebug includeExceptionDetailInFaults="true"/>
              <serviceTimeouts transactionTimeout="00:10:00" />
            </behavior>
          </serviceBehaviors>
        </behaviors>
        <protocolMapping>
          <add binding="basicHttpsBinding" scheme="https" />
        </protocolMapping>
        <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
      </system.serviceModel>
      <system.webServer>
        <modules runAllManagedModulesForAllRequests="true"/>
        <!--
            To browse web app root directory during debugging, set the value below to true.
            Set to false before deployment to avoid disclosing web app folder information.
          -->
        <directoryBrowse enabled="true"/>
        
        <security>
          <requestFiltering>
            <requestLimits maxAllowedContentLength="2147483647" />
          </requestFiltering>
        </security>
    
      </system.webServer>
    
      <system.diagnostics>
        <sources>
          <source name="System.ServiceModel"
                  switchValue="Information, ActivityTracing"
                  propagateActivity="true">
            <listeners>
              <add name="traceListener"
                  type="System.Diagnostics.XmlWriterTraceListener"
                  initializeData= "c:\log\Docs.log" />
            </listeners>
          </source>
        </sources>
      </system.diagnostics>
      
      <applicationSettings>
      </applicationSettings>
    
    </configuration>
    

    Client app app.config

    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
        <startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/></startup>
    
        <system.serviceModel>
            <bindings>
                <basicHttpBinding>
                    <binding name="BasicHttpBinding_IDocsWS" closeTimeout="00:00:59"
                        openTimeout="00:00:59" receiveTimeout="00:00:59" sendTimeout="00:00:59"
                        maxBufferPoolSize="2048000" maxBufferSize="2048000" maxReceivedMessageSize="2147483647"
                        transferMode="Streamed" messageEncoding="Mtom">
                        <readerQuotas maxDepth="2000000" maxStringContentLength="2147483647"
                            maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
                        <security mode="None">
                            <transport clientCredentialType="None" />
                        </security>
                    </binding>
                </basicHttpBinding>
            </bindings>
            <client>
                <endpoint address="http://docserver/DocsWS.svc"
                    binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IDocsWS"
                    contract="Doc.IDocsWS" name="BasicHttpBinding_IDocsWS" />
            </client>
        </system.serviceModel>
    
        <system.diagnostics>
        <sources>
          <source name="System.ServiceModel"
                  switchValue="Information, ActivityTracing"
                  propagateActivity="true">
            <listeners>
              <add name="traceListener"
                  type="System.Diagnostics.XmlWriterTraceListener"
                  initializeData= "c:\log\Docs.log" />
            </listeners>
          </source>
        </sources>
      </system.diagnostics>
      
    </configuration>

    Operation contract

    [OperationContract]
    Stream DownloadAttachmentById(string id);

    Lan


    ZL @ Hypertec

    Monday, August 21, 2017 3:53 PM
  • Just in case, I paste the code for the web method.

    public Stream DownloadAttachmentById(string file_id)
    {
        return new FileStream(@"C:\Data\" + file_id, FileMode.Open);
    }

    Also , I noticed that when the client app is timeout, if I wait long enough let's say 1 hour and click continue button to let the client app continue to run then I can make web method call again, but still will be timeout at 3rd call. It looks like service side blocks the client request at 3rd call.


    ZL @ Hypertec

    Monday, August 21, 2017 10:46 PM
  • Hi ZL,

    To check whether it is related with your service, I suggest you create a new service and client to make a test instead of only creating a new service client. Note, for new service, make a test with the built-in operation instead of copying the operation from previous service.

    To check whether it is related with your operation, I suggest you add a new simple operation with returning “Hello World” string, will this issue exist?

    Do you host your service in IIS? If so, I suggest you try Tracing in IIS to check the request process.

    # Troubleshooting Failed Requests Using Tracing in IIS 7

    https://docs.microsoft.com/en-us/iis/troubleshoot/using-failed-request-tracing/troubleshooting-failed-requests-using-tracing-in-iis

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, August 22, 2017 2:18 AM
  • Hi Edward,

    Thanks for your response! I have tried a service method without returning Stream then there is no issue at all. The issue only occurs when it returns a Stream and the client is not running with service on same machine. I tried enable the service log, but I don't see anything useful. According the timeout exception message, it is timeout in sending a request to server which I understand is that the server is not willing to receive the request with this client app. As I pointed out before, if I launch another client app then it can call the method again but limited at two method call.

    Best regards.

    Lan


    ZL @ Hypertec

    Tuesday, August 22, 2017 5:00 AM
  • Hi ZL,

    Since simple operation works correctly, it seems it is not related with the windows configuration or service configuration.

    Per to your Stream method code, I failed to find anything wrong which will cause this issue.

    Could you share us a simple solution which contains service and client? I will try to reproduce your issue at my side for further trouble shooting.

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Zhang Lan Wednesday, August 23, 2017 2:37 PM
    Wednesday, August 23, 2017 1:38 AM
  • Hi Edward,

    Finally I figured it out. :) I have to close the Stream object at client side. It looks like WCF only allow two Stream object opened at one client app. But the timeout exception really doesn't mean anything and confuse developer. Also I don't know why I can't keep more than two Stream object open at one client app, is there a way to set ? Another thing I don't understand is that if the client app and service app are at same machine then there is no issue at all.

    Not sure how Stream object can be closed or disposed at service side, sometimes the Stream object can be re-read at the client side multiple times. I don't know how the two Stream object at client side and service side communicate with each other. Let's say I close the client side Stream, is the service side Stream will be closed automatically?

    Many thanks !

    Lan


    ZL @ Hypertec

    • Marked as answer by Zhang Lan Wednesday, August 23, 2017 2:37 PM
    Wednesday, August 23, 2017 2:37 PM
  • Hi ZL,

    >>I don't know why I can't keep more than two Stream object open at one client app, is there a way to set ?

    If you are interested on this issue, I would suggest you post a new thread and share us a simple project which could reproduce your issue.

    It is complex to analyze this issue if we could not reproduce it.

    Best Regards,

    Edward


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, August 24, 2017 1:48 AM