locked
Fetching large amounts of text (not binary) data from a WCF 4 REST web service RRS feed

  • Question

  • We would like some advice concerning the proper method for returning large blocks of data from a database application via a WCF 4 REST web service.  (We're using .NET 4 SP1.)  The class below is an example data contract class for a type of object that we would like to enable our clients to fetch from the web service:

    [DataContract]

    public class MyObject

     

    {

       [DataMember]

     

       int

    DataItem1

       {

     

          get { return

    DataItem1; }

     

          set { DataItem1 = value

    ; }

       }

       [DataMember]

     

       string

    DataItem2

       {

     

          get { return

    DataItem2; }

     

          set { DataItem2 = value

    ; }

       }

    }

    The amount of data in any one object is relatively small, but many thousands of objects might be returned in one method call.  For example, a URI might support a GET method which would return all rows from a database table of MyObjects, and there might be a very large number of rows in that table.  The exposed method would have the following URI:

    [WebGet(UriTemplate = "myObjects"

    )]

    List<MyObject> getMyObjects();

     

    We have found that the default behavior of WCF is that the server will simply disconnect once it has serialized the data, regardless of whether or not the client has actually received the full data set.  Is our best option to change client and server code to treat each List<MyObject> parameter (whether input or output) as a stream?  We would prefer not to have to do that.  Ideally we would like to decorate the data contract with something (we don't know what) that would tell WCF that the objects to be serialized might be large and that the web service should use HTTP chunking.  That way our API and our client and server code would remain unchanged.  Am I just dreaming, or is there a way to do that?

     

    Thanks for your help.

     

    Friday, October 14, 2011 10:12 PM

Answers

  • We found the problem.  We had to add an Endpoint Behavior element to our web.config file with the following setting:

    <dataContractSerializer maxItemsInObjectGraph="2147483647"/>

    Once we did that, the problem went away.

     

    • Marked as answer by Yi-Lun Luo Thursday, October 20, 2011 1:19 AM
    Wednesday, October 19, 2011 2:08 PM

All replies

  • Hello, this should not happen. The connection should not be closed until all data has been sent, or a timeout occurs. Can you check if you're encountering a timeout issue? Increase WCF timeout settings, as well as IIS timeout settings (if you're hosting in IIS).
    Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.
    Windows Azure Technical Forum Support Team Blog
    Monday, October 17, 2011 2:25 AM
  • Thank you for the timely response.  I apologize for the delay, but it took us a while to reproduce the problem.  Below is the error detail.  We are still researching this on our side - I hope to have further info within a day or so.

    Event code: 3005

    Event message: An unhandled exception has occurred.

    Event time: 10/18/2011 1:59:31 PM

    Event time (UTC): 10/18/2011 8:59:31 PM

    Event ID: 6fd30e4055354397ab70fd9042982d93

    Event sequence: 15

    Event occurrence: 1

    Event detail code: 0

     

    Application information:

        Application domain: /LM/W3SVC/1/ROOT/KioskManagement.TestClient-3-129634451524602292

        Trust level: Full

        Application Virtual Path: /KioskManagement.TestClient

        Application Path: C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\

        Machine name: CSHQJMILLS3-D

     

    Process information:

        Process ID: 4720

        Process name: w3wp.exe

        Account name: IIS APPPOOL\DefaultAppPool

     

    Exception information:

        Exception type: HttpStageProcessingException

        Exception message:

       at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)

       at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)

       at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)

       at Microsoft.Http.HttpClient.Send(HttpMethod method, Uri uri, RequestHeaders headers, HttpContent content)

       at Microsoft.Http.HttpClient.Send(HttpMethod method, Uri uri)

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.GetGroupCollectionByType(String groupTypeName) in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 688

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.ListBoxDataBind() in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 514

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.OnGroupType_SelectedIndexChanged(Object sender, EventArgs e) in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 97

       at System.Web.UI.WebControls.ListControl.OnSelectedIndexChanged(EventArgs e)

       at System.Web.UI.Page.RaiseChangedEvents()

       at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

     

     

       at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)

       at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)

       at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)

     

    The underlying connection was closed: An unexpected error occurred on a receive.

       at System.Net.HttpWebRequest.GetResponse()

       at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)

     

    Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

       at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)

       at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)

       at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)

       at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)

       at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)

       at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)

       at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)

     

    An existing connection was forcibly closed by the remote host

       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

     

     

     

    Request information:

        Request URL: https://cshqjmills3-d.corp.coinstar.com:443/kioskmanagement.testclient/default.aspx

        Request path: /kioskmanagement.testclient/default.aspx

        User host address: fe80::c67:2a55:2e2c:7474C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\

        User: 

        Is authenticated: False

        Authentication Type: 

        Thread account name: IIS APPPOOL\DefaultAppPool

     

    Thread information:

        Thread ID: 53

        Thread account name: IIS APPPOOL\DefaultAppPool

        Is impersonating: False

        Stack trace:    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)

       at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)

       at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)

       at Microsoft.Http.HttpClient.Send(HttpMethod method, Uri uri, RequestHeaders headers, HttpContent content)

       at Microsoft.Http.HttpClient.Send(HttpMethod method, Uri uri)

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.GetGroupCollectionByType(String groupTypeName) in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 688

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.ListBoxDataBind() in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 514

       at Coinstar.Coin.Server.KioskManagement.TestClient._default.OnGroupType_SelectedIndexChanged(Object sender, EventArgs e) in C:\Code\TFS\Dev\TeamA\Server\Coinstar.Coin.Server.KioskManagement.TestClient\default.aspx.cs:line 97

       at System.Web.UI.WebControls.ListControl.OnSelectedIndexChanged(EventArgs e)

       at System.Web.UI.Page.RaiseChangedEvents()

       at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

     

     

    Tuesday, October 18, 2011 9:11 PM
  • Looks like you're calling the service from an ASP.NET client using the deprecated WCF REST Starter Kit. Do you hit the same issue if you test it from a console client using standard HttpWebRequest or the HttpClient shipped with WCF Web API? Also enable WCF tracing as described in http://msdn.microsoft.com/en-us/library/ms733025.aspx.
    Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.
    Windows Azure Technical Forum Support Team Blog
    Wednesday, October 19, 2011 5:02 AM
  • We found the problem.  We had to add an Endpoint Behavior element to our web.config file with the following setting:

    <dataContractSerializer maxItemsInObjectGraph="2147483647"/>

    Once we did that, the problem went away.

     

    • Marked as answer by Yi-Lun Luo Thursday, October 20, 2011 1:19 AM
    Wednesday, October 19, 2011 2:08 PM
  • Hi..

    In web.config,Endpoint Behavior element Add the following setting :<dataContractSerializer maxItemsInObjectGraph="2147483647"/> Hopefully it will work . Bez using this i am able to featch 1,00,000 .

    Still if it not working then

    1) Increse the buffersize,time out. in web.config file or page level dynamic binding.

    OR

    2) Take a application specfic static variable (int) in <appSeeting> in web.config file.

    <appSetting>

    <Key="ConCommenttimeout" Value="1000"/>....

    if u r use DAL as Entity fw/Ado.net then before doing any db operation u need to assing the CommandTime value  from config file.

    OR

    3) Use Lazy loading ... i.e. phase by phase featching.

    Thank's

    Prasanta Kuamr Pradhan.


    Prasanta De Ultimate http://www.acuvate.com
    Monday, October 24, 2011 9:49 AM