locked
How do I avoid this OData limitation? RRS feed

  • Question

  • For the url: http://services.odata.org/Northwind/Northwind.svc/Employees?$expand=Territories,Orders,Orders/Order_Details&$top=5

    I'm getting:

    Microsoft.Data.OData.ODataException occurred
      Message=The maximum number of bytes allowed to be read from the stream has been exceeded. After the last read operation, a total of 1049051 bytes has been read from the stream; however a maximum of 1048576 bytes is allowed.
      Source=Microsoft.Data.OData
      StackTrace:
           at Microsoft.Data.OData.MessageStreamWrapper.MessageStreamWrappingStream.IncreaseTotalBytesRead(Int32 bytesRead)
           at Microsoft.Data.OData.MessageStreamWrapper.MessageStreamWrappingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
           at Microsoft.Data.OData.BufferingReadStream.Read(Byte[] userBuffer, Int32 offset, Int32 count)
           at System.IO.StreamReader.ReadBuffer(Char[] userBuffer, Int32 userOffset, Int32 desiredChars, Boolean& readToUserBuffer)
           at System.IO.StreamReader.Read(Char[] buffer, Int32 index, Int32 count)
           at System.Xml.XmlTextReaderImpl.ReadData()
           at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
           at System.Xml.XmlTextReaderImpl.ParseAttributes()
           at System.Xml.XmlTextReaderImpl.ParseElement()
           at System.Xml.XmlTextReaderImpl.ParseElementContent()
           at System.Xml.XmlTextReaderImpl.Read()
           at Microsoft.Data.OData.Atom.BufferingXmlReader.ReadInternal(Boolean ignoreInStreamErrors)
           at Microsoft.Data.OData.Atom.BufferingXmlReader.ReadNextAndCheckForInStreamError()
           at Microsoft.Data.OData.Atom.BufferingXmlReader.ReadInternal(Boolean ignoreInStreamErrors)
           at Microsoft.Data.OData.Atom.BufferingXmlReader.Read()
           at System.Xml.XmlReader.SkipSubtree()
           at System.Xml.XmlReader.Skip()
           at Microsoft.Data.OData.Atom.ODataAtomEntryAndFeedDeserializer.FindTypeName()
           at Microsoft.Data.OData.Atom.ODataAtomReader.ReadEntryStart()
           at Microsoft.Data.OData.Atom.ODataAtomReader.ReadAtEntryEndImplementation()
           at Microsoft.Data.OData.ODataReaderCore.ReadImplementation()
           at Microsoft.Data.OData.ODataReaderCore.ReadSynchronously()
           at Microsoft.Data.OData.ODataReaderCore.InterceptException[T](Func`1 action)
           at Microsoft.Data.OData.ODataReaderCore.Read()
           at WindwardReportsDrivers.net.windward.datasource.odata.ODataUtilities.ReadEntryList(String uri, WrCredentials credentials, CONNECT_MODE protocol, IEdmModel model, ODataVersion oDataVersion)
      InnerException: 
    

    Any ideas how to get this data?

    thanks - dave


    Who will win The International Collegiate Programming Championships?

    Wednesday, September 12, 2012 10:26 PM

Answers

All replies

  • ODataMessageReaderSettings.MessageQuotas.MaxRecievedMessageSize

    Thanks,


    Vitek Karas [MSFT]


    Thursday, September 13, 2012 7:49 AM
    Moderator
  • Thank you - that was exactly what was needed.

    Is there a way to tell the server what I am limiting to so it will page as needed to keep it under that limit?

    thanks - dave


    Who will win The International Collegiate Programming Championships?

    Thursday, September 13, 2012 4:56 PM
  • No - this limit is meant to guard the client from malicious servers. It is possible to send valid OData feed which will consume lot of memory to parse. This can be done on the XML level or on the OData level. This limit guards against both.

    Servers have other ways to limit amount of data returned to the client, the reason there is to guard against expensive query execution

    In general a good way to look at OData is to think about it as the "minimum necessary protocol" to do what is needed. We try to keep it that way anyway. It's also stateless (REST...). So the more coupling between client and server, the worse... so to say.

    I can imagine having an HTTP header for this purpose, but I don't think that lot of clients would be interested in it.

    I think you should approach this limit from the security point of view (that's what it's for). Set it to the maximum amount of memory you're willing to give up to process the response. At least roughly (the memory consumption should be linear to the size of the input, cases where it's not are guarded by the other options on the MessageQuotas).

    Thanks,


    Vitek Karas [MSFT]

    Thursday, September 13, 2012 5:31 PM
    Moderator
  • I can see your point but...

    I hit this requesting from Netflix. Netflix is paging, but at a size that is larger than the default. Fundamentally to pull the data from Netflix, I'll make the buffer as large as needed. But since the paging is happening anyways, if I could tell Netflix the maximum data size, it could page appropriately and I would not have to restart with a larger size.

    I'll add it as a suggestion.

    thanks - dave


    Who will win The International Collegiate Programming Championships?

    Thursday, September 13, 2012 5:45 PM
  • I didn't even touch on the complexity to implement such feature on the server. Honestly implementing the cap on bytes on the server in a way that the server still responds (and pages) would be... well really hard. Servers do have caps but they fail the request. The paging cap is much more fuzzy. I understand the frustration with Netflix, they picked a pretty high number for the page size given the size of the Title entity.

    Thanks,


    Vitek Karas [MSFT]

    Thursday, September 13, 2012 6:04 PM
    Moderator
  • Hi,

         Is there a way to get over the same issue in SharePoint hosted app? I am passing the base64 converted content of document in SharePoint to a hosted web service using SP.WebProxy.Invoke (ajax call), sometimes the content length exceeds the limit of 1048576 and I get the below error

    "The maximum number of bytes allowed to be read from the stream has been exceeded. After the last read operation, a total of 1049600 bytes has been read from the stream; however a maximum of 1048576 bytes is allowed"


    Thanks, Ajeet


    Sunday, April 12, 2015 8:22 PM