none
Reading from a HttpWebResponse ResponseStream using a StreamReader issue

    Question

  • I've been having an issue with reading the Response Stream of a HttpWebResponse using a StreamReader.  As far as I can tell it appears to be that the StreamReader tries to read the Response Stream faster than it can be retrieved from the web.

    The code in question is as follows:
        ///
     <summary>
    
    
    
        ///
     Makes a Query where the expected Result is an RDF Graph ie. CONSTRUCT and DESCRIBE Queries
    
    
        ///
     </summary>
    
    
    
        ///
     <param name="sparqlQuery">
    SPARQL Query String</param>
    
    
        ///
     <returns>
    RDF Graph</returns>
    
    
        public
     Graph QueryWithResultGraph(String sparqlQuery)
        {
            try
    
            {
                //Build the Query URI
    
                StringBuilder queryUri = new
     StringBuilder();
                queryUri.Append(this
    ._endpoint.ToString());
                queryUri.Append("?query="
    );
                queryUri.Append(Uri.EscapeDataString(sparqlQuery));
    
                if
     (!this
    ._defaultGraphUri.Equals(String.Empty))
                {
                    queryUri.Append("&default-graph-uri="
    );
                    queryUri.Append(Uri.EscapeUriString(this
    ._defaultGraphUri));
                }
    
                //Make the Query via HTTP
    
                HttpWebResponse httpResponse = this
    .DoQuery(new
     Uri(queryUri.ToString()),false
    );
    
                //Set up an Empty Graph ready
    
                Graph g = new
     Graph();
                g.BaseURI = this
    ._endpoint;
    
                //Parse into a Graph based on Content Type
    
                String ctype = httpResponse.ContentType;
                IRDFReader parser = MIMETypesHelper.GetParser(ctype);
                parser.Load(g, new
     StreamReader(httpResponse.GetResponseStream()));
    
                return
     g;
            }
            catch
     (UriFormatException uriEx)
            {
                //URI Format Invalid
    
                throw
     new
     Exception("The format of the URI was invalid"
    , uriEx);
            }
            catch
     (WebException webEx)
            {
                //Some sort of HTTP Error occurred
    
                throw
     new
     Exception("A HTTP Error occurred"
    , webEx);
            }
            catch
     (RDFException)
            {
                //Some problem with the RDF or Parsing thereof
    
                throw
    ;
            }
            catch
     (Exception)
            {
                //Other Exception
    
                throw
    ;
            }
        }
    
        ///
     <summary>
    
    
    
        ///
     Internal Helper Method which executes the HTTP Requests against the SPARQL Endpoint
    
    
        ///
     </summary>
    
    
    
        ///
     <param name="target">
    URI to make Request to</param>
    
    
        ///
     <param name="sparqlOnly">
    Indicates if only SPARQL Result Sets should be accepted</param>
    
    
        ///
     <returns>
    HTTP Response</returns>
    
    
        private
     HttpWebResponse DoQuery(Uri target, bool
     sparqlOnly)
        {
            //Expect errors in this function to be handled by the calling function
    
    
            //Set-up the Request
    
            HttpWebRequest httpRequest;
            HttpWebResponse httpResponse;
            httpRequest = (HttpWebRequest)WebRequest.Create(target);
    
            //Use HTTP GET/POST according to user set preference
    
            if
     (!sparqlOnly)
            {
                httpRequest.Accept = MIMETypesHelper.HTTPAcceptHeader();
                //For the time being drop the application/json as this doesn't play nice with Virtuoso
    
                httpRequest.Accept = httpRequest.Accept.Replace(","
     + MIMETypesHelper.JSON[0], String.Empty);
            }
            else
    
            {
                httpRequest.Accept = MIMETypesHelper.HTTPSPARQLAcceptHeader();
            }
            httpRequest.Method = this
    ._httpMode;
            httpRequest.Timeout = this
    ._timeout;
    
            //HTTP Debugging
    
            if
     (Options.HTTPDebugging)
            {
                Tools.HTTPDebugRequest(httpRequest);
            }
    
            httpResponse = (HttpWebResponse)httpRequest.GetResponse();
    
            //HTTP Debugging
    
            if
     (Options.HTTPDebugging)
            {
                Tools.HTTPDebugResponse(httpResponse);
            }
    
            return
     httpResponse;
        }
    
    
    The error occurs on the parser.Load() line and is my Tokeniser reporting that it hit the unexpected character code 65535 (which is in fact the -1 indicating end of stream cast to a char).  This error occurs at different points in the Stream each time I run the code indicating some kind of network delay/speed issue.

    I should point out that my Tokenisers and Parsers are valid and correct and pass all of the official test suites for the data formats I'm getting back from my requests.

    Also the data I'm getting back is valid since if I replace the parser.Load() line with the following which copies the response stream into a local memory stream first then it'll work without issue.  Tools.StreamCopy() is a helper function which does a Byte by Byte copy of one stream to another.

                Stream response = httpResponse.GetResponseStream();
                MemoryStream temp = new
     MemoryStream();
                Tools.StreamCopy(response, temp);
                response.Close();
                temp.Seek(0, SeekOrigin.Begin);
                parser.Load(g, new
     StreamReader(temp));
    
    
    So as far I can tell my issue is that I'm reading from the Response Stream too quickly, my Tokenisers are very fast (eg. 100ms or so for a 10k line file which will have maybe 40k distinct tokens in).

    What I want to know is if this is an issue with StreamReader, HttpWebResponse or my usage of them?  And is there any simpler/more elegant way of achieving the work-around I'm using currently?

    For the record I've also posted this on StackOverflow and it's just devolved into an argument with an MVP who says it's definitely my codes fault http://stackoverflow.com/questions/1264952/reading-from-a-httpresponsestream-fails but someone else pointed out that the documentation for TextReader (which underlies StreamReader) says that Read() methods are not guaranteed to return the amount of content you requested and this might be the cause
    • Edited by rav08r Wednesday, August 12, 2009 1:08 PM Clarified question
    Wednesday, August 12, 2009 1:05 PM

Answers

  • Nobody believes you can read too fast from a stream.  Isolate the problem.  Save the NetworkStream content to a file instead of a MemoryStream.  Look at it.  Looks bad, its a stream problem.  Looks good, you have a file you can use to debug the parser.

    Hans Passant.
    Wednesday, August 12, 2009 11:38 PM

All replies

  • It's almost certainly how you're using the StreamReader which is causing your issue.  Can you provide a code example of one of the Tokenizers that is failing specifically where you're reading from the StreamReader?

    Jerry Schulist

    Please remember to mark replies which answer your question as answers and vote for replies which are helpful.
    Wednesday, August 12, 2009 1:51 PM
  • Try reading the Stream before passing it to your parser.

      Stream response = httpResponse.GetResponseStream();
    StreamReader temp = new StreamReader(response);
    temp.ReadToEnd();
    parser.Load(g, temp);

    I think the StreamCopy and closing the response stream is messing you up. Try my suggestion and see what happens. Be aware that the ReadToEnd is a blocking call and might block indefinitely. I think there are timeouts you can set for the StreamReader. Anyways, if the data is fairly large (a meg lets say) it might take a few seconds for the call to finish.
    Wednesday, August 12, 2009 8:30 PM
  • Nobody believes you can read too fast from a stream.  Isolate the problem.  Save the NetworkStream content to a file instead of a MemoryStream.  Look at it.  Looks bad, its a stream problem.  Looks good, you have a file you can use to debug the parser.

    Hans Passant.
    Wednesday, August 12, 2009 11:38 PM