GetResponse blocks until entire response is available

Question

I have an HTTP service I use for testing.   It writes a 1024K response.   More specificially it writes 32K at a time waiting 5 seconds before sending each of the 32 chunks.

On the client side I use HttpWebRequest.GetResponse.   As you might expect,  after 160 seconds the client has the entire request.    The problem I have though is that the GetResponse blocks until the enitre message is received.    However,  if the client and the server are on the same machine,  AND I use localhost,  then the GetResponse returns after the first chunk (which is what I want).   This allows me to stream the response through the process.

I have taken a NETMON trace when the client / server are on different machines.   I can see the HTTP payload packets arriving every five seconds on the client.   But the GetResponse does not unblock

So the question is how do I get GetResponse to not block until the neitre message is read when not using the loopback adapter?

_sjz.

---

sjz

Answer 1

A non blocking request requires using the Async Method.  See the attached webpage

http://msdn.microsoft.com/en-us/library/86wf6409(v=VS.71).aspx

---

jdweng

Answer 2

I am not sure I made the scenario clear.

I totally want to block.   But only until the first byte is written to the response stream by the server.   That should unblock the GetResponse call.   Once that happens I can then start reading the stream.   And to be clear,  this is a very simplified repro.   The real code does an async get of the response.   It too does not return until the full response is received by the client - as opposed to returning when the first byte is received and allowing me to start reading from the stream.

And finally,  the GetResponse{Async} call does complete when on receipt of the first byte when the server is on the same machine and reference via localhost.

---

sjz

Answer 3

Once you send the GetResponse the server is going to download the entire webpage.  The Async method is only going to allow you to start reading the data before the  entire webpage is downloaded.

The GetResponse automatically does a negotiation between the client and server to determine a common mode of downloading the data, to verify any credentials, and to pass a se urity keys necessary to download the data.

I'm not sure how you are going to get the server to transfer only one chunk of data unless you are developing your own protocol between client and server.  I think this is what you are doing.  I think your problem is know when the first chunk is completely receive at the client using Async mode before sending the back the one byte message.  the first chunk need to do one of the following

1) At the end of the first chunk send a termination character like a CR.  The client then knows when the first chunk ias fully received

2) The first chunk can always be a fixed number of bytes.  Then the client know how many characters to receive before send back the one byte message.

3) Include in the 1st chunk a byte count as the first character(s).  then the client know how big the 1st chunk is.

---

jdweng

Answer 4

More details.

The server is simulating a reverse trickle attack when writing to the response.  Instead of writing 1MB in one call,  it is writing to the response stream in a loop - 32K at a time (with intentional multi-second gaps in between).   There is no special protocol.   It is straight HTTP.

My expectation is that GetResponse will stop blocking upon receipt of the first byte written to the response stream.   You don't have the entire response at that time, but you have the status,  and you have the headers.  Then you can read the response stream.   When doing that I would expect the read of the stream to stall and block as well.   Right now I am just trying to understand why the GetResponse{Async} blocks until the ENTIRE response is available.

This works exactly as I describe/expect above WHEN CALLING THROUGH localhost.   But not when using the DNS name or IP address of the machine.

---

sjz

Answer 5

The webpage below has the code shown below.  The code is going to read forever.

http://msdn.microsoft.com/en-us/library/86wf6409(v=VS.71).aspx

// Continue reading data until
                // responseStream.EndRead returns –1.
                IAsyncResult ar = responseStream.BeginRead(
                   rs.BufferRead, 0, BUFFER_SIZE,
                   new AsyncCallback(ReadCallBack), rs);

These two statements are the problem

int read = responseStream.EndRead(asyncResult);
            if (read > 0) 

read will not equal -1 until the connection closes.  You want to terminate after the first chunk.  So either you need to stop after a certain number of characters are received or terminate after a CR (or equivalent end character(s)) is received.

---

jdweng

Answer 6

No I don't want to terminate after the first chunk.   I want to read many many chunks and keep reading until it takes longer than I am willing to wait.   But discussion of the read is orthogonal. 

The GetResponse call never returns until the entire message is present.   That is the problem I am trying to solve.   So forget about the read - for now.   I want to know how to return from {Begin}GetResponse on receipt of the first byte.  Again.  It works for the loopback adapter.   Just not with a DNS name or IP address.

_sjz.

---

sjz

Answer 7

I am not saying terminate.  You did post any code, but if you are using the code on the webpage I referenced the code loops forever.  The Async read will return after each chunk.  You just will never get out of the loop.  Try putting some break points in the code and verify where your code is getting stuck.

---

jdweng

Answer 8

As I said before.  Forget the read.  The code never gets there.  It blocks on the GetResponse{Async} until all the data is available / written to the response by the service.    Once I get out of the GetResponse{Async} then we can talk about loops but until then it is moot.

---

sjz

Answer 9

Hello,

This symptom is because of GetResponse Internal code. If response is fast enough than GetResponse will not block. Try 5 second delay before the first answer(event before headers). I guess GetResponse will block on local host too..

Regards.

Answer 10

As I have mentioned several times,  in the localhost case,  GetResponse returns immediately (does not block) even though there may be megabytes of data still to be written by the service.   The client can then start reading the response stream right away (even though not all the data is there).  In that case the reads start to block - which is expected and what I want.

When using a DNS name or IP address for the same physical server GetResponse blocks until all the data is in process.  Waiting in GetResponse longer totally works,  but I may be transferring megabytes of data.   I want to start reading right away to effectively stream the data as it comes in.

_sjz.

---

sjz

Answer 11

I think youwill need to use wireshark to determine why the interface is being blocked.  I can't tell if the blocking is occuring at the TCP transport level or at the HTTP application level.  The HTTP connection performs a negotiation between the client and server to determine the protocol that is used for the connection. It is possible that the transfer mode that is chosen during the negotiation between the Local Host and Remove Server is different.

For example, HTTP 1.1 allows for chunks whle HTTP 1.0 doesn't allow for chunks.  When connecting to the lcoal host you may be using chunks and when you connect to the DNS you may not be using chunks.  Your DNS server may not have HTTP 1.1 implimented.

---

jdweng

Answer 12

Try this:

http://msdn.microsoft.com/en-us/library/system.net.webrequest.getresponseasync(v=vs.110).aspx

You can also use reactive extensions which is the best way in my opinion.  Why is it best?  Because RX is not a pull based technology.  When you call a method name GetSomething what you are really doing is pulling the data, so if you do not specifically use Asynchronous methodology you will Always block.  But in RX what you do on the client is to subscribe the GetSomething Observable.  That subscription is PUSH based which means that the Observable will push the data into the Client.  It takes a while to get this concept but once you dial-in you will think that it is really cool.

There is another way to do this.  If any method you have anywhere in .NET is blocking and you don't want that to happen.  Simply start a task like this  

//first set up a delegate in your class
delegate void dataready(ObservableCollection<Something> OC);
//next set up an event based on the delegate
public static event dataready OnDataReady

Task.Factory.StartNew((Action)delegate(){
     //do your stuff here
     //Send event when done like this
     If(OnDataReady != null){ OnDataReady(MyData);  }


});

//now on your client class subscribe to the event

MyDataClass.OnDataReady += MyDataClass_OnDataReady;

public void MyDataClass_OnDataReady(ObservableCollection<something> oc){

  //this is the result of your other class sending the data
  //over, but note you have to marshall the data onto this thread
  MyDataClass.Dispatcher.Invoke((Action)delegate(){
       TheCollectionHere = OC;
  }

}

//if TheCollection notifies WPF, then all GUI will be updated
automatically.

---

JP Cowboy Coders Unite!

Answer 13

Yes.   I am enabled tracing through machine.config to see what is going on.   We'll see.

The GetSomething Observable stuff is not relevant.  In truth I am already using tasks and get response async followed by async reads (also in tasks).  It makes no difference.  The async getResponse does not compelte until ALL the data is available in process (for the non-localhost case).

---

sjz