System.Net.Sockets.SocketException: Cannot download BLOB entity!!!

Question

As long as the download task can be completed within 1 minutes, everything is fine. Otherwise, the following exception is thrown:

System.Net.Sockets.SocketException
Status code = System.Net.Sockets.SocketError.ConnectionReset
Message = "An existing connection was forcibly closed by the remote host"

I tried increasing timeout value but the problem persists.

The same problem occurs even when I download the entity through IE7. It will throw this error message after 1 minute: The connection with the server is reset.

Obviously, this means the root problem is not in my code. I suspect there is something wrong with the SDS ??!!!

By the way , my BLOB entity is 4.5 MB, and I only managed to download 1.7 MB.

Answer 1

I followed the code samples in SSDS Primer documentation and managed to perform CRUD for BLOB entities.

However, I have to increase HttpWebRequest.Timeout in order to upload larger BLOBs, otherwise I will get the following error if the upload task cannot be completed within the timeout duration:

System.Net.WebException
  Message = The request was aborted: The request was canceled.
  Status = System.Net.WebExceptionStatus.RequestCanceled


Is there a proper way to solve this problem? Or increasing timeout is the only solution?

Answer 2

Increasing the HttpWebRequest.Timeout value is the only solution I've found in uploading large blobs to Azure Blob Storage.

You might find you need to change other settings to upload very large blobs:

Configuration Adjustments: You must add <httpRuntime maxRequestLength="100000" /> to Web.config's <System.Web> element for 100 MB maximum file size (units are KB).

For uploads larger than 30 MB, change the development machine IIS 7.0's maxAllowedContentLength to 100000000 for 100 MB (See "IIS 7.0: requestLimits Element for requestFiltering (IIS Settings Schema)" (http://msdn.microsoft.com/en-us/library/ms689462.aspx) for information on setting larger file size limits in IIS 7.0. (Units are bytes). Default maximum IIS size is 30 MB.

I don't know what Azure's IIS istance limits are, but I'm about to find out.

The preceding info is from comments in a blob test harness I'm working on.

HTH,

--rj

---

OakLeaf Blog

Answer 3

Besides increasing timeout value, I noticed that I need to set

httpWebRequestObject.KeepAlive = False 

too. Otherwise, another exception will be thrown after 15 - 20 minutes.

By changing both properties, I managed to upload a 93 MB zip file to my SDS container.

But, I am still curious whether I am using the 'correct' approach or not. Is there any other solution?

Answer 4

i, too, am having problems w/ large blobs today.

using my SSDSDeploy.exe utility to upload files, i see a server timeout after 30 seconds. even when my code is adjusted to set a timeout for 60 seconds on the POST request, i still see this timeout/close scoket @ 30 seconds from the remote server.

i don't recall having this problem w/ large files before. i suspect there's something up w/ the SDS servers.

please keep us posted.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 5

i adjusted (upward) both the request timeout value and an internal timeout value for my async callback and was able to upload a file that was more than 4.5MB.

looks to me like perf has changed since the last time i tested uploads (november). is that right?

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 6

These all make sense once you realize that the SDS front end and the clients you are using are all .NET applications.  This means that the traditional changes required to deal with long conversations need to be made.

Evan

Answer 7

Evan:

I'm pretty sure I tested BLOBs using 1,5,10, and later MB payloads w/o seeing the timeout issues back in November.

Have things changed on the server side for SDS?

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 8

I don't have any hard data to say so, but we definitely have different code out there now than we did in November, so it is definitely possible.  Needless to say, we are always trying to increase performance over time, so maybe we went in the wrong direction.

Do you have a succinct sample that shows a performance difference?  Maybe you are doing something our internal performance testing is not measuring.

Evan

Answer 9

Evan:

well, i don't have any repeatable stuff and can't find any data that i kept from my Oct/Nov tests <bummer>.

i modified my code to allow for a param to set the timeout. that will work in the short term. in the long term, i suspect something that catches the timeout and tries again with a longer value will need to be in place.

handling any download timeouts is another issue. not sure what's going on there.

thanks for the follow up. if i find anything concrete, i'll post it here.

hope you and the SDS team have happy new year.

looking forward to cool stuff in 2009.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 10

Evan:

i ran a GET request for a 4.5MB file through WFetch.exe and got the following:

REQUEST: ******************
GET /v1/test1/liveframeworksdkwalkthroughs.zip HTTP/1.1
Host: binaries.data.database.windows.net
Accept: */*
Authorization: Basic ***********************

RESPONSE: **************
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/x-zip-compressed
ETag: 788532
Vary: Accept
x-msft-request-id: f4320f39-89c2-4a10-b8b7-eb1e2cd3ec30
Date: Thu, 01 Jan 2009 01:31:48 GMT

[lots of bits on the wire for a while then...]

0x274c (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.): [sockslib]: recv()

maybe you can use the request id to track this request.  feel free to hijack this authority/container for further testing if you like.

ping me if you need anything else.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 11

Mike,

Just to confirm, do you get that error message after exactly 1 minute too?

So we can now upload a large BLOB but not download it, what a nice error to get on New Year eve.

In terms of performance, I notice the upload speed is always 10 - 15 KB / sec. As for the download speed, usually it is 10 - 15 KB / sec too, but sometimes I can get as high as 50+ KB /sec for a few seconds.

Answer 12

c.c.chai:

i ran the same request (see above) several times and here's what i got:

1 - 67 sec = completed successfully
2 - 70 sec = stalled download incomplete, 122 sec server abort
3 - 69 sec = completed successfully
4 - 73 sec = stalled download incomplete, 133 sec server abort
5 - 65 sec = stalled download incomplete, 121 sec server abort

i stopped there.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 13

all:

i see this is still a problem.  per my previous message in this thread, i have a less than 50-50 chance of successfully downloading a 4.5 MB blob file from SDS servers.

what's the story here?

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 14

Evan/Anil:

this continues to be a problem. for example, i am unable to *download* a 4.5 MB BLOB from SDS due to server disconnections.

i included an example (w/ msft-request-id) and some sample timings on this thread
http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/e907e517-dc2b-45c7-ad74-c5fcfcb1f096

this needs some attention ASAP, OK?

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 15

Mike/CC

Folks here on the team are working on trying to repro this issue.  Once we have something, we shall respond.  I hope to have something in the next couple of days.

Sorry for the delay.   I want to assure you that we are working on your query

Regards

Anil

Answer 16

Anil:

thanks for the update.

feel free to use the authority/container pair (from my repro above) in your own testing. if it turns out to be something about my particular ACE combination, let me know.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 17

Will do Mike

Thanks

Anil

Answer 18

 Hello Folks.

We have looked at the issue and have identified it as a bug.  We are currently in the process of finding a solution for the bug.  Once we have the solution we shall do the needful to fix the issue.  I thank you for bringing this to our attention.  We shall update you when we have the solution and on the next steps.

We appreciate your support

Regards

Anil

Answer 19

Anil:

thanks for the update. good to hear you have it identified and that a fix is in the works.

a few follow up questions:

- can you give us some estimate of the fix time. is this a hot fix? something that will roll into the next "sprint" (and when is that)? something that will be done in hours, days, weeks?

- can you give any additional details on the bug ("files larger than 3MB fail" or "downloads lasting more than 60sec fail" or "PNGs work, but ZIPs fail", etc.)

- can you provide any workaround for the time being ("when uploading files, break any larger files into 3MB sections" or "if you pass the HTTP RANGE header you can download large files in chunks and assemble the final item for delivery to the client", etc.)

looking forward to your reply.

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 20

*bump*

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 21

Anil/Evan/Stan, et al:

It's been a week since SDS team last posted a response on this thread. What's up with the bug fix?  

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 22

*nudge*

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 23

Any update??

Answer 24

This fix for this bug was included as part of the new version rollout yesterday (1/27/09).

Thanks,

Hari

Answer 25

HariSiva:

 Thanks for the update. I'll do some testing and post any issues here in the forum.

 

---

Mike Amundsen [http://amundsen.com/blog/]

Answer 26

I managed to upload and download a 10 MB blob with no problem.

 The upload speed is pretty much the same as last time but the download speed is much slower than last time. 

On average, the download duration is about 3 times the upload duration, which is too slow!!!!

Answer 27

I think the download blob problem is not solved yet.

 

First, the download is slow.

Second, the connection is automatically closed after 2 hours leaving the download incomplete.

Answer 28

c.c.chai,

c.c.chai said:

Second, the connection is automatically closed after 2 hours leaving the download incomplete.

This is expected behavior. We enforce a two hour timeout for blob downloads.

Thanks,

Hari

Answer 29

I have tried downloading a 90 MB blob for 4 times in yesterday and today, the downloads ended at 30 - 34 MB in 2 hours, which is only one third of my blob. So, I think the 2 hours timeout is not a good idea.

 

Again, I am pretty sure the download speed is much slower than before. I used to be able to upload & download at about 1 MB per minute. While the upload speed remains the same, the download speed is simply too slow....