locked
HTTPS downloads failing RRS feed

  • Question

  • User-1468455771 posted

    Hi, I am running Windows Server 2016 / IIS 10, and really hoping someone can help me with an ongoing problem.

    We are having sporadic but persistent problems downloading large files over HTTPS (not HTTP). Here is what I've learned so far:

    1. The problem only happens when downloading files - there are no problems with browsing web pages.

    2. While some downloads work perfectly, others go for a while - usually several MB (sometimes say 20-30MB or more) - but then just stop. I've captured the traffic to the browser, and the data just stops coming from the server with no HTTP error. The browser eventually times out. After it stops, I can resume the download which usually works for another few seconds and then fails again.

    3. This is sporadic and seemingly random: some large downloads work fine, but it's still happening frequently enough to be a problem. Sometimes I can download a large file OK, but then a re-download fails part way through, for no apparent reason.

    4. There are no relevant events in the Event Log. ASP.NET throws a "remote host closed the connection" error, as it does if the user closed their browser. I've confirmed (through traffic capture) that the browsers certainly are not 'disconnecting'. But to the server, it looks as if the browser has disconnected.

    5. The same behavior happens whether serving a static file via IIS or via ASP.NET (using Response.TransmitFile) so I am sure it isn't an ASP.NET issue.


    6. A key thing: it only applies to HTTPS downloads. I've confirmed this by testing. I have tested downloading a file over HTTPS and it fails repeatedly. I then immediately download the same file using HTTP and it works perfectly. I immediately try again via HTTPS and it fails again.  I've even had an HTTP download running perfectly while simultaneously seeing HTTPS downloads fail. I've confirmed the SSL certificate is valid, and the HTTPS downloads often do work, so I doubt anything wrong with the cert. Plus, I tried with a self-signed cert and got the same result.

    7. Large downloads over FTP and RDP work fine (like HTTP) - it's only HTTPS downloads that fail sporadically.

    8. I doubt it is a network card error, because everything works perfectly over HTTP (and other protocols) at same time as an HTTPS download is failing repeatedly.

    9. Have tried with firewall disabled, and get the same result.

    I'm really out of ideas for this, so any suggestions will be gratefully received.

    Thank you!

    Thursday, November 16, 2017 8:25 PM

All replies

  • User-460007017 posted

    Hi kohepu,

    It is not clear why the https wil break the downloading. Maybe you could try to adjust the connection time out for the site level->advanced setting->connection time out.

     In addition, you could try to access application pool advanced setting->try to disable the idle timeout and recycle interval.

    If the steps above not working, then you could try the IIS crypto to change the https setting. If you have any other CA certificate, you could just try a new one.

    https://www.nartac.com/Products/IISCrypto

    Best Regards,

    Yuk Ding

    Friday, November 17, 2017 7:36 AM
  • User-2064283741 posted
    So did your traffic capture show where the issue was occurring?

    You say it looked from the server point of view that the client dropped the connection.

    So the client receives traffic ok but fails to send a request for more packets?!

    If so then maybe it is a client issue
    Friday, November 17, 2017 8:32 AM
  • User-1468455771 posted

    Thank you for the comments! They gave me a clue which led me to the HTTPERR logs, which is recording a number of "Timer_MinBytesPerSecond" errors (explained here: https://support.microsoft.com/en-us/help/820729/error-logging-in-http-apis) that match the times of the disconnects.

    So I think that is the issue - the download drops below 240 bytes/sec (though over what period I don't know), and IIS terminates the connection.

    However, I don't know why it is apparently dropping that slow. It happens at low-traffic time and monitoring shows consistent fast downloads (several MB/s) right up to the time of the disconnecting. It sounds very similar to the issue several people are reporting on Win Server 2016 at: https://forums.iis.net/t/1235830.aspx?minBytesPerSecond+webLimit+setting+not+taking+effect  (see last 3 posts)

    At least I now have an error message to work off....   :)

    Friday, November 17, 2017 11:57 AM
  • User-460007017 posted

    Hi kohepu,

    Have you tried to set the connection time-out and set the minBytesPerSecond to 0 and expand the execution timeout in system.web->execution timeout via configuration editor.

    Best Regards,

    Yuk Ding

    Monday, November 20, 2017 8:51 AM
  • User-1786377522 posted

    Experienced the same issue on Windows 2016 and IIS10 with 5+ Mbytes file downloads (from any browser supporting http/2).

    Found that it is a bug with HTTP/2 on IIS10, and disabling http/2 solved the issue:

    Set:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

    EnableHttp2Tls REG_DWORD to 0

    EnableHttp2Cleartext REG_DWORD to 0

    Create keys if they do not exist.

    Wednesday, November 22, 2017 9:22 AM
  • User-1468455771 posted

    Hi Yuk Ding and Sergei, thanks for your comments.

    Yes, reducing the minBytesPerSecond to 0 seems to have worked (at least for now), but I'd rather not have to disable that based on its purpose.

    This definitely sounds like a bug in Windows 2016 + IIS 10.  How best to report it?

    Thanks

    Wednesday, November 22, 2017 11:51 AM
  • User-460007017 posted

    Hi kohepu,

    I will report this issue.

    Best Regards,

    Yuk Ding

    Wednesday, November 29, 2017 6:09 AM
  • User103788153 posted

    Hi Kohepu,

    I don't think this is a bug for windows server 2016/ IIS 10 with HTTPS.

    I established the similar environment with you: windows server 2016. IIS HTTPS site with self-signed certificate, enable directory browsing so that I could download the website content.

    I tried to download an around 90MB zip file from the HTTPS IIS site, it finished within 30 seconds, no suspension. I repeated this several times, didn't see the issue you mentioned.

    To continue troubleshooting for this issue, I think you could first confirm the HTTP protocol you are using, whether HTTP/1.1 or HTTP/2? There is a new feature - One connection for multiple requests, for HTTP/2.

    If you are using HTTP/2, regarding to this feature, you could capture network trace to check during the downloading process, is it possible the TCP connection is used by other request? or something wrong with the existing TCP connection?

    you could use Network monitor or WireShark for network trace capture.

    If you need more professional assistance from Microsoft, you could try to raise support ticket to Microsoft Customer Service and Support.

    Friday, December 1, 2017 6:48 AM
  • User-2064283741 posted

    As you attention has been arise (maybe I highlighted the wrong thread here) HTTP2 under 2016 seems to have multiple issues.

    Resending info and degrading it to http1.1 and downloads failing and other issues.

    There are many threads about it and I feel this needs to be looked at. Often the advice here is simply disable http2 which is surely the wrong advice.

    Friday, December 1, 2017 7:18 AM
  • User-1530910569 posted

    I have the same problem on Windows 2016 and the problem is related to the HTTP 2 protocol.

    I checked the protocol in IIS Logs as well as I disabled HTTP 2 in the Firefox configuration:

    about:config

    firefoxPref network.http.spdy.enabled false
    firefoxPref network.http.spdy.enabled.http2 false

    The problem disappeared as soon as I switched to HTTP 1.1.

    Now I am considering disabling HTTP 2 on the server level.

    Does anybody have another solution of the problem?

    Monday, December 18, 2017 5:42 PM
  • User-1846777343 posted

    We're experiencing this exact same problem, but we cannot disable HTTPS/2 server-side, as we're leveraging it for optimization for non-static file communication with our server. 

    Has anyone from Microsoft provided an official stance on this, and whether or not it is a bug that is being actively worked on? 

    Wednesday, January 31, 2018 9:44 PM
  • User-1468455771 posted

    Since we disabled HTTP/2 on the server, the problem has gone away.

    Thursday, February 1, 2018 1:12 AM
  • User1480929202 posted

    We have noticed the same issue and disabling HTTP2 was the only workaround.

    In case this is actively investigated by Microsoft, the following ETL trace event may help:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Guid="{dd5ef90a-6398-47a4-ad34-4dcecdef795f}" Name="Microsoft-Windows-HttpService"/>
    <EventID>15</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>1</Task>
    <Opcode>23</Opcode>
    <Keywords>0x8000000000000006</Keywords>
    <TimeCreated SystemTime="2018-01-24T14:25:54.154643800-0800"/>
    <Correlation ActivityID="{800009a4-0002-d700-b63f-84710c7967bb}"/>
    <Execution UserTime="0" KernelTime="24495" ProcessorID="4" ThreadID="4044" ProcessID="4"/>
    <Channel>Microsoft-Windows-HttpService/Trace</Channel>
    <Computer/>
    </System>
    <EventData>
    <Data Name="RequestId">0xD7000002800009A4</Data>
    <Data Name="Reason">SendResponseFailed</Data>
    <Data Name="Status">3221225760</Data>
    </EventData>
    <RenderingInfo Culture="en-US">
    <Level>Error </Level>
    <Opcode>SndError </Opcode>
    <Keywords>
    <Keyword>Flagged on all HTTP events dealing with request processing </Keyword>
    <Keyword>Flagged on all HTTP events dealing with response handling </Keyword>
    </Keywords>
    <Task>HTTP Request Trace Task </Task>
    <Message>Error 3221225760 occurred while sending (corresponding to request ID 0xD7000002800009A4). A TCP Reset will be sent. </Message>
    <Channel>HTTP Service Channel </Channel>
    <Provider>Microsoft-Windows-HttpService </Provider>
    </RenderingInfo>
    </Event>
    

    Error 3221225760 = 0xC0000120
    STATUS_CANCELLED
    The I/O request was canceled.

    Tuesday, February 6, 2018 5:37 PM
  • User-539434273 posted

    Hi all,

    I can confirm this was a defect in the HTTP.sys driver which only affected HTTP/2 channels. The problem was the value of the minBytesPerSecond threshold was being ignored for HTTP/2 downloads only.

    A fix available with the cumulative updates for Windows Server 2016 introduces a new registry key that can be set to overcome this issue.

    1. Install the fix from  https://support.microsoft.com/en-us/help/4093119/windows-10-update-kb4093119
    2. The fix allows a setup of a new registry key called HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters\Http2MinSendRate

      Value must be between 0 and 0xffff, Type is DWORD.

    3. Restart IIS and the HTTP Service (or reboot the system) for the key to take effect

    HTH

    Wednesday, April 18, 2018 9:56 AM
  • User1480929202 posted

    What is the default value of Http2MinSendRate? And what is the recommended value?

    Wednesday, April 18, 2018 10:27 AM
  • User-539434273 posted

    [Edit] please see the answer below as it is more accurate

    Wednesday, April 18, 2018 12:43 PM
  • User-539434273 posted

    To further clarify, I did some research and found out for HTTP/2 connections there are 2 timers involved.

    1. it's the same very same timer that hooks up with the minBytesPerSecond value used in HTTP/1.1 connections. This sets the minimum rate per HTTP stream (it does not consider the HTTP/2 connection as a whole and it would timeout streams individually)

    2. a different timer is used to monitor the whole HTTP/2 connection (that is all the streams that are multiplexed into it). this is set to 150bytes/s by default. After installing the KB the minimum transfer rate for the whole HTTP/2 connection is configurable via the reg key Http2MinSendRate

    Regarding the new reg key, Http2MinSendRate, this works exactly as minBytesPerSecond and is used by HTTP.sys to set the timer timeout. The higher the minimum rate, the more aggressive the timer. The 0xffffffff as a special value to disable the timer was chosen to keep it consistent with the HTTP/1.1 connection min send rate timeout.

    Thursday, April 19, 2018 8:46 AM
  • User-1786377522 posted

    It does not work at all for my server (tried 0xffff , 0xffffffff and 0 values)

    Hi all,

    I can confirm this was a defect in the HTTP.sys driver which only affected HTTP/2 channels. The problem was the value of the minBytesPerSecond threshold was being ignored for HTTP/2 downloads only.

    A fix available with the cumulative updates for Windows Server 2016 introduces a new registry key that can be set to overcome this issue.

    1. Install the fix from  https://support.microsoft.com/en-us/help/4093119/windows-10-update-kb4093119
    2. The fix allows a setup of a new registry key called HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters\Http2MinSendRate

      Value must be between 0 and 0xffff, Type is DWORD.

    3. Restart IIS and the HTTP Service (or reboot the system) for the key to take effect

    HTH

    Thursday, April 19, 2018 11:18 AM
  • User-1846777343 posted

    My testing also indicates that the DWORD value has no impact on the behavior.

    Thursday, April 19, 2018 11:15 PM
  • User-663891294 posted
    Pleanse ensure that the hotfix is installed in order to try a new registry key.
    Tuesday, April 24, 2018 2:09 PM
  • User-1195570443 posted

    Experiencing the same problem here with IIS 10.0.14393.0 on a new Azure VM with Windows Server 2016. Clients downloading fairly large files (200+MB) from our server using Chrome often get "Failed - Network Error" after just a few MBs. Started happening when we went into production with https. Will try disabling HTTP/2 on the server first, to verify that it is the same problem. If it is, I will try the more gentle solution with Http2MinSendRate instead. (I have checked that the KB4093119 is in fact installed on this server. And actually it was done in a Windows Update just a few days ago. Hmmm...)

    Any news about this? If you think you are experiencing the same problem, feel free to share. I need to get to the bottom of this asap, it is bothering the customers on our brand new website a lotfrown

    Tuesday, April 24, 2018 2:37 PM
  • User-1195570443 posted

    UPDATE: Before trying to disable http/2, I decided to run the latest cumulative windows update on the WS 2016. And, hey, the problem seems to be gonesurprised Still need to let the site run for a couple of days before I can say it really is, but after the update and reboot, all downloads from Chrome now seem to run flawlessly - and much faster than beforesmile But if this really was a bug in http.sys IIS 10 / WS 2016, why wasn't there more information about it available? I've sweated over this problem for over a week, and the only clue I could find was in this forum thread... strange.

    I'll let you know when (if) I come to a conclusion.

    Tuesday, April 24, 2018 9:03 PM
  • User-1786377522 posted

    I can confirm that after recent cumulative update HTTP2 works properly, now I can download 100-mbyte file over slow client connection (~150kbytes\sec and ~300ms ping) for 10+ minutes using HTTP2 with no single break. I also removed registry entries I mentioned above. Thanks MS for this fix.

    UPDATE: Before trying to disable http/2, I decided to run the latest cumulative windows update on the WS 2016. And, hey, the problem seems to be gonesurprised Still need to let the site run for a couple of days before I can say it really is, but after the update and reboot, all downloads from Chrome now seem to run flawlessly - and much faster than beforesmile But if this really was a bug in http.sys IIS 10 / WS 2016, why wasn't there more information about it available? I've sweated over this problem for over a week, and the only clue I could find was in this forum thread... strange.

    I'll let you know when (if) I come to a conclusion.

    Thursday, April 26, 2018 12:24 PM
  • User1174230613 posted

    This is old - but I thought id share my experience. I noticed this when we converted from server 2008 to 2016, and only on our classic asp pages. I noticed that I was setting response.buffer to be true, then working with the response headers and such and finally sending response.flush.

    I saw that Chrome would have this problem only on the first call, if I send the call again it works - and thats because it was falling back to http 1.1. Disabling http2 worked but thats not a good solution.

    I added response.end after the response.flush and the problem went away. So take this for what it is worth - for me it seemed to be connected to not ending the response and this worked fine in http 1.1 but somehow in http2 does not.

    Perhaps this can help someone else along the way.

    Thursday, April 25, 2019 4:42 PM
  • User-166041051 posted
    i get a similar erro with iis 8.5 on win 12
    when i download a zip with https it returns an error page:this site can’t be reached the error is:ERR_INVALID_RESPONSE

    1.When i use ip downloaded it directly it works
    2.this error happens sometimes not all the time
    3.after i restart the iis it works fine.

    Can you give me a help with my iis setting?
    Tuesday, July 7, 2020 10:54 PM
  • User771621181 posted

    Hi,

    The same problem exists in windows server 2019 (windows is up to date, there is no updates).

    Albigi can you check if there is a solution for windows server 2019. (kb4093119 update is for windows server 2016. Can you check whether this solution is found in windows server 2019, if not is it known when it will be?)

    Best Regards,

    Milos

    Monday, October 5, 2020 6:24 PM
  • User-306773566 posted

    Hi

    This does not work for us. One thing, we have Windows Server 2019.

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

    EnableHttp2Tls REG_DWORD to 0

    EnableHttp2Cleartext REG_DWORD to 0

    Wednesday, December 16, 2020 9:56 AM
  • User-306773566 posted

    Does it work out for you finally?

    We still get the intermittent error when downloading the files, sometimes they stopped, sometimes not. I’ve checked and tried all the suggestions.

    We are now testing with HTTP (to exclude any certificate issues, however we checked the certificate is ok)

    We have a Windows Server 2019 (1809) Standard, with disabled http/2 (using http/1.1)

    Any solution yet? Any ideas please?

    Thank you.

    Wednesday, December 16, 2020 11:39 AM