IIS sometimes serves the wrong certificate

Question

User1392028997 posted

Hi all,

We run IIS with ARR and NLB on two machines. We followed an installation and setup guide we found online and in general we think the configuration is working now. We have central IIS configuration and a central certificate store setup.

The issue we're facing is that for some (!) requests IIS uses the wrong certificate. This results in an error SSL_ERROR_BAD_CERT_DOMAIN at the client side. The end result abviously is that sometimes pages (or images for instance) don't get loaded.

ARR uses rewrite rules to direct traffic to the webfarm. We have only one clusting IP-address so we use SNI to host multiple websites using HTTPS on the NLB cluster. The certificates are generated using Let's Encrypt and are stored in the central certificate store.

We've verified the configuration in IIS over and over again. At this point we have no idea where to look further, so we hope someone here might be able to give us some pointers.

Thanks in advance!

Kind regards,

Raymond

Answer 1

User690216013 posted

rbrink

We run IIS with ARR and NLB on two machines.

Describe further on your setup please, such as,

• Where the certificate is installed
• ARR HTTPS offloading or not
• Any other certificate mapping (https://docs.jexusmanager.com/tutorials/https-binding.html#background)

Answer 2

User1392028997 posted

Hi lextm,

I was hoping the description of the issue would trigger someone who's seen this before. But I guess I need to explain our environment some more.

We run two physical servers for this. Both run NLB, ARR and IIS so we have combined it all (basically to save resources). NLB is configured to run an active/active cluster in Multicast mode. Filtering mode is set to Multiple host and Affinity to None.

Both machines run a shared IIS configuration of a network share, so configuration is the same. Same goes for all websites: Their files are also stored on a network share. Also we use Central Certificate Store so certificates are stored on a network share as well.

Default webfarm routing rules are not used. We use the following Rewrite Rules:

    <rewrite>
      <globalRules>
        <clear />
        <rule name="Route to webfarm">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
                        <add input="{HTTP_X_ARR_COMPLETED_PHASE}" pattern="1" negate="true" />
          </conditions>
          <serverVariables>
                        <set name="HTTP_X_ORIGINAL_HTTP_HOST" value="{HTTP_HOST}" />
                        <set name="HTTP_X_ARR_COMPLETED_PHASE" value="1" />
          </serverVariables>
          <action type="Rewrite" url="http://MyPoi IIS Server Farm/{R:0}" />
        </rule>
        <rule name="Put HTTP_HOST back">
          <match url=".*" />
          <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
            <add input="{HTTP_X_ARR_COMPLETED_PHASE}" pattern="1" />
          </conditions>
          <serverVariables>
            <set name="HTTP_HOST" value="{HTTP_X_ORIGINAL_HTTP_HOST}" />
          </serverVariables>
          <action type="None" />
        </rule>
      </globalRules>
      <allowedServerVariables>
        <add name="HTTP_HOST" />
      </allowedServerVariables>
    </rewrite>

I don't know about any other certificate mapping (or what you mean by that). All websites are configured to use SNI and have a corresponding certificate in the Central Certificate Store (generated by Let's Encrypt).

If there is anything else you need, please let me know and I'll share it.

Thanks again,

Raymond

Answer 3

User1392028997 posted

Hi all,

So I finally found the root cause of the issue. My tip for everyone else facing iisues where IIS serves the wrong certificate: check your certificate bindings! 

By that I don't mean the website bindings you see in IIS Manager, but the certificate bindings you can see when you run netsh http show sslcert from an elevated command prompt. Do this on all webservers in your farm (and ARR cluster nodes if applicable) and compare the output!

I found the following certificate binding (the first one in the list) suspicious as it was only listed on one of the servers and it was not actually related to a binding of the application with the specified Application ID {4dc3e181-e14b-4a21-b022-59fc669b0914}. I was able to determine the actual website by looking at the other certificate bindings with the same Application ID btw.

The invalid certificate binding

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 97065df6db457d673b93b888faca3dxxxxxxxxxx
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : Root
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled
    Reject Connections           : Disabled
    Disable HTTP2                : Not Set

Solution

It was probably not the simplest way to do it, but to resolve the issue I opened the bindings of the formentioned website in IIS Manager on the affected server. I then modified (set a fixed IP), saved, undid my modification (IP back to *) and saved again. After I did that for each of the bindings I re-run netsh http show sslcert to confirm that invalid certificate binding was removed. That was the case. Please note both our servers were sharing their IIS Configuration in a central place...

There was no need te reboot IIS or the server for that matter. Right after undoing and saving the bindings again, the issue was gone. 

Last but not least I used Firefox to test this issue. Hitting Ctrl-F5 on a troubled website would more ofter result in errors than in any other browser, as far as I could tell. Investigating each request (not just page, but images, scripts, etc.) showed failing request practically with every refresh.

I sure hope someone else facing the same issue will find this post someday and will be helped by it.

Cheers,
Raymond