Rule works on one server but does not work on another, pull my hair out RRS feed

  • Question

  • User-1328420699 posted


    Okay so I have a Windows Server 2016 box and Windows 10 Enterprise Oct 2018 update box, both utilising real IIS and the same version of the web site installed (MS .NET MVC, SQL Server etc..).

    On our site we often get hit by bots that don't respect the robots.txt file and will spam the site with thousands of calls a minute. To prevent this we've added the following rewrite rule that should allow Googlebot and Bingbot but any other request whose user agent string mentions one of the terms in the rules expression will be immediately aborted.

    		<rule name="Bot Blocking" stopProcessing="true">
    			<match url=".*" />
    			<conditions logicalGrouping="MatchAll">
    				<add input="{HTTP_USER_AGENT}" pattern="^((?!(GoogleBot|BingBot)).)*$" />
    				<add input="{HTTP_USER_AGENT}" pattern="^.*(bot|crawler|kicks|scanner|spider).*$" />
    			<action type="AbortRequest" />

    The problem is on the Windows Server 2016 box, it is not aborting the requests when it mentions, for example, "crawler" in the user agent string. The Windows 10 IIS box works as expected. 

    I found this post https://stackoverflow.com/questions/20637592/iis-url-rewrite-works-one-old-server-but-not-on-new and added the rewrite portion from the answer to the applicationhost.config file it indicates inside the <system.webserver> element but the Server 2016 box still fails to apply the rule correctly.

    I'm at a total loss as to why there is a difference. Any ideas of why this isn't working or how to fix it would be very much appreciated.



    [UPDATE 1] If I RDP into the server having the issue, load IIS manager, edit the rule and use the test pattern dialog and paste in a user agent string that should be blocked the result is a match. Given the problematic IIS server can match the rule using it's own test then why the hell does it not match when the browser sends the exact same text as the user agent? Urgh! I hate these issues the most.

    [UPDATE 2] The server was wrong, it's Windows Server 2016 where it's failing.

    Thursday, April 4, 2019 12:37 PM

All replies

  • User-2064283741 posted

    The best way to try and solve this or any URLrewrite problem is to use failed request tracing.


    Then you will see more clues as to what failed and why.

    Thursday, April 4, 2019 6:52 PM
  • User-1328420699 posted

    Using the link provided I'm running into issues since the rewrite option is not available as one to create trace log for.

    The follow on link in the article you linked to that explains how to solve the missing rewrite log category.

    The config file bit it suggests doesn't work as there is no traceDefinitionProviders section defined and if I put the whole section in from the article IIS fails to start at all.

    The repair option is currently outstanding because the location/file the URL rewrite module was installed from is no longer accessible so I next need to find it on an MS site and try repairing. Alternatively using the web platform installer to uninstall/reinstall it. But haven't had time to do that yet. 

    Wednesday, April 10, 2019 12:25 PM
  • User-2064283741 posted

    You will need to install tracing to really see what is happening. And compare dev to prod with the same requests in a side-by-side test.

    Other than that have you tried to recreate the exact conditions on prod and dev?

    e.g. you have a actual user agent that fails on prod but works on dev? (using for example some user-agent switcher in the browser)

    Do ANY get blocked?! (that is probably tricky to tell it might be better initially to pass back a status code like 403 or something (maybe with a unique subcode) instead of an abort request so you can see them in the logs.)


    Wednesday, April 10, 2019 5:27 PM