IIS SEO Toolkit Not Crawling Sites w/ TLS 1.0 Disabled?

Question

User458895643 posted

I can't seem to crawl sites that have TLS 1.0 disabled, e.g. lelabofragrances.com

...But I don't know of anOTHER site w/ TLS 1.0 disabled to try, so I can't confirm it's all non-TLS 1.0-friendly sites, or just that one.

I already have this update, which might allow TLS 1.1/1.2 for IIS SEO Toolkit?
https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in

Any thoughts?

More to the point: do you know of any other sites w/ TLS 1.0 disabled I could try to crawl?

Quick way to see which protocols are enabled on a site:

https://cryptoreport.websecurity.symantec.com/checker/

Thanks!

-Dan

Answer 1

User458895643 posted

UPDATE: found one other site w/ TLS 1.0 disabled. Can't crawl it either.

community.optymyze.com

...So, what's up?

Is there any way to get IIS crawler to crawl a site w/out TLS 1.0 enabled?

Answer 2

User458895643 posted

ping

Answer 3

User-795994785 posted

Hi, did you find a solution to crawl a site that has TLS 1.0 disabled?

Answer 4

User-9504584 posted

I can confirm that if a site has 1.0, 1.2 and 1.3 enabled it'll work fine, only if 1.0 is disabled will this tool not work. I've installed 4.7.x current .NET and it had no effect. 

Any thoughts from the peanut gallery? This may be an older tool but nothing out there does as good a technical SEO scan IMO.

Answer 5

User31539575 posted

Hi, same issue here, any fix available?

Answer 6

User1485604992 posted

Hi,

I found a way using Fiddler.

First install it, then go to Rules > Customize Rules, and in the Script editor add the following in the OnBeforeRequest() function at the top:

if (oSession.HTTPMethodIs("CONNECT") && oSession.HostnameIs("www.yourdomain.com")) {
  oSession["x-OverrideSslProtocols"] = " ssl3;tls1.0;tls1.1;tls1.2";
}

Answer 7

User-1428548577 posted

Can you please explain how this Fiddler configuration is supposed to fix the problem in IIS SEO Toolkit?

Answer 8

User690216013 posted

Can you please explain how this Fiddler configuration is supposed to fix the problem in IIS SEO Toolkit?

Fiddler works as a proxy, so the network diagram becomes,

Crawler ---(connection A)---> Fiddler ---(connection B)---> your site

As the crawler only accepts TLS 1.0, you can configure connection A to support TLS 1.0, while connection B can be TLS 1.1 and above.

Answer 9

User2146767023 posted

Hi,

I found a way using Fiddler.

<g class="gr_ gr_9 gr-alert gr_gramm gr_inline_cards gr_run_anim Punctuation only-ins replaceWithoutSep" id="9" data-gr-id="9">First</g> install it, then go to Rules > Customize Rules, and in the Script editor add the following in the OnBeforeRequest() function at the top:

if (oSession.HTTPMethodIs("CONNECT") && oSession.HostnameIs("www.yourdomain.com")) {
  oSession["x-OverrideSslProtocols"] = " ssl3;tls1.0;tls1.1;tls1.2";
}

Tried this it is still not working I am getting an error: 

The link to 'https://www.....com/' has resulted in an error.
Details: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Web.Management.SEO.Crawler.UrlDownloader.OnGetContent(CrawlerProcessContext context)

Answer 10

User435951521 posted

any progress to this issue?

How did you guys solve it?

I was using the <g class="gr_ gr_58 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" id="58" data-gr-id="58">seo</g> tool till August 2018 but now it stopped working and gives me " the <g class="gr_ gr_132 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" id="132" data-gr-id="132">url</g> for <g class="gr_ gr_152 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar only-ins doubleReplace replaceWithoutSep" id="152" data-gr-id="152">hyperlink</g> is broken"

Answer 11

User-1460986047 posted

Hi Guys

Hope this is not off topic - the resulting error might be the same but maybe the cause here is different - also unfortunately dont have a solution, rather looking for it myself too..

Anyway I have come across multiple sites where I get a single row (when crawling with IIS SEO toolkit) and its the "the url for hyperlink is broken" result 

Always qwondered whats up and came across this: https://weblogs.asp.net/jeffwids/iis7-search-engine-optimization-toolkit-does-not-like-html5-doctype

And in the comments there: "Actually, it's not the HTML5 tag that's causing SEO grief, it's the extra whitespace before/above the tag that is the problem. ..." <- That seems to be true in all the cases I have come across.

So would anyone with any IIS SEO TOOLKIT customization experience know how to fix this simple but irritatting issue?

Answer 12

User2146767023 posted

For those who might have the same issue.

It is not encouraging that MS has completely abandoned it and also not released the source code. There is no better solution in the market except for a couple of paid solutions that is not as intuitive as this tool.

I had the same issue and have come up with a solution that works. It involves, using the IIS-SEO dll and then calling the remote server with tls 1.2. I am still using the UI created by Microsoft IIS but planning to replace it with my own when I get time or help to write it.

Here is the link with instructions: https://github.com/Officeclip/IIS-SEOCrawler

Answer 13

User-1460986047 posted

Hi jojobar

Thanks for the reply - it sounds promising.

To be hinest not entirely sure how it will work, wil investigate and try it. Wanted to make sure I thanked you first incase i never return to this forum again (..abandonment of products does that to communities ..)

Anyway, in case you see this, I also wanted to ask you a question which may become obvious as I start investigating what you provided but may help others to move faster:

I am running IIS on company machine (a virtual machine in this case) which means some limitations exist in how I am able to confifgure/ install sofwatware on that machine - would you expect this to be a possible issue when it comes to implementing your solution?

And thanks again - you are my hereo (and no this isnt aprils fools despite the date) - much appreciated, its sad a very decent tool goes to waste if we cant get past things like these

S.