User986200253 posted
I think maybe this is what you're looking for?
"You can point the site analysis tool in the Toolkit at the top-level URL of your web-site, and
it will automatically crawl your site like a search engine would – following each link, reviewing the HTML on each page, and generating a report of every SEO or content violation it finds on the site - along with suggestions on how to fix each of them.
The SEO toolkit also makes it easy to optimize which content on your Web site gets indexed by search engines. You can manage robots.txt files, which search engine crawlers use to understand which URLs are excluded from the crawling process. You can also manage
sitemaps, which provide URLs for crawling to search engine crawlers. You can also use the SEO Toolkit to provide additional metadata about the URL, such as last modified time, which search engines take into account when calculating relevancy in search results."
From
Getting Started With the SEO Toolkit.
