locked
Create a site map for my asp.net mvc web site , to show it inside the search engine results RRS feed

  • Question

  • User-540818677 posted

    I have created a web site using asp.net mvc-5 . and I have hosted it to one of the hosting provider and I register the web site with Google app.

    Now after 1.5 months of publishing the web site online it starts to be returned by search engines mainly Google and Bing. But the search result will not show my web site links (such as contact us, our projects, our team, etc). now I read that I need to create a site map for my web site as follow:-

    1. I need to create a robots.txt inside the root of my asp.net mvc and I need to specify the location of the site map as follow:-

    User-agent: *
    Disallow:
    
    Sitemap: http://mywebsite.com/sitemap

    2. Then I need to create an xml which list my main links.

    But I have the following 2 questions regarding creating and submitting a site map:-

    1. If I do the above steps (creating robots.txt and xml site map), then will search engines be able to understand the site map automatically, or I will have to submit the site map to Google and to Bing separately ?

    2. Is there any change that my site map will be returned in the search result without creating a sitemap.xml . I mean can search engine automatically extract my site urls and create a site map ?

    Thanks

    Sunday, January 17, 2016 2:36 AM

Answers

All replies

  • User1124521738 posted

    the main reason to create a site map is to link in content that you want to make sure gets crawled, you can also indicate how often is should be re-checked and provide a last updated timestamp and weights for importance. 

    in earlier days on the internet, many folks used DHTML for menus, and today, many people (still) use flash, neither lend themselves well to being crawled, so a sitemap was a way to enumerate critical content you wanted to make sure gets included by the search engines.  you don't have to make it sitemap.xml, you can do /sitemap as a route or /sitemap.ashx and have the output be generated programmatically rather than having to use a static file that you manually update. 

    If you have plain ol' anchortags/hyperlinks the search engines can crawl your content by getting all the interlinking content, but a sitemap is a way to highlight what content you see as important, and the search engines can check it first to see what you broadcast as new, rather than it having to perform constant crawls to see if there is something new there.

    you can see how I've handled this in an mvc project if you visit https://github.com/ninianne98/CarrotCakeCMS-MVC/blob/master/CMSAdmin/Controllers/CmsContentController.cs see SiteMap() and https://github.com/ninianne98/CarrotCakeCMS-MVC/blob/148edcabc6a41203542cd5e8238478e06bc8c475/CMSCore/SiteContent/SiteMapHelper.cs

    see also https://support.google.com/webmasters/answer/156184?hl=en

    Sunday, January 17, 2016 4:59 AM
  • User-540818677 posted

    If you have plain ol' anchortags/hyperlinks the search engines can crawl your content by getting all the interlinking content, but a sitemap is a way to highlight what content you see as important, and the search engines can check it first to see what you broadcast as new, rather than it having to perform constant crawls to see if there is something new there

    First thanks a lot for your valuable reply... now I start understanding this process having say this; then in my case I can not find a reason to even have a robots.txt file.. as my website is an informative web site about our company and I am ok with having search engine crawl all my web pages without any "disallow",,, so can you adivce on these 2 questions:-

    1. what I will get if I have a robots.txt compared to not having it?since in the robots.txt i can list any disallow pages to not be crawled which do not apply in my case, also since site map is not necessary so i do not have to mention the url to find the site map...

    2. second question what will be the benefits of having sitemap compared to not having a site map, as in my case i have my menu build up using anchortags/hyperlinks and i do not use flash ot dhtml  ?

    Sunday, January 17, 2016 3:10 PM
  • User-219423983 posted

    Hi john,

    what I will get if I have a robots.txt compared to not having it?

    About the robots.txt file, you should first know that the purpose of it is to keep crawlers out of certain parts of your website. Not having one should result in all your content being indexed. For more things, you could have a look at the following links.

    http://serverfault.com/questions/154820/what-happens-if-a-website-does-not-have-a-robots-txt-file

    https://developers.google.com/webmasters/control-crawl-index/docs/faq?hl=en#h01

    https://www.koozai.com/blog/search-marketing/a-brief-guide-to-robots-txt-and-five-mistakes-to-avoid/

    what will be the benefits of having sitemap compared to not having a site map

    Using sitemaps has many benefits, not only easier navigation and better visibility by search engines. Sitemaps offer the opportunity to inform search engines immediately about any changes on your site. Of course, you cannot expect that search engines will rush right away to index your changed pages but certainly the changes will be indexed faster, compared to when you don't have a sitemap. For more things about this, you could read the below link carefully, especially the section “Why Use a Sitemap”.

    http://www.webconfs.com/importance-of-sitemaps-article-17.php

    Best Regards,

    Weibo Zhang

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, January 28, 2016 8:31 AM