locked
How to Fetch Data from external Website and store that data in database dynamically RRS feed

  • Question

  • I am doing a Project in ASP.Net where i am using c# language for back end coding. Here i get problem in fetch data from external website. Actually i don't know  how to do that, also i don't know about how to use json api's in c# for fetching data from another website. If i have api's of any website then how do i implement in c#. Can you please help me.
    Friday, August 19, 2016 11:13 AM

Answers

  • First, check if that other websites TOS even allows automation.

    Secondly, ideally use any provided Web API instead. JSON, Rest, XML - does not matter.
    While you can use Webscraping, doing so is terribly ineffective.
    Also if they provide a API that means that part you can use without question.

    If you have to do Website scraping, the usual approach is the HTML Agility Pack. Or some similar library that can interpret HTML.
    So ideally just learn using JSON. Just google "JSON from .NET" should get you someplace usefull.

    Friday, August 19, 2016 11:48 AM
  • Screen scraping is a challenging process. Ultimately you'll end up having to use WebClient (or friends) to download the response.  If the response is JSON then you can use any JSON serializer to get the raw data (such as NewtonSoft). If the response is HTML then you'll need to parse the HTML. .NET doesn't have support for this but there are a couple of libraries available including HtmlAgilityPack. Unfortunately you still have a lot of work to do but it does help.

    Screen scraping is too large of a topic to talk about in the forums.  I suggest you read up on how to do it.

    Michael Taylor
    http://www.michaeltaylorp3.net

    Friday, August 19, 2016 3:00 PM

All replies

  • First, check if that other websites TOS even allows automation.

    Secondly, ideally use any provided Web API instead. JSON, Rest, XML - does not matter.
    While you can use Webscraping, doing so is terribly ineffective.
    Also if they provide a API that means that part you can use without question.

    If you have to do Website scraping, the usual approach is the HTML Agility Pack. Or some similar library that can interpret HTML.
    So ideally just learn using JSON. Just google "JSON from .NET" should get you someplace usefull.

    Friday, August 19, 2016 11:48 AM
  • Screen scraping is a challenging process. Ultimately you'll end up having to use WebClient (or friends) to download the response.  If the response is JSON then you can use any JSON serializer to get the raw data (such as NewtonSoft). If the response is HTML then you'll need to parse the HTML. .NET doesn't have support for this but there are a couple of libraries available including HtmlAgilityPack. Unfortunately you still have a lot of work to do but it does help.

    Screen scraping is too large of a topic to talk about in the forums.  I suggest you read up on how to do it.

    Michael Taylor
    http://www.michaeltaylorp3.net

    Friday, August 19, 2016 3:00 PM