none
Get Request to nasdaq.com not functioning RRS feed

  • Question

  • Hello, I am having a terrible time trying to capture data from a nasdaq website and am hoping someone knows what is happening. This situation is easy to describe. I can capture data from a finance.yahoo website but the same code fails when I try to capture data from a nasdaq website.

    Is nasdaq screening C# get requests?

    public static async Task Testing() { ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; using (HttpClient client = new HttpClient()) { var url1 = string.Format(https://old.nasdaq.com/symbol/msft/financials?query=income-statement);

    // this program will not get the data from the old.nasdaq website.

    var url1 = string.Format(https://finance.yahoo.com/quote/msft/balance-sheet?p=msft);

    // this program will get the data from the finance.yahoo website

    Task<string> ISA_Task = client.GetStringAsync(url1); string dls = await ISA_Task;

    Console.WriteLine(dls);

    Console.ReadKey(); } }



    Robert Meyer

    Sunday, November 10, 2019 5:47 PM

All replies

  • For one thing, it is very likely they want to inhibit you from getting their data without paying for it. So of course it is not easy.

    Second, many websites generate content from a script. So if you get the raw HTML then you do not get the data. Even the script is likely in a separate file.



    Sam Hobbs
    SimpleSamples.Info


    Sunday, November 10, 2019 6:46 PM
  • Thanks for your thoughts Sam however the website in question is freely accessed via https. I can access it in my browser but cannot access it via my C# application. Again the website in question is:

    https://old.nasdaq.com/symbol/msft/financials?query=income-statement


    Robert Meyer

    Sunday, November 10, 2019 7:53 PM
  • If the data is freely available then why scrape it? Why not get the data using whatever interface (web service, API or whatever) that they offer? Such as:

    Except those services cost, right? That is what I mean; they want you to pay for the data unless you view it in their website.



    Sam Hobbs
    SimpleSamples.Info

    Sunday, November 10, 2019 8:14 PM
  • The web service I use to access the data is I.E and I must say that trying to access 5000 individual tickers one click at a time takes too long. C# is the best way to access the data in a timely manner but for some reason I am unable to scrape what I want because the website isn't accessible. I know it can be done, just am having problems getting it done. So after several months of reading, studying, testing, and pulling my hair out I'm reaching out for some experienced help.

    Robert Meyer

    Sunday, November 10, 2019 9:56 PM
  • The web service I use to access the data is I.E and I must say that trying to access 5000 individual tickers one click at a time takes too long.

    Do you understand what a web service is? There are no clicks used by a web service.



    Sam Hobbs
    SimpleSamples.Info

    Sunday, November 10, 2019 10:04 PM
  • Did you not understand my meaning when I said I was using I.E.

    Robert Meyer

    Sunday, November 10, 2019 10:16 PM
  • Then you do not understand what web services are. They use HTTP, the thing that existed before IE did. So IE supports them.

    Go to REST test test... in IE. The left side has a default URL. Click Ajax request. You will get JSON in the right side. It works in IE.

    That is an interactive interface but REST is intended for use in programs without human interaction. A program can submit a request (as in the left side) and get JSON back (as in the right side) that can be processed by the program. It can all be done without human interaction.

    Try the following also.

    • https://jsonplaceholder.typicode.com/todos/1
    • https://jsonplaceholder.typicode.com/posts/42

    The Nasdaq Basic Real-Time Prices API says it is accessible via WebSocket and a REST Web API.



    Sam Hobbs
    SimpleSamples.Info

    Monday, November 11, 2019 12:17 AM
  • My original question still remains - can someone help me understand why I can scrape the yahoo website and not the nasdaq website?

    Robert Meyer

    Monday, November 11, 2019 3:21 AM
  • I answered that already. It is speculation but general enough that it is nearly certainly relevant.

    This is getting nowhere; I will unsubscribe from this.



    Sam Hobbs
    SimpleSamples.Info

    Monday, November 11, 2019 4:14 AM
  • Hi BobbyBob711, 

    Thank you for posting here.

    According to your description, I make a test based on your code and reproduce your problem.

    ‘HttpClient.GetStringAsync’ method cannot get effective response from the old nasdaq website.

    Therefore, the problem is caused by the website, and as Simple Samples suggested, you can use interfaces they provide to get the data.

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, November 11, 2019 9:45 AM
    Moderator
  • I can tell some are becoming annoyed with my original question because it seems I am seeking information that nasdaq offers via an API however this is not entirely accurate. Simply said - I am seeking to data mine the fundamentals and as best as I can tell there is not one simple solution to this challenge. Scrapping the data one web page at a time is  the best route to go and nasdaq.com has the best fundamental data.

    Given that Microsoft spends billions of dollars per year developing software solutions means C# can definitely be used to accomplish this nasdaq challenge. I know it can be done but I don't possess the .Net skills to gett'er done.

    I have used Fiddler to examine the http headers being sent  to finance.yahoo and old.nasdaq and I see nothing that stands out that is preventing me from scraping old.nasdaq into a usuable variable. Does anyone know why my browser can access old.nasdaq but my C# console app cannot?


    Robert Meyer

    Tuesday, November 12, 2019 4:14 AM
  • Hi BobbyBob711,

    Thanks for your feedback.

    There is no problem in C# code, and the different HTML pages between old nasdaq and finance yahoo website is the key to the problem which may be caused by the original design of website.

    Best Regards,

    Xingyu Zhao



    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, November 13, 2019 8:58 AM
    Moderator
  • The site itself is really messed up. Even in the browser it is reporting lots of errors. Curl works fine for it which indicates that it is probably returning a header that the client code is trying to do something with. You can refer to this older doc about the various options that are used by the clients. From there I would use Fiddler to monitor a curl call (which the PS version uses GetWebRequest) to see what it is sending and receiving. Then replicate that in the handler for your client call.

    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, November 13, 2019 4:01 PM
    Moderator
  • Thank you Mr. Michael for your thoughts - I am investigating...

    Robert Meyer

    Friday, November 15, 2019 4:20 AM