none
Web Client Download so slow RRS feed

  • Question

  • Program to retrieve stock prices:
    Form -

    Code -

    Option Strict On
    Imports System.Net
    
    Public Class Form1
        Private Function YahooToDate(Yahoo As String) As String
            Return DateAdd(DateInterval.Second, CDbl(Yahoo), New Date(1970, 1, 1)).ToString("yyyyMMdd")
        End Function
        Private Function DateToYahoo(DateToFix As Date) As String
            Dim Date1 As New Date(1970, 1, 1)
            Return (DateDiff(DateInterval.Day, Date1, DateToFix) * 86400).ToString
        End Function
        Private Function TrimData(DataIN As String) As String
            Dim ReturnString As String = ""
            Dim Startpoint As Integer = DataIN.IndexOf(":{" & Chr(34) & "prices" & Chr(34) & ":[{")
            If Startpoint = -1 Then Return ("Start Phrase Not Found - Bad Date ?")
            Dim Endpoint As Integer = DataIN.IndexOf("}]", Startpoint)
            If Endpoint = -1 Then Return ("End Phrase not found - Should not happen")
            ReturnString = DataIN.Substring(Startpoint, Endpoint - Startpoint)
            Return ReturnString.Replace(":{" & Chr(34) & "prices" & Chr(34) & ":[{", "")
        End Function
        Private Sub BtnTest_Click(sender As System.Object, e As System.EventArgs) Handles BtnTest.Click
            Dim reply As String
            TxtResults.Text = "Start at " & Now.ToLongTimeString & vbNewLine
            Using client As WebClient = New WebClient()
                client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36")
                Dim TheURL As String
                TheURL = "https://finance.yahoo.com/quote/" & TxtTicker.Text & "/history?period1=" & DateToYahoo(DTPDate.Value) & _
                                                            "&period2=" & DateToYahoo(DateAdd(DateInterval.Day, 1, DTPDate.Value)) & _
                                                            "&interval=1d&filter=history&frequency=1d"
                TxtResults.AppendText(TheURL & vbNewLine)
                Try
                    reply = client.DownloadString(TheURL).Replace(Chr(10), vbNewLine.ToCharArray)
                    TxtResults.AppendText("Download Finished " & Now.ToLongTimeString & vbNewLine)
                    TxtResults.AppendText(TrimData(reply) & vbNewLine)
                    TxtResults.AppendText("Trimming Finished " & Now.ToLongTimeString & vbNewLine)
                Catch ex As Exception
                    MessageBox.Show("Server returned error:" & vbNewLine & ex.Message, "Error retreiving Data", MessageBoxButtons.OK, MessageBoxIcon.Error)
                    reply = "No Data for the Date"
                End Try
            End Using
        End Sub
    
        Private Sub Form1_Load(sender As System.Object, e As System.EventArgs) Handles MyBase.Load
            DTPDate.Value = DateAdd(DateInterval.Day, -1, Now)
        End Sub
    End Class
    
    It works but here it takes 40 + seconds to download the page, if I paste the URL into a browser, it loads and displays in a second or two. Tried various user agent strings, no difference. Obviously I'd like to know if and how it can be speeded up but also, it is that slow for others? - I have a fast connection (300/20 MBit).

    Here is a typical run:
    Start at 2:32:36 PM
    https://finance.yahoo.com/quote/AAPL/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d
    Download Finished 2:33:20 PM
    "date":1493645400,"open":145.10000610351562,"high":147.1999969482422,"low":144.9600067138672,"close":146.5800018310547,"volume":33602900,"unadjclose":146.5800018310547
    Trimming Finished 2:33:20 PM

    I haven't pursued cleaning up the output yet due to the slowness. FYI - Indexes must be prefaced with "^":
    "DJI" will not work, must be "^DJI"

    Thursday, May 18, 2017 9:50 PM

Answers

  • Frank - Yes, Yahoo HAD a web-based API. As best as I can tell, It has been removed.

    Using your GetStringFromURL results:

    Start at 4:37:41 PM
    https://finance.yahoo.com/quote/AAPL/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d
    Download Finished 4:38:25 PM
    **Data removed for brevity**
    Trimming Finished 4:38:25 PM

    Took 44 seconds. . .

    That's incredible; I timed it and it was less than two seconds.

    *I don't know...*


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    • Marked as answer by Devon_Nullman Friday, May 19, 2017 11:10 PM
    Thursday, May 18, 2017 11:46 PM

All replies

  • Can't really test since your code depends on controls and data which we don't have access to.

    Lloyd Sheen

    Thursday, May 18, 2017 9:59 PM
  • Devon,

    You might want to reconsider how you're getting it:

    Option Strict On
    Option Explicit On
    Option Infer Off
    
    Imports System.Net
    Imports System.IO
    
    Public Class Form1
        Private Sub Form1_Load(sender As System.Object, _
                               e As System.EventArgs) _
                               Handles MyBase.Load
    
            Const url As String = _
                "https://finance.yahoo.com/quote/AAPL/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d"
    
            Dim sw As New Stopwatch
            sw.Start()
    
            Dim returnValue As String = GetStringFromURL(url)
    
            sw.Stop()
    
            Stop
    
        End Sub
    
        Private Function GetStringFromURL(ByVal url As String) As String
    
            Dim retVal As String
            Dim sb As New System.Text.StringBuilder
    
            Try
                Dim request As WebRequest = WebRequest.Create(url)
    
                Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)
                    Using dataStream As Stream = response.GetResponseStream
                        Using rdr As New StreamReader(dataStream)
                            sb.Append(rdr.ReadToEnd)
                        End Using
                    End Using
                End Using
    
                retVal = sb.ToString
    
            Catch ex As Exception
                retVal = Nothing
            End Try
    
            Return retVal
    
        End Function
    
    End Class

    If you then use HTML AgilityPack, you should be able to parse through the returned string more easily than you're currently doing.

    *****

    I thought that Yahoo had a Web-based API?


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, May 18, 2017 10:54 PM
  • Frank - Yes, Yahoo HAD a web-based API. As best as I can tell, It has been removed.

    Using your GetStringFromURL results:

    Start at 4:37:41 PM
    https://finance.yahoo.com/quote/AAPL/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d
    Download Finished 4:38:25 PM
    **Data removed for brevity**
    Trimming Finished 4:38:25 PM

    Took 44 seconds. . .
    Results with a stopwatch added:

    https://finance.yahoo.com/quote/^DJI/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d
    Download Finished in 43.495 seconds
    Data was here
    Trimming Finished in 0.012 Seconds
    
    Did you get significantly faster results ?


    • Edited by Devon_Nullman Thursday, May 18, 2017 11:53 PM Added Content
    Thursday, May 18, 2017 11:41 PM
  • Frank - Yes, Yahoo HAD a web-based API. As best as I can tell, It has been removed.

    Using your GetStringFromURL results:

    Start at 4:37:41 PM
    https://finance.yahoo.com/quote/AAPL/history?period1=1493596800&period2=1493683200&interval=1d&filter=history&frequency=1d
    Download Finished 4:38:25 PM
    **Data removed for brevity**
    Trimming Finished 4:38:25 PM

    Took 44 seconds. . .

    That's incredible; I timed it and it was less than two seconds.

    *I don't know...*


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    • Marked as answer by Devon_Nullman Friday, May 19, 2017 11:10 PM
    Thursday, May 18, 2017 11:46 PM
  • Hi Devon_Nullman,

    Even though GlobalProxySelection.GetEmptyWebProxy() has become deprecated, this is the only way I was able to speed up the WebClient requests.

    More detailed info, please refer to GlobalProxySelection.GetEmptyWebProxy Method ()

    Hope it is helpful to you.

    Best Regards,

    Cherry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Friday, May 19, 2017 5:39 AM
    Moderator
  • Frank - thanks, your comment made me think it was something computer related so I copied the project to another pc and sure enough, each request was around one second. I then realized that I had just installed a new AVG. Disabled the "shield" and firewall and that did the trick, called AVG and they fixed the issue.

    • Proposed as answer by Cor Ligthert Saturday, May 20, 2017 4:32 AM
    Friday, May 19, 2017 11:12 PM
  • Thanks Cherry - If you look at my response to Frank, it was the security software I had just upgraded on my PC. All is fixed now.

    Friday, May 19, 2017 11:19 PM
  • Frank - thanks, your comment made me think it was something computer related so I copied the project to another pc and sure enough, each request was around one second. I then realized that I had just installed a new AVG. Disabled the "shield" and firewall and that did the trick, called AVG and they fixed the issue.

    Ah ok

    Makes sense

    :)


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Saturday, May 20, 2017 12:24 AM
  • Frank - thanks, your comment made me think it was something computer related so I copied the project to another pc and sure enough, each request was around one second. I then realized that I had just installed a new AVG. Disabled the "shield" and firewall and that did the trick, called AVG and they fixed the issue.

    Devon,

    Would you not mark this reply as well as answer, otherwise it gives confusion by those who read the answer.


    Success
    Cor

    Saturday, May 20, 2017 4:34 AM
  • Frank - thanks, your comment made me think it was something computer related so I copied the project to another pc and sure enough, each request was around one second. I then realized that I had just installed a new AVG. Disabled the "shield" and firewall and that did the trick, called AVG and they fixed the issue.

    Devon,

    Would you not mark this reply as well as answer, otherwise it gives confusion by those who read the answer.


    Success
    Cor

    I don't understand - you want me to mark my own comment as an answer ?

    Saturday, May 20, 2017 5:31 AM
  • Yes, you get no points, however it becomes beneath the one from Frank in the answers and if somebody finds this because they have a likewise problem like you. They see your main problem was your viruschecker/firewall

    Success
    Cor

    Saturday, May 20, 2017 8:20 AM
  • I don't understand - you want me to mark my own comment as an answer ?

    It does make one wonder why Cor would insert himself into a thread where he otherwise has no interest.

    It's ... quite curious...


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Saturday, May 20, 2017 8:54 AM