none
vb.net download file from url with credentials FedEx.com RRS feed

  • Question

  • I am trying to download delivery information from FedEx's website.  I have to pass login credentials before I can download file.  Using WebBrowser method I can login and download file but must deal with Windows UI SaveAs dialog box.  I don't like this method as it's very sloppy,  I have dealt with that using sendkeys but this posses a issue with running .net application in a windows task  scheduler that has UI manipulation to fully automate.  Sample code below:

    Public Class Form1
    
        Dim ctr As Integer
    
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    
            btnGet_Click(sender, e)
    
        End Sub
    
        Private Sub btnGet_Click(sender As Object, e As EventArgs) Handles btnGet.Click
    
            If System.IO.File.Exists("C:\Temp\FedExInSight_Inbound.csv") = True Then
                My.Computer.FileSystem.DeleteFile("C:\Temp\FedExInSight_Inbound.csv")
            End If
    
            ctr = 10
            Timer1.Interval = 1000
    
            btnGet.Enabled = False
            'WebBrowser1.AllowNavigation = True
            WebBrowser1.ScriptErrorsSuppressed = True
            WebBrowser1.Navigate("https://www.fedex.com/insight/manifest/download_post.jsp")
    
            Timer1.Start()
    
        End Sub
    
    
        Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
    
            If WebBrowser1.Url.ToString = "about:blank" Then
                Exit Sub
            End If
    
            WebBrowser1.Document.GetElementById("username").SetAttribute("value", "MyUserName")
            WebBrowser1.Document.GetElementById("password").SetAttribute("value", "MyP@ssw0rd")
            WebBrowser1.Document.GetElementById("login").InvokeMember("click")
    
        End Sub
    
    
        Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
    
            If ctr = 1 Then
                SendKeys.Send("{TAB}")
                SendKeys.Send("{TAB}")
                SendKeys.Send("{TAB}")
                SendKeys.Send("{ENTER}")
                Timer1.Stop()
                ctr = 5
                Timer2.Start()
                Exit Sub
            Else
                lblTimer.Text = "Wait: " & ctr - 1 & " s"
            End If
            ctr = ctr - 1
        End Sub
    
        Private Sub Timer2_Tick(sender As Object, e As EventArgs) Handles Timer2.Tick
    
            If ctr = 1 Then
                SendKeys.Send("C:\Temp\FedExInSight_Inbound.csv")
                SendKeys.Send("{ENTER}")
                Timer2.Stop()
                btnGet.Enabled = True
                lblTimer.Text = "Downloaded!"
                WebBrowser1.Navigate("about:blank")
                Exit Sub
            Else
                lblTimer.Text = "Wait: " & ctr - 1 & " s"
            End If
            ctr = ctr - 1
        End Sub
    
    End Class

    So with Code above is it possible to suppress dialog box so it can be ran in windows task scheduler?  

    Now I think the preferred method would be using a WebClient but the very simple code below does not work and just download the page's source.  

            Dim rDate As String = DateTime.Now.Date.ToString("MMddyy")
    
            Try
    
                Dim client As New WebClient()
                client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)")
    
    
                Dim params = New NameValueCollection
                params.Add("username", "MyUserName")
                params.Add("password", "MyP@ssw0rd")
    
                ' login to website
                'client.UploadValues("https://www.fedex.com/insight/login/index.jsp", params)
                client.UploadValues("https://www.fedex.com/insight/manifest/manifest.jsp", params)
                'client.UploadValues("https://www.fedex.com/insight/entrance/pre_entrance_post.jsp", params)
                'client.UploadValues("https://www.fedex.com/insight/entrance/entrance_post.jsp", params)
                'client.UploadValues("https://www.fedex.com/fcl/logon.do", params)
    
    
                ' download report
                Dim responseData = client.UploadValues("https://www.fedex.com/insight/manifest/download_post.jsp", params)
                File.WriteAllBytes("FedExInSight_Inbound-" & rDate & ".csv", responseData)
    
    
                ' logout
                client.UploadString("https://www.fedex.com/insight/home/logout.jsp", "")
    
            Catch ex As Exception
                MsgBox(ex.ToString)
            End Try

    I think the issue is in the logon redirect?  in debugging using Chrome F12  shows it does a redirect to https://www.fedex.com/fcl/logon.do (302 Moved Temporarily)  but if that is the login URL you get a Forbidden when trying from browser.  really don't know where to go from here. any help is greatly appreciated.  

    Link of someone with same issue: https://stackoverflow.com/questions/28434295/vb-net-downloading-a-file-without-popups 


    Work Smarter Not Harder





    Thursday, May 17, 2018 5:36 PM

Answers

  • I'm sorry, but this practice is against the terms of use of the Fed-Ex website (and most websites for that matter).

    "The use of non-authorized scripting technologies to obtain information from fedex.com or submit information through fedex.com is strictly prohibited."

    You will need to utilize one of the official automation methods available from their developer website.

    It is against the policies of this forum to assist with code that would violate another company's usage policy.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    • Marked as answer by jimbrown.net Thursday, May 17, 2018 6:01 PM
    Thursday, May 17, 2018 5:45 PM
    Moderator

All replies

  • I'm sorry, but this practice is against the terms of use of the Fed-Ex website (and most websites for that matter).

    "The use of non-authorized scripting technologies to obtain information from fedex.com or submit information through fedex.com is strictly prohibited."

    You will need to utilize one of the official automation methods available from their developer website.

    It is against the policies of this forum to assist with code that would violate another company's usage policy.


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    • Marked as answer by jimbrown.net Thursday, May 17, 2018 6:01 PM
    Thursday, May 17, 2018 5:45 PM
    Moderator
  • FWIW, their site is likely working against the code you are using specifically to prevent this kind of access.

    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Thursday, May 17, 2018 5:46 PM
    Moderator