locked
How to read and give value to a website with http web requests? RRS feed

  • Question

  • Hi Everybody,

    I want to set and get value to/from a webpage in this case:

    test page

    I would like to go through this problem without using any webbrowser, also because the IE version is not supported by the website. I dont want to change the version of IE by regedit, so I hope there is a way to use HTTP Web Requests.

    Thanks

    Monday, August 31, 2020 11:56 PM

All replies

  • Learn how to work with developer tools of your web browser to figure out things like element id's and so on then apply it via HtmlAgility pack. This code gives me the TextArea (first put some text into the text area) and the sole link on the page.

    See examples and docs here https://html-agility-pack.net/

    Imports System.Net
    Imports HtmlAgilityPack
    Imports HtmlDocument = HtmlAgilityPack.HtmlDocument
    
    Public Class Form1
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            Using client = New WebClient()
                ' Download the HTML
                Dim html As String = client.DownloadString("https://primapad.com/music_all")
    
                ' Now feed it to HTML Agility Pack:
                Dim doc As New HtmlDocument()
                doc.LoadHtml(html)
    
                ' Now you could query the DOM. For example you could extract
                ' all href attributes from all anchors:
                For Each link As HtmlNode In doc.DocumentNode.SelectNodes("//a[@href]")
                    Dim href As HtmlAttribute = link.Attributes("href")
                    If href IsNot Nothing Then
                        Console.WriteLine(href.Value)
                    End If
                Next link
    
    
                Dim node As HtmlNode = doc.DocumentNode.SelectSingleNode("//*[@id=""conteudo""]")
                Dim value As String = If(node Is Nothing, "Error, id not found", node.InnerHtml)
    
                Console.WriteLine($"Value [{value}]")
            End Using
    
        End Sub
    End Class
    


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Tuesday, September 1, 2020 1:23 AM
  • Hi Karen,

    thanks for your answer. 

    Using html agility pack wasn't on my ideas, but if it brings it to work then I'll be happy.

    I've just tried your code in a new project, I downloaded Nuget project html agility pack, i did add it as a reference and, with a single button, I use your code. 

    It just bring me to nothing. Nothing appear and no error code either. 

    I thought It 'll make appear the textarea text. Am I wrong?

    Thanks,

    Mattia Fanti

    Wednesday, September 2, 2020 12:41 PM
  • Hi Mattia Fanti,

    Thanks for your feedback.

    I make a test based on Karen's code, and successfully get the value in the notepad.

    Code:

                Using client = New WebClient()
                Dim html As String = client.DownloadString("https://primapad.com/music_all")
    
                Dim doc As New HtmlDocument()
                doc.LoadHtml(html)
    
                Dim node As HtmlNode = doc.DocumentNode.SelectSingleNode("//*[@id=""conteudo""]")
                Dim value As String = If(node Is Nothing, "Error, id not found", node.InnerHtml)
    
                MessageBox.Show($"{value}")
    

    Result of my test:

    The code searchs the first node that matches "id = "conteudo"", and then get the inner html text in it.

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, September 3, 2020 6:03 AM
  • Hi Xingyu Zhao,

    thanks for your answer. Endeed, I'm sorry, I've just tried again today and it gives me back the textarea value!

    Unluckly, I have to mention, I'm realizing this way is freezing the form for not less than 2 seconds.. There is any way to let it run smoothly?

    Thanks

    Mattia Fanti

    EDIT1: Would it be possible to suggest me an examples that drives me to an answer through Html agility pack examples about in case I want to change the value of Textarea by software?

    Still thanks

    • Edited by Mattia Fanti Thursday, September 3, 2020 10:44 AM
    Thursday, September 3, 2020 10:33 AM
  • Hi Mattia Fanti,

    >>There is any way to let it run smoothly?

    You can use async method:

        Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            Dim task As Task = Task.Run(Sub()
                                            Using client = New WebClient()
                                                Dim html As String = client.DownloadString("https://primapad.com/music_all")
    
                                                Dim doc As New HtmlDocument()
                                                doc.LoadHtml(html)
    
                                                Dim node As HtmlNode = doc.DocumentNode.SelectSingleNode("//*[@id=""conteudo""]")
                                                Dim value As String = If(node Is Nothing, "Error, id not found", node.InnerHtml)
    
                                                MessageBox.Show($"{value}")
                                            End Using
                                        End Sub)
            Await task
        End Sub

    >>Would it be possible to suggest me an examples that drives me to an answer through Html agility pack examples about in case I want to change the value of Textarea by software?

    What does 'drive me to an answer' mean? Do you want to detect real-time change the 'value'? 

    Best Regards,

    Xingyu Zhao



    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.



    Friday, September 4, 2020 1:55 AM
  • Hi Xingyu,

    I'm trying to detect every changing in real time and if possible, to change the value of textarea via software.

    Thanks

    Mattia Fanti

    Friday, September 4, 2020 2:52 AM
  • Hi Mattia Fanti,

    >>I'm trying to detect every changing in real time and if possible, to change the value of textarea via software.

    It is impossible that simply modify the ‘server’ data from the ‘client’.

    Besides, you can refer to the following code to change the downloaded html.

            '...
            Dim node As HtmlNode = doc.DocumentNode.SelectSingleNode("//*[@id=""conteudo""]")
            node.InnerHtml = TextBox1.Text
            Dim value As String = If(node Is Nothing, "Error, id not found", node.InnerHtml)
            '...

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Friday, September 4, 2020 7:07 AM
  • Hi Xingyu,

    definitely I don't need to change the downloaded html if I can't send back any update to the textarea.

    I really didn't know I can't set value to the textarea through http web requests!

    So something like this is impossibile with http web requests?

     WebBrowser1.Document.All("conteudo").SetAttribute("value", "Hello World")

    Thanks

    Friday, September 4, 2020 11:35 AM
  • Hi Mattia Fanti,

    For more efficient responses about this question, you can consider posting it in Developer Community forum. Thanks. 

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, September 7, 2020 1:56 AM