locked
Doubts on how to consume data from an html page using json. RRS feed

  • Question

  • User662762443 posted

    Guys, how can I consume data from an html page that contains a table of information through a json?

    Saturday, May 1, 2021 1:39 PM

All replies

  • User475983607 posted

    Guys, how can I consume data from an html page that contains a table of information through a json?

    Your question is confusing.  As you asking how to populate an HTML table from data JSON?  If so, use AJAX.  

    When asking for assistance, always share your source code.  It help to clarify the intent.

    Saturday, May 1, 2021 2:33 PM
  • User662762443 posted

    Hi mgebhard, I need to read and show the return data found in this url that i added in the implementation. But it doesn't work. Do you have any idea how I could be doing?

    This is my code
    using System;
    using System.Net.Http;
    using System.Web.UI;
    
    namespace ProjetoJsonAthenas
    {
        public partial class _Default : Page
        {
            HttpClient client;
            Uri dateUri;
            protected void Page_Load(object sender, EventArgs e)
            {
                if (!Page.IsPostBack)
                {
                    getAll();
                }
            }
            public _Default()
            {
                if (client == null)
                {
                    client = new HttpClient();
                    client.BaseAddress = new Uri("https://developer.athenas.online/table.html");
                    client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
                }
            }
    
            private void getAll()
            {
                // chamando a api pela url
                System.Net.Http.HttpResponseMessage response = client.GetAsync("api/athenas").Result;
    
                //se retornar com sucesso busca os dados
                if (response.IsSuccessStatusCode)
                {
                    //pegando o cabeçalho
                    dateUri = response.Headers.Location;
    
                    //Pegando os dados do Rest e armazenando na variável usuários
                    var nomes = response.Content.ReadAsAsync<IEnumerable<Names>>().Result;
    
                    //preenchendo a lista com os dados retornados da variável
                    grdDateAthenas.DataSource = names; //grdDateAthenas -> This is the name I put on the grid as identification on the site.master page
                    //
                    grdDateAthenas.DataSource = id;
                    grdDateAthenas.DataSource = Email;
                    grdDateAthenas.DataSource = category;
                    grdDateAthenas.DataBind();
                }
    
                //Se der erro na chamada, mostra o status do código de erro.
                else
                    Response.Write(response.StatusCode.ToString() + " - " + response.ReasonPhrase);
            }
        }
    }

    This is my master site.

    <%@ Master Language="C#" AutoEventWireup="true" CodeBehind="Site.master.cs" Inherits="ProjetoJsonAthenas.SiteMaster" %>
    
    <!DOCTYPE html>
    
    <html lang="pt">
    <head runat="server">
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title><%: Page.Title %> - Data extracted using Json</title>
    
        <asp:PlaceHolder runat="server">
            <%: Scripts.Render("~/bundles/modernizr") %>
        </asp:PlaceHolder>
    
        <webopt:BundleReference runat="server" Path="~/Content/css" />
        <link href="~/favicon.ico" rel="shortcut icon" type="image/x-icon" />
    
    </head>
    <body>
        <form runat="server">
            <div class="row">
                <asp:GridView ID="grdDataAthenas" runat="server" AutoGenerateColumns="false">
                    <Columns>
                        <asp:BoundField DataField="id" HeaderText="id" />
                        <asp:BoundField DataField="Name" HeaderText="Name" />
                        <asp:BoundField DataField="Email" HeaderText="Email" />
                        <asp:BoundField DataField="Category" HeaderText="Category" />
                    </Columns>
                </asp:GridView>
            </div>
    
            <div class="container body-content">
                <asp:ContentPlaceHolder ID="MainContent" runat="server">
                </asp:ContentPlaceHolder>
                <hr />
                <footer>
                    <p>&copy; <%: DateTime.Now.Year %></p>
                </footer>
            </div>
    
        </form>
    </body>
    </html>
    

    And here is my default.aspx page:

    <%@ Page Title="Home Page" Language="C#" MasterPageFile="~/Site.Master" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="ProjetoJsonAthenas._Default" %>
    
    <asp:Content ID="BodyContent" ContentPlaceHolderID="MainContent" runat="server">
    </asp:Content>
    

    Saturday, May 1, 2021 7:34 PM
  • User475983607 posted

    The design does not make sense.  The HttpClient base address is set to an html page while the call is to api/athenas.  As shown the code throws a 404 (Not found).

    If your goal is to read an HTML table then I recommend the HTML Agility Pack.

    https://html-agility-pack.net/

    Saturday, May 1, 2021 7:51 PM
  • User662762443 posted

    mgebhard, I couldn’t think of a simple and practical way to do this json implementation, wouldn’t you have an example of how I could do it? My instructor asked me to do it using the visual studio tool.

    Sunday, May 2, 2021 9:17 PM
  • User662762443 posted

    I needed an example of how to implement using this HTML Agility Pack.

    Sunday, May 2, 2021 10:03 PM
  • User475983607 posted

    mgebhard, I couldn’t think of a simple and practical way to do this json implementation, wouldn’t you have an example of how I could do it? My instructor asked me to do it using the visual studio tool.

    There is no JSON anywhere in your samples.  I'm not sure what you're asking.  Perhaps contact your instructor for assistance.  That's what instructor are for...

    I needed an example of how to implement using this HTML Agility Pack.

    All the examples you need are in the support documentation. 

    https://html-agility-pack.net/

    Sunday, May 2, 2021 10:33 PM
  • User1535942433 posted

    Hi rtaVix,

    Do you mean you need to use html agility pack scrape URL? If you need to do this,you could follow below example:

    static void Main(string[] args)
        {
            List<string> linksToVisit = ParseLinks("https://www.facebook.com");
        }
    
    public static List<string> ParseLinks(string urlToCrawl)
        {
    
            WebClient webClient = new WebClient();
    
            byte[] data = webClient.DownloadData(urlToCrawl);
            string download = Encoding.ASCII.GetString(data);
    
            HashSet<string> list = new HashSet<string>();
    
            var doc = new HtmlDocument();
            doc.LoadHtml(download);
            HtmlNodeCollection nodes =    doc.DocumentNode.SelectNodes("//a[@href]");
    
                foreach (var n in nodes)
                {
                    string href = n.Attributes["href"].Value;
                    list.Add(GetAbsoluteUrlString(urlToCrawl, href));
                }
            return list.ToList();
        }

    More details,you could refer to below article:

    https://html-agility-pack.net/knowledge-base/34582769/html-agility-pack-url-scraping---getting-full-html-link

    Best regards,

    Yijing Sun

    Monday, May 3, 2021 5:29 AM
  • User662762443 posted

    yij sun, I understand, but how do I get the data return from this page with json after implementing the html agility pack? Well, I received the following procedure to create an automated program that enters the page: https://developer.athenas.online/table.html capture the data from the table and return a json.

    Monday, May 3, 2021 11:19 AM
  • User475983607 posted

    The requirement you shared does not make sense.  JSON is typically consumed by a JavaScript application or returned from a REST service.  Your sample code contains no JavaScript or a REST service.  The code you shared is Web Forms trying to populate a GridView from a data source.  Usually the data source is a ASP.NET dataset in ASP.NET Web Forms.

    My best guess is you do not understand the requirements and need to contact the instructor for clarification.   

    Monday, May 3, 2021 12:52 PM
  • User662762443 posted

    I did understand the requirements, yes, I even made contact with my instructor, but he mentions the same requirements described above, in short it did not help me at all, so I am on the forum asking for help. But I got to start something and it didn't take off.

    Monday, May 3, 2021 1:02 PM
  • User753101303 posted

    Hi,

    Seems a surprising assignment. You really clarified "consume data from an html page that contains a table of information through a json" ?

    IIMO it would make more sense if the excercise is to render an HTML page that looks the same but whose data is coming from a json fle.

    If you really have to read this HTML page, extract data and expoe that as json, it is simple enough so that you don't have to use the HTML  Agility Pack (the idea when you start is to learn about solving each problem onestep  after the other to reach your final goal rarther than to learn about using librairies or advanced features).

    In the real workd #1 happens mich more often than  #2 so I wonder if you really understood what your intrusctor told to do...

    If #2, you want to read the HTML page as a string, extract data out of this (simple string handling should be enough) and rhen expose those data as web service (if you have seen this already ?)

    Monday, May 3, 2021 1:26 PM
  • User662762443 posted

    PatriceSC, I understood clearly, but as I am a beginner, I even tried to start something, but it gives 404 error, because I was not doing it according to the suggestion presented by the colleague previously, I tried to adapt the suggestion, but it did not go ahead. If you have an example of how I can do it, thank you. I posted the code earlier as requested, but answering your question, I did understand the point of the question, but how should I do and progress, I am not getting it.

    Monday, May 3, 2021 1:38 PM
  • User475983607 posted

    I understood clearly, but as I am a beginner, I even tried to start something, but it gives 404 error, because I was not doing it according to the suggestion presented by the colleague previously, I tried to adapt the suggestion, but it did not go ahead. If you have an example of how I can do it, thank you. I posted the code earlier as requested, but answering your question, I did understand the point of the question, but how should I do and progress, I am not getting it.

    This community does not understand the requirement either.  We are asking you to clarify.  If you are unable to clarify the requirements then how do expect this community to provide an answer?  Asking the same confusing question over and over is not helpful.

    Anyway, I built a sample based on your code Web Forms code and the HTML Agility Pack.  Keep in mind, there's no JSON because you have not provided any clue where JSON is being used.

    <%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="WebFormsDemo.HtmlAgility.Default" %>
    
    <!DOCTYPE html>
    
    <html xmlns="http://www.w3.org/1999/xhtml">
    <head runat="server">
        <title></title>
    </head>
    <body>
        <form id="form1" runat="server">
            <div>
                <asp:GridView ID="GridView1" runat="server"></asp:GridView>
            </div>
        </form>
    </body>
    </html>
    using HtmlAgilityPack;
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Net;
    using System.Web;
    using System.Web.UI;
    using System.Web.UI.WebControls;
    
    namespace WebFormsDemo.HtmlAgility
    {
        public class EmailModel
        {
            public int Codigo { get; set; }
            public string Nome { get; set; }
            public string Email { get; set; }
            public int Categoria { get; set; }
        }
        public partial class Default : System.Web.UI.Page
        {
            protected void Page_Load(object sender, EventArgs e)
            {
                if(!Page.IsPostBack)
                {
                    GridView1.DataSource = ConvertHtmlTableToType();
                    GridView1.DataBind();
                }
            }
    
    
            protected List<EmailModel> ConvertHtmlTableToType()
            {
                List<EmailModel>  items = new List<EmailModel>();
                WebClient webClient = new WebClient();
                string page = webClient.DownloadString("https://developer.athenas.online/table.html");
    
                HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
                doc.LoadHtml(page);
    
                foreach (HtmlNode rows in doc.DocumentNode.SelectNodes("//table/tbody"))
                {
                    foreach (HtmlNode row in rows.SelectNodes("tr"))
                    {
                        EmailModel item = new EmailModel()
                        {
                            Codigo = int.Parse(row.SelectNodes("td")[0].InnerText),
                            Nome = row.SelectNodes("td")[1].InnerText,
                            Email = row.SelectNodes("td")[2].InnerText,
                            Categoria = int.Parse(row.SelectNodes("td")[3].InnerText)
                        };
                        items.Add(item);
                    }
                }
                return items;
            }
    
        }
    }

    Monday, May 3, 2021 2:55 PM
  • User662762443 posted

    I tried to do it that way too. It's wrong?

    const JSONTable = []
    
    await fetch("https://developer.athenas.online/table.html").then((response) => {
        return response.text()
    }).then((html) => {
        const parser = new DOMParser();
        const doc = parser.parseFromString(html, 'text/html');
      
        const items = doc.querySelectorAll('tr');
        items.forEach((props, key) => {
          if (key === 0) return
                
          const values = props.querySelectorAll('td')  
          JSONTabela.push({
              Code: values[0].innerHTML.trim(),
              Name: values[1].innerHTML.trim(),
              Mail: values[2].innerHTML.trim(),
              Category: values[3].innerHTML.trim()
            })
        })
    }).catch((err) => {
       console.error(err)
    })
    
    console.log(JSONTable)

    Monday, May 3, 2021 3:18 PM
  • User475983607 posted

    I tried to do it that way too. It's wrong?

    Have you tried the code?  Does it work?  

    Keep in mind, the developer.athenas.online domain does not support CORS.  The fetch script must originate from the same domain as developer.athenas.online otherwise the browser will throw a CORS error.  I assume it is very unlikely that your script originates from the same domain. 

    Again, the requirements make no logical sense.  This community cannot help you if you are unable to clarify the requirements.  

    Monday, May 3, 2021 3:36 PM
  • User475983607 posted

    Another option is use Web API to parse the html and return JSON.  At this point we don't even know what kind of application you are allowed to build.  The only code you've shared in Web Forms and JavaScript and mostly like the JavaScript approach will not work.  Plus your JavaScript has bugs. 

        public class HtmlTableController : ApiController
        {
            // GET: HtmlTable
           
            public List<EmailModel> Get()
            {
                List<EmailModel> items = new List<EmailModel>();
                WebClient webClient = new WebClient();
                string page = webClient.DownloadString("https://developer.athenas.online/table.html");
    
                HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
                doc.LoadHtml(page);
    
                foreach (HtmlNode rows in doc.DocumentNode.SelectNodes("//table/tbody"))
                {
                    foreach (HtmlNode row in rows.SelectNodes("tr"))
                    {
                        EmailModel item = new EmailModel()
                        {
                            Codigo = int.Parse(row.SelectNodes("td")[0].InnerText),
                            Nome = row.SelectNodes("td")[1].InnerText.Trim(),
                            Email = row.SelectNodes("td")[2].InnerText.Trim(),
                            Categoria = int.Parse(row.SelectNodes("td")[3].InnerText)
                        };
                        items.Add(item);
                    }
                }
                return items;
            }
        }

    Results

    [
      {
        "Codigo": 1,
        "Nome": "Jorge da Silva",
        "Email": "jorge@terra.com.br",
        "Categoria": 1
      },
      {
        "Codigo": 2,
        "Nome": "Flavia Monteiro",
        "Email": "flavia@globo.com",
        "Categoria": 2
      },
      {
        "Codigo": 3,
        "Nome": "Marcos Frota Ribeiro",
        "Email": "ribeiro@gmail.com",
        "Categoria": 2
      },
      {
        "Codigo": 4,
        "Nome": "Raphael Souza Santos",
        "Email": "rsantos@gmail.com",
        "Categoria": 1
      },
      {
        "Codigo": 5,
        "Nome": "Pedro Paulo Mota",
        "Email": "ppmota@gmail.com",
        "Categoria": 1
      },
      {
        "Codigo": 6,
        "Nome": "Winder Carvalho da Silva",
        "Email": "winder@hotmail.com",
        "Categoria": 3
      },
      {
        "Codigo": 7,
        "Nome": "Maria da Penha Albuquerque",
        "Email": "mpa@hotmail.com",
        "Categoria": 3
      },
      {
        "Codigo": 8,
        "Nome": "Rafael Garcia Souza",
        "Email": "rgsouza@hotmail.com",
        "Categoria": 3
      },
      {
        "Codigo": 9,
        "Nome": "Tabata Costa",
        "Email": "tabata_costa@gmail.com",
        "Categoria": 2
      },
      {
        "Codigo": 10,
        "Nome": "Ronil Camarote",
        "Email": "camarote@terra.com.br",
        "Categoria": 1
      },
      {
        "Codigo": 11,
        "Nome": "Joaquim Barbosa",
        "Email": "barbosa@globo.com",
        "Categoria": 1
      },
      {
        "Codigo": 12,
        "Nome": "Eveline Maria Alcantra",
        "Email": "ev_alcantra@gmail.com",
        "Categoria": 2
      },
      {
        "Codigo": 13,
        "Nome": "João Paulo Vieira",
        "Email": "jpvieria@gmail.com",
        "Categoria": 1
      },
      {
        "Codigo": 14,
        "Nome": "Carla Zamborlini",
        "Email": "zamborlini@terra.com.br",
        "Categoria": 3
      }
    ]

    Monday, May 3, 2021 4:12 PM