none
Azure Data Factory HTTP connector to parse a webpage

    Question

  • A bit new to using Azure for ETL and Machine Learning.

    I want to parse a webpage and convert it into a labeled catalog of structured data, to which I can apply ML transforms.

    I was reading up the Azure documentation on HTTP Connector but I am unclear on a stepwise process to do so using either the Azure Data factory UI or scripts.

    Can Azure Data Factory be used to such a parsing task and if yes, is there clear documentation on how the Azure Data Factory UI can be used to do so?

    Friday, July 13, 2018 6:47 AM

All replies

  • Hi Nirshuk,

    If I understand right, you want to convert the data in a webpage into the structured data, and run ML transformation on it?

    Maybe you could firstly load data into Azure Data Lake Store (HTTP connector to AzureDataLakeStore connector), and then run an U-SQL script to do the transformation.

    Please reference the post: 

    https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-data-lake-analytics 

    https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-get-started-portal

    Monday, July 16, 2018 3:41 AM