none
Data From Rest API RRS feed

  • Question

  • I need to extract data from enterprise systems through REST API that return JSON data as Response. I have some specific query around this

    1. Which Activity should i use within ADF to extract data from REST API.
    2. Can i use ADF to extract both Historical and Incremental data from Rest API
    3. If there is no Parameter within REST API that supports both Historical and Incremental, how often do i need to call the API from ADF or should i go for Event Hub to get data into target systems.
    4. How do i know if the data in source system is changed from the Rest API

    Please help with best possible design in Azure for handling REST API..


    • Edited by MS_Novice Tuesday, February 12, 2019 9:24 AM formatting
    Tuesday, February 12, 2019 9:23 AM

All replies

  • Please suggest any approach for the above requirement. How can i achieve it through Azure

    Wednesday, February 13, 2019 7:17 AM
  • Hi there,

    One way to go about this is by using event hubs in conjunction with Azure Blob Storage or Azure Data Lake Store ( on which you can perform analytics later). To read more, please refer:

    https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview

    Unfortunately, with REST API as source, there's no way to track updates and incrementally copy data.

    The best way would be to set up event hubs and set up an event based trigger. To read more, please refer :

    https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger


    MSDN

    Wednesday, February 13, 2019 12:35 PM
    Owner
  • Thanks! Chirag for the details. Definitely i'll try to explore the above options..

    Wednesday, February 13, 2019 12:54 PM
  • Hi MS_NOVICE,

    Did you get a chance to try out the solution ? Does it work well for you ?


    MSDN

    Friday, February 15, 2019 6:58 AM
    Owner
  • Hi Chirag,

    There is some change in the requirement. Below is what i have highlighted

    • All the enterprise data will be available through Custom Service Bus and there are supporting REST API .
    • There is a need for real time and historical (analytical) type of reporting.

    Based on the above requirement i have some questions, also i have included a high level Solution diagram please see if this will be feasible.

    1.    Query 1: For the initial run how will i perform historical load.

    2.    Query 2: For the subsequent/incremental load i can use Event Hub. Will that be correct assumption?

    Saturday, February 16, 2019 2:49 PM
  • Hi MS_NOVICE,

    Sorry for the delayed response. 

    For the initial run, you can use Azure Data Factory to directly copy from the REST API, the historical data to Azure Data Lake Store.

    For subsequent runs, you can use event hubs to feed data into blob storage or Azure Data Lake Store. Your solution is definitely feasible :).


    MSDN

    Monday, February 25, 2019 11:33 AM
    Owner