locked
Issue with incremental data loading Using REST API in Azure Data Factory RRS feed

  • Question

  • Hi Team,

    I am unable to set the proper condition in relative URL by passing parameters.I am implementing incremental (delta)data loading using REST API.My source is REST API and Sink is SQL database

    for example i am setting condition like IN relative URL --- @concat('?id=', dataset().id, '&date>',dataset().date).

    i am getting the duplicate data,not getting incremental data. Also after executing the pipeline,if i am triggering pipeline again data is loading again which should not load if there is no incremental data.According to me ">" condition is not working.

    pipeline flow- LOOKUP+ForEach then Foeach have Copy+SP activity( for updating last load date)

    Could you please let me know if anyone has implemented the same.I am looking forward your response.Thanks!

    Friday, February 7, 2020 4:16 PM

All replies

  • Hello Pankaj, could you please tell us which REST API this is?  Is there an public facing API we can test against?
    Friday, February 7, 2020 6:12 PM
  • Hi Martin,

    It is confidential,Sorry i can't share with anyone.Thanks!

    Saturday, February 8, 2020 5:49 PM
  • I have seen REST API's implemented with very different behaviors.  Some simple CRUD, others more like OData.  For how to work with a specific REST API, you would need to find that API's documentation and work from that.

    Some might support incremental, some might not.

    Thursday, February 13, 2020 10:52 PM
  • Hi Martin,

    Thanks for your response.

    For performing incremental load i am using TimeRange parameter based on that doing incremental load. for example :passing Parameter at Dataset level

    1.id

    2. TimeRangeFrom

    3. TimeRangeTo 

    RelativeURL:-  @concat('?id=',dataset().id,'&TimeRangeFrom=',dataset().TimeRangeFrom,'&TimeRangeTo=',dataset().TimeRangeTo)
    and  executing  the pipeline and incremental data is loading but after that once again i am executing the pipeline,Data is loading again that means condition is not satisfying properly because after loading incremental data pipeline should not load the data again.
    Could you please suggest me what i need to do now.?
    Martin, Could you please share that link where you have seen implementation different kind of behavior of REST APIs.Thanks!

    Friday, February 14, 2020 6:27 AM