none
Perform Pagination for REST API - ADF

    Question

  • Hi,

    I'm consuming REST API using ADF which has pagination implemented, response does contain section in document which says how many total rows are there and how many rows each page has, using which can decide for how many pages I have to Run.

    However I'm not sure how to read this data within ADF and perform this operation.

    My current setup looks like (this does not have pagination and just gets first page)

    SQL task to get Businesses for which I have to call API, then for each look which iterated over each business and calls API

    This returns first page and I store than in blob storage

    Now if I have to do pagination, i think for-loop need to have another for loop which is for pagination

    Am I on right track or is there better way of achieving this pagination?

    Also how do I read those json page section which has pagination detail.

    Thanks,

    Dilkush


    Thanks, Dilkush Patel Microsoft SQL Developer Support Microsoft Corporation

    Friday, August 10, 2018 1:52 PM

All replies

  • ADF will release a new connector, generic REST connector, which is able to automatically fetch data from all pages.  I believe that's what you want.
    Tuesday, August 14, 2018 6:41 AM
  • Any idea when will ADF be releasing this connector?

    Currently I just see HTTP connector, if some connector with pagination support is going to be released soon it would be great

    Thanks,

    Dilkush


    Thanks, Dilkush Patel Microsoft SQL Developer Support Microsoft Corporation

    Tuesday, August 14, 2018 1:56 PM
  • The Rest connector may be released at the end of this month. Thanks.
    Wednesday, August 15, 2018 1:17 AM
  • I'm facing the same issue. Is there any news on the generic REST api?
    Tuesday, November 6, 2018 8:10 PM