ADF - REST Connector - Pagination Rules - Dynamic Content RRS feed

  • Question

  • Hi, I'm new to ADF and am exploring pulling data from a RESTful API that uses pagination using REST connector. I can pull data from the first result page, but I don't seem to be able to work out how to correctly configure pagination to pull the full data set returned. Part of the challenge is that the API used does not conform to basic assumptions of how ADF REST operates according to documentation. I've been through ADF documentation, MSDN forums, O'Reilly Learning platform and Google and could not find any useful answers to my question. 

    I'm trying to pull data from Zendesk Support API using Incremental Export endpoint:

    This endpoint helpfully provides a return value of 'next_page' URL that could be passed into AbsoluteURL variable under Pagination Rules, however, instead of returning null when it reaches the last page, it returns URL to be used for next scheduled export. ADF REST docs state: 'The connector will stop iterating when it gets HTTP status code 204 (No Content), or any of the JSONPath expression in "paginationRules" returns null.' Effectively my call to Zendesk API gets stuck in an endless loop and then errors. I could use another return value 'count' to determine if the returned page is full or partial. If a partial page is returned, this signals exit condition. 

    I was trying to create a dynamic expression for AbsoluteURL to return 'next_page' if full page returned and null if partial. I don't seem to figure out how to do this. Is this even possible? Are there any other approaches I can try? There seem to be some suggestions of using ForEach with HTTP Connector, but this assumes I have a list of items to pass into ForEach, which I don't (and haven't yet found a way to dynamically create this from other available Zendesk API endoints). 

    Any pointers would be much appreciated. 
    Saturday, February 16, 2019 8:01 PM

All replies

  • Hello InsightDudeUK.  Thank you for bringing this to our attention.  I will investigate the issue and reply when I have more information.
    Thursday, February 21, 2019 1:09 AM
  • Hi there,

    Regarding this question: 

    I was trying to create a dynamic expression for AbsoluteURL to return 'next_page' if full page returned and null if partial

    Dynamic expression value generated from other dynamic parameter/constant value in the pipeline while the value you seek for needs to be determined inside the Rest connector itself. So I'm afraid this won't help in your scenario. 


    Thursday, February 21, 2019 2:59 AM
  • Thank you for the insight Wang Zhang.  With this in mind, I do have a possible work around to try.

    Given that one pipeline can call another pipeline, and the parameters needed are generated during the page export,

    what about a sort of tail-recursion?  Say you have two pipelines.  One is used to pull the first page, and will call the second pipeline with the necessary parameters.  The second pipeline pulls the second page and then generates the parameters to call the next page.  The second pipeline calls another instance of itself passing the generated parameters, halting on your 'incomplete page condition'.

    This way you do not need a list of urls that a for-each would require.

    This is just an idea.  I'll need to verify that there are no prohibitions against a pipeline calling a new instance of itself.

    Does this inspire you?

    Thursday, February 21, 2019 10:17 PM
  • Okay.  Looks like "circular dependencies are not allowed", so recursion is out.  However, there is the "until" action.

    The documentation says it provides the functionality of a "do-loop".  This means that the code block is executed before checking the conditional.  Inside the 'until' activities, save the next page value to a variable. Have the result of your 'incomplete page condition' be saved to a variable.  Then use this in the conditional.

    Thursday, February 21, 2019 10:34 PM
  • I got same requirment. I am calling VSTS (Azure Dev Ops API) which never returns null or page not found for nextLink but returns isLastBatch true if not more records. Any solution?
    Sunday, November 17, 2019 9:39 PM
  • Any solution if nextPage never return null instead another parameter of API returns true for lastPage? 
    Sunday, November 17, 2019 9:40 PM
  • Hello Daya Dubey.  Are you still in need of assistance?
    Tuesday, November 26, 2019 6:43 PM
  • Yes please. Rest API for Azure Dev Ops returns two parameter one is nextLink which never returns null but isLastBatch which returns true if no more records in page. ADF only works if nextLink returns null, which never happens in my case. Also output of copy pipeline is metadata information not the actual output of the copy command so I can't get the value of isLastBatch.
    Wednesday, November 27, 2019 10:53 AM
  • @Daya Dubey, at present, I see no inbuilt way to accomplish this with the Copy Activity (REST dataset) alone.

    I had the idea to use a dynamic expression for some logic on the parameters, but unfortunately it appears the dynamic expression is evaluated before the '$.isLastBatch'.

    Since the pagination feature isn't working for us right now, lets work around it.  Put both a copy activity + web activity, inside a loop.  Both activities must target the same endpoint.  The copy activity would move the data, while the web activity would expose those parameters.  By exposing the parameters, we can store them in a variable to use in the next iteration's input and the loop's conditional.

    Tuesday, December 3, 2019 2:44 AM
  • We have not received a response from you.  Are you still facing the issue?  If you found a solution, would you please share it here with the community?  Otherwise, let us know and we will continue to engage with you on the issue.
    Thursday, December 5, 2019 7:29 PM