locked
Use relative url in pagination rule in the REST connector RRS feed

  • Question

  • Hi all,

    I'm trying to fetch data from Confluence using a REST connector in Azure Data Factory. Pagination doesn't seem to work. The response is:

    {
        "results": [ ... ],
        "start": 0,
        "limit": 25,
        "size": 25,
        "totalSize": 33,
        "cqlQuery": "type=page",
        "searchDuration": 92,
        "_links": {
            "base": "https://x.atlassian.net/wiki",
            "context": "/wiki",
            "next": "/rest/api/search?next=true&limit=25&start=25&cql=type=page",
            "self": "https://x.atlassian.net/wiki/rest/api/search?cql=type=page"
        }
    }

    The docs state:

    AbsoluteUrl: Indicates the URL to issue the next request. It can be either absolute URL or relative URL.

    However setting AbsoluteUrl to $._links.next does not work. And I can't seem to concat $._links.base with $._links.next.

    Please help, thanks!

    Merijn

    Tuesday, March 31, 2020 11:38 AM

All replies

  • Hi Merijn,

    Can you also share the Structure of your URL ? If it's in the form :

    https://reqres.in/api/users?page=1

    https://reqres.in/api/users?page=2

    https://reqres.in/api/users?page=3

    and so on, where (1,2,3...) come as an attribute in the response (Say, nextUrl), you can try this :

    • Make the baseUrl in the linked service as https://reqres.in/api/users?page=1, and leave relativeUrl in dataset as empty.
    • change the paginationRules as "QueryParameters.page": "$.nextUrl"

    Hope this helps.

    Tuesday, March 31, 2020 11:56 AM
  • Hi, thanks for your help.

    The structure of the url is: https://x.atlassian.net/wiki/rest/api/search?cql=type=page

    And the next url in the response is relative: /rest/api/search?next=true&limit=25&start=25&cql=type=page

    So the structure is different than your example.

    If I could concat $._links.base with $._links.next I think it would work, but I can't get it to work...

    Thanks again.

    Tuesday, March 31, 2020 1:27 PM
  • Hi there,

    Unfortunately as of now the REST connector doesn't support expressions or calculations in the relative URL. If you have relatively less data and don't mind running the Copy activity multiple times, you can chain a set variable activity and use a parameterized dataset for the REST source.

    Here's an article I wrote on how to parameterize a dataset. 

    Note - You can use an Until activity or a ForEach activity to do multiple iterations and to break once there's no data.

    Thursday, April 2, 2020 5:01 AM