locked
ADF Rest Pagination w/ Relative URL RRS feed

  • Question

  • I'm trying to utilize the REST type connection in ADFv2 to fetch data from a RESTful service. The service returns the data in pages. The paging information is returned as part of the json response and consists of a relative url for the next fetch and header value. Based on the documentation it doesn't appear that this pattern is supported by the REST connection. Instead, the REST connection requires an absolute url, query parameter, and/or header. So, my question: is there a way to build the absolute url in an expression? Something like:

    {
        "name": "MyTestREST",
        "properties": {
                "type": "RestResource",
                "typeProperties": {
                    "relativeUrl": "customers?size=50",
                    "paginationRules": {
                        "AbsoluteUrl": "@concat(<LinkedServiceBaseURL>,$.links.next.uri",
         "Headers['MS-ContinuationToken']": "$.continuationToken"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "MyRestService",
                    "type": "LinkedServiceReference"
                }
        }
    }

    I'd need to substitute the '<LinkedServiceBaseURL>' into the AbsolutionUrl value. Any ideas?

    Wednesday, January 23, 2019 10:35 PM

All replies

  • Hi Pierre,

    The REST connector for ADF doesn't support Relative URLs and there's no direct way to create an Absolute URL by using an attribute from the JSON response. 

    I am working internally with the ADF team to help us more on this. I will update you as soon as I hear from them.


    MSDN

    Thursday, January 24, 2019 12:42 PM
  • Hi Pierre,

    I guess parameterize linked service could also satisfy your requirement, right?

    Rest linked service parameterization is not supported in UI, but you could modify the code. Please follow this doc.

    https://docs.microsoft.com/en-us/azure/data-factory/parameterize-linked-services

    Thursday, January 24, 2019 12:47 PM
  • Hi Pierre,

    Can you also share the Structure of your URL ? If it's in the form :

    https://reqres.in/api/users?page=1

    https://reqres.in/api/users?page=2

    https://reqres.in/api/users?page=3

    and so on, where (1,2,3...) come as an attribute in the response (Say, nextUrl), you can try this :

    • Make the baseUrl in the linked service as https://reqres.in/api/users?page=1, and leave relativeUrl in dataset as empty.
    • change the paginationRules as "QueryParameters.page": "$.nextUrl


    MSDN

    Thursday, January 24, 2019 1:09 PM
  • The REST API I am using is the Microsoft Partner Center REST API (https://docs.microsoft.com/en-us/partner-center/develop/partner-center-rest-api-reference). Paging consists of a query parameter and a header with token. So, if I'm calling 'https://api.partnercenter.microsoft.com/v1/customers/{{customer-tenant-id}}/subscriptions/{{subscription-id}}/utilizations/azure' to get the next page of data I would need to call: 'https://api.partnercenter.microsoft.com/v1/customers/{{customer-tenant-id}}/subscriptions/{{subscription-id}}/utilizations/azure?seek_operation=next' plus add the header: 'MS-ContinuationToken:{{token}}'. The JSON returned from the first call includes the token and a relative URL (e.g. '/azure?seek_operation=next') that has the query parameter.
    Friday, January 25, 2019 8:01 PM
  • Please try with the following settings in your pipeline:

    1. In REST lined service, set the URL as "https://api.partnercenter.microsoft.com/v1/customers/{{customer-tenant-id}}/subscriptions/{{subscription-id}}/utilizations/"  (better to end with '/')

    2. In REST dataset, set relativeUrl as "azure"  (do NOT start with '/')

    3. In REST dataset, set paginationRules as

    • "AbsoluteUrl": "$.nextUrl"
    • "Headers.MS-ContinuationToken": "$.nextToken"

    "$.nextUrl" and "$.nextToken" are just example because I don't know how the JSON response look like.

    Monday, February 11, 2019 7:59 AM
  • Thanks for the response. However, this doesn't work. The "$.nextUrl" is relative and not absolute.
    Wednesday, February 27, 2019 7:45 PM
  • Any Updates on thip topic?

    Friday, August 2, 2019 1:50 PM
  • Thanks for the response. However, this doesn't work. The "$.nextUrl" is relative and not absolute.
    What's the error message you saw?
    Monday, August 5, 2019 10:15 AM
  • Hi, I have a requirement to query activity runs using pipeline run id and use the below URL

    https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/queryActivityruns?api-version=2018-06-01.

    The response is in multiple pages as when I copy the response to the database, I only see first 100 rows. I get a continuationToken in the response. What is the url for next page? How do I pass the continuation token to get the next page

    • Edited by ArDe97 Monday, October 28, 2019 11:47 PM
    Monday, October 28, 2019 11:28 PM
  • Hi there,

    Since this thread is too old, can you please create a new forum thread as your issue is also not the exact same as the above one?

    Creating a new thread will get the issue more visibility across the community.

    Tuesday, October 29, 2019 6:04 AM
  • created a new thread.


    Tuesday, October 29, 2019 1:51 PM