locked
REST API Call without Pagination RRS feed

  • Question

  • Hi,

    unfortunately my REST interface does not support pagination. I get a lot of data, so I need to load it into Batchsizes (2000 Rows).
    How can I use ADF to increase this value with expression?
    I think I can do this by always increasing the skip value by the top value until EOF.
    example
    http: // my / resturl / orderDataSet $ format = json & $ skip = 0 & top = $ 2,000?
    http: // my / resturl / orderDataSet $ format = json & $ skip = 2000 & $ top = 2000?
    http: // my / resturl / orderDataSet $ format = json & $ skip = 4000 & $ top = 2000?
    http: // my / resturl / orderDataSet $ format = json & $ skip = 6000 & $ top = 2000?
      ...
    Question: How can I always increase the skip value in a loop?
    or are there other Options? Is the result then 1 file in the Azure Data Lake (json) - or more?
    thanks

    Regards

    Nicole  :)


    Answer from MSDN


    • Edited by Nicole44 Thursday, June 13, 2019 6:01 PM
    Thursday, June 13, 2019 5:48 PM

All replies

  • I think we can achieve by doing the following , for clarity I am putting the screenshot 

    The ouThe

    You should have the output like .


    {
        "url": " http: // my / resturl / orderDataSet $ format = json & $ skip =  0 & top = $    2000?",
        "method": "GET",
        "headers": {}
    }
    {
        "url": " http: // my / resturl / orderDataSet $ format = json & $ skip =  2000& top = $    4000?",
        "method": "GET",
        "headers": {}
    }

    Let me know how its goes .


    Thanks Himanshu


    Thursday, June 20, 2019 1:42 AM