locked
Split data into parts RRS feed

  • Question

  • Hi all!
    I have an Azure Data Factory.
    Inside it, I have an Azure Data Flow that prepares my data.
    The next step - is sending this data to API.
    But API has a limitation - no more than 100 rows in a request.
    So, I know how to send web-request from Azure Data Factory(Web Activity).
    But how to split data from data flow by 100 rows?
    Can you help me, please?

    Monday, November 4, 2019 1:45 PM

Answers

  • Hello compasses and thank you for your question.  To better help you, can you tell me what the result of your Data Flow is?  Does it write to SQL or BLOB or something else?  Could you also tell me approximately how many records you will have, and if it will change or stay constant.

    One idea I had was to use Data Flow's partitioning feature to split into files of 100 rows each.  I do not know anything about your data, so I cannot suggest anything specific to your data.  However you could add a column, a simple row-number.  Then you could use the Fixed Range partition type to group together like 1-99, 100-199, 200-299.

    Monday, November 4, 2019 10:58 PM

All replies

  • Hello compasses and thank you for your question.  To better help you, can you tell me what the result of your Data Flow is?  Does it write to SQL or BLOB or something else?  Could you also tell me approximately how many records you will have, and if it will change or stay constant.

    One idea I had was to use Data Flow's partitioning feature to split into files of 100 rows each.  I do not know anything about your data, so I cannot suggest anything specific to your data.  However you could add a column, a simple row-number.  Then you could use the Fixed Range partition type to group together like 1-99, 100-199, 200-299.

    Monday, November 4, 2019 10:58 PM
  • Hi,

    My Data Flow writes data to BLOB. 

    I have about 100000 records. They are constant after the writing to BLOB.

    A row looks like:

    Name, Surname, Id

    I'll try to follow your advice. Thank you!


    • Edited by compasses Tuesday, November 5, 2019 10:30 AM
    Tuesday, November 5, 2019 9:56 AM
  • Thank you for the response.  Please let me know (and mark as answered) if it works out for you.

    Let us know if you run into trouble.

    Tuesday, November 5, 2019 7:38 PM
  • Thank you!

    Your advice works!

    Wednesday, November 6, 2019 5:24 PM
  • Great!  Thank you.
    Thursday, November 7, 2019 5:16 PM