locked
Why daily copy task from Table Storage empty CSV file? RRS feed

  • Question

  • I have Azure Table Storage(1000 rows) as input and Azure Data Lake Storage as output using Copy Data(Preview)

    Runs in once works fine and data is copied to CSV file. All rows looks good.

    However scheduled tasks for running every day result that file is each day empty. It this bug or does copy task behaves so that if there are new table storage rows then it removed every rows from CSV file and inserts 0 row?

    What is correct way to implement that all rows in table rows eventually are copied to CSV file on Azure Data Lake Storage? I don't mind having one blob file for each day.


    Kenny_I

    Saturday, December 10, 2016 9:56 AM

Answers

  • Incremental copy / append to destination is not supported. It would go against the design of a robust, distributed, idempotent system.

    If you want to create a task that will combine multiple files into one, use Data Lake Analytics. Make sure you recreate the whole output file from the entire set of input files, this will make your system robust and idempotent.

    • Marked as answer by Kenny_I Wednesday, December 14, 2016 7:33 AM
    Tuesday, December 13, 2016 5:25 PM

All replies

  • My destination Data lake CSV file is empty after scheduled Copy task is run.

    It seem like ADF task delete all rows and insert new rows of certain period.

    How to make ADF to copy data incrementally? For example it could copy rows from previous day without no deleting any rows from destination file.         

    Is this correct pipeline code to edit? How to modify?

    "type": "AzureTableSource",
                            "azureTableSourceQuery": "$$Text.Format('Timestamp gt datetime\\'{0:yyyy-MM-ddTHH:mm:ssZ}\\' and Timestamp le datetime\\'{1:yyyy-MM-ddTHH:mm:ssZ}\\'', WindowStart, WindowEnd)"


    Kenny_I

    Tuesday, December 13, 2016 7:05 AM
  • Incremental copy / append to destination is not supported. It would go against the design of a robust, distributed, idempotent system.

    If you want to create a task that will combine multiple files into one, use Data Lake Analytics. Make sure you recreate the whole output file from the entire set of input files, this will make your system robust and idempotent.

    • Marked as answer by Kenny_I Wednesday, December 14, 2016 7:33 AM
    Tuesday, December 13, 2016 5:25 PM