none
Copying data needs to include empty folders

    Question

  • I'm using Azure Data Factory to recursively copy data from a DLS in region X to a DLS in region Y. Copy behavior is set to preserve the hierarchy. One major disadvantage is that it doesn't copy empty folders (+100 empty folders) which are required.

    ADLcopy doesn't support region to region copy so it's not an option.

    The use of Azure Storage Explorer to first download the data doesn't download empty folders as well.

    What are my options?


    Thursday, May 31, 2018 9:05 AM

Answers

  • Based this doc, https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-store, ADF currently doesn't support copy empty folders.


    recursive Indicates whether the data is read recursively from the sub folders or only from the specified folder. Note when recursive is set to true and sink is file-based store, empty folder/sub-folder will not be copied/created at sink.
    Allowed values are: true (default), false
    Thursday, May 31, 2018 2:19 PM

All replies

  • In your input dataset, if you have specified format or structure section, please remove it and try again. Because if you don’t specify format, ADF will do copy as is.
    Thursday, May 31, 2018 9:55 AM
  • Thanks for the reply. I specified binary copy since I want the files/folders to be copied as-is.
    Thursday, May 31, 2018 11:47 AM
  • Based this doc, https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-store, ADF currently doesn't support copy empty folders.


    recursive Indicates whether the data is read recursively from the sub folders or only from the specified folder. Note when recursive is set to true and sink is file-based store, empty folder/sub-folder will not be copied/created at sink.
    Allowed values are: true (default), false
    Thursday, May 31, 2018 2:19 PM