none
Copy multiple containers from Storage Account

    Question

  • Hi all,

    When selecting the Azure Blob Storage as Linked Service you can select a File path. This is a folder in the Storage Account. 

    But I would like to copy more than one container as we have many.

    In this article they explain that you might use .\ or .\\ to copy all the underlying containers. This worked in the v1 version but in v2 I get an error. This article describes that wildcards are not supported.

    Is it possible to copy multiple containers?


    Basic Cloud | Twitter


    • Edited by RalJans Friday, August 31, 2018 7:07 AM
    Friday, August 31, 2018 7:05 AM

Answers

  • Hi RalJans,

    1. your foreach activity items should be:

    @activity('Get Metadata1').output.childItems  

    Because this is the array you need to iterate.

    2. in your copy activity, instead of reusing the dataset of get metadata activity, you need create a new dataset for your source. And a dataset parameter for it. Let's name it containerName. Then your foldPath should reference that parameter.

    3. Then in your copy activity, you should pass item().name to the dataset. 

    • Marked as answer by RalJans Sunday, September 2, 2018 12:25 AM
    Saturday, September 1, 2018 9:59 AM

All replies

  • You could use a get metadata activity to list all the containers first and then use a foreach activity to iterate the containers.
    Friday, August 31, 2018 7:17 AM
  • Thanks for your quick reply.

    Sounds like an option but how can I list all the containers? As Get Metadata requires a Dataset and this Dataset (blob account) is limited to only one container.


    Basic Cloud | Twitter

    Friday, August 31, 2018 11:21 AM
  • Yes. GetMetaData activity can list connections. In the dataset, you just keep both folder path and filename empty. You could use the "debug" button in the UI to have a quick try. 
    Friday, August 31, 2018 11:24 AM
  • Thanks

    I have made some progress based on your idea. But I still need some more guidance on how to set it up.
    I have the Get Metadata step and it gives me the output. The Name value displays all the containers.

    I also have a foreach step and within that step I have an activity Copy Data

    The Copy Data contains a Source (the storage account with an empty File Path), a Sink that contains the destination storage account. The Mapping tab is empty and Settings is set to Auto.

    Could you elaborate how to use the output from the Get Metadata (i think the value is @activity(‘Get Metadata1’).output.itemName) to the source of the Copy Data activity?


    Basic Cloud | Twitter

    Friday, August 31, 2018 2:04 PM
  • Use @item().name

    in the Copy Data in your loop

    that gives you the name of the files 


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Friday, August 31, 2018 2:16 PM
  • If you are looking for PLAN B take a look at Logic Apps

    you can copy from one blob to another using Logic apps from ADF


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Friday, August 31, 2018 2:18 PM
  • Just make sure you have selected the Child Items in your Get Metadata Activities

    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Friday, August 31, 2018 2:28 PM
  • If you are looking for PLAN B take a look at Logic Apps

    you can copy from one blob to another using Logic apps from ADF


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Thanks, but I think I have seen in Logic Apps you also need to specify a container. My question was about copying multiple containers in one step. If there is an alternative that would be great but I think Data Factory is fine and I am almost there.


    Basic Cloud | Twitter

    Friday, August 31, 2018 2:33 PM
  • Just make sure you have selected the Child Items in your Get Metadata Activities

    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    I have and I am getting the correct output: Link

    I am wondering if it should be item.name() or @activity(‘Get Metadata1’).output.Name and where to put that in the loop. Like explained here


    Basic Cloud | Twitter

    Friday, August 31, 2018 2:38 PM
  • I now have this JSON as pipeline: LINK

    But it fails on Activity ForEach1 failed: Activity failed because an inner activity failed


    Basic Cloud | Twitter

    Friday, August 31, 2018 3:00 PM
  • Hi RalJans,

    1. your foreach activity items should be:

    @activity('Get Metadata1').output.childItems  

    Because this is the array you need to iterate.

    2. in your copy activity, instead of reusing the dataset of get metadata activity, you need create a new dataset for your source. And a dataset parameter for it. Let's name it containerName. Then your foldPath should reference that parameter.

    3. Then in your copy activity, you should pass item().name to the dataset. 

    • Marked as answer by RalJans Sunday, September 2, 2018 12:25 AM
    Saturday, September 1, 2018 9:59 AM
  • Thanks ! 

    I now have working solution !


    Basic Cloud | Twitter

    Sunday, September 2, 2018 12:25 AM