none
File Copy from DMG on premise to Azure Data Lake

    Question

  • Hi,

    I am trying to copy source files from DMG on premise location to Azure Data Lake.  I want to be able to qualify the copy so that I only pick up certain files. Part of what I need to do is found in the following article, but I really need the ability to specify a qualifier as to which files to pull like *.CSV for example.

    https://social.msdn.microsoft.com/Forums/en-US/f9a75798-29e3-4b30-995a-b25c9b9dded9/pipeline-to-copy-on-premise-file-system-to-azure-data-lake-store?forum=AzureDataLake

    So for example, I would like to copy all files of extension CSV, but leave files of type PPT.  Is there a way to do this? 

    Thanks!

    David

    Thursday, March 17, 2016 6:05 PM

All replies

  • If you write an upload script using either the SDK or with the Powershell extension, you can write the script in a way to look for only files for a given extension.

    Michael Rys

    • Proposed as answer by Michael Amadi Friday, April 8, 2016 6:55 AM
    Monday, March 21, 2016 6:02 PM
    Moderator
  • Michael,

    Thanks for you reply! Would this code execute outside of the context of the ADF or would you do this in ADF Custom Activity or how else? If outside of the context of the ADF does it even go through DMG anymore using the SDK? PowerShell?  If it is outside do you execute using Azure Batch?

    Thanks!

    Monday, March 28, 2016 9:55 PM
  • Oh... you can also use ADF to move data into the Azure Data Lake Store directly. Sorry I missed that aspect of your question.


    Michael Rys

    • Proposed as answer by Michael Amadi Friday, April 8, 2016 6:55 AM
    Thursday, March 31, 2016 6:12 AM
    Moderator