Make a copy of Data Lake Store file from one ADLS path to another? RRS feed

  • Question

  • I'd like to copy files from one data lake path to another. Is this possible with the API? For example I am thinking:

    1. Use DataLakeStoreFileSystemManagementClient.FileSystem.Open to open the stream.

    2. CopyTo new stream.

    3. Then use DataLakeStoreFileSystemManagementClient.FileSystem.Create to recreate the file in a different location.

    Will this work?  Is there another way?


    Monday, May 9, 2016 5:36 AM

All replies

  • Hi,

    Are you open to using other tools or do you just want to use APIs?

    You could use the code-free copy wizard (https://channel9.msdn.com/Blogs/Windows-Azure/Azure-Data-Factory-code-free-copy-wizard) to do quick copy between your ADLS locations. You can also schedule the copy.

    Sachin Sheth
    Program Manager
    Azure Data Lake.

    Tuesday, May 10, 2016 5:25 PM
  • Hi Sachin,

    Thanks for your reply.  Needs to be API based - I need to do special processing and filtering of the files as a binary copy before they come into the system, and then arrange them in a proper staging directory, so that I can assure my refinement process that it is ingesting the proper file format with the proper ADF pipeline/USQL code.


    Wednesday, May 11, 2016 2:57 AM
  • Hi, have you tried using the WebHDFS APIs - https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-rest-api/?

    These APIs allow you to read and upload data to ADLS (among other operations).

    Please let us know if that helped.

    Monday, May 16, 2016 9:06 PM