none
AdlCopy - Copy failure error

    Question

  • Hi,

    I have an adhoc need to copy some files from a Blob Storage to a Data Lake Store. I trying to do this with AdlCopy. The command I used is as follows.

    AdlCopy /Source https://myblob.blob.core.windows.net/mycontainer/myfolder/2017/05/01/ /dest adl://mydatalake.azuredatalakestore.net/myfolder/2017/05/01/ /sourcekey myblobstoragekey /Pattern *.json

    When I execute the above command, the copy job start and identifies the number of files to be copied. But after sometime aborts with the below error.

    Initializing Copy.
    Files from 1 to 25. Copy Started.\
    Copy Failed: No write permission to the specified location.
    JobId: 9999xxx999x99x99

    I have SuperUser privilege to the Data Lake Store account, but still I am getting the above error. Have anyone encountered this error?

    with regards,
    Parthiban


    Wednesday, May 10, 2017 9:10 AM

Answers

  • Looks like you don't have write access in the ADLS folder. Access privileges to the ADLS account and access privileges to individual files/folders are separate. Having admin privilege to the account means you can create and modify the account. Individual folders and files have separate ACLs for read/write.

    You probably want to use the wizard in the Azure portal to add permissions for a user. You can find that under the ADLS resource's Data Explorer tab. Once you are in data explorer you can see an "Access" tab that gives you the controls. 

    You can find an explanation at:

    https://blogs.msdn.microsoft.com/azuredatalake/2016/07/31/introducing-file-and-folder-acls-for-azure-data-lake-store/ 

    and

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-access-control

    Friday, May 12, 2017 9:38 AM

All replies

  • Looks like you don't have write access in the ADLS folder. Access privileges to the ADLS account and access privileges to individual files/folders are separate. Having admin privilege to the account means you can create and modify the account. Individual folders and files have separate ACLs for read/write.

    You probably want to use the wizard in the Azure portal to add permissions for a user. You can find that under the ADLS resource's Data Explorer tab. Once you are in data explorer you can see an "Access" tab that gives you the controls. 

    You can find an explanation at:

    https://blogs.msdn.microsoft.com/azuredatalake/2016/07/31/introducing-file-and-folder-acls-for-azure-data-lake-store/ 

    and

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-access-control

    Friday, May 12, 2017 9:38 AM
  • Hi,

    Usually a SuperUser should have all the privileges to the Data Lake Store account. So you should *not* have to specify ACLs for that user on the folders/files. However, there is a bug in AdlCopy due to which this is not the case and we are working on fixing it shortly.

    In the meanwhile, please follow the instructions provided by Omid. You will need to explicitly set ACLs on the destination so that the SuperUser is able to copy over the data using adlcopy. Sorry for the inconvenience.

    Thanks,
    Sachin Sheth
    Azure Data Lake team.

    Monday, May 15, 2017 8:08 PM
  • Thanks Omid. I explicitly applied the read, write and execute permissions and it worked. I'll mark this as answer.

    with regards,
    Parthiban

    Wednesday, May 17, 2017 4:07 PM
  • Thanks for the update, Sachin. I have done as Omid suggested.

    BTW, please let us know once the fix is ready.

    with regards,
    Parthiban


    Wednesday, May 17, 2017 4:11 PM