Preserve ACL between ADSL Gen 2 when using a data factory copy data pipeline

Question

Hi,

as a native solution is not ready yet to backup ADLS Gen 2, I did create a second ADLS Gen2 (with a cool access to reduce the cost) and executed a Data Factory copy data pipeline.

I did try to preserve ACL but didn't figured out how to preserves those ones..

For exemple I did try the following options with no success.

1. "preserve": [ "Attributes" ]

2. "preserve": [ "ACL", "Owner", "Group" ]

Actually I did get the following error with the second option :

Copy with preserving ACL is not supported from AzureBlobFS to AzureBlobFS

Please find below a sample code of a copy data pipeline.

{
    "name": "bck",
    "properties": {
        "activities": [
            {
                "name": "refined",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "BinarySource",
                        "storeSettings": {
                            "type": "AzureBlobFSReadSettings",
                            "recursive": true
                        }
                    },
                    "sink": {
                        "type": "BinarySink",
                        "storeSettings": {
                            "type": "AzureBlobFSWriteSettings",
                            "copyBehavior": "PreserveHierarchy"
                        }
                    },
                    "enableStaging": false,
                    "preserve": [
                        "Attributes"
                    ]
                },
                "inputs": [
                    {
                        "referenceName": "adls_ref",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "adlsbck_ref",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": []
    }
}

Regards,

James

Answer 1

Hi James,

Did you have a look at the following docs :

https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-preserve-metadata

https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-store#preserve-acls-to-data-lake-storage-gen2

Hope this helps.

Answer 2

Yes I did, actually it does concern ADLS gen 1 to ADLS gen 2.

I did the following tests trhoug a Data Factory copy data pipeline :

- ADLS gen 1 to ADLS gen 2 : could preserve ACL (see https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-store#preserve-acls-to-data-lake-storage-gen2)

- ADLS gen 2 to ADLS gen 1: couldn't preserve ACL (the option could be set on the json code but it's erased at each run)

- ADLS gen 2 to ADLS gen 2: couldn't preserve ACL (the error message is : Copy with preserving ACL is not supported from AzureBlobFS to AzureBlobFS) 

Answer 3

Hi James,

I had a conversation with the internal team and they confirmed that preserving ACLs is only enabled for ADLS Gen1 to ADLS Gen2 copies.

I would recommend you to provide feedback at the feedback forum. All the feedback you share, is closely monitored by the Data Factory Product team and implemented in future releases.

Also, Regarding service general availability, I would suggest to keep an eye on Azure updates. 

Azure updates provide information about important Azure product updates, roadmap, and announcements.

Answer 4

Thank you for the confirmation, I did add my vote on this feedback : https://feedback.azure.com/forums/270578-data-factory/suggestions/34048126-copy-acl-and-privileges-in-data-lake-store

Will be great to see this feature's coming soon.

Regards,

James

Answer 5

Great! I have also notified the team and they're checking if they can implement it.