locked
Azure Data Factory S3 Bucket Connectivity - Subfolder RRS feed

  • Question

  • Hello,

    Using Azure data factory, I am trying to connect to my S3 bucket but I am unable to connect.  Are there any limitations regarding where the bucket is located?  Otherwise, does Azure support subfolders?  I have a bucket, not in the root, that I am able to access using a variety of tools, just not Azure.  Here's my bucket IAM permissions and location (example with bogus domain):

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket"
                ],
                "Condition": {
                    "StringLike": {
                        "s3:prefix": "testing/main/*"
                    }
                },
                "Resource": [
                    "arn:aws:s3:::testing.com"
                ]
            },
            {
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::testing.com/testing/main/*"
                ]
            }
        ]
    }

    Here's my Azure example Json (not sure how to define the bucket):

    {
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
    "location": {
    "value": "East US"
    },
    "factoryName": {
    "value": "[concat('adfs3toazureblob', uniqueString(resourceGroup().id, deployment().name))]"
    },
    "sourceAmazonS3AccessKeyId": {
    "value": "xxxxxxx"
    },
    "sourceAmazonS3SecretAccessKey": {
    "value": "xxxxxxxxxx"
    },
    "s3BucketName": {
    "value": <what goes here?>
    },
    "s3BucketRegion": {
    "value": {
    "regionName": "US East (N. Virginia)",
    "region": "us-east-1"
    }
    },
    "targetBlobStorageAccountNameOrConnectionString": {
    "value": test
    },
    "targetBlobStorageContainer": {
    "value": test
    }
    }
    }

    Please let me know if you need more information.

    Thanks!

    Wednesday, May 29, 2019 8:52 PM

All replies

  • Hi Cloudking1,

    You will need to provide the following permissions :

    • s3:GetObject and s3:GetObjectVersion for Amazon S3 Object Operations.
    • s3:ListBucket or s3:GetBucketLocation for Amazon S3 Bucket Operations.
    • s3:ListAllMyBuckets is also required for Copy activities.

    For more information, please refer this doc.

    Please try assigining the above permissions to see if it resolves the issue. Else, we can gladly continue to dive deeper into the issue.


    Thursday, May 30, 2019 7:24 AM
  • Thank you very much for the quick reply.  I can't assign the "s3:ListAllMyBuckets" to this bucket unfortunately.  Does AzCopy require the same permissions?  I thought I could possibly use that instead.
    Thursday, May 30, 2019 5:30 PM