none
ADF V2 connectivity to AWS S3 Bucket is failing

    Question

  • I got the AWS S3 bucket Access Key and Secret Key and when i try to create linked service, i get connectivity error. When i use AWS command line utility i was able to connect and perform all actions like aws s3 ls, cp, mv etc. I went ahead and created the data set and while i browse for files i get below error -

    UserError: Access to location of bucket  is forbidden, check whether 's3:ListBucket' or 's3:GetBucketLocation' permission granted to the AWS account.'Type=,Message=Access Denied,Source=,', activityId: 07f7f6f8-eb08-43a3-adc8-1886c89267e3

    Wednesday, May 16, 2018 2:17 AM

All replies

  • Can you double check if you have whitelisted Azure IP range on your S3 account?  While performing the copy, ADF will be trying to read data from S3 and S3 needs to configured to allow requests from this range of IPs to perform a read on the data.
    Wednesday, May 16, 2018 8:34 AM
    Moderator
  • Thanks Wang, my ADF is in USEAST2 and i got the microsoft Data Center ip list for useast2, which is as below. Should i send this list to S3 Bucket provider?

      <Region Name="useast2">
        <IpRange Subnet="13.68.0.0/17" />
        <IpRange Subnet="13.77.64.0/18" />
        <IpRange Subnet="20.36.128.0/17" />
        <IpRange Subnet="20.186.0.0/17" />
        <IpRange Subnet="20.186.128.0/18" />
        <IpRange Subnet="20.190.131.0/24" />
        <IpRange Subnet="20.190.192.0/18" />
        <IpRange Subnet="23.100.64.0/21" />
        <IpRange Subnet="23.101.32.0/20" />
        <IpRange Subnet="23.101.80.0/20" />
        <IpRange Subnet="23.101.144.0/20" />
        <IpRange Subnet="23.102.96.0/19" />
        <IpRange Subnet="23.102.204.0/22" />
        <IpRange Subnet="23.102.208.0/20" />
        <IpRange Subnet="40.65.192.0/18" />
        <IpRange Subnet="40.67.128.0/19" />
        <IpRange Subnet="40.70.0.0/18" />
        <IpRange Subnet="40.70.64.0/20" />
        <IpRange Subnet="40.70.80.0/21" />
        <IpRange Subnet="40.70.128.0/17" />
        <IpRange Subnet="40.75.0.0/19" />
        <IpRange Subnet="40.75.64.0/18" />
        <IpRange Subnet="40.77.128.128/25" />
        <IpRange Subnet="40.77.129.0/24" />
        <IpRange Subnet="40.77.130.0/25" />
        <IpRange Subnet="40.77.132.0/24" />
        <IpRange Subnet="40.77.136.48/28" />
        <IpRange Subnet="40.77.163.0/24" />
        <IpRange Subnet="40.77.166.160/27" />
        <IpRange Subnet="40.77.167.0/24" />
        <IpRange Subnet="40.77.168.0/24" />
        <IpRange Subnet="40.77.170.0/24" />
        <IpRange Subnet="40.77.175.96/27" />
        <IpRange Subnet="40.77.177.0/24" />
        <IpRange Subnet="40.77.178.0/23" />
        <IpRange Subnet="40.77.182.0/28" />
        <IpRange Subnet="40.77.182.32/27" />
        <IpRange Subnet="40.77.184.0/25" />
        <IpRange Subnet="40.77.224.128/25" />
        <IpRange Subnet="40.77.228.0/24" />
        <IpRange Subnet="40.77.233.0/24" />
        <IpRange Subnet="40.77.234.192/27" />
        <IpRange Subnet="40.77.240.0/25" />
        <IpRange Subnet="40.77.245.0/24" />
        <IpRange Subnet="40.77.248.0/25" />
        <IpRange Subnet="40.77.251.0/24" />
        <IpRange Subnet="40.78.208.48/28" />
        <IpRange Subnet="40.78.220.0/24" />
        <IpRange Subnet="40.79.0.0/21" />
        <IpRange Subnet="40.79.8.0/27" />
        <IpRange Subnet="40.79.8.32/28" />
        <IpRange Subnet="40.79.8.64/27" />
        <IpRange Subnet="40.79.8.96/28" />
        <IpRange Subnet="40.79.9.0/24" />
        <IpRange Subnet="40.79.16.0/20" />
        <IpRange Subnet="40.79.32.0/20" />
        <IpRange Subnet="40.79.48.0/27" />
        <IpRange Subnet="40.79.48.32/28" />
        <IpRange Subnet="40.79.49.0/24" />
        <IpRange Subnet="40.79.56.0/21" />
        <IpRange Subnet="40.79.64.0/20" />
        <IpRange Subnet="40.79.80.0/21" />
        <IpRange Subnet="40.79.90.0/24" />
        <IpRange Subnet="40.79.91.0/28" />
        <IpRange Subnet="40.79.92.0/24" />
        <IpRange Subnet="40.79.93.0/28" />
        <IpRange Subnet="40.79.94.0/24" />
        <IpRange Subnet="40.79.95.0/28" />
        <IpRange Subnet="40.82.4.0/22" />
        <IpRange Subnet="40.84.0.0/17" />
        <IpRange Subnet="40.87.168.0/22" />
        <IpRange Subnet="40.90.130.160/27" />
        <IpRange Subnet="40.90.132.128/26" />
        <IpRange Subnet="40.90.136.0/28" />
        <IpRange Subnet="40.90.137.32/27" />
        <IpRange Subnet="40.90.138.160/27" />
        <IpRange Subnet="40.90.140.160/27" />
        <IpRange Subnet="40.90.140.192/27" />
        <IpRange Subnet="40.90.144.64/26" />
        <IpRange Subnet="40.90.145.32/27" />
        <IpRange Subnet="40.90.145.64/27" />
        <IpRange Subnet="40.90.146.192/27" />
        <IpRange Subnet="40.90.148.96/27" />
        <IpRange Subnet="40.91.12.16/28" />
        <IpRange Subnet="40.91.12.48/28" />
        <IpRange Subnet="40.91.12.64/26" />
        <IpRange Subnet="40.91.12.128/28" />
        <IpRange Subnet="40.91.12.160/27" />
        <IpRange Subnet="40.91.12.208/28" />
        <IpRange Subnet="40.91.12.240/28" />
        <IpRange Subnet="40.91.13.64/27" />
        <IpRange Subnet="40.91.13.96/28" />
        <IpRange Subnet="40.91.13.128/27" />
        <IpRange Subnet="40.91.13.240/28" />
        <IpRange Subnet="40.91.14.0/24" />
        <IpRange Subnet="40.123.0.0/17" />
        <IpRange Subnet="40.126.3.0/24" />
        <IpRange Subnet="52.109.4.0/22" />
        <IpRange Subnet="52.114.136.0/21" />
        <IpRange Subnet="52.114.180.0/23" />
        <IpRange Subnet="52.115.48.0/22" />
        <IpRange Subnet="52.115.52.0/23" />
        <IpRange Subnet="52.136.29.0/24" />
        <IpRange Subnet="52.138.80.0/21" />
        <IpRange Subnet="52.138.96.0/19" />
        <IpRange Subnet="52.143.192.0/24" />
        <IpRange Subnet="52.147.160.0/19" />
        <IpRange Subnet="52.167.0.0/16" />
        <IpRange Subnet="52.177.0.0/16" />
        <IpRange Subnet="52.179.128.0/17" />
        <IpRange Subnet="52.184.128.0/19" />
        <IpRange Subnet="52.184.160.0/21" />
        <IpRange Subnet="52.184.168.0/28" />
        <IpRange Subnet="52.184.168.80/28" />
        <IpRange Subnet="52.184.168.96/27" />
        <IpRange Subnet="52.184.168.128/28" />
        <IpRange Subnet="52.184.169.0/24" />
        <IpRange Subnet="52.184.170.0/24" />
        <IpRange Subnet="52.184.176.0/20" />
        <IpRange Subnet="52.184.192.0/18" />
        <IpRange Subnet="52.225.128.0/21" />
        <IpRange Subnet="52.225.136.0/27" />
        <IpRange Subnet="52.225.136.32/28" />
        <IpRange Subnet="52.225.136.64/28" />
        <IpRange Subnet="52.225.137.0/24" />
        <IpRange Subnet="52.225.192.0/18" />
        <IpRange Subnet="52.232.151.0/24" />
        <IpRange Subnet="52.232.160.0/19" />
        <IpRange Subnet="52.232.192.0/18" />
        <IpRange Subnet="52.239.156.0/24" />
        <IpRange Subnet="52.239.157.0/25" />
        <IpRange Subnet="52.239.157.128/26" />
        <IpRange Subnet="52.239.157.192/27" />
        <IpRange Subnet="52.239.172.0/22" />
        <IpRange Subnet="52.239.184.0/25" />
        <IpRange Subnet="52.239.184.160/28" />
        <IpRange Subnet="52.239.184.192/27" />
        <IpRange Subnet="52.239.185.32/27" />
        <IpRange Subnet="52.239.185.64/27" />
        <IpRange Subnet="52.239.192.0/25" />
        <IpRange Subnet="52.239.192.160/27" />
        <IpRange Subnet="52.239.192.192/26" />
        <IpRange Subnet="52.239.198.0/25" />
        <IpRange Subnet="52.239.198.160/27" />
        <IpRange Subnet="52.239.198.192/26" />
        <IpRange Subnet="52.239.206.0/24" />
        <IpRange Subnet="52.239.207.0/27" />
        <IpRange Subnet="52.239.207.32/28" />
        <IpRange Subnet="52.239.207.64/26" />
        <IpRange Subnet="52.239.207.128/26" />
        <IpRange Subnet="52.239.222.0/23" />
        <IpRange Subnet="52.242.64.0/18" />
        <IpRange Subnet="52.245.44.0/24" />
        <IpRange Subnet="52.245.45.0/25" />
        <IpRange Subnet="52.245.45.128/28" />
        <IpRange Subnet="52.245.45.160/27" />
        <IpRange Subnet="52.245.45.192/26" />
        <IpRange Subnet="52.245.46.0/27" />
        <IpRange Subnet="52.245.46.48/28" />
        <IpRange Subnet="52.245.46.64/28" />
        <IpRange Subnet="52.245.46.112/28" />
        <IpRange Subnet="52.245.46.128/28" />
        <IpRange Subnet="52.245.46.160/27" />
        <IpRange Subnet="52.245.46.192/27" />
        <IpRange Subnet="52.245.46.224/28" />
        <IpRange Subnet="52.247.0.0/17" />
        <IpRange Subnet="52.250.128.0/18" />
        <IpRange Subnet="52.251.0.0/17" />
        <IpRange Subnet="52.252.0.0/17" />
        <IpRange Subnet="52.253.64.0/20" />
        <IpRange Subnet="52.253.148.0/23" />
        <IpRange Subnet="52.253.154.0/23" />
        <IpRange Subnet="52.254.0.0/18" />
        <IpRange Subnet="52.254.64.0/19" />
        <IpRange Subnet="52.254.96.0/20" />
        <IpRange Subnet="52.254.112.0/21" />
        <IpRange Subnet="65.52.108.0/23" />
        <IpRange Subnet="65.52.110.0/24" />
        <IpRange Subnet="65.55.44.16/28" />
        <IpRange Subnet="65.55.44.32/27" />
        <IpRange Subnet="65.55.44.64/27" />
        <IpRange Subnet="65.55.44.96/28" />
        <IpRange Subnet="65.55.44.128/27" />
        <IpRange Subnet="65.55.60.188/30" />
        <IpRange Subnet="65.55.105.0/26" />
        <IpRange Subnet="65.55.105.96/27" />
        <IpRange Subnet="65.55.105.224/27" />
        <IpRange Subnet="65.55.106.0/26" />
        <IpRange Subnet="65.55.106.64/27" />
        <IpRange Subnet="65.55.106.128/26" />
        <IpRange Subnet="65.55.107.48/28" />
        <IpRange Subnet="65.55.107.64/27" />
        <IpRange Subnet="65.55.108.0/24" />
        <IpRange Subnet="65.55.209.128/26" />
        <IpRange Subnet="65.55.211.32/27" />
        <IpRange Subnet="65.55.213.64/26" />
        <IpRange Subnet="65.55.213.128/26" />
        <IpRange Subnet="65.55.217.0/24" />
        <IpRange Subnet="65.55.219.32/27" />
        <IpRange Subnet="65.55.219.128/25" />
        <IpRange Subnet="104.44.88.32/27" />
        <IpRange Subnet="104.44.88.96/27" />
        <IpRange Subnet="104.44.91.96/27" />
        <IpRange Subnet="104.44.93.160/27" />
        <IpRange Subnet="104.44.94.48/28" />
        <IpRange Subnet="104.44.95.208/28" />
        <IpRange Subnet="104.46.0.0/21" />
        <IpRange Subnet="104.46.96.0/19" />
        <IpRange Subnet="104.46.192.0/20" />
        <IpRange Subnet="104.47.200.0/21" />
        <IpRange Subnet="104.208.128.0/17" />
        <IpRange Subnet="104.209.128.0/17" />
        <IpRange Subnet="104.210.0.0/20" />
        <IpRange Subnet="131.253.12.208/28" />
        <IpRange Subnet="131.253.12.224/30" />
        <IpRange Subnet="131.253.13.16/29" />
        <IpRange Subnet="131.253.13.48/28" />
        <IpRange Subnet="131.253.13.72/29" />
        <IpRange Subnet="131.253.13.80/29" />
        <IpRange Subnet="131.253.13.96/30" />
        <IpRange Subnet="131.253.14.16/28" />
        <IpRange Subnet="131.253.14.208/28" />
        <IpRange Subnet="131.253.14.224/28" />
        <IpRange Subnet="131.253.15.8/29" />
        <IpRange Subnet="131.253.15.16/28" />
        <IpRange Subnet="131.253.24.0/28" />
        <IpRange Subnet="131.253.24.192/26" />
        <IpRange Subnet="131.253.34.224/27" />
        <IpRange Subnet="131.253.38.0/27" />
        <IpRange Subnet="131.253.38.128/26" />
        <IpRange Subnet="134.170.221.0/24" />
        <IpRange Subnet="137.116.0.0/18" />
        <IpRange Subnet="137.116.64.0/19" />
        <IpRange Subnet="137.116.96.0/22" />
        <IpRange Subnet="157.55.7.128/26" />
        <IpRange Subnet="157.55.10.192/26" />
        <IpRange Subnet="157.55.11.128/25" />
        <IpRange Subnet="157.55.37.0/24" />
        <IpRange Subnet="157.55.38.0/24" />
        <IpRange Subnet="157.55.48.0/24" />
        <IpRange Subnet="157.55.50.0/25" />
        <IpRange Subnet="157.55.55.100/30" />
        <IpRange Subnet="157.55.55.104/29" />
        <IpRange Subnet="157.55.55.136/29" />
        <IpRange Subnet="157.55.55.144/29" />
        <IpRange Subnet="157.55.55.160/29" />
        <IpRange Subnet="157.56.2.128/25" />
        <IpRange Subnet="157.56.3.0/25" />
        <IpRange Subnet="191.236.192.0/18" />
        <IpRange Subnet="191.237.128.0/18" />
        <IpRange Subnet="191.239.224.0/20" />
        <IpRange Subnet="193.149.64.0/21" />
        <IpRange Subnet="199.30.18.0/23" />
        <IpRange Subnet="199.30.20.0/24" />
        <IpRange Subnet="199.30.22.0/24" />
        <IpRange Subnet="199.30.28.64/26" />
        <IpRange Subnet="199.30.28.128/25" />
        <IpRange Subnet="199.30.29.0/24" />

    Wednesday, May 16, 2018 3:06 PM
  • The S3 Bucket Provider ( Weather Source) came back and said that there are no IP restrictions.

    Wednesday, May 16, 2018 3:13 PM
  • Can you run this command from AWS CLI? I recently went through this same error and had to get updated permissions to run the get-bucket-location command.

    aws s3api get-bucket-location --bucket yourBucketName

    Wednesday, May 16, 2018 9:23 PM
  • You mentioned you had access through the CLI, but without knowing details of your CLI environment I cannot say if that is definitive proof your user account has the correct permissions.  For example, if you are running the CLI on an instance which has an instance profile attached to it, your CLI environment could be granted permissions from an IAM role:

    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html

    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

    I would check all permissions listed here:

    https://docs.microsoft.com/en-us/azure/data-factory/connector-amazon-simple-storage-service#required-permissions

    Make sure those permissions encompass all the S3 resources you would like to access.  The AWS IAM user account you are using must have those permissions assigned directly to it, or your AWS IAM user account must inherit those permissions through an IAM group:

    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html

    An example IAM policy for S3 can be found here:

    https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-overview.html#access-control-resources-manage-permissions-basics

    Wednesday, May 16, 2018 10:42 PM
    Moderator
  • I getting access denied when i run this command from CLI
    Thursday, May 17, 2018 6:14 PM
  • I have a call with S3 Bucket Provider to see if he can provide below necessary permission - 

    • s3:GetObject and s3:GetObjectVersion for Amazon S3 Object Operations.
    • s3:ListBucket or s3:GetBucketLocation for Amazon S3 Bucket Operations.
    • Since we are using the Data Factory Copy Wizard, s3:ListAllMyBuckets is also required.

    I will keep you posted 

    Thursday, May 17, 2018 6:18 PM