none
Need help with : How to pass header to Rest Dataset GET request in Azure Data Factory RRS feed

  • Question

  • I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here
    https://docs.microsoft.com/en-us/azure/data-factory/connector-rest & https://www.alexvolok.com/2019/adfv2-rest-api-part1-oauth2/

    The API which I'm trying to connect uses OAuth2 so I need to first get the access token and then do a get request to get actual data.

    Below are the steps which I'm following

    1. Creating a Web HTTP request in the pipeline and passing the client_ID, client secret, username, password and grant type in the body of the request. When I debug the pipline I do get the Access_token which I need in step 2.

    2. In Step two I have a copy activity which uses the output(access_token) from web to authenticate the second REST GET request but this is where I'm facing a lot of issues. The code which I'm using is "@concat('Bearer ', activity('GetAccessToken').output.access_token)"

    3. In step 3 I have two datasets and 2 Linked services, Dataset 1 is a REST dataset which has the base url and relative url which is linked to the REST linked service and secondly the sink dataset is connected to AZURE datalake storage.

    4. In Source Dataset I'm passing additional header Authorization = @concat('Bearer ', activity('GetAccessToken').output.access_token) and ideally since the API which I want to call will return empty if no parameters are send so I pass in the parameters inside the "Request Body" is that even correct? the Request body would look something like this "start_date=2020/07/17&end_date=2020/07/18".

    5. The sink is a simple Json dataset stored in DataLake.

    When I try to debug I get the error as below


    But I'm getting the below error

        {
          "errorCode": "2200",
          "message": "Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 401 indicates failure. { \"Error\": { \"Message\":\"Authentication failed: Invalid headers\", \"Server-Time\":\"2020-07-27T06:59:24\", \"Id\":\"6AAF87BC-5634-4C28-8626-810A19B86BFF\" } },Source=Microsoft.DataTransfer.ClientLibrary,'",
          "failureType": "UserError",
          "target": "CopyDataFromAPI",
          "details": []
        }
    I've tried in Postman and Python and the get request works fine there.

    Please advise if I'm doing anything wrong.
    Tuesday, July 28, 2020 3:11 PM