locked
copy activity, REST dataset, additional headers RRS feed

  • Question

  • Hi
    I am trying to use REST as source dataset in Data factory 'Copy data' activity.
    The server requirement is to send an Accept header of application/vnd.api+json and a Content Type header of application/vnd.api+json with each request. 
    Whatever I put as Accept header in Additional headers i always get response:
    "The HttpStatusCode 406 indicates failure. {"errors":[{"title":"Non JSON API Compliant Request Format","detail":"Invalid Accept Header: 'application/json'. JSON requests must comply with the jsonapi.org specification - supplying Accept and Content-Type headers with a value of 'application/vnd.api+json'.","code":"1030","status":"406"}]}"

    I am able to pull this data via Postman or using Web activity or Python or.... 
    I would like to use REST becouse I need pagination

    Please advise
    Thursday, November 21, 2019 7:53 PM

Answers

  • Adam, after much troubleshooting, I have resolved the issue I was having.  The Rest API connector ONLY accepts json and ignores any Accepts additionHeader arguments you pass.  The only way to request a different format is to use the Http connection object. 

    In that connection/dataset type, you specify the additionalHeaders in a very simple, non-json format, such as "key1:value1\nkey2:value2\nkey3:value3".  The Http additionalHeaders field is a textarea, not a field for each additional header key/value pair like the Rest dataset.  So, if you use the dynamic content option to substitute any values, do something like I did, including carriage returns between key/value pairs.  If you try to put the "\n" new line character, ADF adds an additional "\" to it.  Here is how I did my dynamic content:

    @{concat('Accept: text/csv
    Content-Type: application/json
    Authentication: Bearer ', variables('BearerToken'))}

    Hope this helps!

    Monday, November 25, 2019 4:54 AM

All replies

  • Hi Adam,

    I have reached out to internal team to get more info about your query. Will get back to you once I have an update from the team.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 22, 2019 8:34 PM
  • I am having the same issue.  When I send a request with proper headers in Postman, I get expected response.  When I send same headers in ADF, using Rest dataset/connection pair in copy activity, I repeatedly get "Requested representation format is invalid" (error 406).  Also, it appears that Additional Headers have been dropped as properties from the Rest Dataset interface.  I had to clone an existing Rest Dataset from our library that had them and set to get a full config form.
    • Edited by CaConklin Monday, November 25, 2019 3:34 AM
    Monday, November 25, 2019 3:32 AM
  • Adam, after much troubleshooting, I have resolved the issue I was having.  The Rest API connector ONLY accepts json and ignores any Accepts additionHeader arguments you pass.  The only way to request a different format is to use the Http connection object. 

    In that connection/dataset type, you specify the additionalHeaders in a very simple, non-json format, such as "key1:value1\nkey2:value2\nkey3:value3".  The Http additionalHeaders field is a textarea, not a field for each additional header key/value pair like the Rest dataset.  So, if you use the dynamic content option to substitute any values, do something like I did, including carriage returns between key/value pairs.  If you try to put the "\n" new line character, ADF adds an additional "\" to it.  Here is how I did my dynamic content:

    @{concat('Accept: text/csv
    Content-Type: application/json
    Authentication: Bearer ', variables('BearerToken'))}

    Hope this helps!

    Monday, November 25, 2019 4:54 AM
  • Hi CaConklin, thanks much for sharing your findings. 

    Hi Adam Zawadzki, as CaConklin mentioned REST connector only supports "application/json" as "Accept" settings in additional headers.

    If you have any feedback regarding this, I would suggest you to please share your idea/suggestion in ADF user voice forum. All the feedback shared in this forum are monitored and reviewed by Azure Data Factory engineering team and will take appropriate action.

    ADF feedback forum: https://feedback.azure.com/forums/270578-azure-data-factory

    Once you have created the feedback in the forum, please do share the link here, so that other members of the community with similar idea could up-vote/comment your suggestion, which would help to increase the priority of the request.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, November 26, 2019 1:37 AM
  • Hi CaConklin<abbr class="affil" style="border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;margin:0px;outline:0px;padding:0px;display:inline-block;vertical-align:top;color:#000000;"></abbr> 

    Thank you for your reply and confirming that REST connector ignores accept additional header.

    HTTP solution works fine except pagination

    Regards,

    Adam

    Tuesday, November 26, 2019 8:35 AM