locked
Http File - How to use relative url with dynamic date content (ADF v2) RRS feed

  • Question

  • In Data Factory V2, I have an Http file (dataset) that I have configured as a data input.
    I have set a Base URL and I also want to add a relative URL with dynamic dates.

    My pipeline is working if I harcode my dates in the relative URL field, like this:

    @concat('?from_date=', '2018-05-29','&to_date=', '2018-05-29')

    However I get an error if I to try to use the formatDateTime or adddays functions:

    @concat('?from_date=', adddays(utcnow(),-1,'yyyy-mm-dd'),'&to_date=', adddays(utcnow(),-1,'yyyy-mm-dd'))
    @concat('?from_date=',formatDateTime(utcnow(),'yyyy-mm-dd'),'&to_date=',formatDateTime(utcnow(),'yyyy-mm-dd'))
    Here is the error that I get:
    {
        "errorCode": "2200",
        "message": "ErrorCode=UserErrorFailedToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http source file.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=System,'",
        "failureType": "UserError",
        "target": "ANameIGave"
    }
    I wonder what I am doing wrong and how I can debug this further. (For example I don't know how to find what is the final relative url that is generated in the end. I only see this error).
    Wednesday, May 30, 2018 6:54 PM

Answers

  • Thanks for your answer!  I finally found out what the problem was.
    The problem was not related to missing string interpolatio expression, but was simply because my date format was incorrect. 
    The format 'yyyy-mm-dd' was incorrect for the month part. It should be in capital letters like this 'yyyy-MM-dd'.

    The correct relative url to pass the date of yesterday in my url looks like this:

    @concat('?from_date=', adddays(utcnow(),-1,'yyyy-MM-dd'),'&to_date=', adddays(utcnow(),-1,'yyyy-MM-dd'))

    • Marked as answer by Yabadabadou Thursday, May 31, 2018 2:31 PM
    Thursday, May 31, 2018 2:31 PM

All replies

  • Could you try string interpolation expression. Which means adding {} around your expression except @. Like this: @{concat(***)}
    Thursday, May 31, 2018 12:27 AM
  • Thanks for your answer!  I finally found out what the problem was.
    The problem was not related to missing string interpolatio expression, but was simply because my date format was incorrect. 
    The format 'yyyy-mm-dd' was incorrect for the month part. It should be in capital letters like this 'yyyy-MM-dd'.

    The correct relative url to pass the date of yesterday in my url looks like this:

    @concat('?from_date=', adddays(utcnow(),-1,'yyyy-MM-dd'),'&to_date=', adddays(utcnow(),-1,'yyyy-MM-dd'))

    • Marked as answer by Yabadabadou Thursday, May 31, 2018 2:31 PM
    Thursday, May 31, 2018 2:31 PM