locked
Using pipeline expressions to provide dataflow parameter values RRS feed

  • Question

  • I have a dataflow that has two parameters. 

    In the pipeline that calls the dataflow, the parameters page shows the two dataflow parameters.

    When I click in the cell for the dataflow parameter value, it brings up the dataflow expression editor. 

    I'd really like to use a pipeline expression to fill a dataflow parameter value (it seems kinda self-referencing to use a dataflow expression to fill a dataflow parameter).

    At some point in the last week, I think I remember that when I clicked in the cell, I got an option to choose the pipeline expression editor or the dataflow expression editor, then there was an icon at the end of the cell showing what kind of expression it was. I can't seem to get that back. Am I losing my mind? 

    Mike

    Wednesday, September 4, 2019 9:19 PM

Answers

  • Aha! I figured it out.  It seems the data type you choose in the dataflow parameter, can restrict which options you have for expressions.  In testing this I created a string, a timestamp, and a short.

    For the string type, I had the option of Data Flow expression or Pipeline expression.

    For the timestamp and the short both, I was immediately taken to the Data Flow expression.

    The explanation that comes to mind, is that some data types are not available in regular pipeline parameters.  When working with a datetime in regular pipelines, it is stored in string type variables/parameters, and validation is not applied.  Inside a pipeline expression conversions are carried out, and at that point some validations are enacted.

    In contrast, the Data Flow has the native timestamp data type.

    We can still work with this.  I am working on a work-around now.

    Wednesday, September 4, 2019 10:53 PM
  • You will need to use a Data Flow parameter type of String in order to use the option of pipeline parameters.

    https://docs.microsoft.com/en-us/azure/data-factory/parameters-data-flow

     Note

    To use pipeline control flow expressions, your data flow parameter must be of type string.

    Friday, September 6, 2019 8:37 PM

All replies

  • Hello Mike and thank you for your inquiry.  You are not losing your mind.  I recall seeing the same option.

    Trying now, I see:

    Are you not seeing the same when clicking on the text box?

      The steps I took were, creating new pipeline, and execute Data Flow activity, using the 'create new flow' option when choosing a flow.  I created parameter inside the flow without assigning any default value.  Then I come back to the pipeline view and clicked to see the above.

    Wednesday, September 4, 2019 10:12 PM
  • Aha! I figured it out.  It seems the data type you choose in the dataflow parameter, can restrict which options you have for expressions.  In testing this I created a string, a timestamp, and a short.

    For the string type, I had the option of Data Flow expression or Pipeline expression.

    For the timestamp and the short both, I was immediately taken to the Data Flow expression.

    The explanation that comes to mind, is that some data types are not available in regular pipeline parameters.  When working with a datetime in regular pipelines, it is stored in string type variables/parameters, and validation is not applied.  Inside a pipeline expression conversions are carried out, and at that point some validations are enacted.

    In contrast, the Data Flow has the native timestamp data type.

    We can still work with this.  I am working on a work-around now.

    Wednesday, September 4, 2019 10:53 PM
  • My idea for a work-around failed.  Have I at least helped clear some confusion?
    Thursday, September 5, 2019 10:50 PM
  • You will need to use a Data Flow parameter type of String in order to use the option of pipeline parameters.

    https://docs.microsoft.com/en-us/azure/data-factory/parameters-data-flow

     Note

    To use pipeline control flow expressions, your data flow parameter must be of type string.

    Friday, September 6, 2019 8:37 PM