locked
Data Factory V2 - Can't use dataset parameter RRS feed

  • Question

  • Hi,

    I have a strange problem. I want to control the name of my output file with a parameter.

    1. If I hard code the output file name it works.
    2. If I create a pipeline parameter and use the code @{pipeline().parameters.fileparamPipe} it works
    3. If I create a dataset parameter and use the code @{dataset().fileParam} it won't even compile

    The parameters are set to a very simple name while testing. For example Filetest.csv

    Since I am iterating over a colletion of files I want to set a dataset parameter for each output file.

    The problem is also there for http requests (which is my input)

    Does anyone know where I go wrong here? 


    best wishes /Carl

    Friday, March 2, 2018 8:11 AM

Answers

  • Confirmed on my side, space in the dataset with parameters will lead to errors. I'll notify the team as a bug and see if we can fix them.

    Thanks

    Thursday, March 15, 2018 4:12 AM

All replies

  • The dataset parameter works fine on my side. Would you share your dataset's config? How you defined your parameter and used?
    Friday, March 2, 2018 9:25 AM
  • Hi, absolutely.

    here is the dataset

    {
        "name": "Blob Bob Productmapping Channel",
        "properties": {
            "linkedServiceName": {
                "referenceName": "MimerDevBlobStorage",
                "type": "LinkedServiceReference"
            },
            "parameters": {
                "fileParam": {
                    "type": "String",
                    "defaultValue": "MIXPAK.csv"
                }
            },
            "type": "AzureBlob",
            "structure": [
                {
                    "name": "channeltmsid",
                    "type": "String"
                },
                {
                    "name": "channelid",
                    "type": "String"
                }
            ],
            "typeProperties": {
                "format": {
                    "type": "TextFormat",
                    "columnDelimiter": "\u001e",
                    "treatEmptyAsNull": true,
                    "firstRowAsHeader": false
                },
                "fileName": {
                    "value": "@{dataset().fileParam}",
                    "type": "Expression"
                },
                "folderPath": "sourcefiles/bob/productmapping/channel"
            }
        }
    }
    I would like to set the dataset parameter from the activity with 

    @{item().ProductCode} but at this point the input from the activity is not set

    Thanks!



    best wishes /Carl

    Friday, March 2, 2018 10:23 AM
  • Thanks Carl, the dataset's config looks correct. Could you share your pipeline's config? Where did you encounter this error, on powershell, SDK, or UI?
    Monday, March 5, 2018 5:57 AM
  • Hi again, sorry for not replying earlier.

    I have used mainly the UI. Using powershell was an option but part of the code was lost after deploying so that is not an option now.

    I was able to create my datasets from scratch and changing the pipeline/activity datasets to these new ones and then it worked.

    The code was exactly the same except for the dataset names.

    And now, after a lot of effort I now have a working pipeline.

     

    However, when I try to change the names of datasets to more describing ones, the pipeline stops working. I am doing this in the GUI and using the Debug button. The error message is the same as above.

    best regards

    /Carl


    best wishes /Carl

    Monday, March 12, 2018 7:37 AM
  • Hi Carl,

    May I ask the new name you used for the dataset which caused the error? Maybe some characters are not supported on dataset name.

    Thanks

    Tuesday, March 13, 2018 5:53 AM
  • Hi,

    so here is the code for the dataset.

    {
        "name": "bobx",
        "properties": {
            "linkedServiceName": {
                "referenceName": "Bob",
                "type": "LinkedServiceReference"
            },
            "parameters": {
                "param": {
                    "type": "String",
                    "defaultValue": "TVG007"
                }
            },
            "type": "HttpFile",
            "structure": [
                {
                    "name": "channeltmsid",
                    "type": "String"
                },
                {
                    "name": "channelid",
                    "type": "String"
                }
            ],
            "typeProperties": {
                "format": {
                    "type": "JsonFormat",
                    "filePattern": "setOfObjects",
                    "jsonNodeReference": "$.channels",
                    "jsonPathDefinition": {
                        "channeltmsid": "tmsId",
                        "channelid": "id"
                    }
                },
                "relativeUrl": {
                    "value": "/bob/api/public/channel-list?serviceId=@{dataset().param}",
                    "type": "Expression"
                },
                "requestMethod": "Get"
            }
        }
    }

    And I want to change the name to "Bob channellist channel".


    best regards

    /carl


    best wishes /Carl

    Tuesday, March 13, 2018 7:18 AM
  • Hi again,

    just for fun I removed the spaces and renamed the dataset to "Bobchannellistchannel".

    This works.

    Which is puzzling because I have other datasets with space that actually work also.

    best regards

    /Carl


    best wishes /Carl

    Tuesday, March 13, 2018 7:26 AM
  • Is other datasets with space in name also have parameters? I'm thinking that the space in name might have caused errors in parsing datasets' parameters.
    Tuesday, March 13, 2018 8:20 AM
  • Confirmed on my side, space in the dataset with parameters will lead to errors. I'll notify the team as a bug and see if we can fix them.

    Thanks

    Thursday, March 15, 2018 4:12 AM