locked
Tumbling Windows. Reset Filename in JSON and Republished but the old name still appears in the data Lake RRS feed

  • Question

  • Hi, I recently created a Tumbling Windows Example. After running successfully I decided to set the file name as a dynamic filename. I already had one that worked so I used this. 

    I Published the change and assumed that the new file name would be created but its still adding data to the old file name

    Here is the new JSON script

    {
        "name": "CopyFactWindowourData",
        "properties": {
            "activities": [
                {
                    "name": "Copy_84q",
                    "type": "Copy",
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [
                        {
                            "name": "Destination",
                            "value": "@{concat('/trainingData/', formatDateTime(pipeline().parameters.windowStart,'yyyy'),'/',formatDateTime(pipeline().parameters.windowStart,'MM'),'/',formatDateTime(pipeline().parameters.windowStart,'dd'),'/', formatDateTime(pipeline().parameters.windowStart,'HH'))}/@concat('Sales_',formatDateTime(adddays(utcnow(),-1),'yyyyMMddHHmm'),'.csv’)"
                        }
                    ],
                    "typeProperties": {
                        "source": {
                            "type": "SqlSource",
                            "sqlReaderQuery": {
                                "value": "SELECT [ID] , [DateTimeEvent], [Dim1Key], [Dim2Key],[Fact1],[Fact2]\nFROM [dbo].[FactWindowHour]\nwhere [DateTimeEvent] >= '@{formatDateTime(pipeline().parameters.windowStart, 'yyyy-MM-dd HH:mm' )}' \nAND [DateTimeEvent] < '@{formatDateTime(pipeline().parameters.windowEnd, 'yyyy-MM-dd HH:mm' )}'\n",
                                "type": "Expression"
                            }
                        },
                        "sink": {
                            "type": "AzureDataLakeStoreSink"
                        },
                        "enableStaging": false
                    },
                    "inputs": [
                        {
                            "referenceName": "SourceDataset_84q",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "DestinationDataset_84q",
                            "type": "DatasetReference",
                            "parameters": {
                                "cw_folderPath": {
                                    "value": "@{concat('/trainingData/', formatDateTime(pipeline().parameters.windowStart,'yyyy'),'/',formatDateTime(pipeline().parameters.windowStart,'MM'),'/',formatDateTime(pipeline().parameters.windowStart,'dd'),'/', formatDateTime(pipeline().parameters.windowStart,'HH'))}",
                                    "type": "Expression"
                                }
                            }
                        }
                    ]
                }
            ],
            "parameters": {
                "windowStart": {
                    "type": "String"
                },
                "windowEnd": {
                    "type": "String"
                }
            }
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }
    

    The old filename isnt here any more so Im wondering why it is still saving data to the old TumblingWindows.csv name?

    Just as an extra. User Properties Destination. If you auto generate it goes back to the original TumblingWindows.csv file name and I have to reset it manually. Its taking that information from somewhere but I cant see where.


    Debbie


    Wednesday, January 23, 2019 10:54 AM

Answers

  • Figured it out. I needed to change it in the data set so it affects everything else. 


    Debbie

    • Marked as answer by Debbie Edwards Wednesday, January 23, 2019 11:31 AM
    Wednesday, January 23, 2019 11:31 AM

All replies

  • Figured it out. I needed to change it in the data set so it affects everything else. 


    Debbie

    • Marked as answer by Debbie Edwards Wednesday, January 23, 2019 11:31 AM
    Wednesday, January 23, 2019 11:31 AM
  • Glad to hear that your issue is resolved. Appreciate for sharing the resolution, this would certainly benefit other community members.

    Wednesday, January 23, 2019 12:15 PM