locked
Tumbling Window Configuration RRS feed

  • Question

  • Hi

    Please help. I am having trouble with my Tumbling Window Trigger configuration along with this pipeline....is the time in UTC or CST or what? Please see attachment . I set the Tumbling Window Delay to fire off the pipeline every 15 mins (which is easy setting it up) but nothing is happening when the 15 mins Tumbling Window comes around (and I have read the documentation :-) ).  Also please see my JSON...

    Thanks so much!
    Mike


    Mike Kiser


    • Edited by EMKISER Tuesday, November 19, 2019 4:21 AM
    Tuesday, November 19, 2019 4:17 AM

Answers

  • Triggers uses UTC time zone. 

    Did you mean, you don't see the pipeline executions in Monitor? Do cross check if you have any filters in Monitor section which needs to be removed. 


    If the response helped, do "Mark as answer" or upvote it
    - Vaibhav

    • Marked as answer by EMKISER Wednesday, November 20, 2019 12:00 AM
    Tuesday, November 19, 2019 5:12 AM
  • Hi Mike,

    Yes the time is in UTC as shown in the below screenshot :

    From the JSON time, I see that the start time is 19th November UTC around 2 AM. The way it works is that if you give the start time as now or a time in the past, it will start immediately. So can you please try setting the start time to a time in the past or now to see if that works. 

    Tuesday, November 19, 2019 9:28 AM
  • Yes, it is copying files but the trigger run is showing up in monitoring; the pipeline is not. I have tried everything; it's not the filter...

    Mike


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:19 PM
    Wednesday, November 20, 2019 12:01 AM
  • Hello again!

    I have worked on this most of the day and nothing makes sense; I have the Tumbling Windows working at 15 min intervals; when I run my pipeline in Debug the pipeline runs and it says it has copied over 9 files. When I look in my Blob there is ONE file; when I look at the Trigger Runs in the monitor, it shows successful Tumbling Window runs; BUT when I look at my PIPELINE runs there are NONE. Please see all the attachments. Does anyone have any idea why I am seeing this? This should not be that difficult.

    Thanks so much!
    Mike Kiser

    {
        "name": "CopyNewFilesByLastModifiedDate1",
        "properties": {
            "description": "Copy new and changed files only by using LastModifiedDate",
            "activities": [
                {
                    "name": "CopyNewFiles",
                    "description": "Copy new and changed files only by using LastModifiedDate",
                    "type": "Copy",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [
                        {
                            "name": "Source",
                            "value": "@{pipeline().parameters.FolderPath_Source}/"
                        },
                        {
                            "name": "Destination",
                            "value": "Poller2Sink"
                        }
                    ],
                    "typeProperties": {
                        "source": {
                            "type": "BlobSource",
                            "recursive": false
                        },
                        "sink": {
                            "type": "JsonSink",
                            "storeSettings": {
                                "type": "AzureBlobStorageWriteSettings"
                            },
                            "formatSettings": {
                                "type": "JsonWriteSettings",
                                "quoteAllText": true
                            }
                        },
                        "enableStaging": false,
                        "translator": {
                            "value": "@pipeline().parameters.FolderPath_Destination",
                            "type": "Expression"
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "AzureBlobStorageDataSourceStore1",
                            "type": "DatasetReference",
                            "parameters": {
                                "FolderPath": {
                                    "value": "@pipeline().parameters.FolderPath_Source",
                                    "type": "Expression"
                                },
                                "LastModified_From": {
                                    "value": "@pipeline().parameters.LastModified_From",
                                    "type": "Expression"
                                },
                                "LastModified_To": {
                                    "value": "@pipeline().parameters.LastModified_To",
                                    "type": "Expression"
                                }
                            }
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "ScadaBlobs",
                            "type": "DatasetReference"
                        }
                    ]
                },
                {
                    "name": "Copy Data1",
                    "type": "Copy",
                    "dependsOn": [
                        {
                            "activity": "CopyNewFiles",
                            "dependencyConditions": [
                                "Skipped"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "source": {
                            "type": "AzureTableSource",
                            "azureTableSourceIgnoreTableNotFound": false
                        },
                        "sink": {
                            "type": "SqlSink",
                            "writeBatchSize": 10000,
                            "sqlWriterStoredProcedureName": "[dbo].[TestProcedure]",
                            "sqlWriterTableType": "TestTable"
                        },
                        "enableStaging": false,
                        "translator": {
                            "type": "TabularTranslator",
                            "columnMappings": {
                                "PartitionKey": "Column1",
                                "RowKey": "Column2",
                                "Timestamp": "Column3"
                            }
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "AzureStorageTable1",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "AzureSqlTable1",
                            "type": "DatasetReference"
                        }
                    ]
                }
            ],
            "parameters": {
                "FolderPath_Source": {
                    "type": "String",
                    "defaultValue": "/<myfolder>/<sub_folder>/"
                },
                "FolderPath_Destination": {
                    "type": "String",
                    "defaultValue": "/<myfolder>/<sub_folder>/"
                },
                "LastModified_From": {
                    "type": "String",
                    "defaultValue": "2019-02-01T00:00:00Z"
                },
                "LastModified_To": {
                    "type": "String",
                    "defaultValue": "2019-06-01T00:00:00Z"
                }
            },
            "annotations": []
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:18 PM
    Wednesday, November 20, 2019 1:03 AM
  • Hi Mike,

    • First, can you please make sure that your linked service is pointing to the right directory and maybe throw in the file name as well if you want to just copy one file.
    • In your pipeline runs, can you please remove the annotation filters and see if that shows the results? I want to understand if there's a problem with the filtering logic.
    • Debug runs are not shown up in pipeline runs. You will see all pipeline runs that are either triggered manually or by a trigger.
    • I also want to know if you have multiple triggers pointing to the same pipeline ? In one of the screenshots you have BT182 as a trigger and you have a modified tumbling trigger in one. This might be a possible cause for trigger fails.

    Hope this helps.

    Wednesday, November 20, 2019 8:39 AM
  • Yes, it is copying files but the trigger run is showing up in monitoring; the pipeline is not. I have tried everything; it's not the filter...

    Please try removing highlighted filter, I guess that's the reason only "Inner Pipeline" is listed there.


    If the response helped, do "Mark as answer" or upvote it
    - Vaibhav

    • Marked as answer by EMKISER Wednesday, November 20, 2019 3:46 PM
    Wednesday, November 20, 2019 8:58 AM
  • Thanks, Vaibhav,

    You were correct; it was the filters. However, when I copy from Blob to Blob using LastModified with Tumbling Windows, it deletes everything in the Blob folder. Any ideas?

    Thanks again!
    Mike


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:18 PM
    Wednesday, November 20, 2019 3:48 PM

All replies

  • Triggers uses UTC time zone. 

    Did you mean, you don't see the pipeline executions in Monitor? Do cross check if you have any filters in Monitor section which needs to be removed. 


    If the response helped, do "Mark as answer" or upvote it
    - Vaibhav

    • Marked as answer by EMKISER Wednesday, November 20, 2019 12:00 AM
    Tuesday, November 19, 2019 5:12 AM
  • Hi Mike,

    Yes the time is in UTC as shown in the below screenshot :

    From the JSON time, I see that the start time is 19th November UTC around 2 AM. The way it works is that if you give the start time as now or a time in the past, it will start immediately. So can you please try setting the start time to a time in the past or now to see if that works. 

    Tuesday, November 19, 2019 9:28 AM
  • Yes, it is copying files but the trigger run is showing up in monitoring; the pipeline is not. I have tried everything; it's not the filter...

    Mike


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:19 PM
    Wednesday, November 20, 2019 12:01 AM
  • Hello again!

    I have worked on this most of the day and nothing makes sense; I have the Tumbling Windows working at 15 min intervals; when I run my pipeline in Debug the pipeline runs and it says it has copied over 9 files. When I look in my Blob there is ONE file; when I look at the Trigger Runs in the monitor, it shows successful Tumbling Window runs; BUT when I look at my PIPELINE runs there are NONE. Please see all the attachments. Does anyone have any idea why I am seeing this? This should not be that difficult.

    Thanks so much!
    Mike Kiser

    {
        "name": "CopyNewFilesByLastModifiedDate1",
        "properties": {
            "description": "Copy new and changed files only by using LastModifiedDate",
            "activities": [
                {
                    "name": "CopyNewFiles",
                    "description": "Copy new and changed files only by using LastModifiedDate",
                    "type": "Copy",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [
                        {
                            "name": "Source",
                            "value": "@{pipeline().parameters.FolderPath_Source}/"
                        },
                        {
                            "name": "Destination",
                            "value": "Poller2Sink"
                        }
                    ],
                    "typeProperties": {
                        "source": {
                            "type": "BlobSource",
                            "recursive": false
                        },
                        "sink": {
                            "type": "JsonSink",
                            "storeSettings": {
                                "type": "AzureBlobStorageWriteSettings"
                            },
                            "formatSettings": {
                                "type": "JsonWriteSettings",
                                "quoteAllText": true
                            }
                        },
                        "enableStaging": false,
                        "translator": {
                            "value": "@pipeline().parameters.FolderPath_Destination",
                            "type": "Expression"
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "AzureBlobStorageDataSourceStore1",
                            "type": "DatasetReference",
                            "parameters": {
                                "FolderPath": {
                                    "value": "@pipeline().parameters.FolderPath_Source",
                                    "type": "Expression"
                                },
                                "LastModified_From": {
                                    "value": "@pipeline().parameters.LastModified_From",
                                    "type": "Expression"
                                },
                                "LastModified_To": {
                                    "value": "@pipeline().parameters.LastModified_To",
                                    "type": "Expression"
                                }
                            }
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "ScadaBlobs",
                            "type": "DatasetReference"
                        }
                    ]
                },
                {
                    "name": "Copy Data1",
                    "type": "Copy",
                    "dependsOn": [
                        {
                            "activity": "CopyNewFiles",
                            "dependencyConditions": [
                                "Skipped"
                            ]
                        }
                    ],
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "source": {
                            "type": "AzureTableSource",
                            "azureTableSourceIgnoreTableNotFound": false
                        },
                        "sink": {
                            "type": "SqlSink",
                            "writeBatchSize": 10000,
                            "sqlWriterStoredProcedureName": "[dbo].[TestProcedure]",
                            "sqlWriterTableType": "TestTable"
                        },
                        "enableStaging": false,
                        "translator": {
                            "type": "TabularTranslator",
                            "columnMappings": {
                                "PartitionKey": "Column1",
                                "RowKey": "Column2",
                                "Timestamp": "Column3"
                            }
                        }
                    },
                    "inputs": [
                        {
                            "referenceName": "AzureStorageTable1",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "AzureSqlTable1",
                            "type": "DatasetReference"
                        }
                    ]
                }
            ],
            "parameters": {
                "FolderPath_Source": {
                    "type": "String",
                    "defaultValue": "/<myfolder>/<sub_folder>/"
                },
                "FolderPath_Destination": {
                    "type": "String",
                    "defaultValue": "/<myfolder>/<sub_folder>/"
                },
                "LastModified_From": {
                    "type": "String",
                    "defaultValue": "2019-02-01T00:00:00Z"
                },
                "LastModified_To": {
                    "type": "String",
                    "defaultValue": "2019-06-01T00:00:00Z"
                }
            },
            "annotations": []
        },
        "type": "Microsoft.DataFactory/factories/pipelines"
    }


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:18 PM
    Wednesday, November 20, 2019 1:03 AM
  • Hi Mike,

    • First, can you please make sure that your linked service is pointing to the right directory and maybe throw in the file name as well if you want to just copy one file.
    • In your pipeline runs, can you please remove the annotation filters and see if that shows the results? I want to understand if there's a problem with the filtering logic.
    • Debug runs are not shown up in pipeline runs. You will see all pipeline runs that are either triggered manually or by a trigger.
    • I also want to know if you have multiple triggers pointing to the same pipeline ? In one of the screenshots you have BT182 as a trigger and you have a modified tumbling trigger in one. This might be a possible cause for trigger fails.

    Hope this helps.

    Wednesday, November 20, 2019 8:39 AM
  • Yes, it is copying files but the trigger run is showing up in monitoring; the pipeline is not. I have tried everything; it's not the filter...

    Please try removing highlighted filter, I guess that's the reason only "Inner Pipeline" is listed there.


    If the response helped, do "Mark as answer" or upvote it
    - Vaibhav

    • Marked as answer by EMKISER Wednesday, November 20, 2019 3:46 PM
    Wednesday, November 20, 2019 8:58 AM
  • Thanks, Vaibhav,

    You were correct; it was the filters. However, when I copy from Blob to Blob using LastModified with Tumbling Windows, it deletes everything in the Blob folder. Any ideas?

    Thanks again!
    Mike


    Mike Kiser

    • Marked as answer by EMKISER Saturday, November 23, 2019 7:18 PM
    Wednesday, November 20, 2019 3:48 PM