none
ADF COPY Can't change defaulted Activity times in Pipeline RRS feed

  • Question

  • When creating an ADF COPY from a SQL Server VM into an Azure SQL Database, The COPY pipeline had a start time of 6:00am. However, the final script it generated adds activity periods meaning the COPY activities run at 00:00 CET (1:00 GMT).

    Why is this and how can I change it ?

    I tried a few things but basically there is nowhere to add individual Activity times or an Offset in the creation wizard.

    Also, once you get into JSON scripts, if you add an Offset or time change to the Pipeline script then it complains that it is in conflict with the Output Dataset (The Activity schedule does not match the schedule of the output Dataset. Activity: 'StoredProcedureActivityTemplate'. Dataset: 'OutputDatasetStoPro'.).

    But if you change the Output Dataset script then it says that it is not allowed to do that (Updating the availability section of a Dataset is not supported).

    Below is a snipit from the Pipeline script showing the activity periods.

    Any advice is welcome and thanks in advance.

    "start": "2018-09-06T06:00:00Z",

    "end": "2019-12-31T00:00:00Z",

    "isPaused": false,

    "runtimeInfo": {

        "deploymentTime": "2018-09-07T13:30:35.9855693Z",

        "activePeriodSetTime": "2018-09-07T13:30:36.5535017Z",

        "pipelineState": "Running",

        "activityPeriods": {

            "activity-0-[stg_schema]_[Table1]->[target_schema]_[Table1]": {

                "start": "2018-09-06T00:00:00Z",

                "end": "2019-12-31T00:00:00Z"

            },

            "activity-1-[stg_schema]_[Table2]->[target_schema]_[Table2]": {

                "start": "2018-09-06T00:00:00Z",

                "end": "2019-12-31T00:00:00Z"

            },

            "activity-2-[stg_schema]_[Table3]->[target_schema]_[Table3]": {

                "start": "2018-09-06T00:00:00Z",

                "end": "2019-12-31T00:00:00Z"

    Tuesday, September 11, 2018 5:02 PM

All replies

  • Hi,

    Seems the issue of updating the availability is still not fixed. 

    https://feedback.azure.com/forums/270578-data-factory/suggestions/13115547-azure-data-factory-add-ability-to-update-dataset

    A workaround I can come up with is to use the "clone" button to duplicate another dataset and change your pipeline to reference the new dataset. 

    And you could also consider to use ADF V2. It provides tumbling window trigger and schedule trigger which are easy to understand and use. 

    https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger

    https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-schedule-trigger


    Wednesday, September 12, 2018 1:21 PM