none
Date Time Filter

    Question

  • Hi Team,

    Currently I am trying to implement incremental load via ADF. Every time I run my copy Activity I see the data gets doubled to azure table storage. That is if there are 50 rows in my source table and If I run twice my job from Source to Staging(Azure Table Storage)  has 100 rows. When I try to filter I am getting errors and none of the rows are return.

    DateTime present in Table Storage is like this
    2018-10-02T18:30:00+00:00. But in source time format is not coming.
      "azureTableSourceQuery": {
                                "value": "CREATEDDATE gt  @{activity('Lookup1').output.firstRow.LASTRUNDATE} and CREATEDDATE lt @{activity('Lookup1').output.firstRow.CURRENTDATE}",
                                "type": "Expression"
                            }


    Monday, October 8, 2018 1:32 AM

All replies

  • 1. Both your source and sink are azure table storage, right?

    2. Please follow this doc link.

    If your source column is of DateTime type, you should change your query as following:

    "azureTableSourceQuery": {
                                "value": "CREATEDDATE gt  datetime'@{activity('Lookup1').output.firstRow.LASTRUNDATE}' and CREATEDDATE lt datetime'@{activity('Lookup1').output.firstRow.CURRENTDATE}"',
                                "type": "Expression"
                            }

    3. If above doesn't work, please provided the failure activity id and the error message.

    Wednesday, October 10, 2018 3:12 AM