locked
Data factory - Insert type of replace and column mapping help RRS feed

  • Question

  • We have a data factory that is pumping data from on prem SQL Server to Azure Storage table.The data pipeline is scheduled to run once a day and next day it runs it duplicates the rows which are duplicates. To remedy this, we updated the pipeline to include the following so that we can use a column as unique identifer and replace the row in the next run.
    "sink": {
                            
    "type": "AzureTableSink",

    "azureTableRowKeyName":"ColumnName",
        
    "azureTableInsertType": "replace",
    "writeBatchSize": 300,

    "writeBatchTimeout": "02:00:00"
      
     }

    I added this JSON below to add a translator column mapping to map the ColumnName and that produces a error:
    Copy activity met invalid parameters: 'Source', Detailed message: Column mapping with empty structure is not supported. Structure could be defined in Table definition.
    Parameter name: Source.

    JSON added for mapping
    "translator": {
                            
    "type": "TabularTranslator",

    "columnMappings": "ColumnName: ColumnName"
                        }

    Appreciate your input on where to add the mapping to fix the error.

    Thanks.

    Sreeni

    Friday, November 4, 2016 8:41 PM

All replies

  • Hi Sreeni,

    Firstly, for you case, I saw that you have specify azureTableRowKeyName as the column 'ColumnName', there is no need to further add column mapping for this column 'ColumnName'.

    Secondly, to correct map other columns except partitionkey and rowkey, yes, you should leverage column mapping feature, with structure specified for both source & sink. Please refer to the section Column Mapping Samples under Move data to and from Azure Table using Azure Data Factory for more details.

    Monday, November 7, 2016 3:21 AM