none
data flow in azure RRS feed

  • Question

  • Hi,

    i am trying to tranfer data to sink in dataflow from 'select columns' .The column that i have derived in previous process is giving null values in the sink.

    1. i have checked datatype mapping .i has mapped properly.

    2. The datatype of dervied column is decimal,if i am going to change the size e.g decimal(6,4)->decimal(5,3) or decimal(8,3) ,it is giving an error:    'collectPreviewData failure on job=9c36f2a2-cc25-4903-8802-f04bc3aff9c2, jobState=Failed com.microsoft.dataflow.Issues: DF-SINK-005 - Column(s) [inr_convrted_currency] do not exist or their datatypes do not match the store datatypes  '.

    Monday, November 25, 2019 9:41 AM

All replies

  • Hello ashwini_8 and thank you for your question.

    I have attempted to repro your issue (Dataflow script included below) and did not run into any issues.  I suspect I am doing something different from what you are doing.  If you can provide a simplified and/or redacted version of your Dataflow's script, I will attempt to repro using that.

    Note: for this sink I created new dataset, with 'Import Schema' set to None.

    source(output(
    		Id as integer,
    		Date as timestamp
    	),
    	allowSchemaDrift: true,
    	validateSchema: false,
    	isolationLevel: 'READ_UNCOMMITTED',
    	format: 'table') ~> source1
    source1 derive(decimal1 = toDecimal(
        right(
            toString(
                currentTimestamp()
            )
        ,5),
    6,4)) ~> DerivedColumn1
    DerivedColumn1 derive(decimal2 = toDecimal(decimal1,5,3)) ~> DerivedColumn2
    DerivedColumn2 select(mapColumn(
    		dec2 = decimal2
    	),
    	skipDuplicateMapInputs: true,
    	skipDuplicateMapOutputs: true) ~> Select1
    Select1 sink(allowSchemaDrift: true,
    	validateSchema: false,
    	partitionBy('hash', 1)) ~> sink1

    Tuesday, November 26, 2019 12:24 AM
    Moderator
  • Can you share with us the the target database table schema definition? At a minimum, what is the data type of the column that are trying to store 'inr_convrted_currency'?
    Tuesday, November 26, 2019 9:50 PM
  • @ashwini_8 , please respond or we will assume you are no longer blocked.
    Monday, December 2, 2019 6:38 PM
    Moderator