Dataflow: sha2 result toBinary anomaly RRS feed

  • Question

  • Hi

    I am using the sha2 function in a derived column and the result is an hexadecimal string such as 


    I created to 2 derived columns the first:

    iif(isNull($$), null()+'',concat('0x',sha2(256, $$)))

    and second:

    toBinary(sha2(256, $$))

    and sent the output to a CSV file and a parquet

    In the CSV I see what is expected



    but in the parquet one I have



    How could that be?



    Friday, August 23, 2019 1:23 PM

All replies

  • Hello Paul and thank you for your inquiry.  This looks like quite a puzzle!

    Parquet is the file format which contains multiple encodings (i.e. integers stored as integers, not characters representing integers), right?  What does your output schema look like?  Are there any transformations after the derived columns or sink side rules?

    While you check that, I'm going to try setting up a reproduction.  I am assuming that all settings other than file format and location are identical.

    Friday, August 23, 2019 10:15 PM
  • Paul, I attempted to reproduce your anomaly.  Everything checked out on my end.  Did you find anything?
    Monday, August 26, 2019 8:26 PM
  • Paul, are you still in need of assistance?
    Tuesday, August 27, 2019 6:18 PM