locked
Conversion from csv to parquet - Keep null values RRS feed

  • Question

  • Hi,

    I have problem with that null values are not retained when converting a csv file to parquet format. I have tried to force source dataset to retain null values without success.

    Csv (Original File)

    Parquet

    treatEmptyAsNull in source dataset(csv file)

    Anyone have an idea how to keep null values in the parquet file?

    Kind Regards

    AL_T

    Wednesday, November 27, 2019 7:16 AM

All replies

  • Data preview is not a true representation of your data. If you need to handle NULL and empty values use dataflow to classify them appropriately.
    • Edited by dataflowuser Wednesday, November 27, 2019 6:28 PM
    • Proposed as answer by dataflowuser Wednesday, November 27, 2019 6:28 PM
    • Unproposed as answer by AL_T Thursday, November 28, 2019 10:08 AM
    Wednesday, November 27, 2019 6:27 PM
  • Hi dataflowuser.

    Ok, but I have also imported the data to a sql table and can see that the null values are converted to empty strings from null, so I'm pretty sure that the conversion from csv to parquet converts the null values to empty strings.

    I guess that you are referring to mapping data flows? I don't want to use mapping data flows for such a simple conversion, there must be a way to maintain nulls in the data factory.

    Thanks.

    AL_T

     
    Thursday, November 28, 2019 10:08 AM
  • Ok, but I have also imported the data to a sql table and can see that the null values are converted to empty strings from null

    While loading to SQL, you see empty instead of NULLs? I couldn't reproduce this. 

    Please post csv dataset json code and have you configured anything in this box? like empty or a space in it? 


    If the response helped, do "Mark as answer" and upvote it
    - Vaibhav

    Thursday, November 28, 2019 10:20 AM