locked
Delimiter and quote character issue RRS feed

  • Question

  • Hi ,

    Could any one suggest on this scenario please.

    -> I have a pipe delimiter .csv file in ADLS when i am trying to read it from sql dw if a column value coming as pipe then its saying too many columns.

    -> To get rid of this issue we have kept Quote character as " in ADF and generating the file so that all strings are enclosed by "

    -> Now records are rejecting whose column value is having " or ""

    -> Later I tried quote character as unicode \u001c where the strings will enclosed by this character and declared in the file format STRING_DELIMITER as \u001c 

    -> but still it is not considering this and rejecting the records , means ignoring STRING_DELIMITER

    -> Finally I tried my delimiter itself as \u001c which is a unicode and declared the FIELD_TERMINATOR = \u001c but surprisingly its not taking this at all and failing 

    Now question is STRING_DELIMETER or FIELD_TERMINATOR can be taken as \u001c ? and STRING_DELIMITER is nothing but enclosed character only ? is there any solution to resolve this ?

    Help me to resolve this .

    Thanks in advance 

    --Poornima M

     

    Tuesday, November 13, 2018 5:06 PM

All replies

  • You may be able to use multiple characters as delimiter to avoid this issue ie: ~|~ or something that won't show up in your data as consecutive characters. 
    Tuesday, November 13, 2018 5:56 PM
  • Hi ,

    But ADF is not allowing multiple characters as delimiters .

    --Poornima M

    Tuesday, November 13, 2018 6:04 PM