none
Copy data activity suddenly changes file extension. RRS feed

  • Question

  • My copy activity used to copy files in ADLS gen1 from the delimited text to parquet format with all having the same filenames (ending with .csv). However, I found out today that this behavior has changed and is now copies the files, but changes the file extensions. So all *.csv files became *.parquet files. This change in behavior has caused my pipelines to break. 

    How often do these behavior changes occur? How can I prepare my pipelines on these changes? Is it possible to run your published pipelines on fixed/old versions of ADF so they won't break when any update happen?

    Update: I have posted a feedback request to have the ability to fix your ADF version on published pipelines.

    see link

    Monday, September 9, 2019 12:39 PM

Answers

  • Hi Michael Hogervorst,

    Sorry for the inconvenience. This is due to recent release for Copy activity behavior change by ADF product team. A Copy to parquet format but generate the file name as ".csv", this is not the correct behavior. This enhancement targets to fix this behavior. This kind of behavior change happens rarely, unless a bug is identified/reported.

    We are still waiting for a response from ADF team, about announcements or notify customers about behavior change before it goes to production. We will keep posted once we have an update. Meanwhile please feel free share your feedback in ADF user voice forum about the same. 

    Since you have pipelines depend on this behavior,  now, if you would like to switch your pipeline to previous (old) behavior, please provide us your subscription id and we can pass the info to ADF engineering team to whitelist your subcription to fall back to previous behavior. But in your scenario, generating ".parquet" file for parquet format should be the right behavior.  

    If you would like to switch your pipeline to previous (old) behavior, please share below details to AzCommunity[at]Microsoft[dot]com.

    Subject of the email: <Azure Data Factory: Copy data activity suddenly changes file extension>
    Thread URL: <https://social.msdn.microsoft.com/Forums/en-US/fa7df897-4303-475c-8449-97d7cfbac9e1/copy-data-activity-suddenly-changes-file-extension?forum=AzureDataFactory>

    Subscription ID:  <your subscription id>
    Data Factory Name: <DF Name>
    Pipeline Name: <Your ADF pipeline name>



    Let us know once the email is sent.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, September 10, 2019 2:14 AM
    Moderator

All replies

  • Hi Michael Hogervorst

    We have reached out to internal team to get more info about this behavior change. Will keep posted once we have an update.

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.


    Tuesday, September 10, 2019 12:00 AM
    Moderator
  • Hi Michael Hogervorst,

    Sorry for the inconvenience. This is due to recent release for Copy activity behavior change by ADF product team. A Copy to parquet format but generate the file name as ".csv", this is not the correct behavior. This enhancement targets to fix this behavior. This kind of behavior change happens rarely, unless a bug is identified/reported.

    We are still waiting for a response from ADF team, about announcements or notify customers about behavior change before it goes to production. We will keep posted once we have an update. Meanwhile please feel free share your feedback in ADF user voice forum about the same. 

    Since you have pipelines depend on this behavior,  now, if you would like to switch your pipeline to previous (old) behavior, please provide us your subscription id and we can pass the info to ADF engineering team to whitelist your subcription to fall back to previous behavior. But in your scenario, generating ".parquet" file for parquet format should be the right behavior.  

    If you would like to switch your pipeline to previous (old) behavior, please share below details to AzCommunity[at]Microsoft[dot]com.

    Subject of the email: <Azure Data Factory: Copy data activity suddenly changes file extension>
    Thread URL: <https://social.msdn.microsoft.com/Forums/en-US/fa7df897-4303-475c-8449-97d7cfbac9e1/copy-data-activity-suddenly-changes-file-extension?forum=AzureDataFactory>

    Subscription ID:  <your subscription id>
    Data Factory Name: <DF Name>
    Pipeline Name: <Your ADF pipeline name>



    Let us know once the email is sent.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, September 10, 2019 2:14 AM
    Moderator
  • Hi KranthiPakala-MSFT,

    I've already changed my pipelines to the new behavior and I made them fool-proof for file-extensions changes. A switch to old behavior is not necessary any more, but it is good to know that this is possible for your pipelines. 

    The update indeed makes more sense and I am happy to see the ADF team working thoroughly on the product. An update report could be useful, but I would most likely not read it each time. So I guess I'll just have carry on and hope that the few behavioral changes doesn't impact my pipelines too much.

    I will consider posting a new idea on the feedback site to ask for a custom ADF-version selection in the pipelines. That can be useful and more accessible than an email.

     

    Tuesday, September 10, 2019 6:44 AM
  • Hi Michael Hogervorst,

    Thanks for your response and yes, ADF engineering team is continuously putting their efforts to the improve the product. 

    Appreciate for taking time and acknowledging this enhancement. Also if you can share the feedback link here, it would be helpful for other members/users to up-vote your idea/suggestion you have shared in the feedback forum. 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, September 10, 2019 6:35 PM
    Moderator