Azure Data Factory U-SQL pipeline creates a random folder that have similar name to a target output file RRS feed

  • Question

  • I am currently using Azure Data Factory V2.  When I execute a U-SQL pipeline, that pools together multiple CSV files and output a single CSV file to Azure Data Lake, it randomly creates a folder that is named similarly to the actual output file I create.

    For instance, if the U-SQL pipeline is meant to create "customer_interaction.csv", sometimes Azure will create "customer_interaction.csv" and "customer_interaction.csv---e522659f-638a-42ed-a453-96d9774398a1#N" folder(the GUID looking string is not the run id of that pipeline, and I could not find where it comes from), which has empty file named like "D2B099F5-28C0-4094-86E9-E43D195028D5-3_0_1.dtf#N".

    I went through the logs on Azure Data Analytics, and found no error or issue related to that activity.  I am suspecting this is some kind of temporary file that is made by Azure system, but I am not sure about that at this point.  Does anybody know why such folder and file are created?

    Wednesday, November 4, 2020 8:34 PM