locked
Files produced in previous activity not picked up by next activity - is that expected or a bug? RRS feed

  • Question

  • I have a Data Factory pipeline with several consecutive activities. The first ones gather event data using U-SQL scripts and stores results in csv-files(in Data Lake Storage Gen1). These activities works fine and correctly does not proceed until the previous one has succeeded.

    But later down the pipeline I have a U-SQL script that merges results over several days. It use a date pattern to collect files to operate on (like this "DailyBuckets/{date:yyyy}/{date:MM}/{date:dd}/{*}") and then later a SELECT to only pick up the last 30 days of files.

    But we have now noticed when the pipeline runs it misses the file produced in the activity before, this days daily result. If I run the merge U-SQL script manually later on the same day will merge data from all files including the one created the same day.

    I would expect that a succeeded U-SQL script would mean all produced output like files in the data lake storage to be ready and can be consumed by the next activities but this does not seems to be the case, or have I stumbled upon a bug?

    Thursday, April 16, 2020 9:55 AM

All replies

  • Hello Karl , 

    Thanks for the question . 

    Can you please share the  JSON for the ADF , please ?


    Thanks Himanshu
    ================
    If you think my answer helped you , please click "mark as answer " , this will be help other community members to get to the resolution faster
    Alone, we can do so little; together, we can do so much

    Monday, April 20, 2020 2:58 PM
  • Hello  Karl , 

    We have not received a response from you.  Are you still facing the issue?  If you found a solution, would you please share it here with the community?  Otherwise, let us know and we will continue to engage with you on the issue.


    Thanks Himanshu
    ================
    If you think my answer helped you , please click "mark as answer " , this will be help other community members to get to the resolution faster
    Alone, we can do so little; together, we can do so much

    Wednesday, April 22, 2020 7:35 PM