none
Different fileset wildcards in the same folder

    Question

  • Let's say I have a folder with 2 different file types in the same folder.  One has the file name that looks like "{id}_M.tsv and the other looks like "{id}_P.tsv".

    Id like to be able to process both of those files in a single U-SQL script, but it looks like I can't. Here's a small example script of what I'd like to do.

    DECLARE @matchFiles = "/folder/{*}_M.tsv";
    DECLARE @pathFiles = "/folder/{*}_P.tsv";

    @paths =
        EXTRACT Col1 string
        FROM @pathFiles
        USING Extractors.Tsv();
    OUTPUT @paths TO "/folder/readps.tsv" USING Outputters.Tsv();

    @matches =
        EXTRACT Col1 string
        FROM @matchFiles
        USING Extractors.Tsv();
    OUTPUT @matches TO "/folder/readms.tsv" USING Outputters.Tsv();

    When this runs locally, only one of the file sets will have any files. The other file set gets replaced with Empty Input in the job graph. And when I run this locally, the order of the extractions determines which will have the files.

    Is there any way to get it to read in both file sets? 

    I know I can get this working by separating the files into different folders, but do I have to? 

    Thanks,

    Matt

    Friday, January 19, 2018 9:33 PM

Answers

  • Hi Matt

    Is that occurring on the cluster or in local run? We recently fixed this issue in local run. I am not sure if it has surfaced in the latest version yet, but if you encounter it on local run and have not downloaded the latest version of the tooling, please do so.

    Best regards

    Michael


    Michael Rys

    • Marked as answer by mdpurecars Friday, January 19, 2018 11:01 PM
    Friday, January 19, 2018 10:02 PM
    Moderator

All replies

  • Hi Matt

    Is that occurring on the cluster or in local run? We recently fixed this issue in local run. I am not sure if it has surfaced in the latest version yet, but if you encounter it on local run and have not downloaded the latest version of the tooling, please do so.

    Best regards

    Michael


    Michael Rys

    • Marked as answer by mdpurecars Friday, January 19, 2018 11:01 PM
    Friday, January 19, 2018 10:02 PM
    Moderator
  • Yes, it’s when running locally. I hadn’t tried it on a cluster because it wasn’t running locally :) I’ll make sure I have the latest version of the tooling and give it another go. Thanks, Matt
    Friday, January 19, 2018 11:01 PM