none
U-SQL to aggregate files from different folders (Day folders. 01, 02 etc) Wildcard for 2 numbers

    Question

  • I have files in the following folder structure in my data lake (Example)

    2019

          01

              01

              02

              03 

    etc

    My U-SQL Script wants to aggregate all the files in January folder

    DECLARE @now DateTime = DateTime.Now;
    //The data is taken from yesterday so we needs to use yesterdays date
    DECLARE @yesterday  = @now.AddDays(-1);
    DECLARE @nowString string = @yesterday.ToString("yyyyMM");
    DECLARE @year string = @yesterday.ToString("yyyy");
    DECLARE @month string = @yesterday.ToString("MM");
    
    DECLARE @sourceFolderPath string = "/Salesdemo/";
    DECLARE @sourceFilePrefix string = "Sales_";
    
    DECLARE @fileExtension string = ".csv";
    
    
    
    DECLARE @salesFile  string = @sourceFolderPath + @year + "/" + @month + "/" + "05"  + "/"  + @sourceFilePrefix + @nowString + "{*}" + @fileExtension;
    

    So this gives me /Salesdemo/2019/01/05/Sales_20190105.csv

    I basically need to get rid of "05" and replace with a wildcard so it takes files from every folder

    I tried replacing with {*} like the wildcard in the file name (And although it works)  I think it could be better by wildcarding to 2 digits.



    Debbie

    Monday, January 7, 2019 3:42 PM

Answers

All replies