none
Load recent files in USQL

    Question

  • Hi All,

    I am having one folder, where I am getting files on daily basis.

    let's say \input\ folder have 100 files, tomorrow again 100 new files loaded.

    I don't want to load all files again in

    is there anything which can help us to load all the files in  which has been arrived today only

    Thanks & advance


    Thankx &amp; regards, Vipin <g class="gr_ gr_27 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" data-gr-id="27" id="27">jha</g> MCP



    • Edited by Vipin jha Wednesday, April 18, 2018 3:31 PM AA
    Wednesday, April 18, 2018 12:45 PM

All replies

  • Is there any thing we can do to load only latest file  like 

    date 01/04/2018  having 10 files

    date 10/04/2018  having 15 files

    date 15/04/2018  having 12 files

    todays got 5 files , so I want to load only tofays 5 files 


    Thankx &amp; regards, Vipin jha MCP

    Wednesday, April 18, 2018 12:57 PM
  • If you load the data using the date in the path, you can use filesets in U-SQL. E.g. (caution: pseudo-code): 

    @data =
    EXTRACT ...., loaddate DateTime
    FROM "/input/{loaddate:yyyy}/{loaddate:MM}/{loaddate:dd}/{*}.csv"
    USING Extractors.Csv();

    @data =
    SELECT * FROM @data WHERE loaddate>DateTime.Now.AddDays(-1);

    If that is not feasible, we just have added a new feature that allows you to filter based on the created or modified date:

    @data = 
        EXTRACT 
          ...,
         modified_date = FILE.MODIFIED(), 
         created_date = FILE.CREATED()
        FROM "/input/{*}.csv"
        USING Extractors.Csv();
    
    @data =
    SELECT *
    FROM @data
    WHERE created_date > DateTime.Now.AddDays(-1);
    


    Michael Rys

    Wednesday, April 18, 2018 5:11 PM
    Moderator