none
Using filepath in U-SQL Query with static file set

    Question

  • In one of my U-Sql queries I need to use information of the filepath (folder and file) because it contains essential information about the data. I know that this is possible by using virtual columns in a pattern like this:

    @data = EXTRACT folder string, filename string, field1 string, field2 string
    FROM "/{folder}/{filename}.csv"
    USING Extractors.Csv();

    However I cannot use patterns in my query. I have to be able to read a specific set of files which can be located in different folders. The folders can contain thousands of files and therefore it's not possible to read the data from all files and filter the data in a later stage in the query.

    Therefore, my queries refer to a static set of files like this:

    @data = EXTRACT folder string, filename string, field1 string, field2 string
    FROM 	"/folder1/file1.csv",
    		"/folder3/file2.csv",
    		"/folder5/file3.csv",
    USING Extractors.Csv();

    But now, the fields "folder" and "file" remain empty. It seems logical I guess, because there is no definition for the folder/file column in the FROM clause)

    Is there another way of adding data from the filename/path in a query with a static set of files?



    • Edited by Simon_Z Friday, January 6, 2017 3:00 PM
    Friday, January 6, 2017 3:00 PM

All replies