none
Alternative of U-SQL filedate in Databricks.

    Question

  • In U-SQL by giving startDate and endDate we can extract all the logs from folders in data lake store by giving path : 

    path = "/myData/myLogs/{filedate:yyyy}.{filedate:MM}.{filedate:dd}/
    log.{filedate:yyyy}.{filedate:MM}.{filedate:dd}.{filedate:HH}.csv"


    Folders in Data lake store:

    /myData/myLogs/2016.01.01/2016.01.01.00.csv

    /myData/myLogs/2016.01.01/2016.01.01.00.csv

    .
    .

    /myData/myLogs/2016.11.12/2016.11.12.23.csv

    How can I read all these files in Dataframe? knowing that startTime and endTime will be different each time.


    Tuesday, October 30, 2018 4:43 PM

All replies