none
How do I reference a local data root using the adl:// convention? RRS feed

  • Question

  • I have a U-SQL script that I parameterize to run on different environments, mixing and matching sometimes files stored in blob (wasb://) storage and adls (adl://) - usually I am reading from blob files and writing to adls files.

    So in the Azure environment I can build a string using variables that ends-up looking like this:

    adl://adlsname.azuredatalakestore.net/00000000000000000000000000000000/change/hedges/inbound/filename.ext

    (where "adlsname" is my actual ADLS name)

    QUESTION:  when I want to use my script to test locally against (Local-machine), how do I build that first part of the URI?  I've tried:

    adl://Local-machine.azuredatalakestore.net/ ...

    but that is an incorrect format that errors when I submit it locally.  I know I can just say "/00000000000000000000000000000000/change/hedges/inbound/filename.ext" but I have a template in which I construct the URI.

    Thanks for any suggestions!


    Bill Blakey

    Thursday, October 11, 2018 11:33 PM

All replies

  • That depends where you keep the data locally. Note that local run cannot access data in Azure. So you either place the data into the so called local data root or some custom place on your local disk.

    In the first case, just use the relative URI instead of the absolute URI (I would expect that you can templatize that).

    In the second case, you can use local file paths, e.g., c:\path\todata\000000000000\...



    Michael Rys

    Friday, October 12, 2018 12:51 AM
    Moderator
  • Thank you Michael.  In case it spurs any other ideas, below is the snippet from the top portion of my U-SQL script to develop/test a stored procedure execution, I'm building up the parameter values to pass to it.  My process is to run/test locally first (Local-machine) then to the Azure ADLA - it supports files in/out in blob storage (registered to the ADLA resource account) and/or ADLS.  Locally this resulting @infolder and @infile are invalid as I showed.  Then once it tests good I move the file into my U-SQL Database Project from the U-SQL Project within the same Solution.

    DECLARE @accountid string = "12345";
    DECLARE @product = "change";
    DECLARE @domain = "hedges";
    DECLARE @subdomain = "orders";
    DECLARE @filename string = "2018OCT.csv";
    
    // in
    DECLARE @infilename string = @filename;
    DECLARE @storageType_in string = "adl";
    DECLARE @storageAcct_in string = "(Local-machine)";
    DECLARE @storageSuffix_in string = (@storageType_in == "wasb") ? ".blob.core.windows.net" : ".azuredatalakestore.net";
    DECLARE @baseURI_in string = (@storageType_in == "wasb") 
                                    ? @storageType_in + "://" + @accountid + "@" + @storageAcct_in + @storageSuffix_in
                                    : @storageType_in + "://" + @storageAcct_in + @storageSuffix_in + "/" + @accountid;
    DECLARE @infolder string = @baseURI_in + "/" + @product + "/" + @domain + "/";
    DECLARE @infile string = @infolder + "inprocess/summary" + "/" + @infilename; 
    


    Bill Blakey

    Friday, October 12, 2018 3:24 PM
  • (Replying to my own Reply - ha!)

    So I now just added that (Local-machine) condition and it works fine:

    DECLARE @storageSuffix_in string = 	(@storageAcct_in == "(Local-machine)") ? "" :
    									(@storageType_in == "wasb") ? ".blob.core.windows.net" : ".azuredatalakestore.net";
    DECLARE @baseURI_in string =	(@storageAcct_in == "(Local-machine)") ? @accountid :
    								(@storageType_in == "wasb") 
    									? @storageType_in + "://" + @accountid + "@" + @storageAcct_in + @storageSuffix_in
    									: @storageType_in + "://" + @storageAcct_in + @storageSuffix_in + "/" + @accountid;
    


    Bill Blakey

    Friday, October 12, 2018 3:57 PM