none
U-SQL script unable to extract file from local Azure Storage Emulator

    Question

  • I'm able to run my U-SQL script which is a simple EXTRACT of a csv file both:

    a) locally using Data Lake Store local path (ie @"/input/185-Dataset_Valid.csv")

    b) on Azure ADLA referencing the file in blob storage (using the usual "wasb://" prefix)

    However, I downloaded and am running the local Azure Storage Emulator:

    Windows Azure Storage Emulator 5.4.0.0 command line tool
    IsRunning: True
    BlobEndpoint: http://127.0.0.1:10000/
    QueueEndpoint: http://127.0.0.1:10001/
    TableEndpoint: http://127.0.0.1:10002/

    When I change my U-SQL script to point to the local blob storage account like this:

    DECLARE @INPUT_FILE string = @"http://127.0.0.1:10000/devstoreaccount1/11111111111/caf/trades/inbound/185-Dataset_Valid.csv";

    I get this error when Submitting:

    E_STORE_USER_FILENOTFOUND: File not found or access denied: http://127.0.0.1:10000/devstoreaccount1/11111111111/caf/trades/inbound/185-Dataset_Valid.csv.
    Description: Unable to obtain information about this file.
    Resolution: Ensure the file with exact casing exists (paths are case-sensitive) and the account has read permission to the file.    

    In both the VS2017 Server Explorer and the Azure Storage Explorer I can see the path and file as I have it above.  I've copy and pasted the URL a few times to make sure of no typos, but no luck.

    I know in Azure ADLA I have to register my blob storage account, but I do not know of a way to do that for the local ADLA.

    How do I read and write files locally with U-SQL and the local Azure Storage Emulator?

    Thanks in advance for any suggestions!

    Wednesday, May 16, 2018 6:02 PM

Answers

  • Dear Bill

    This scenario is not supported. Local Run currently only supports accessing files on SMB mounted shares/disks (e.g., C:/ or D:/ paths) where you set the data root property (if you want relative access).

    Please file a feature request at http://aka.ms/adlfeedback if that is something you would like us see consider.


    Michael Rys

    Wednesday, May 16, 2018 10:24 PM
    Moderator

All replies

  • Dear Bill

    This scenario is not supported. Local Run currently only supports accessing files on SMB mounted shares/disks (e.g., C:/ or D:/ paths) where you set the data root property (if you want relative access).

    Please file a feature request at http://aka.ms/adlfeedback if that is something you would like us see consider.


    Michael Rys

    Wednesday, May 16, 2018 10:24 PM
    Moderator
  • I dont think you will be able to achieve it. Here are the list of possible file inputs that you can use. I dont see an option to access the local azure storage emulator files when using ADLA via VS2017.
    Wednesday, May 16, 2018 11:03 PM
    Moderator