none
UNION ALL + Custom Extractor doesn't work locally ?

    Question

  • I get a bug when using UNION ALL statement in U-SQL locally when data come from a custom extractor (Cognition assemblies).

    Compilation succeeds but input is "empty". So execution do nothing.

    @filesX =
        EXTRACT FileName string,
                Data byte[]
        FROM @pathA
        USING new Cognition.Vision.ImageExtractor();
    @filesY =
        EXTRACT FileName string,
                Data byte[]
        FROM @pathB
        USING new Cognition.Vision.ImageExtractor();

    @allFiles = SELECT FileName, Data, "XXX" AS Origin FROM @filesX UNION ALL SELECT FileName, Data, "YYY" AS Origin FROM @filesY;


    It works when I submit that code on ADLA service.

    So I use a IF ... ELSE ... END; as a workaround.


    http://blog.djeepy1.net

    Sunday, August 20, 2017 5:18 PM

All replies

  • Running U-SQL extensions (like Cognitive toolkit) in the local environment is not supported today. In your Azure account you explicitly push a button to deploy the necessary libraries and have them registered in your U-SQL database before you want to use these. A similar process does not exist for downloading these libraries and setting them up locally. If you feel this is needed feel free to vote for it at on the feedback site.

    There is also a workaround you can use outline in this blog on workaround for advanced analytics in local run.

    I am also curious to know more about your current setup. It sounds like you are getting past compilation to get a runtime error in the local run. How did you get to that with the cognitive API calls? You can contact me at omafnan@microsoft to share details.

    Omid

    Monday, August 21, 2017 7:55 AM
  • Hi Omid,

     

    I used your workaround without knowing it J. I downloaded resources from usqlext and registered it locally.

    And it works very well expect the UNION ALL operator.

     

    I do not have any compilation or runtime error. Input remains empty ; files are not extracted.


    http://blog.djeepy1.net

    Tuesday, August 22, 2017 4:48 AM
  • I tried to replicate your scenario and it seemed to work ok for me. Perhaps you can check my steps below and see if there is something different in the script or setup.

    I used a slightly different method to set up the libraries locally. I used the database export feature (recently added) to copy portions of the database down to my machine and import to local account. You can access this from the Server Explorer context menu (see below). I chose just the assemblies for being exported and auto imported to my local account. 

    I copied some images into the folder pointed to by my DATAROOT setting for local runs.  I then created a sample script in my environment and ran it locally. I chose a subset of text strings from the UNION ALL to export to TSV. I did get results coming out. The script is below.

    REFERENCE ASSEMBLY ImageCommon;
    REFERENCE ASSEMBLY FaceSdk;
    REFERENCE ASSEMBLY ImageEmotion;
    REFERENCE ASSEMBLY ImageTagging;
    REFERENCE ASSEMBLY ImageOcr;

    DECLARE @pathA string = "{FileName}.jpg";
    DECLARE @pathB string = "other/{FileName}.jpg";
    @filesX =
        EXTRACT  FileName string,
                Data byte[]
        FROM @pathA
        USING new Cognition.Vision.ImageExtractor();
    @filesY =
        EXTRACT FileName string,
                Data byte[]
        FROM @pathB
        USING new Cognition.Vision.ImageExtractor();
    @allFiles =
        SELECT FileName,
               Data,
               "XXX" AS Origin
        FROM @filesX
        UNION ALL
        SELECT FileName,
               Data,
               "YYY" AS Origin
        FROM @filesY;
    @out =
        SELECT FileName, Origin
        FROM @allFiles;

    Tuesday, August 22, 2017 8:38 AM
  • It builds. It runs. But input is empty (so is output)

    Can you confirm your output contains all filenames ?


    http://blog.djeepy1.net

    Tuesday, August 22, 2017 12:10 PM
  • My output does contain all the file names. Can you share your actual script/project with me? Again, you can contact me at omafnan@microsoft
    Wednesday, August 23, 2017 2:12 AM