none
C++ Wrapper in registered assembly.

    Question

  • Hi! I need to use C++ Wrapper in my library for U-SQL job. I have created a C# library project and I have added new reference to C++ wrapper .dll. Wrapper works correctly in C# console application. In my library I use a wrapper method in my extractor class(only for testing). After building a library in C# and registration of the new assembly to my local default database (master), my job throws exception:

    "An unhandled exception from user code has been reported when invoking the method 'Constructor' on the user type 'Microsoft.Analytics.Interfaces.IExtractor'","description":"Unhandled exception from user code: \"Could not load file or assembly 'TestWrapper, Version=1.0.5941.40099, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified"

    Is this possible to use wrappers in U-SQL Scripts? How can i do it correctly?

    Thursday, April 7, 2016 11:19 PM

Answers

  • The C++ dll needs to be made available inside each vertex for the code to work. You have two options to achieve that:

    1. You use the ADDITIONAL_FILES option with CREATE ASSEMBLY (or the additional files option in the Register Assembly option in VS) to add the file. Then everytime you REFERENCE the assembly, the file will be made available. See https://msdn.microsoft.com/en-us/library/azure/mt621364.aspx.
    2. You upload the file into your ADLS account and then use DEPLOY RESOURCE "path to dll"; to deploy it in the script. That will place the file into the user code's working directory.

    The first one is probably better unless you want to version the C++ differently.


    Michael Rys

    Friday, April 8, 2016 12:11 AM
    Moderator
  • Could You advise me how use correctly own C++ wrapper. I have no idea why i cant run my scripts (locally and on DLA Service). I create my Wrapper using tutorial from this site: https://drthitirat.wordpress.com/2013/06/03/use-c-codes-in-a-c-project-wrapping-native-c-with-a-managed-clr-wrapper/

    Here is my script code:

    DROP ASSEMBLY IF EXISTS [CustomExtractors];
    
    CREATE ASSEMBLY [CustomExtractors]
    FROM @"C:\Users\Krzys\Documents\visual studio 2013\Projects\USQLWrapper\CustomExtractors\bin\Debug\CustomExtractors.dll"
    WITH ADDITIONAL_FILES =
         (
             @"C:\Users\Krzys\Documents\visual studio 2013\Projects\USQLWrapper\CustomExtractors\bin\Debug\ClassLibrary1.dll"
         );
    
    REFERENCE ASSEMBLY [CustomExtractors];
    
    @result =
        EXTRACT name string,
                sequence string
        FROM @"E:/local_path/fasta2gb"
        USING CustomExtractors.ExtractorsFactory.GetFastaExtractor();
    
    
    OUTPUT @result
    TO @"E:/local_path/test.txt"
    USING Outputters.Text();

    Edit:

    I discovered that my projects had different architectures, and when i set everything to x64, locally everything works correctly.

    Saturday, April 9, 2016 8:29 PM
  •  

    Regarding this error, please make sure you use 64 bit binary as U-SQL runtime is 64 bit.

    An attempt was made to load a program with an incorrect format

    Wednesday, May 4, 2016 4:51 AM

All replies

  • Never use U-SQL job before, according my previous experience in other scenario of using platform invoke. There are 2 possible reasons:

    1. Path: The C++ dll can't be located. in your test console application, the current directory is where the .EXE located. Usually the C++ dll is placed in same folder. so the runtime can find C++ dlls. 

    But in the U-SQL job the current directory maybe not the folder where is the wrapper class.  A solution is add the path of C++ dll to the path environment variable. Refer following for details:

    Specify the search path for DllImport in .NET

    2. Permission: in console application, the account to running the application the one current logged in which is usually administrator. But for U-SQL job, the account may be network service or some other low privilege account. The solution is use Administrator to run U-SQL job or give the account which is running U-SQL job full permission the the C++ dll folder. 

    Thursday, April 7, 2016 11:44 PM
  • The C++ dll needs to be made available inside each vertex for the code to work. You have two options to achieve that:

    1. You use the ADDITIONAL_FILES option with CREATE ASSEMBLY (or the additional files option in the Register Assembly option in VS) to add the file. Then everytime you REFERENCE the assembly, the file will be made available. See https://msdn.microsoft.com/en-us/library/azure/mt621364.aspx.
    2. You upload the file into your ADLS account and then use DEPLOY RESOURCE "path to dll"; to deploy it in the script. That will place the file into the user code's working directory.

    The first one is probably better unless you want to version the C++ differently.


    Michael Rys

    Friday, April 8, 2016 12:11 AM
    Moderator
  • Hi! I registered new assembly with additional files option, but now script throw exception:

    "An unhandled exception from user code has been reported when invoking the method 'Constructor' on the user type 'Microsoft.Analytics.Interfaces.IExtractor'","description":"Unhandled exception from user code: \"Could not load file or assembly 'TestWrapper, Version=1.0.5941.40099, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised.","resolution":"Make sure the bug in the user code is fixed.","helpLink":"","details":"==== Caught exception System.BadImageFormatException\n\n   w Cust\n   w ScopeEngine.ManagedUDO<1>.{ctor}(ManagedUDO<1>* , basic_string<char\\,std::char_traits<char>\\,std::allocator<char> >* argv, Int32 argc) w c:\\users\\krzys\\documents\\visual studio 2013\\projects\\usqlwrapper\\usqlwrapper\\bin\\debug\\47e537cdf4cadd10\\wrappertest_5c263bb694afdbf3\\__scopecodegenengine__.dll.cpp:wiersz 214","internalDiagnostics":""}"

    I  have tried also register assembly using "managed dependencies" and then place REFERENCE ASSEMBLY to C++ wrapper in U-SQL script but exception is the same.


    Friday, April 8, 2016 9:56 AM
  • Could You advise me how use correctly own C++ wrapper. I have no idea why i cant run my scripts (locally and on DLA Service). I create my Wrapper using tutorial from this site: https://drthitirat.wordpress.com/2013/06/03/use-c-codes-in-a-c-project-wrapping-native-c-with-a-managed-clr-wrapper/

    Here is my script code:

    DROP ASSEMBLY IF EXISTS [CustomExtractors];
    
    CREATE ASSEMBLY [CustomExtractors]
    FROM @"C:\Users\Krzys\Documents\visual studio 2013\Projects\USQLWrapper\CustomExtractors\bin\Debug\CustomExtractors.dll"
    WITH ADDITIONAL_FILES =
         (
             @"C:\Users\Krzys\Documents\visual studio 2013\Projects\USQLWrapper\CustomExtractors\bin\Debug\ClassLibrary1.dll"
         );
    
    REFERENCE ASSEMBLY [CustomExtractors];
    
    @result =
        EXTRACT name string,
                sequence string
        FROM @"E:/local_path/fasta2gb"
        USING CustomExtractors.ExtractorsFactory.GetFastaExtractor();
    
    
    OUTPUT @result
    TO @"E:/local_path/test.txt"
    USING Outputters.Text();

    Edit:

    I discovered that my projects had different architectures, and when i set everything to x64, locally everything works correctly.

    Saturday, April 9, 2016 8:29 PM
  • I tried to use all those options in my project and locally all this solutions work correctly, but when I tried to use them on DLA i have always the same exception.

    jobError,2016-04-13 02:52:31 PDT,<br/>Vertex SV1_Extract[0][1].v1 {8A0CDE10-F0DE-4B33-9EA2-1D092F5AF530} failed <br/><br/>Error:<br/>Vertex user code error<br/><br/>exitcode=CsExitCode_StillActive Errorsnippet=<br/>{"diagnosticCode":195887134&#45;"severity":"Error"&#45;"component":"RUNTIME"&#45;"source":"User"&#45;"errorId":"E_RUNTIME_USER_UNHANDLED_EXCEPTION_FROM_USER_CODE"&#45;"message":"An unhandled exception from user code has been reported when invoking the method 'Extract' on the user type 'CustomExtractors.FastaExtractor'"&#45;"description":"Unhandled exception from user code: \"Could not load file or assembly 'TestWrapper.dll' or one of its dependencies. The specified module could not be found.\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised."&#45;"resolution":"Make sure the bug in the user code is fixed."&#45;"helpLink":""&#45;"details":"==== Caught exception System.IO.FileNotFoundException\n\n at CustomExtractors.FastaExtractor.<Extract>d__0.MoveNext()\n at ScopeEngine.SqlIpExtractor<ScopeEngine::CosmosInput&#45;Extract_0_Data0>.GetNextRow(SqlIpExtractor<ScopeEngine::CosmosInput\\&#45;Extract_0_Data0>* &#45; Extract_0_Data0* output) in d:\\data\\ccs\\jobs\\8f7774f9-20c9-4ed8-bd63-f2314c15c95f_v0\\sqlmanaged.h:line 1705"&#45;"internalDiagnostics":""}

    First I tried add to data lake store my C# library and wrapper and  invoke this script:

    DROP ASSEMBLY IF EXISTS [CustomExtractors];
    
    CREATE ASSEMBLY [CustomExtractors]
    FROM @"adl://mgrdatalakestore.azuredatalakestore.net/Libs/CustomExtractors.dll"
    WITH ADDITIONAL_FILES =
         (
             @"adl://mgrdatalakestore.azuredatalakestore.net/Libs/TestWrapper.dll"
         );
    
    REFERENCE ASSEMBLY [CustomExtractors];
    
    @result =
        EXTRACT name string,
                sequence string
        FROM @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/fasta2gb"
        USING CustomExtractors.ExtractorsFactory.GetFastaExtractor();
    
    
    OUTPUT @result
    TO @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/tetsowy.txt"
    USING Outputters.Text();

    Later I used additional file option in the Register Assembly options in Visual Studio and i used this script:

    @result =
        EXTRACT name string,
                sequence string
        FROM @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/fasta2gb"
        USING CustomExtractors.ExtractorsFactory.GetFastaExtractor();
    
    
    OUTPUT @result
    TO @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/tetsowy.txt"
    USING Outputters.Text();

    I tried also use Menaged Dependencies option in the Register Assembly for this script:

    REFERENCE ASSEMBLY [TestWrapper];
    REFERENCE ASSEMBLY [CustomExtractors];
    
    @result =
        EXTRACT name string,
                sequence string
       // FROM @"E:/local_path/fasta2gb"
        FROM @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/fasta2gb"
        USING CustomExtractors.ExtractorsFactory.GetFastaExtractor();
    
    
    OUTPUT @result
    //TO @"E:/local_path/testowy.txt"
    TO @"adl://mgrdatalakestore.azuredatalakestore.net/Fastq/tetsowy.txt"
    USING Outputters.Text();

    All scripts throws IO.FileNotFound exception? What happens? All projects are compiled for x64 platform.



    Wednesday, April 13, 2016 11:50 AM
  • One item I have not mentioned here is that you only get a very light-weight set of C# assemblies deployed with your default runtime. For any other assembly that you need, you either need to add it with a REFERENCE SYSTEM ASSEMBLY (if you want to pick up the latest and correct version of the system assembly), or add it as ADDITIONAL FILE.

    In your example, you probably need to add at least the C++ runtime as well.

    Furthermore, you need to make sure that the path you load the code from is indeed available in the correct location. Since I don't know how you call the code, you may specify the wrong path?


    Michael Rys

    Tuesday, April 19, 2016 6:32 AM
    Moderator
  •  

    Regarding this error, please make sure you use 64 bit binary as U-SQL runtime is 64 bit.

    An attempt was made to load a program with an incorrect format

    Wednesday, May 4, 2016 4:51 AM