none
U-SQL Customized Extractor fail to load dependency assembly

    Question

  • Hi all.

    We built a customized extractor to extract the zip files in a Data Lake U-SQL project. The extractor uses a third party dependency (Ionic.zip) to unzip the data. We have uploaded (Register Assembly) both the extractor assembly and its dependency Ionic.zip to the master database Assemblies folder. From Visual Studio we can see both files uploaded successfully.

    But the jobs using the extractor failed with the error that complains could not load the dependency assembly Ionic.zip.

    Any idea on why this error happens? Any help appreciated.

     

     

    DESCRIPTION

    Unhandled exception from user code: "Could not load file or assembly 'Ionic.Zip, Version=1.9.1.6006, Culture=neutral, PublicKeyToken=edbe51ad942a3f5c' or one of its dependencies. The system cannot find the file specified." The details includes more information including any inner exceptions and the stack trace where the exception was raised.

    RESOLUTION

    Make sure the bug in the user code is fixed.

     

    DETAILS

    ==== Caught exception System.IO.FileNotFoundException

       at Virsae.DataLakeAnalytics.CMSCHIExtractor.GetReplayZipMemoryStream(Stream zipStream)

       at Virsae.DataLakeAnalytics.CMSCHIExtractor.<Extract>d__1.MoveNext()

       at ScopeEngine.SqlIpExtractor<ScopeEngine::CosmosInput,Extract_0_Data0>.GetNextRow(SqlIpExtractor<ScopeEngine::CosmosInput\,Extract_0_Data0>* , Extract_0_Data0* output) in d:\data\ccs\jobs\7aef568b-81fe-4f94-b5b6-e7f3cced697a_v0\sqlmanaged.h:line 1849

     

    ERROR

    E_RUNTIME_USER_UNHANDLED_EXCEPTION_FROM_USER_CODE

     

    MESSAGE

    An unhandled exception from user code has been reported when invoking the method 'Extract' on the user type 'Virsae.DataLakeAnalytics.CMSCHIExtractor'

    Both extractor and its dependency uploaded


    Monday, September 26, 2016 10:47 PM

Answers

  • Eventually we gave up the third party zip component and use the .net built-in zip library. 

    BTW, the Data Lake extractor can automatically unzip the file if it is compressed with GZipStream and has .gz as the file name extension.


    Saturday, October 1, 2016 7:30 AM
  • Once you register it as a managed dependency file, you still need to add the REFERENCE ASSEMBLY statement to your script to make it available.


    Michael Rys

    • Marked as answer by Roger Rong NZ Friday, October 28, 2016 7:40 PM
    Friday, October 28, 2016 7:02 PM
    Moderator

All replies

  • Eventually we gave up the third party zip component and use the .net built-in zip library. 

    BTW, the Data Lake extractor can automatically unzip the file if it is compressed with GZipStream and has .gz as the file name extension.


    Saturday, October 1, 2016 7:30 AM
  • Is Ionic.Zip a .Net assembly? Did you reference it in the script then?

    If it is not a .Net assembly you need to register it as a dependent additional file.

    See this blog post for some discussions of similar scenarios.


    Michael Rys

    Wednesday, October 19, 2016 10:37 PM
    Moderator
  • Thanks your response Michael.

    The IonicZip is a .net assembly. We did not reference it in the U-SQL script. It looks like this is the reason of the exception.

    It is registered as a managed dependence file.

    Roger

     
    Friday, October 28, 2016 6:19 PM
  • Once you register it as a managed dependency file, you still need to add the REFERENCE ASSEMBLY statement to your script to make it available.


    Michael Rys

    • Marked as answer by Roger Rong NZ Friday, October 28, 2016 7:40 PM
    Friday, October 28, 2016 7:02 PM
    Moderator
  • Hi Michael,

    I'm new to .net world. I'm trying to reuse a 3rd party api created to extract data from ORC files using USQL.

    Link where the code is present - https://github.com/sindhudweep/Orcneas/issues/6

    When i try to run the USQL job im getting the below error. Have i missed any assemblies etc. pls guide me through.

    i tried to run a usql script which will extract 3.5GB of compressed data(approx 81 million records) in ORC format into a csv file.

    when i run the usql script it gives the below error message.Can u pls help me out in this ??

    **DESCRIPTION
    Unhandled exception from user code: "Capacity must be positive.
    Parameter name: capacity" The details includes more information including any inner exceptions and the stack trace where the exception was raised.
    RESOLUTION
    Make sure the bug in the user code is fixed.
    DETAILS
    ==== Caught exception System.ArgumentOutOfRangeException

    at System.IO.MemoryStream..ctor(Int32 capacity)
    at Orcneas.Core.OrcExtractor.GetSeekableStream(IUnstructuredReader input, SeekableInputStreamHackMode inputStreamHackMode)
    at Orcneas.Core.OrcExtractor.d__4.MoveNext()
    at ScopeEngine.SqlIpExtractorScopeEngine::CosmosInput,Extract_0_Data0.GetNextRow(SqlIpExtractorScopeEngine::CosmosInput\,Extract_0_Data0* , Extract_0_Data0* output) in d:\data\ccs\jobs\604c4a84-fa51-4cc8-9472-6dc480f13734_v0\sqlmanaged.h:line 1924
    ERROR
    E_RUNTIME_USER_UNHANDLED_EXCEPTION_FROM_USER_CODE
    MESSAGE
    An unhandled exception from user code has been reported when invoking the method 'Extract' on the user type 'Orcneas.Core.OrcExtractor'**

    Thanks,

    Bharath.C

    Wednesday, September 6, 2017 1:02 PM