Microsoft.ML porting away from .netcore to .net frameworks 4.7.2 RRS feed

  • Question

  • After moving the working code for the githubticketclassifier, a decision tree using the microsoft.ml lightGbm decision tree trainer I had great results and made my own dataset. I then converted it to .net 4.7.2 frameworks in a console environment and it still worked ok. just had some closing of the console I found could be avoided using messagbox popups. Anyway next I tried calling from the autodesk.revit api to send some features and it couldnt finish the model it would create the dataset from the tsv files, however then it would throw an exception. So nothing worked I tried adding it directly to my class that gets called by the revit api and i recieved the following error as soon a i invoke the main method. 

    Could not load file or assembly 'System.Memory, Version=, Culture=neutral,

    PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. 

    The system cannot find the file specified.

    This is installed and is the latest version, 4.5.4. I looked at the app.config bindings and i see this...

        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <assemblyIdentity name="System.Memory" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />

    i tried changing to the newer version and it made no difference.

    Monday, September 21, 2020 10:11 AM

All replies

  • IMO you can try to reinstall that library from NuGet. The latest version 4.5.4 still have support for .NET Framework v4.6.1, which should run fine on 4.7.2 projects.

    For it to run on .NET Framework v4.6.1 it does need some external dependencies, which NuGet should take care for you. You may check whether the followings exists for your project, because if the required libraries do not exists, the DLL will refuse to load and you'll see that error.


    Note that you can't just copy the DLL from .NET Core to full .NET Framework project and expect them to run, because they contains link to different libraries and more importantly, system related ones. This NuGet package contains 3 versions of System.Memory.dll, for .NET v4.6.1/.NET Standard v1.1/.NET Standard 2.0 respectively.

    Tuesday, September 22, 2020 1:38 AM
  • @Cheong,

    So i was able to run the console executable from my current revit api class all in .net frameworks 4.7.2, if i can convert that to a dll or something and pass and return values, then i would be happy. 

    regarding the problkems with embedding the microsoft.ml code into the revit api classes directly, the debugger breaked on...

    nd the problem was called  TextLoader and its in the Microsoft.ML.Data assembly

             case PropertyInfo property:
                        if (!InternalDataKindExtensions.TryGetDataKind(property.PropertyType.IsArray ? property.PropertyType.GetElementType() : property.PropertyType, out dk))
                            throw Contracts.Except($"Property {memberInfo.Name} is of unsupported type.");

    I was wondering if I caused the problems because i didnt use the correct method of handing the paths to the data loader.

    • Edited by Favouritism Tuesday, September 22, 2020 10:45 AM
    Tuesday, September 22, 2020 10:12 AM
  • If you use "dotnet run" to run your console executable, then it uses .NET Core runtime.

    If you use "Self-contained" option to publish your .NET Core project, then all the dependencies are already included in your .EXE file.

    When you see there is a dependency of NuGet package with name .UnSafe on it, it means this will contain low level code that will allow the library function on that particular platform, which is/are not included in the original namespace of the library. So you had better get that.

    As of .NET Core 3+, it is no longer "just a subset of .NET Framework full version" so you should install whatever the library package tell you would be missing if you plan to run on a particular runtime.

    Tuesday, September 22, 2020 11:28 AM
  • So should I create a .net core project and see what dependencies are listed for microsoft.ML?

    Tuesday, September 22, 2020 12:05 PM
  • When a library is added to .NET Core project, it's dependency is different to a .NET Framework full version project because the functions offered by BCL is different.

    You should add with package manager like NuGet that will help you sort out the dependencies automatically.

    Tuesday, September 22, 2020 2:19 PM
  • I am getting closer, I managed to call from a x64 winform project the githubissue dt class library

    i added the nuget packages piecemeal and the nuget manager added this special class for me...


    Then i tried with Autodesk revit api which is where is need it and i got this exception. Do you think its also missing this transformer class from Microsoft.ML???

    Revit Errors...

    See the end of this message for details on invoking 
    just-in-time (JIT) debugging instead of this dialog box.

    ************** Exception Text **************
    System.FormatException: Parsing failed with an exception: The type initializer for 'System.MemoryExtensions' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.MemoryExtensions' threw an exception. ---> System.MissingMethodException: Method not found: 'IntPtr System.Runtime.CompilerServices.Unsafe.ByteOffset(!!0 ByRef, !!0 ByRef)'.
       at System.MemoryExtensions.MeasureStringAdjustment()
       at System.MemoryExtensions..cctor()
       --- End of inner exception stack trace ---
       at Microsoft.ML.Data.TextLoader.Parser.ParseRow(RowSet rows, Int32 irow, Helper helper, Boolean[] active, String path, Int64 line, String text)
       at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
       at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
       --- End of inner exception stack trace ---
       at Microsoft.ML.Data.TextLoader.Cursor.<ParseParallel>d__33.MoveNext()
       at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
       at Microsoft.ML.Data.RootCursorBase.MoveNext()
       at Microsoft.ML.Transforms.ValueToKeyMappingTransformer.Train(IHostEnvironment env, IChannel ch, ColInfo[] infos, IDataView keyData, ColumnOptionsBase[] columns, IDataView trainingData, Boolean autoConvert)
       at Microsoft.ML.Transforms.ValueToKeyMappingTransformer..ctor(IHostEnvironment env, IDataView input, ColumnOptionsBase[] columns, IDataView keyData, Boolean autoConvert)
       at Microsoft.ML.Transforms.ValueToKeyMappingEstimator.Fit(IDataView input)
       at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
       at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
       at GitHubIssueClassification.DecisionTree.BuildAndTrainModel(IDataView trainingDataView, IEstimator`1 pipeline) in C:\Users\OCCIDENTALIS\Source\Repos\GitHubIssueClassification\DecisionTree.cs:line 56
       at GitHubIssueClassification.DecisionTree.TrainAndMakePredictions(String FamilyName, String Description) in C:\Users\OCCIDENTALIS\Source\Repos\GitHubIssueClassification\DecisionTree.cs:line 38
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

    Thursday, September 24, 2020 11:09 AM
  • Is the System.Runtime.CompilerServices.Unsafe.dll I mentioned exists in the same folder of the project?

    If it is, the application should be able to find the System.Runtime.CompilerServices.Unsafe.ByteOffset() method it offers.

        <member name="M:System.Runtime.CompilerServices.Unsafe.ByteOffset``1(``0@,``0@)">
          <summary>Determines the byte offset from origin to target from the given references.</summary>
          <param name="origin">The reference to origin.</param>
          <param name="target">The reference to target.</param>
          <typeparam name="T">The type of reference.</typeparam>
          <returns>Byte offset from origin to target i.e. <paramref name="target" /> - <paramref name="origin" />.</returns>

    Friday, September 25, 2020 2:43 AM
  • This method .byteOffset(); in the compiler.unsafe asembly can be found in the code of both the calling method of the revit api external command class and in the microsoft.ML class. you can also see the binding for the assembly here...

    The revit api is a bit backward and doesnt allow multi-threading, however I am nbot trying to do anything extraordinary only call from a form invoked from the main method of an external command in the Revit api.

    <?xml version="1.0" encoding="utf-8"?>
            <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <assemblyIdentity name="System.Runtime.CompilerServices.Unsafe" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />
            <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />
            <assemblyIdentity name="System.Memory" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />
            <assemblyIdentity name="System.Collections.Immutable" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />
            <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
            <bindingRedirect oldVersion="" newVersion="" />

    Friday, September 25, 2020 11:21 AM
  • So Let me explain what I am doing i am using an output type of class library.

    This class library is an assembly that the revit api loads from a path in a manifest file that the revit application looks for on opening.

    Once opened you can invoke this external command so it executes through a revit api interface.

    I am invooking a windows form that calls an another project with the microsoft.ml decision tree code.

    Do you think i would be better off using a console output type fro the microsfot.ml and sending arguments to it? from here i save the prediction to a text file and the addin keeps refreshing until a result is returned by the decion tree model.

    Friday, September 25, 2020 12:54 PM
  • Agreed using a console project would allow you to have simpler way to debug your program, as your .NET Framework 4.X version and .NET Core version can run with same argument so you can just rule out possibility for introducing variance of behavior while you port it to WinForm. When the logic is stabilized you can always launch a new thread to run the Microsoft.ML codes, and it should produce similar behavior as if you run it under Console project.
    Monday, September 28, 2020 3:40 AM