ML.NET in Unity - c#

I cannot figure out how to use ML.NET in Unity.
What I did:
Converted my project to be compatible with framework 4.x.
Converted api compatibility level to framework 4.x.
Made assets/plugins/ml folder and droped in Microsoft.ML apis with corresponding xmls.
Marked all ml.dlls platform settings to be only 86_64 compatible (this was redundant).
I can now:
Call ML apis and create MlContext, TextLoader, and do the training of a model. When a model is trained I can also evaluate the trained model, but...
I cannot:
When trying to get a prediction out of the model I get an error:
github comment on issue from 28.12.18 (there is also a whole project attached there, you can see the code there)
The same code works in visual studio solution.
public float TestSinglePrediction(List<double> signal, MLContext mlContext, string modelPath)
{
ITransformer loadedModel;
using (var stream = new FileStream(modelPath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
loadedModel = mlContext.Model.Load(stream);
}
var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);
var abstractSignal = new AbstractSignal()
{
Sig1 = (float)signal[0],
Sig2 = (float)signal[1],
Sig3 = (float)signal[2],
Sig4 = (float)signal[3],
Sig5 = (float)signal[4],
Sig6 = (float)signal[5],
Sig7 = (float)signal[6],
Sig8 = (float)signal[7],
Sig9 = (float)signal[8],
Sig10 = (float)signal[9],
Sig11 = (float)signal[10],
Sig12 = (float)signal[11],
Sig13 = (float)signal[12],
Sig14 = (float)signal[13],
Sig15 = (float)signal[14],
Sig16 = (float)signal[15],
Sig17 = (float)signal[16],
Sig18 = (float)signal[17],
Sig19 = (float)signal[18],
Sig20 = (float)signal[19],
RfdX = 0
};
var prediction = predictionFunction.Predict(abstractSignal);
return prediction.RfdX;
}
This is the method that returns an error line:
var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);

Starting with Unity 2018.1, unity can target .net 4.x. So you would need to set the .net version to .NET 4.x Equivalent, or .net standard 2.0 (https://blogs.unity3d.com/2018/03/28/updated-scripting-runtime-in-unity-2018-1-what-does-the-future-hold/) and make sure you add your dll to the project as a reference in visual studio. If you don't add it as a reference, then visual sudio doesn't know about it.

As Nick said in his post**, it should work with Unity if you follow those steps.
However, at the time I am writing this post, the ML.NET team has not yet done comprehensive testing with Unity, so it's not completely surprising that it's not working out of the box. This issue has been opened on the ML.NET Github repository. I suggest keeping an eye on that issue for the status of Unity support.
** Nick:
Starting with Unity 2018.1, unity can target .net 4.x. So you would need to set the .net version to .NET 4.x Equivalent, or .net standard 2.0 (https://blogs.unity3d.com/2018/03/28/updated-scripting-runtime-in-unity-2018-1-what-does-the-future-hold/) and make sure you add your dll to the project as a reference in visual studio. If you don't add it as a reference, then visual sudio doesn't know about it.

As follows is a bit modyfied Iris Example from https://learn.microsoft.com/en-us/dotnet/machine-learning/tutorials/iris-clustering (that one does not work anymore due to some ML API changes)
First make sure that you have the latest .net version installed and that your Unity version is at least 2019.2.0f1 (this was a preview version) or higher.
Creste a new unity project. Create a Plugins folder inside your Assets folder. Import all ML .Net APIs into that folder (Might be a foolish thing to do, but I have forehand created a visual studio soution and added all those APIs to that solution via nuget, and than just copied those dll files to Assets/Plugins folder in my unity project).
In Assets folder create an Data folder and paste iris.data file from https://github.com/dotnet/machinelearning/blob/master/test/data/iris.data into it.
Create a script named MLuTest and paste into it the following code:
public class MLuTest : MonoBehaviour{
static readonly string _dataPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "iris.data");
static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "IrisClusteringModel.zip");
MLContext mlContext;
void Start()
{
Debug.Log("starting...");
mlContext = new MLContext(seed: 0);
IDataView dataView = mlContext.Data.ReadFromTextFile<IrisData>(_dataPath, hasHeader: false, separatorChar: ',');
string featuresColumnName = "Features";
var pipeline = mlContext.Transforms
.Concatenate(featuresColumnName, "SepalLength", "SepalWidth", "PetalLength", "PetalWidth")
.Append(mlContext.Clustering.Trainers.KMeans(featuresColumnName, clustersCount: 3));//read and format flowery data
var model = pipeline.Fit(dataView);//train
using (var fileStream = new FileStream(_modelPath, FileMode.Create, FileAccess.Write, FileShare.Write))//save trained model
{
mlContext.Model.Save(model, fileStream);
}
var predictor = mlContext.Model.CreatePredictionEngine<IrisData, ClusterPrediction>(model);//predict
IrisData Setosa = new IrisData
{
SepalLength = 5.1f,
SepalWidth = 3.5f,
PetalLength = 1.4f,
PetalWidth = 0.2f
};
Debug.Log(predictor.Predict(Setosa).PredictedClusterId);
Debug.Log("...done predicting, now do what u like with it");
}
}
public class IrisData
{
[LoadColumn(0)]
public float SepalLength;
[LoadColumn(1)]
public float SepalWidth;
[LoadColumn(2)]
public float PetalLength;
[LoadColumn(3)]
public float PetalWidth;
}
public class ClusterPrediction
{
[ColumnName("PredictedLabel")]
public uint PredictedClusterId;
[ColumnName("Score")]
public float[] Distances;
}
This should work right out of the box ... well it did for me. Where you could mess up is when getting api files, they could be different version from mine or just some .net framework compatible. So get the content of my Plugins folder (mind you all those apis may not be necesery, do the cherrypicking yourselve):https://github.com/dotnet/machinelearning/issues/1886
It used to be (in previous unity versions) that some player settings had to be changed but i did not have to do it. But anhoo here are mine:
I hope this helps, since Unity update 19.2 I have not had any problems mentioned in previous posts in this thread.

Related

Accessing Unity package version at runtime

I'm working on a custom analytics system (which is a custom package) and it would be super helpful to know which version of this package is integrated in the unity app when I'm looking into the data.
Is there any quick solution to retrieve the pacakge version at runtime ?
As said what you could do is using a ScriptableObject just for storing the version for the runtime later.
You could e.g. do it like
// See https://docs.unity3d.com/Manual/PlatformDependentCompilation.html
#if UNITY_EDITOR
using UnityEditor;
#endif;
public class PackageVersion : ScriptableObject
{
// See https://docs.unity3d.com/ScriptReference/HideInInspector.html
[HideInInspector] public string Version;
#if UNITY_EDITOR
// Called automatically after open Unity and each recompilation
// See https://docs.unity3d.com/ScriptReference/InitializeOnLoadMethodAttribute.html
[InitializeOnLoadMethod]
private static void Init()
{
// See https://docs.unity3d.com/ScriptReference/Compilation.CompilationPipeline-compilationFinished.html
// Removing the callback before adding it makes sure it is only added once at a time
CompilationPipeline.compilationFinished -= OnCompilationFinished;
CompilationPipeline.compilationFinished += OnCompilationFinished;
}
private static void OnCompilationFinished()
{
// First get the path of the Package
// This is quite easy since this script itself belongs to your package's assemblies
var assembly = typeof(PackageVersion).Assembly;
// See https://docs.unity3d.com/ScriptReference/PackageManager.PackageInfo.FindForAssembly.html
var packageInfo = PackageManager.PackageInfo.FindForAssembly();
// Finally we have access to the version!
var version = packageInfo.version;
// Now to the ScriptableObject instance
// Try to find the first instance
// See https://docs.unity3d.com/ScriptReference/AssetDatabase.FindAssets.html
// See https://docs.unity3d.com/ScriptReference/PackageManager.PackageInfo-assetPath.html
var guid = AssetDataBase.FindAssets($"t:{nameof(PackageVersion)}", packageInfo.assetPath). FirstOrDefault();
PackageVersion asset;
if(!string.isNullOrWhiteSpace(guid))
{
// See https://docs.unity3d.com/ScriptReference/AssetDatabase.GUIDToAssetPath.html
var path = AssetDatabase.GUIDToAssetPath(guid);
// See https://docs.unity3d.com/ScriptReference/AssetDatabase.LoadAssetAtPath.html
asset = AssetDatabase.LoadAssetAtPath<PackageVersion>(path);
}
else
{
// None found -> create a new one
asset = ScriptableObject.CreateInstance<PackageVersion>();
asset.name = nameof(PackageVersion);
// make it non editable via the Inspector
// See https://docs.unity3d.com/ScriptReference/HideFlags.NotEditable.html
asset.hideFlags = HideFlags.NotEditable;
// Store the asset as actually asset
// See https://docs.unity3d.com/ScriptReference/AssetDatabase.CreateAsset.html
AssetDataBase.CreateAsset(asset, $"{packageInfo.assetPath}/{nameof(PackageVersion)}");
}
asset.Version = version;
// See https://docs.unity3d.com/ScriptReference/EditorUtility.SetDirty.html
EditorUtility.SetDirty(asset);
}
#endif
}
Now wherever needed you can just reference that PackageVersion object and access the version on runtime.
Note: Typed on smartphone but I hope the idea gets clear

Call Swagger generator for plugins

i want to generate the swagger document for plugins.
I point the endpoint for the api to a plugincontroller. In this i have a method to create the documentation for a particular version. While loading the plugin all items are already registered in the swagger tooling.
(somehow the new documents don't get picked up by the swagger middleware that is why i need this workaround.)
[HttpGet("api/plugins/swaggerdoc/{version}")]
public IActionResult GetSwaggerDoc(string version)
{
SwaggerDocument gen = new SwaggerGenerator(apiDescriptionGroupCollectionProvider, schemaRegistryFactory, Swagger.SwaggerElements.GeneratorOptions.SwaggerGeneratorOptions).GetSwagger(version);
return Ok(gen);
}
but it fails to generate the document properly. It shows to much information about the properties. e.g.
"parameters":[
{
"name":"api-version",
"in":"query",
"description":null,
"required":false,
"type":"string",
"format":null,
"items":null,
"collectionFormat":null,
"default":null,
"maximum":null,
"exclusiveMaximum":null,
"minimum":null,
"exclusiveMinimum":null,
"maxLength":null,
"minLength":null,
"pattern":null,
"maxItems":null,
"minItems":null,
"uniqueItems":null,
"enum":null,
"multipleOf":null
}
how can i resolve this issue?
I found a solution:
All elements needed can be retrieved through dependency injection, or make a static reference in the startup to get a hold on it. Like generatoroptions.
public PluginsController(IActionDescriptorChangeProvider changeProvider, IOptions<MvcJsonOptions> mvcJsonOptionsAccessor, ISchemaRegistryFactory schemaRegistryFactory, IApiDescriptionGroupCollectionProvider apiDescriptionGroupCollectionProvider)
{
this.changeProvider = changeProvider;
this.schemaRegistryFactory = schemaRegistryFactory;
this.apiDescriptionGroupCollectionProvider = apiDescriptionGroupCollectionProvider;
this.mvcJsonOptionsAccessor = mvcJsonOptionsAccessor;
}
the implementation of the method will be something like this:
SwaggerDocument gen = new SwaggerGenerator(apiDescriptionGroupCollectionProvider, schemaRegistryFactory, Swagger.SwaggerElements.GeneratorOptions.SwaggerGeneratorOptions).GetSwagger(version);
var jsonBuilder = new StringBuilder();
var _swaggerSerializer = SwaggerSerializerFactory.Create(mvcJsonOptionsAccessor);
using (var writer = new StringWriter(jsonBuilder))
{
_swaggerSerializer.Serialize(writer, gen);
return Ok(jsonBuilder.ToString());
}
This will work for the 4.0.1 version of Swashbuckle.AspNetCore... The upcomping version of Swashbuckle.AspNetCore (5.x) will have to have another implementation because of the support of openapi 2 and 3.

Xamarin with PCLStorage - Where to place SQLite3 file?

I am reading 'Xamarin in Action' by Manning Publications and working on Countr example. I am using below code to retrieve Sqlite3 file as has been mentioned in chapter 7 of book.
var local = FileSystem.Current.LocalStorage.Path;
var datafile = PortablePath.Combine(local, "counters.db3");
connection = new SQLiteAsyncConnection(datafile);
connection.GetConnection();
I wanted to add one more database which has static content that I already prepared. I need to place this database inside app's assets or resources and read it when app loads. I realized that on Windows, above code is looking for database at "C:\Users\\AppData\Local\Microsoft.TestHost.x86\15.6.2\". For testing Model, ViewModel I placed the database file at that location and that helped. However, I wanted to know what is right place to put database file and how to access it so I can test in Android and iPhone emulators?
Just use Environment.GetFolderPath and store the database the location Android and IOS gives your app to store data without further permissions:
string dbPath = Path.Combine (
Environment.GetFolderPath (Environment.SpecialFolder.Personal),
"counters.db3");
You can find more information on this article:
https://developer.xamarin.com/guides/android/data-and-cloud-services/data-access/part-3-using-sqlite-orm/
In Visual Studio, Xamarin Solution by default has multiple projects -
App.Core that has libraries needed for designing Model and ViewModels
App.Droid that has Android SDK libraries to build view for Android devices
App.iOS that has iOS SDK libaries to build View for Apple devices
It is not possible to add below code in App.Core library as that project has no information about App's file system. Only App.Droid and App.iOS projects carry this information.
string dbPath = Path.Combine (
Environment.GetFolderPath (Environment.SpecialFolder.Personal),
"counters.db3");
Since we want to copy a database with pre-existing content to LocalDirectory, Setup.cs in App.Droid and App.iOS must be altered. For example, for App.Droid, below are the steps needed (This is one way and there are of course alternatives)
Place Sqlite DB in App.Droid/Assets folder
Create a class file (let's call it AndroidConfiguration.cs) and add a method which I call copyElementDatabaseToLocalStorage() (Code for this method can be found after these list of steps)
In SetUp.cs, call copyElementDatabaseToLocalStorage() method
Code for copyElementDatabaseToLocalStorage Method:
using System;
using Android.App;
using System.IO;
using PCLStorage;
namespace App.Droid.AndroidSpecificSetters
{
public class AndroidConfiguration
public async void copyElementDatabaseToLocalStorage()
{
string localStorage = FileSystem.Current.LocalStorage.Path;
string databaseLocation = Path.Combine(localStorage, "counters.db3");
try
{
if (!File.Exists(databaseLocation))
{
using (var binaryReader = new BinaryReader(Application.Context.Assets.Open("counters.db3")))
{
using (var binaryWriter = new BinaryWriter(new FileStream(databaseLocation, FileMode.Create)))
{
byte[] buffer = new byte[2048];
int length = 0;
while ((length = binaryReader.Read(buffer, 0, buffer.Length)) > 0)
{
binaryWriter.Write(buffer, 0, length);
}
}
}
}
} catch (Exception e)
{
...
}
}
}
}
Code Inside App.Droid Setup.cs constructor
public Setup(Context applicationContext) : base(applicationContext)
{
AndroidConfiguration androidConfiguration = new AndroidConfiguration();
androidConfiguration.copyElementDatabaseToLocalStorage();
}

How can I have assemblies (dll) for local packages in the NuGet v3 API?

Introduction
Hello everyone, I found a way to find the local NuGet packages using the NuGet v3 API. I don't know if I do it the correct way, but it works... There isn't a lot of documentation. The only interesting documentation I found is this blog. So, I don't know if I do it the correct way. I try to find some issues about my problem on the official NuGet GitHub. I didn't find anything interesting even on Google.
So, I kept my code and I tried to find a way to get the assemblies (dlls) when I search local packages with NuGet v3 API. I try a lot of NuGet classes, but nothing seems to return what I expected. Some functions return the package information, but the AssemblyReferences property is missing.
I know in the NuGet v2 API, the search functions return a IPackage. Now, in the v3, it is returning a IPackageSearchMetadata or a LocalPackageInfo depending on what functions you are calling. So, they changed the returned object.
So, is there a way to get the AssemblyReferences like the IPackage in the Nuget V2 API?
I don't know if it's an issue but, I posted on it on GitHub.
Code example using the Nuget V2 API (working)
var rootPath = #"pathWhereNugetPackageAre";
LocalPackageRepository repo1 = new LocalPackageRepository(rootPath);
var newtonsoft = repo1.FindPackagesById("Newtonsoft.Json").First();
//return the package and the assembly property because it's v2
Code example using the Nuget V3 API (what I tried)
var rootPath = #"pathWhereNugetPackageAre";
var logger = new Logger();
List<Lazy<INuGetResourceProvider>> providers = new List<Lazy<INuGetResourceProvider>>();
providers.AddRange(Repository.Provider.GetCoreV3());
FindLocalPackagesResourceV2 findLocalPackagev2 = new FindLocalPackagesResourceV2(rootPath);
var packageFound = findLocalPackagev2.GetPackages(logger, CancellationToken.None)
.FirstOrDefault();
//found, but missing a lot of informations...
FindLocalPackagesResourceV3 findLocalPackagev3 = new FindLocalPackagesResourceV3(rootPath);
var packageNotFound = findLocalPackagev3.GetPackages(logger, CancellationToken.None)
.FirstOrDefault();
// I don't know why it is null. It should be found
NuGetv3LocalRepository nugetV3LocalRepo = new NuGetv3LocalRepository(rootPath);
var newtonsoftToFound = nugetV3LocalRepo.FindPackagesById("Newtonsoft.Json")
.FirstOrDefault();
// return empty I don't know why...
var supportedFramework = new[] { ".NETFramework,Version=v4.6" };
var searchFilter = new SearchFilter(true)
{
SupportedFrameworks = supportedFramework,
IncludeDelisted = false
};
PackageSource localSource = new PackageSource(rootPath);
SourceRepository localRepository = new SourceRepository(localSource, providers);
PackageSearchResource searchLocalResource = await localRepository
.GetResourceAsync<PackageSearchResource>();
var packageFound3 = await searchLocalResource
.SearchAsync("Newtonsoft.Json", searchFilter, 0, 10, logger, CancellationToken.None);
var thePackage = packageFound3.FirstOrDefault();
// found but missing the assemblies property
// Logger class
public class Logger : ILogger
{
private List<string> logs = new List<string>();
public void LogDebug(string data)
{
logs.Add(data);
}
public void LogVerbose(string data)
{
logs.Add(data);
}
public void LogInformation(string data)
{
logs.Add(data);
}
public void LogMinimal(string data)
{
logs.Add(data);
}
public void LogWarning(string data)
{
logs.Add(data);
}
public void LogError(string data)
{
logs.Add(data);
}
public void LogInformationSummary(string data)
{
logs.Add(data);
}
public void LogErrorSummary(string data)
{
logs.Add(data);
}
}
I know I can list all dlls file using the DirectoryInfo, but I'm searching a NuGet class that does the logic.
#emgarten He answered my question on GitHub
For finding packages in local folders take a look at: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Protocol/Utility/LocalFolderUtility.cs
Note that v2 and v3 folders expect different structures and files to be present, so if a package is not returned it is likely because it isn't in the expected location or additional files beyond the nupkg were not in place.
You can read a package using PackageArchiveReader which gives you data similar to IPackage.
You can use the PackageArchiveReader for extracting informations on a NuGet package. You can use GetFiles(), or others functions (many functions are available). In my case, I used GetReferenceItems(), because it's grouped by target frameworks. So it's more interesting for getting the right assemblies with a target framework.
Note : If you having troubles finding packages with GetPackageV2() and GetPackageV3(), be sure that you have a valid V2 structure folder or V3 structure folder. I compared my NuGet folder with my NuGet Visual Studio folder and some files were missing (*.nuspec, *nupkg.sha512...). It should be this missing that the V3 search didn't work. I compared my NuGet folder with my last project (NuGet v2) and folders and files were the same. So, I used the GetPackageV2().
Hope this will help!

How to persist a solution with projects and documents using Roslyn's AdhocWorkspace?

I've been trying to persist a new solution containing a project and a simple cs file with the following code but nothing get saved to the disk. Am I doing something wrong or is Roslyn not the tool to be used to generate solutions project and files?
class Program
{
static void Main(string[] args)
{
var workspace = new AdhocWorkspace();
var solutionInfo = SolutionInfo.Create(SolutionId.CreateNewId(),
VersionStamp.Create(),
#"C:\Seb\SebSol.sln");
var projectInfo = ProjectInfo.Create(ProjectId.CreateNewId(),
VersionStamp.Create(),
"SebProj",
"SebProj.dll",
LanguageNames.CSharp,
#"C:\Seb\SebSol\SebProj\SebProj.csproj");
workspace.AddSolution(solutionInfo);
workspace.AddProject(projectInfo);
var sourceText = SourceText.From("public class A { }");
workspace.CurrentSolution.AddDocument(DocumentId.CreateNewId(projectInfo.Id), "ClassA.cs", sourceText);
workspace.TryApplyChanges(workspace.CurrentSolution);
}
}
You're looking for MsBuildWorkspace, which can actually update sln and csproj files in MSBuild format on disk.
Other than that class, Roslyn APIs are completely agnostic to project formats such as MSBuild.

Categories