We Keep Coding

Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

I just started using Kafka and hit the following rookie error:
'Value cannot be null.
Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.'
It happens when trying to send a class object, ActMessage object, rather then the a simple string that comes with the example. The line of code that raises the erros is:
using (var p = new ProducerBuilder<Null, ActMessage>(config ).Build()
I am using the .net client.
My understanding is that i need to use one of the default serializes in the first type parameter, one that come with Kafka client, as explained here, but can't find them on this .net package.
I guess i could build one but that would be a waste of time.
Here a reproducible example:
public class ActMessage {
public int SomeId {get;set;}
public string SomeContent {get;set;}
}
class Tester {
void send(){
var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
using (var p = new ProducerBuilder<Null, ActMessage>(config).Build()) //throws error here
{
var dr = p.ProduceAsync("news", new Message<Null, ActMessage>
{
Value = new ActMessage { SomeId = 1, SomeContent="hi" },
}
).Result;
}
}
}

I suggest checking out the working examples/ dir in that repo to see working code that you can copy into your own projects.
If you have your own class, you need to implement the ISerializer and IDeserializer interfaces.
Or you can use the built-in ones
However, an alternative is to use Avro
This requires writing an Avro schema file, then using avrogen to create your class, not manually write it. E.g.
dotnet tool install --global Apache.Avro.Tools
avrogen -s User.avsc .
Then you must always add some ValueSerializer in Kafka clients in order to send data

How can I add AutoMapper Profiles to my mapper config using a for loop?

I want to load a bunch of automapper profiles of referenced libraries, without having to type each one out by hand.
I'm trying to take the following steps:
Get all profiles from referenced assemblies
Add profiles to mapper config
Register mapper for DI
Step 1 works, but something goes wrong in step 2.
Current code:
public static class WebApiConfig
{
public static void Register(HttpConfiguration config)
{
var container = new UnityContainer();
var assemblyNames = Assembly.GetExecutingAssembly().GetReferencedAssemblies()
.Where(a => a.Name.StartsWith("OKL_KPS"));
var assemblies = assemblyNames.Select(an => Assembly.Load(an));
var loadedProfiles = new List<Type>();
foreach (var assembly in assemblies)
{
var assemblyProfiles = assembly.ExportedTypes.Where(type => type.IsSubclassOf(typeof(Profile)));
loadedProfiles.AddRange(assemblyProfiles);
}
var mapconfig = new MapperConfiguration(cfg =>
{
// Magic should happen here
foreach (var profile in loadedProfiles)
{
var resolvedProfile = container.Resolve(profile) as Profile;
cfg.AddProfile(resolvedProfile);
}
});
container.RegisterInstance<IMapper>(mapconfig.CreateMapper());
config.DependencyResolver = new UnityResolver(container);
//routes here
}
}
I also tried cfg.AddProfile((Profile)Activator.CreateInstance(profile.AssemblyQualifiedName, profile.Name).Unwrap());, but this returns the assembly name of the service I'm using it in, not the name of the library where the profile is from.
Edit
The assemblies aren't loading during the register step. To hack this there's a Dummy class in each library which are initialised before registering the profiles. Optimal solution is not needing these dummy classes, otherwise it would be cleaner to add each profile explicitly.
I also tried adding the ExportAttribute to the profile, but that didn't work either.

You scan loaded assemblies on available properties using LINQ queries. Something like this should work:
var profiles = AllClasses.FromLoadedAssemblies().
Where(type => typeof(Profile).IsAssignableFrom(type));
//add profiles to config
var mapconfig = new MapperConfiguration(cfg =>
{
// Magic should happen here
foreach (var profile in profiles)
{
var resolvedProfile = container.Resolve(profile) as Profile;
cfg.AddProfile(resolvedProfile);
}
});
//register mapper using config
container.RegisterInstance<IMapper>(mapconfig.CreateMapper());

You can add profiles by listing your assembly instance, assembly name or type.
Using names:
var mapconfig = new MapperConfiguration(cfg =>
{
cfg.AddProfiles("Foo.YourProject.API");
cfg.AddProfiles("Foo.YourProject.Service");
cfg.AddProfiles("Foo.YourProject.Repository");
...
});
Also check official documentation for more information.

Within the documenation is described on how to use AutoMapper with the Microsoft DI framework. And there it simply forwards to the corresponding NuGet package and an article that describes how it searches through the current application domain for all the open types and profiles to load them.
In your classes you then have to simply inject an IMapper imapper into the constructor which then simply does what you expect.
The only next caveat I ran into was, that not all my assemblies where loaded before I called services.AddAutoMapper() in my ConfigureServices() method. But for this case I simply added a simple helper method, which will be called before DI starts to do its work:
public static void LoadAllLocalAssemblies()
{
var entryAssembly = Assembly.GetEntryAssembly();
var location = entryAssembly.Location;
var path = Path.GetDirectoryName(location);
var files = Directory.EnumerateFiles(path, "*.dll");
foreach (var file in files)
{
try
{
Assembly.LoadFrom(file);
}
catch
{
}
}
}
After that all assemblies are loaded into the current app domain and the NuGet package will resolve all classes that are derived from Profile. My handlers get an IMapper mapper injected within the constructor and within my method I can call mapper.Map<MyDestination>(mySource) and it works as expected.
No dummy classes, interfaces or whatsoever is needed.

ML.NET in Unity

I cannot figure out how to use ML.NET in Unity.
What I did:
Converted my project to be compatible with framework 4.x.
Converted api compatibility level to framework 4.x.
Made assets/plugins/ml folder and droped in Microsoft.ML apis with corresponding xmls.
Marked all ml.dlls platform settings to be only 86_64 compatible (this was redundant).
I can now:
Call ML apis and create MlContext, TextLoader, and do the training of a model. When a model is trained I can also evaluate the trained model, but...
I cannot:
When trying to get a prediction out of the model I get an error:
github comment on issue from 28.12.18 (there is also a whole project attached there, you can see the code there)
The same code works in visual studio solution.
public float TestSinglePrediction(List<double> signal, MLContext mlContext, string modelPath)
{
ITransformer loadedModel;
using (var stream = new FileStream(modelPath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
loadedModel = mlContext.Model.Load(stream);
}
var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);
var abstractSignal = new AbstractSignal()
{
Sig1 = (float)signal[0],
Sig2 = (float)signal[1],
Sig3 = (float)signal[2],
Sig4 = (float)signal[3],
Sig5 = (float)signal[4],
Sig6 = (float)signal[5],
Sig7 = (float)signal[6],
Sig8 = (float)signal[7],
Sig9 = (float)signal[8],
Sig10 = (float)signal[9],
Sig11 = (float)signal[10],
Sig12 = (float)signal[11],
Sig13 = (float)signal[12],
Sig14 = (float)signal[13],
Sig15 = (float)signal[14],
Sig16 = (float)signal[15],
Sig17 = (float)signal[16],
Sig18 = (float)signal[17],
Sig19 = (float)signal[18],
Sig20 = (float)signal[19],
RfdX = 0
};
var prediction = predictionFunction.Predict(abstractSignal);
return prediction.RfdX;
}
This is the method that returns an error line:
var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);

Starting with Unity 2018.1, unity can target .net 4.x. So you would need to set the .net version to .NET 4.x Equivalent, or .net standard 2.0 (https://blogs.unity3d.com/2018/03/28/updated-scripting-runtime-in-unity-2018-1-what-does-the-future-hold/) and make sure you add your dll to the project as a reference in visual studio. If you don't add it as a reference, then visual sudio doesn't know about it.

As Nick said in his post**, it should work with Unity if you follow those steps.
However, at the time I am writing this post, the ML.NET team has not yet done comprehensive testing with Unity, so it's not completely surprising that it's not working out of the box. This issue has been opened on the ML.NET Github repository. I suggest keeping an eye on that issue for the status of Unity support.
** Nick:
Starting with Unity 2018.1, unity can target .net 4.x. So you would need to set the .net version to .NET 4.x Equivalent, or .net standard 2.0 (https://blogs.unity3d.com/2018/03/28/updated-scripting-runtime-in-unity-2018-1-what-does-the-future-hold/) and make sure you add your dll to the project as a reference in visual studio. If you don't add it as a reference, then visual sudio doesn't know about it.

As follows is a bit modyfied Iris Example from https://learn.microsoft.com/en-us/dotnet/machine-learning/tutorials/iris-clustering (that one does not work anymore due to some ML API changes)
First make sure that you have the latest .net version installed and that your Unity version is at least 2019.2.0f1 (this was a preview version) or higher.
Creste a new unity project. Create a Plugins folder inside your Assets folder. Import all ML .Net APIs into that folder (Might be a foolish thing to do, but I have forehand created a visual studio soution and added all those APIs to that solution via nuget, and than just copied those dll files to Assets/Plugins folder in my unity project).
In Assets folder create an Data folder and paste iris.data file from https://github.com/dotnet/machinelearning/blob/master/test/data/iris.data into it.
Create a script named MLuTest and paste into it the following code:
public class MLuTest : MonoBehaviour{
static readonly string _dataPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "iris.data");
static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "IrisClusteringModel.zip");
MLContext mlContext;
void Start()
{
Debug.Log("starting...");
mlContext = new MLContext(seed: 0);
IDataView dataView = mlContext.Data.ReadFromTextFile<IrisData>(_dataPath, hasHeader: false, separatorChar: ',');
string featuresColumnName = "Features";
var pipeline = mlContext.Transforms
.Concatenate(featuresColumnName, "SepalLength", "SepalWidth", "PetalLength", "PetalWidth")
.Append(mlContext.Clustering.Trainers.KMeans(featuresColumnName, clustersCount: 3));//read and format flowery data
var model = pipeline.Fit(dataView);//train
using (var fileStream = new FileStream(_modelPath, FileMode.Create, FileAccess.Write, FileShare.Write))//save trained model
{
mlContext.Model.Save(model, fileStream);
}
var predictor = mlContext.Model.CreatePredictionEngine<IrisData, ClusterPrediction>(model);//predict
IrisData Setosa = new IrisData
{
SepalLength = 5.1f,
SepalWidth = 3.5f,
PetalLength = 1.4f,
PetalWidth = 0.2f
};
Debug.Log(predictor.Predict(Setosa).PredictedClusterId);
Debug.Log("...done predicting, now do what u like with it");
}
}
public class IrisData
{
[LoadColumn(0)]
public float SepalLength;
[LoadColumn(1)]
public float SepalWidth;
[LoadColumn(2)]
public float PetalLength;
[LoadColumn(3)]
public float PetalWidth;
}
public class ClusterPrediction
{
[ColumnName("PredictedLabel")]
public uint PredictedClusterId;
[ColumnName("Score")]
public float[] Distances;
}
This should work right out of the box ... well it did for me. Where you could mess up is when getting api files, they could be different version from mine or just some .net framework compatible. So get the content of my Plugins folder (mind you all those apis may not be necesery, do the cherrypicking yourselve):https://github.com/dotnet/machinelearning/issues/1886
It used to be (in previous unity versions) that some player settings had to be changed but i did not have to do it. But anhoo here are mine:
I hope this helps, since Unity update 19.2 I have not had any problems mentioned in previous posts in this thread.

How can I have assemblies (dll) for local packages in the NuGet v3 API?

Introduction
Hello everyone, I found a way to find the local NuGet packages using the NuGet v3 API. I don't know if I do it the correct way, but it works... There isn't a lot of documentation. The only interesting documentation I found is this blog. So, I don't know if I do it the correct way. I try to find some issues about my problem on the official NuGet GitHub. I didn't find anything interesting even on Google.
So, I kept my code and I tried to find a way to get the assemblies (dlls) when I search local packages with NuGet v3 API. I try a lot of NuGet classes, but nothing seems to return what I expected. Some functions return the package information, but the AssemblyReferences property is missing.
I know in the NuGet v2 API, the search functions return a IPackage. Now, in the v3, it is returning a IPackageSearchMetadata or a LocalPackageInfo depending on what functions you are calling. So, they changed the returned object.
So, is there a way to get the AssemblyReferences like the IPackage in the Nuget V2 API?
I don't know if it's an issue but, I posted on it on GitHub.
Code example using the Nuget V2 API (working)
var rootPath = #"pathWhereNugetPackageAre";
LocalPackageRepository repo1 = new LocalPackageRepository(rootPath);
var newtonsoft = repo1.FindPackagesById("Newtonsoft.Json").First();
//return the package and the assembly property because it's v2
Code example using the Nuget V3 API (what I tried)
var rootPath = #"pathWhereNugetPackageAre";
var logger = new Logger();
List<Lazy<INuGetResourceProvider>> providers = new List<Lazy<INuGetResourceProvider>>();
providers.AddRange(Repository.Provider.GetCoreV3());
FindLocalPackagesResourceV2 findLocalPackagev2 = new FindLocalPackagesResourceV2(rootPath);
var packageFound = findLocalPackagev2.GetPackages(logger, CancellationToken.None)
.FirstOrDefault();
//found, but missing a lot of informations...
FindLocalPackagesResourceV3 findLocalPackagev3 = new FindLocalPackagesResourceV3(rootPath);
var packageNotFound = findLocalPackagev3.GetPackages(logger, CancellationToken.None)
.FirstOrDefault();
// I don't know why it is null. It should be found
NuGetv3LocalRepository nugetV3LocalRepo = new NuGetv3LocalRepository(rootPath);
var newtonsoftToFound = nugetV3LocalRepo.FindPackagesById("Newtonsoft.Json")
.FirstOrDefault();
// return empty I don't know why...
var supportedFramework = new[] { ".NETFramework,Version=v4.6" };
var searchFilter = new SearchFilter(true)
{
SupportedFrameworks = supportedFramework,
IncludeDelisted = false
};
PackageSource localSource = new PackageSource(rootPath);
SourceRepository localRepository = new SourceRepository(localSource, providers);
PackageSearchResource searchLocalResource = await localRepository
.GetResourceAsync<PackageSearchResource>();
var packageFound3 = await searchLocalResource
.SearchAsync("Newtonsoft.Json", searchFilter, 0, 10, logger, CancellationToken.None);
var thePackage = packageFound3.FirstOrDefault();
// found but missing the assemblies property
// Logger class
public class Logger : ILogger
{
private List<string> logs = new List<string>();
public void LogDebug(string data)
{
logs.Add(data);
}
public void LogVerbose(string data)
{
logs.Add(data);
}
public void LogInformation(string data)
{
logs.Add(data);
}
public void LogMinimal(string data)
{
logs.Add(data);
}
public void LogWarning(string data)
{
logs.Add(data);
}
public void LogError(string data)
{
logs.Add(data);
}
public void LogInformationSummary(string data)
{
logs.Add(data);
}
public void LogErrorSummary(string data)
{
logs.Add(data);
}
}
I know I can list all dlls file using the DirectoryInfo, but I'm searching a NuGet class that does the logic.

#emgarten He answered my question on GitHub
For finding packages in local folders take a look at: https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Protocol/Utility/LocalFolderUtility.cs
Note that v2 and v3 folders expect different structures and files to be present, so if a package is not returned it is likely because it isn't in the expected location or additional files beyond the nupkg were not in place.
You can read a package using PackageArchiveReader which gives you data similar to IPackage.
You can use the PackageArchiveReader for extracting informations on a NuGet package. You can use GetFiles(), or others functions (many functions are available). In my case, I used GetReferenceItems(), because it's grouped by target frameworks. So it's more interesting for getting the right assemblies with a target framework.
Note : If you having troubles finding packages with GetPackageV2() and GetPackageV3(), be sure that you have a valid V2 structure folder or V3 structure folder. I compared my NuGet folder with my NuGet Visual Studio folder and some files were missing (*.nuspec, *nupkg.sha512...). It should be this missing that the V3 search didn't work. I compared my NuGet folder with my last project (NuGet v2) and folders and files were the same. So, I used the GetPackageV2().
Hope this will help!

BundleTransformer.Less inject variables depending on context/request

We would like the use the bundling mechanism of System.Web.Optimization in combination with the Less transformer.
The problem is that the same application/server serves pages for different branded websites. So depending on the 'SiteContext' the same .less files are used but different values should be used by the .less variables. So we want the (re)use the same less files but with different variables depending on the context of the request.
I tried a couple of different theories:
In all 3 cases I setup different bundles depending on the SiteContext.
1 inject an #import directive with the themed variables by using a custom VirtualPathProvider that intercepts the variables.less file.
So I have:
the styling file eg: header.less (imports the variables file)
the variables file: variables.less
a themed variables file: variables-theme.less (injected in variables.less via the VirtualPathProvider)
This is not working because the BundleTransformer cache sees this as the same file and doesn't know about the SiteContext. The cache key is based on the Url of the IAsset and we cannot influence this behavior.
2 Replace the variables.less import by variables-themed.less with an custom transformer that runs before the Less transformer.
Again no luck, same caching issues.
And as a side effect, the extra transformer was not called in debug because the assets are not bundled but called individually by the LessAssetHandler. This could be solved by writing your own AssetHandler that calls all required transformers.
3 create themed Asset names that are resolved by a custom VirtualPathProvider
Eg. Add header-themeX.less to the bundle, this file doesn't exist but you resolve this file to header.less and use method 2 to set the correct variables file import. (replace the import of the variables.less to the themed version).
Once again no luck. I think this could solve the caching issue if it wasn't for the Bundle.Include(string virtualPath) that does a File.Exists(path) internally. It doesn't pass via the CustomVirtualPathProvider.
Am I looking to deep to solve this?
All ideas are welcome, I can imagine that this will become a problem to more and more people as the System.Web.Optimization library gets more popular...
Keep in mind that:
we have a lot of .less/css files
we will have 5 or so themes
we like to keep things working in visual studio (that is why header.less has a ref. to variables.less)
Thanks for any feedback.

Michael!
You use the Microsoft ASP.NET Web Optimization Framework and the Bundle Transformer in multi-tenant environment, so you need to replace some components of the System.Web.Optimization and create own versions of the debugging HTTP-handlers (see «Problem: LESS file imports are added to BundleResponse.Files collection» discussion). As far as I know, Murat Cakir solve all these problems in the SmartStore.NET project.
In the Bundle Transformer there are 2 ways to inject of LESS-variables:
Look a properties GlobalVariables and ModifyVariables of LESS-translator:
using System.Collections.Generic;
using System.Web.Optimization;
using BundleTransformer.Core.Builders;
using BundleTransformer.Core.Orderers;
using BundleTransformer.Core.Transformers;
using BundleTransformer.Core.Translators;
using BundleTransformer.Less.Translators;
public class BundleConfig
{
public static void RegisterBundles(BundleCollection bundles)
{
var nullBuilder = new NullBuilder();
var nullOrderer = new NullOrderer();
var lessTranslator = new LessTranslator
{
GlobalVariables = "my-variable='Hurrah!'",
ModifyVariables = "font-family-base='Comic Sans MS';body-bg=lime;font-size-h1=50px"
};
var cssTransformer = new CssTransformer(new List<ITranslator>{ lessTranslator });
var commonStylesBundle = new Bundle("~/Bundles/BootstrapStyles");
commonStylesBundle.Include(
"~/Content/less/bootstrap-3.1.1/bootstrap.less");
commonStylesBundle.Builder = nullBuilder;
commonStylesBundle.Transforms.Add(cssTransformer);
commonStylesBundle.Orderer = nullOrderer;
bundles.Add(commonStylesBundle);
}
}
Create a custom item transformation:
using System.Text;
using System.Web.Optimization;
public sealed class InjectContentItemTransform : IItemTransform
{
private readonly string _beforeContent;
private readonly string _afterContent;
public InjectContentItemTransform(string beforeContent, string afterContent)
{
_beforeContent = beforeContent ?? string.Empty;
_afterContent = afterContent ?? string.Empty;
}
public string Process(string includedVirtualPath, string input)
{
if (_beforeContent.Length == 0 && _afterContent.Length == 0)
{
return input;
}
var contentBuilder = new StringBuilder();
if (_beforeContent.Length > 0)
{
contentBuilder.AppendLine(_beforeContent);
}
contentBuilder.AppendLine(input);
if (_afterContent.Length > 0)
{
contentBuilder.AppendLine(_afterContent);
}
return contentBuilder.ToString();
}
}
And register this transformation as follows:
using System.Web.Optimization;
using BundleTransformer.Core.Orderers;
using BundleTransformer.Core.Bundles;
public class BundleConfig
{
public static void RegisterBundles(BundleCollection bundles)
{
var nullOrderer = new NullOrderer();
const string beforeLessCodeToInject = #"#my-variable: 'Hurrah!';";
const string afterLessCodeToInject = #"#font-family-base: 'Comic Sans MS';
#body-bg: lime;
#font-size-h1: 50px;";
var commonStylesBundle = new CustomStyleBundle("~/Bundles/BootstrapStyles");
commonStylesBundle.Include(
"~/Content/less/bootstrap-3.1.1/bootstrap.less",
new InjectContentItemTransform(beforeLessCodeToInject, afterLessCodeToInject));
commonStylesBundle.Orderer = nullOrderer;
bundles.Add(commonStylesBundle);
}
}
Both ways have disadvantage: the injection of LESS-variables does not work in debug mode.