We Keep Coding

Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

I just started using Kafka and hit the following rookie error:
'Value cannot be null.
Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.'
It happens when trying to send a class object, ActMessage object, rather then the a simple string that comes with the example. The line of code that raises the erros is:
using (var p = new ProducerBuilder<Null, ActMessage>(config ).Build()
I am using the .net client.
My understanding is that i need to use one of the default serializes in the first type parameter, one that come with Kafka client, as explained here, but can't find them on this .net package.
I guess i could build one but that would be a waste of time.
Here a reproducible example:
public class ActMessage {
public int SomeId {get;set;}
public string SomeContent {get;set;}
}
class Tester {
void send(){
var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
using (var p = new ProducerBuilder<Null, ActMessage>(config).Build()) //throws error here
{
var dr = p.ProduceAsync("news", new Message<Null, ActMessage>
{
Value = new ActMessage { SomeId = 1, SomeContent="hi" },
}
).Result;
}
}
}

I suggest checking out the working examples/ dir in that repo to see working code that you can copy into your own projects.
If you have your own class, you need to implement the ISerializer and IDeserializer interfaces.
Or you can use the built-in ones
However, an alternative is to use Avro
This requires writing an Avro schema file, then using avrogen to create your class, not manually write it. E.g.
dotnet tool install --global Apache.Avro.Tools
avrogen -s User.avsc .
Then you must always add some ValueSerializer in Kafka clients in order to send data

How to get instances of a class using Reflection?

How to get instances of a class using C# Reflection?.
For example,
public class Sample
{
}
Sample s = new Sample();
Sample s2 = new Sample();
Sample s3 = new Sample();
How to get these three instances of sample class using reflection?

Using System.Activator.CreateInstance you can create instances of a class using reflection. For example...
System.Type type = typeof(Sample);
object obj = Activator.CreateInstance(type);
In this example obj is your newly created instance. There's few overloads of this method, more information in this MSDN Documentation
Edit Based On Comment
If what you want is to be able to retrieve all instances of a class, I guess this is not possible using managed code. You will need to use unmanaged code to either profile the managed heap using the Profiling API or use the HeapWalk function to enumerating all objects allocated in the managed heap.

var s = (Sample)Activator.CreateInstance(Sample);
var s2 = (Sample)Activator.CreateInstance(Sample);
var s3 = (Sample)Activator.CreateInstance(Sample);

.NET assemblies: understanding type visibility

I am trying to reproduce something that System.Xml.Serialization already does, but for a different source of data.
For now task is limited to deserialization only.
I.e. given defined source of data that I know how to read. Write a library that takes a random type, learns about it fields/properties via reflection, then generates and compiles "reader" class that can take data source and an instance of that random type and writes from data source into the object's fields/properties.
here is a simplified extract from my ReflectionHelper class
public class ReflectionHelper
{
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, XDataReader reader);
}
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = GetCSharpName(t);
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] {source});
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<EntityField> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + #".Generated
{
public class " + readerClassName + #" : ReflectionHelper.FieldReader<" + className + #">
{
public void Fill(" + className + #" e, XDataReader reader)
{
" + w.GetStringBuilder().ToString() + #"
}
}
}
";
}
}
and the calling code:
class Program
{
static void Main(string[] args)
{
ReflectionHelper.GetFieldReader<Foo>();
Console.ReadKey(true);
}
private class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
}
The dynamic compilation of course fails because Foo class is not visible outside of Program class. But! The .NET XML deserializer somehow works around that - and the question is: How?
After an hour of digging System.Xml.Serialization via Reflector I came to accept that I lack some kind of basic knowledge here and not really sure what am I looking for...
Also it is entirely possible that I am reinventing a wheel and/or digging in a wrong direction, in which case please do speak up!

You don’t need to create a dynamic assembly and dynamically compile code in order to deserialise an object. XmlSerializer does not do that either — it uses the Reflection API, in particular it uses the following simple concepts:
Retrieving the set of fields from any type
Reflection provides the GetFields() method for this purpose:
foreach (var field in myType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
// ...
I’m including the BindingFlags parameter here to ensure that it will include non-public fields, because otherwise it will return only public ones by default.
Setting the value of a field in any type
Reflection provides the function SetValue() for this purpose. You call this on a FieldInfo instance (which is returned from GetFields() above) and give it the instance in which you want to change the value of that field, and the value to set it to:
field.SetValue(myObject, myValue);
This is basically equivalent to myObject.Field = myValue;, except of course that the field is identified at runtime instead of compile-time.
Putting it all together
Here is a simple example. Notice you need to extend this further to work with more complex types such as arrays, for example.
public static T Deserialize<T>(XDataReader dataReader) where T : new()
{
return (T) deserialize(typeof(T), dataReader);
}
private static object deserialize(Type t, XDataReader dataReader)
{
// Handle the basic, built-in types
if (t == typeof(string))
return dataReader.ReadString();
// etc. for int and all the basic types
// Looks like the type t is not built-in, so assume it’s a class.
// Create an instance of the class
object result = Activator.CreateInstance(t);
// Iterate through the fields and recursively deserialize each
foreach (var field in t.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
field.SetValue(result, deserialize(field.FieldType, dataReader));
return result;
}
Notice I had to make some assumptions about XDataReader, most notably that it can just read a string like that. I’m sure you’ll be able to change it so that it works with your particular reader class.
Once you’ve extended this to support all the types you need (including int? in your example class), you can deserialize an object by calling:
Foo myFoo = Deserialize<Foo>(myDataReader);
and you can do this even when Foo is a private type as it is in your example.

If I try to use sgen.exe (the standalone XML serialization assembly compiler), I get the following error message:
Warning: Ignoring 'TestApp.Program'.
- TestApp.Program is inaccessible due to its protection level. Only public types can be processed.
Warning: Ignoring 'TestApp.Program+Foo'.
- TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
Assembly 'c:\...\TestApp\bin\debug\TestApp.exe' does not contain any types that can be serialized using XmlSerializer.
Calling new XmlSerializer(typeof(Foo)) in your example code results in:
System.InvalidOperationException: TestApp.Program+Foo is inaccessible due to its protection level. Only public types can be processed.
So what gave you the idea that XmlSerializer can handle this?
However, remember that at runtime, there are no such restrictions. Trusted code using reflection is free to ignore access modifiers. This is what .NET binary serialization is doing.
For example, if you generate IL code at runtime using DynamicMethod, then you can pass skipVisibility = true to avoid any checks for visibility of fields/classes.

I've been working a bit on this. I'm not sure if it will help but, anyway I think it could be the way. Recently I worked with Serialization and DeSerealization of a class I had to send over the network. As there were two different programs (the client and the server), at first I implemented the class in both sources and then used serialization. It failed as the .Net told me it had not the same ID (I'm not sure but it was some sort of assembly id).
Well, after googling a bit I found that it was because the serialized class was on different assemblies, so the solution was to put that class in a independent library and then compile both client and server with that library. I've used the same idea with your code, so I put both Foo class and FieldReader class in a independent library, let's say:
namespace FooLibrary
{
public class Foo
{
public string Field1 = null;
public int? Field2 = null;
}
public abstract class FieldReader<T>
{
public abstract void Fill(T entity, IDataReader reader);
}
}
compile it and add it to the other source (using FooLibrary;)
this is the code I've used. It's not exactly the same as yours, as I don't have the code for GetCSharpName (I used t.Name instead) and XDataReader, so I used IDataReader (just for the compiler to accept the code and compile it) and also change EntityField for object
public class ReflectionHelper
{
public static FieldReader<T> GetFieldReader<T>()
{
Type t = typeof(T);
string className = t.Name;
string readerClassName = Regex.Replace(className, #"\W+", "_") + "_FieldReader";
object[] fields = new object[10];
string source = GetFieldReaderCode(t.Namespace, className, readerClassName, fields);
CompilerParameters prms = new CompilerParameters();
prms.GenerateInMemory = true;
prms.ReferencedAssemblies.Add("System.Data.dll");
prms.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().GetModules(false)[0].FullyQualifiedName);
prms.ReferencedAssemblies.Add(t.Module.FullyQualifiedName);
prms.ReferencedAssemblies.Add("FooLibrary1.dll");
CompilerResults compiled = new CSharpCodeProvider().CompileAssemblyFromSource(prms, new string[] { source });
if (compiled.Errors.Count > 0)
{
StringWriter w = new StringWriter();
w.WriteLine("Error(s) compiling {0}:", readerClassName);
foreach (CompilerError e in compiled.Errors)
w.WriteLine("{0}: {1}", e.Line, e.ErrorText);
w.WriteLine();
w.WriteLine("Generated code:");
w.WriteLine(source);
throw new Exception(w.GetStringBuilder().ToString());
}
return (FieldReader<T>)compiled.CompiledAssembly.CreateInstance(readerClassName);
}
private static string GetFieldReaderCode(string ns, string className, string readerClassName, IEnumerable<object> fields)
{
StringWriter w = new StringWriter();
// write out field setters here
return #"
using System;
using System.Data;
namespace " + ns + ".Generated
{
public class " + readerClassName + #" : FieldReader<" + className + #">
{
public override void Fill(" + className + #" e, IDataReader reader)
" + w.GetStringBuilder().ToString() +
}
}";
}
}
by the way, I found a tiny mistake, you should use new or override with the Fill method, as it is abstract.
Well, I must admit that GetFieldReader returns null, but at least the compiler compiles it.
Hope that this will help you or at least it guides you to the good answer
regards

XmlSerializer Deserialize failures

I have wsdl from third party server. Ran svcutil and ended up wih a set of
XmlNode AMethod(object Request);
methods. There is a separate 100 page pdf describing response/request objects for each method
My thought was wrap web methods and use XmlSerializer to return strongly typed objects. Returned xml looks like this (i removed soap headers):
<Response xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:type="ResponseExt"
xmlns="http://www.thirdparty.com/lr/">
<Code>0</Code>
<Message>SUCCESS</Message>
<SessionId>session_token</SessionId>
</Response>
Looked simple. Created a class(from document/wire captures):
[XmlRoot("Response")]
//EDIT added XmlType
[XmlType("ResponseExt", Namespace = "http://www.thirdparty.com/lr/")]
public class MyClass {
public string Code {get; set;}
public string Message {get; set;}
public string SessionId {get; set;}
}
Processing time:
//XmlNode node = xml from above
XmlSerializer serializer = new XmlSerializer(typeof(MyClass));
XmlNodeReader reader = new XmlNodeReader(node);
Myclass myclass = serializer.Deserialize(reader) as MyClass
Last line is where it blows up with inner exception message: The specified type was not recognized: name='ResponseExt', namespace='http://www.thirdparty.com/lr/', at <Response xmlns=''>.
I can't figure out how to make Serializer happy and what exactly these two mean
xsi:type="ResponseExt"
xmlns="http://www.thirdparty.com/lr/
As always any advice and pointer are appreciated
EDIT: Accepted answer below.
I was still getting exception, until i found this, hopefully it'll save someone some time.
I started to work backwards. Captured xml on the wire. Deserialized to my created classes with correct attributes: worked like a charm. Tried again from webservice - exception. For some reason XmlSerializer doesn't recognize ResponseExt.
XmlSerializer serializer = new XmlSerializer(typeof(Response));
XmlNode node = (XmlNode)results[0];
XmlDocument doc = new XmlDocument();
doc.LoadXml(node.OuterXml); //reload node
XmlNodeReader reader = new XmlNodeReader(doc.FirstChild); //there is only one node
Response rsp = serializer.Deserialize(reader) as Response; //works
EDIT: underlying issue wsdl file was not complete. After spending 2 days on this and finding this (ugly) workaround, third-party vendor provided complete WSDL with all types that deserialize without errors.

Why are you manually deserializing XML, when you have WSDL ?
If you have WSDL, use the svcutil.exe tool, or the wsdl.exe tool, to generate proxy classes and DTOs for the XML messages being sent and received on the wire.
The point of a web services toolkit, or "stack" is to provide this for you, so that you don't have to author classes and XML serialization code by hand.
Did you try this? Did you try to run the WSDL through one of those tools? Or did you try to "Add web reference" in Visual Studio?
After updating the question, I suggest that you modify the WSDL, rather than write custom code. You can produce a custom WSDL for the service, which will correctly generate the proxy classes you want. If you don't need all 100 methods (or however many there are), then leave them out. If you want a custom object from a method, then define a complexType that corresponds to that object. This is much simpler and more reliable than hand-authoring XML deserialization code for each method.
If you don't like that idea, and want to stick with manually writin the XML deserialization code, then you need to do two things:
attach a namespace to the XmlRoot attribute.
change the name of your class to ResponseExt, and derive it from a class called Response. Decorate that Response class with an XmlInclude attribute. This aligns the use of the Xml Serializer with the xsi:type used in the XML fragment.
It looks like this in code:
[XmlRoot("Response", Namespace="http://www.thirdparty.com/lr/")]
public class ResponseExt : Response {
}
[XmlRoot("Response", Namespace="http://www.thirdparty.com/lr/")]
[XmlInclude(typeof(ResponseExt))]
public class Response {
public string Code {get; set;}
public string Message {get; set;}
public string SessionId {get; set;}
}
public class XsiType
{
public static void Main(string[] args)
{
try
{
string filename = "XsiType.xml";
XmlSerializer s1 = new XmlSerializer(typeof(Response));
ResponseExt r = null;
using(System.IO.StreamReader reader= System.IO.File.OpenText(filename))
{
r= (ResponseExt) s1.Deserialize(reader);
}
var builder = new System.Text.StringBuilder();
var xmlws = new System.Xml.XmlWriterSettings { OmitXmlDeclaration = true, Indent= true };
using ( var writer = System.Xml.XmlWriter.Create(builder, xmlws))
{
//s1.Serialize(writer, r, ns);
s1.Serialize(writer, r);
}
string xml = builder.ToString();
System.Console.WriteLine(xml);
}
catch (System.Exception exc1)
{
Console.WriteLine("Exception: {0}", exc1.ToString());
}
}
}
related: How can I force the use of an xsi:type attribute?

c# create an instance of an object from string

I have a string variable contain:
string classCode = "public class Person { public string Name{get;set;} }";
How can I create an instance of an object from the classCode ?
like
object obj = CreateAnInstanceAnObject(classCode);

You'll need to use CodeDom to compile an in-memory assembly, and then use reflection to create the type.
Here's a sample article on MSDN that walks through the process of code generation.
Once you've compiled the code, you can use Activator.CreateInstance to create an instance of it.

Building on the answers from above, here is a working demo to generate, compile and instantiate a class from an in-memory assembly:
namespace DynamicCompilation
{
using System;
using System.CodeDom;
using System.CodeDom.Compiler;
using System.Reflection;
using Microsoft.CSharp;
internal static class Program
{
private static void Main()
{
var ccu = new CodeCompileUnit();
var cns = new CodeNamespace("Aesop.Demo");
cns.Imports.Add(new CodeNamespaceImport("System"));
var ctd = new CodeTypeDeclaration("Test")
{
TypeAttributes = TypeAttributes.Public
};
var ctre = new CodeTypeReferenceExpression("Console");
var cmie = new CodeMethodInvokeExpression(ctre, "WriteLine", new CodePrimitiveExpression("Hello World!"));
var cmm = new CodeMemberMethod
{
Name = "Hello",
Attributes = MemberAttributes.Public
};
cmm.Statements.Add(cmie);
ctd.Members.Add(cmm);
cns.Types.Add(ctd);
ccu.Namespaces.Add(cns);
var provider = new CSharpCodeProvider();
var parameters = new CompilerParameters
{
CompilerOptions = "/target:library /optimize",
GenerateExecutable = false,
GenerateInMemory = true
};
////parameters.ReferencedAssemblies.Add("System.dll");
var results = provider.CompileAssemblyFromDom(parameters, ccu);
if (results.Errors.Count == 0)
{
var t = results.CompiledAssembly.GetType("Aesop.Demo.Test");
var inst = results.CompiledAssembly.CreateInstance("Aesop.Demo.Test");
t.InvokeMember("Hello", BindingFlags.Public | BindingFlags.Instance | BindingFlags.InvokeMethod, null, inst, null);
}
Console.ReadLine();
}
}
}

Simple put you cannot do this in one line as you are attempting. It is possible to create an instance of an existing class via it's name and one of the overloads of Activator.CreateInstance.
What you are trying to achieve here though is quite different. You are attempting to both 1) define a new class type and 2) create an instance of it. Defining new metadata in the running process dynamically is very difficult to achieve with static languages like C#. It requires a significant amount of work that can't easily be put into a StackOverflow answer.

The following project should guide you in what your trying to accomplish:
RunTime Code Compilation
However, if you are attempting to write code at runtime, you may want to rethink your architecture. You may be creating more of a headache for yourself than you need to be.
What are you trying to accomplish by creating this object?