I want to write a custom serializer for kafka in CSharp. I've searched a lot and I couldn't find a good reference on how to write custom serializer for kafka in a dotnet language.
Any thoughts on how to write one with dotnet?
I just find the answer to this question.
The library to use for kafka in dotnet is provided by confluent.
Kafka .NET Client
There should be a serialization class implementing the interface :
Confluent.Kafka.ISerializer<T>
Normally we should create the producer via ProducerBuilder class :
Confluent.Kafka.ProducerBuilder<TKey, TValue>
There is a method to this class to set serializers for both key and value.
It is a little different than the original documentation for custom serializer in Kafka which instructing to set the name of the custom serializer class in the producer configuration.
Following is the code to set the value serializer:
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
ClientId = "thiPC"
};
var producerBuilder = new ProducerBuilder<Null, Customer>(config);
producerBuilder.SetValueSerializer(new CustomerSerializer()); // <------ The Magic Code
var producer = producerBuilder.Build();
The sample code for the Serialization class is like this;
public class CustomerSerializer : Confluent.Kafka.ISerializer<Customer>
{
public byte[] Serialize(Customer data, SerializationContext context)
{ ..........}
}
There is nothing important for the Customer class in my example it is a normal class just to hold the customer properties.
In serialization class, you should return a byte[] in the Serialize method which is obvious!
I hope this would be useful for folks implementing Kafka with dotnet.
Related
I am using NSwag to generate C# rest client for my asp.net core web API. At the moment, I just need to generate the client interfaces and not the classes themselves. I tried the following settings to generate just C# client interfaces but it does not generate nor classes neither interfaces.
GenerateClientClasses = false
GenerateClientInterfaces = true
Is there anything wrong with my settings?
Also, Is there any way to extend or change the generated code of the client interfaces. For example how I can annotate the client interface methods with some custom attributes? For example:
public partial interface IGetEmployeeByIdClient
{
// How to add the following custom attributes to the generated client interface method
[MyCustomerAttribute("/api/v1/GetEmployeeById/{id}"] )
System.Threading.Tasks.Task<GetEmployeeByIdQueryResult> GetEmployeeByIdAsync(string id);
}
This does not look intuitive but I managed to get just interfaces with following configuration.
var settings = new TypeScriptClientGeneratorSettings
{
GenerateClientClasses = false,
GenerateDtoTypes = true,
GenerateClientInterfaces = true,
TypeScriptGeneratorSettings =
{
TypeStyle = NJsonSchema.CodeGeneration.TypeScript.TypeScriptTypeStyle.Interface
}
};
I have a .net core 3.0 preview 6 MVC application and API.
In the API, I am using a third party class library (that I can't change) which defines the class properties as Pascal cased with the JsonProperty, PropertyName snaked cased eg...
public class Company
{
[JsonProperty(PropertyName = "company_name")]
public string CompanyName { get; set; }
more properties ….
}
The problem is that when I supply these via the api they reach the MVC app as Camel case (the default for .net core 3)... and then can't be Deserialized back the to the class model.
Not matter what I try, the API always produces camel cased JSon, eg. the property above will be called companyName.
I tried,
options.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver { NamingStrategy = new CamelCaseNamingStrategy { OverrideSpecifiedNames = true } };
options.SerializerSettings.ContractResolver = new DefaultContractResolver { NamingStrategy = new DefaultNamingStrategy { OverrideSpecifiedNames = true } };
I've tried NamingStrategy = null on both camel and default ContractResolver. Also tried setting the NamingStrategy to Snake
But nothing changes the outputted Json, it's always camelcased.
I can see the resulting string is camel cased by using ReadAsStringAsync in the MVC app... I when I use JsonConvert.DeserializeObject, the properties are always null, because neither the name or Json PropertyName match the names in the resulting string.
Is this a bug in .net core previews or am missing something else?
Thanks Mustafa, your suggested duplicate is kinda the same issue with kinda the same solutions that I've already tried i.e. changing the setting of the ContractResolver / NamingStrategy to different values.... however, My issue is that none of the suggested solutions appear to have any effect on the API response it always comes back as camelCased.
Interestingly, when I change the NamingStrategy to say Snake, Swagger shows the schema as set (i.e. snake) but the actual output is still camelCased!!!
Also, I have no control over the base classes so I can't change the names / json properties of the classes I'm attempting to transmit.
Microsoft.AspNetCore.Mvc.NewtonsoftJson
Doesn't come up default.
Try to install this nuget package manually to your service project. That worked for me.
Try this and remove all JsonProperty attributes.
From now if you don't specify any JsonPropertyy it will act like this CompanyName like company_name or ProPertyName1 like pro_perty_name1. In this examples will explain the idea of the property name.
And be sure add this configuration to bottom of the ConfigureServices method, it may overwritten by another things i dont know.
services.AddMvc().AddJsonOptions(options =>
{
options.SerializerSettings.ContractResolver = new DefaultContractResolver { NamingStrategy = new SnakeCaseNamingStrategy() };
});
Not really sure where the issue was but had a feeling it was something to do with the mix of Newtonsoft.Json, Json.Net, Swagger and the fact that I was using the Microsoft.AspNet.WebApi.Client to get the HttpContent.ReadAsAsync….all having different Json's
So, I decided to start again with a real simple app and api using the new System.Text.Json included in .Net Core preview (and none of the other libraries). Also not using the HttpContent.ReadAsAsync but instead reading the response as a string and then deserializing with the new library (System.Text.Json)
Doing this I had exactly the same issue …. neither the property name or Json PropertyName match the names in the api returned string i.e class property name = "CompanyName" and Json PropertyName = "company_name" and the api supplied json name = "companyName". So the value isn't set when Deserializing.
However, in the new System.Text.Json options I'm able to specify PropertyNameCaseInsensitive = true, which fixes my problem, now companyName does equal CompanyName and the class model values are set correctly when Deserializing.
So my api call methods end up looking like this...
using HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, string.Format("Companies?aCompanyName={0}", aCompanyName));
using HttpResponseMessage response = await Client.SendAsync(request);
string content = await response.Content.ReadAsStringAsync();
if (response.IsSuccessStatusCode == false)
{
throw new ApiException
{
StatusCode = (int)response.StatusCode,
Content = content
};
}
_JsonOptions = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true
};
return JsonSerializer.Deserialize<IEnumerable<Company>> (content, _JsonOptions);
I did attempt to set the JsonSerializerOptions globally in the startup class but this didn't work.
I've transferred this approach to all my http calls in my app, removed all references to Newtonsoft and it all works.
I came across this issue when converting my api from .netcore 2.2 to .netcore 3.
My api was returning responses converted to camelCase even though my models were PascalCase.
In startup.cs:
.netcore 2:
services.AddMvc()
.AddJsonOptions(options => options.SerializerSettings.ContractResolver = new DefaultContractResolver());
.netcore 3:
// keeps the casing of models when serializing to json (default is converting to camelCase)
services.AddMvc()
.AddJsonOptions(options => options.JsonSerializerOptions.PropertyNamingPolicy = null);
This means you don't need to import newtonsoft.json.
Recently I found an annoying different behavior between net35 and net40 NewtonSoft Json library. For net40, the serialized payload is good. But for net35, the serialized payload includes annoying value k__BackingField.
Here is a sample code to repro the issue:
// Notice that there is no serializable attribute
public class SamplePayload
{
public Guid Id { get; set; }
}
static void Main(string[] args)
{
var writeStream = new MemoryStream();
var formatter = new JsonMediaTypeFormatter();
formatter.WriteToStreamAsync(typeof(SamplePayload), new SamplePayload(), writeStream, null, null).Wait();
Console.WriteLine(System.Text.Encoding.UTF8.GetString(writeStream.ToArray()));
}
If reference a net40/net45 Json library, the serialized payload is like "Id" which is expected. But with net35 library, the serialized payload includes "k__BackingField".
I'm wondering why there is such a behavior difference? Is it a defect in NewtonSoft Json library, or a by-design behavior? If it is latter, what's the best practice to avoid such issue?
.NET 3.5 is very old and k__BackingField is there by design AFAIK. Getting rid of it is well-documented. You need to check the version and when it includes that field, apply the solution linked here.
I have a WebAPI endpoint that implements two different versions of the API (legacy and new). The legacy endpoints use a specific Serializer that has all objects serialized as lower case words with underscores, the v2 endpoint uses camel cased property names. For example, V1 = "document_type" and V2 = "documentType"
This is currently achieved using controller specific attributes to define the serialization, like so:
public class CamelCaseControllerConfiguration : Attribute, IControllerConfiguration
{
public void Initialize(HttpControllerSettings controllerSettings, HttpControllerDescriptor controllerDescriptor)
{
controllerSettings.Formatters.JsonFormatter.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
controllerSettings.Formatters.JsonFormatter.SerializerSettings.Converters.Add(new StringEnumConverter());
}
}
This all works fine when called from a client via REST, but the documentation generated by Swagger always shows the property names using the legacy serializer settings. Any suggestions on configuring swashbuckle to serialize each version properly?
as far as i know swagger use first Formatters settings that can find. so if you use this:
controllerSettings.Formatters.Insert(0, new JsonMediaTypeFormatter { SerializerSettings = { ContractResolver = new CamelCasePropertyNamesContractResolver() } });
your documentation generated by Swagger will be fine.
swagger is a very good library and i hope they can fix this problem soon.
I am currently having 2 issues in service stack. I am currently trying to build a service to imitate an existing server software. This requires a few things that i am having issues with.
This is using a self hosted servicestack instance and latest version
I need to have service on "/" that takes no paramters.
All my services need to return results using a customer XML serializer not the data contact one no matter what is in the accept header. (currently return html representation of DTO)
For issue 1 i have been using [FallbackRoute("/")] which is working but then no matter what i do i can't get my custom serializer to be used.
For issue 2 i made a custom serializer using the dotnet xml serializer that will generate the output i need and registered it as a ContentTypeFilters. I then manually set the response type header but this did not trigger my serializer. This is really starting to drive me nuts as i need to implement about 20 services and i can't even get the simple root service working let alone the rest of them.
Basically my XML is in a format the DataContract serializer can't handle and the url's and content must be an exact match for the existing system.
It looks like both issue 1 and issue 2 are really the same issue; Your custom serialiser isn't getting called. This is either an issue with registering your serialiser, returning the content type or both. Below shows how you should set it up. Using ServiceStack v4:
Register your custom serialiser:
In your AppHost Configure method you need to register your custom XML serialiser:
StreamSerializerDelegate serialize = (request, response, stream) => {
// Replace with appropriate call to your serializer and write the output to stream
var myCustomSerializer = new MyCustomSerializer(response);
stream.write(myCustomerSerializer.getResult());
};
StreamDeserializerDelegate deserialize = (type, fromStream) => {
// Implement if you expect to receive responses using your type
throw new NotImplementedException();
};
// Register these methods to run if content type 'application/xml' is sent/received
ContentTypes.Register("application/xml", serialize, deserialize);
Set the return content type:
In your service you need to set the return content type, so the serialiser knows to run. You can do this either by adding an attribute on each method than needs to use this type, or if all your methods return this type you can configure it as the default.
Per method basis:
You can use the AddHeader attribute with the ContentType parameter. i.e:
public class TestService : Service
{
[AddHeader(ContentType = "application/xml")]
public TestResponse Get(RootRequest request)
{
return new TestResponse { Message = "Hello from root" };
}
}
All methods return this type:
You can set the default content type in the AppHost Configure method. i.e:
public override void Configure(Funq.Container container)
{
SetConfig(new HostConfig {
DebugMode = true,
DefaultContentType = "application/xml"
});
}
Fully working demo app
The demo is a self hosted console app, that takes a request to the root / or to /Test and returns a custom serialised response.
Hope this helps.