We Keep Coding

How to use AvroSerializer without a schema registry

I am trying to write a unit test that verifies that adding a new property to an Avro schema is backwards compatible.
First I took the Avro generated .cs model and saved it as MyModelOld.cs and renamed the class inside to MyModelOld.
Then I re-ran Avro gen against the avsc file with the new property.
What I'm trying to do is this:
var schemaRegistry = -> something that doesn't require a running docker image <-;
var deserializerOld = new AvroDeserializer<MyModelOld>(schemaRegistry);
var serializerNew = new AvroSerializer<MyModel>(schemaRegistry);
var myModel = new MyModel() {...};
var myModelBytes = await serializerNew.SerializeAsync(myModel, new());
var myModelOld = await deserializerOld.DeserializeAsync(myModelBytes, false, new());
// Check properties...
Then I was going to go the opposite direction and check that the new property uses the specified default value.
The problem I'm having is what to use for the schema registry. I don't want to have a docker image running for these tests because I don't think it shouldn't be necessary.
I've tried a mock of ISchemaRegistry, but it appears to need a fully functional class in order for serialize/deserialize to work.
I could probably walk through the logic for CachedSchemaRegistryClient and try to munge it to work, but before I do so I'd like to find out if someone knows of an ISchemaRegistry implementaion that would work for my use case.
Has anyone tried to write tests to validate backwards compatibility of Avro schema updates?
If so, how did you go about doing so?
Thanks.

I ended up doing it this way:
private ISchemaRegistryClient NewTestRegistry(string topic)
{
// Code to mock SchemaRegistry taken from:
// https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/test/Confluent.SchemaRegistry.Serdes.UnitTests/SerializeDeserialize.cs
Dictionary<string, int> store = new Dictionary<string, int>();
var schemaRegistryMock = new Mock<ISchemaRegistryClient>();
#pragma warning disable CS0618 // Type or member is obsolete
schemaRegistryMock.Setup(x => x.ConstructValueSubjectName(topic, It.IsAny<string>()))
.Returns($"{topic}-value");
schemaRegistryMock.Setup(x => x.RegisterSchemaAsync($"{topic}-value", It.IsAny<string>(), It.IsAny<bool>()))
.ReturnsAsync((string topic, string schema, bool normalize) =>
store.TryGetValue(schema, out int id) ? id : store[schema] = store.Count + 1
);
#pragma warning restore CS0618 // Type or member is obsolete
schemaRegistryMock.Setup(x => x.GetSchemaAsync(It.IsAny<int>(), It.IsAny<string>()))
.ReturnsAsync((int id, string format) =>
new Schema(store.Where(x => x.Value == id).First().Key, null, SchemaType.Avro)
);
return schemaRegistryMock.Object;
}
[TestMethod]
public async Task BackwardsCompatible()
{
var topic = "MyCoolTopic";
var schemaRegistry = NewTestRegistry(topic);
var context = new SerializationContext(MessageComponentType.Value, topic);
var deserializerOld = new AvroDeserializer<MyModelOld>(schemaRegistry);
var serializerNew = new AvroSerializer<MyModel>(schemaRegistry);
var myModel = new MyModel() { /* Set properties */};
var myModelBytes = await serializerNew.SerializeAsync(myModel, context);
var myModelOld = await deserializerOld.DeserializeAsync(myModelBytes, false, context);
// Check properties...
}
[TestMethod]
public async Task ForwardsCompatible()
{
// Similar to the above test.
}

If you want to test schemas, you don't need Kafka-related serializers; just use raw Avro C# library.
Alternatively, look at the existing tests
var config = new SchemaRegistryConfig { Url = "irrelevanthost:8081" };
var src = new CachedSchemaRegistryClient(config);
Assert...(src... );

"Please ensure OpenTelemetry is configured via Steeltoe extension methods" error?

I'm getting this error every time that the endpoint /actuator/prometheus is being call.
I don't have any idea what could be the problem?
This is the initialization code (I'm sure that openTelemetryEndpoint variable has a value):
builder.Services.AddAllActuators();
builder.Services.AddPrometheusActuator();
// OpenTelemetry configuration
var openTelemetryServiceName = Environment.GetEnvironmentVariable("OTEL_SERVICE_NAME");
var openTelemetryEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT");
if (!string.IsNullOrWhiteSpace(openTelemetryEndpoint))
{
// Configure metrics
builder.Services.AddOpenTelemetryMetrics(b =>
{
b.AddHttpClientInstrumentation();
b.AddAspNetCoreInstrumentation();
b.AddMeter(openTelemetryServiceName + "-metrics");
b.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(openTelemetryEndpoint);
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
});
});
// Configure tracing
builder.Services.AddOpenTelemetryTracing(b =>
{
b.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(openTelemetryServiceName));
b.AddHttpClientInstrumentation();
b.AddAspNetCoreInstrumentation();
b.AddSource(openTelemetryServiceName + "-activity-source");
b.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(openTelemetryEndpoint);
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
});
});
// Configure logging
builder.Logging.AddOpenTelemetry(b =>
{
b.IncludeFormattedMessage = true;
b.IncludeScopes = true;
b.ParseStateValues = true;
b.AddOtlpExporter(options =>
{
options.Endpoint = new Uri(openTelemetryEndpoint);
options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
});
b.AddConsoleExporter();
});
}

Marcelo,
The error message is saying that you need to use the Steeltoe extension for Metrics: `AddOpenTelemetryMetricsForSteeltoe to get the desired functionality. Steeltoe internally uses OpentelemetryMetrics for its own exports for example /Metrics, /Prometheus and also Wavefront Exporter. If you want to in addition to these add an OLTP exporter you would in addition have to use the extension methods to add the additional configuration.

UpdateDefinition with multiple fields not working

I try to build an update request using the C# mongodb driver.
I would like to update a field only if he is not null. Here is the following code :
public void Replace(Core.Entity.Project project)
{
var filter = Builders<Entity.Project>.Filter.Eq(x => x.ProjectNumber, project.ProjectNumber);
var update = BuildUpdateRequest(project);
_mongoCollection.UpdateOne(filter, update);
}
private UpdateDefinition<Entity.Project> BuildUpdateRequest(Core.Entity.Project project)
{
var builder = Builders<Entity.Project>.Update;
var update = builder.Set(x => x.LastUpdateDate, DateTime.UtcNow);
if (!string.IsNullOrEmpty(project.UserId))
{
update.Set(x => x.UserId, project.UserId);
}
}
Unfortunately when I run my unit/integration tests, the data is not updated.
Do you have a better way to use this objects ?

Ok it was easy. Fixed it with :
update = update.Set(x => x.UserId, project.UserId);

Failed to register Instance in AutoFac?

I want to use the existing mock instance of my project in AutoFac. I do not want to rewrite my PROD code. So I found something AutoFac which is not working. I think I am missing something.
I have tried below code.
public AboutTideEditorMockTest () {
aboutTideService = new AboutTideEditorService (iAboutTideEditorRepository.Object, exceptionLogServiceMock.Object);
aboutTideServiceWithNullParam = new AboutTideEditorService (null, exceptionLogServiceMock.Object);
}
//This is my test case
[FactWithAutomaticDisplayName]
public void Test1 () {
var cb = new ContainerBuilder ();
var studyLoaderMock = new Mock<IAboutTideEditorService> ().Object;
var studyLoaderMock1 = iAboutTideEditorRepository.Object;
var studyLoaderMock2 = exceptionLogServiceMock.Object;
cb.RegisterInstance (studyLoaderMock).As<IAboutTideEditorService> ();
cb.RegisterInstance (studyLoaderMock1).As<IAboutTideEditorRepository> ();
cb.RegisterInstance (studyLoaderMock2).As<IExceptionLogService> ();
var container = cb.Build ();
using (var scope = container.BeginLifetimeScope ()) {
var component = scope.Resolve<AboutTideEditorService> ();
responseData = component.AddAboutTideContent (applicationUser, aboutTide);
Assert.Equal (ProcessStatusEnum.Invalid, responseData.Status);
}
}
I want to use the existing mock instance that I am passing to "RegisterInstance". When I am trying to debug my test case I am getting "responseData" null. I am not able to go inside in AddAboutTideContent.

You are not setting up the mock return value and you need to resolve IAboutTideEditorService rather than AboutTideEditorService.
You also need to generate the mocks differently. There is no need to change the production code though!
Do it like this:
[FactWithAutomaticDisplayName]
public void Test1() {
var cb = new ContainerBuilder();
var studyLoaderMock = new Mock<IAboutTideEditorService>();
var studyLoaderMock1 = new Mock<IAboutTideEditorRepository>(); // you don't need that when resolving only IAboutTideEditorService
var studyLoaderMock2 = new Mock<IExceptionLogService>(); // you don't need that when resolving only IAboutTideEditorService
cb.RegisterInstance(studyLoaderMock.Object).As<IAboutTideEditorService>();
cb.RegisterInstance(studyLoaderMock1.Object).As<IAboutTideEditorRepository>(); // you don't need that when resolving only IAboutTideEditorService
cb.RegisterInstance(studyLoaderMock2.Object).As<IExceptionLogService>(); // you don't need that when resolving only IAboutTideEditorService
var container = cb.Build();
studyLoaderMock
.Setup(x => x.AddAboutTideContent(It.IsAny<YourTypeHereForParameterA>,
It.IsAny<YourTypeHereForParameterB>)
.Returns(new MyResponseDataType()); // add the right types here necessary, I can't tell which types they are because I am not seeing the functions code
using (var scope = container.BeginLifetimeScope()) {
var component = scope.Resolve<IAboutTideEditorService>(); // changed to IAboutTideEditorService
responseData = component.AddAboutTideContent(applicationUser, aboutTide);
Assert.Equal(ProcessStatusEnum.Invalid, responseData.Status);
}
}
Your function call was returning null because that's the default behavior of a mock with Moq = MockBehavior.Loose. If you want a function of a mock to return a specific value for non explicit or explicit parameters, you have to call Setup(delegate) and Returns(objectInstance) or Returns(Func<ObjectType>).
In general your test-setup doesn't make much sense. You are basically only registering mocks with the Autofac-Container which makes the container itself irrelevant for your tests. Using IoC for tests is usually only required when you are directly testing against the implementations rather than mocks. Those tests are called Integration-Tests.
It would make more sense like this:
[FactWithAutomaticDisplayName]
public void Test1() {
var cb = new ContainerBuilder();
var studyLoaderMock1 = new Mock<IAboutTideEditorRepository>();that when resolving only IAboutTideEditorService
var studyLoaderMock2 = new Mock<IExceptionLogService>();
var studyLoader = new AboutTideEditorService(studyLoaderMock1.Object, studyLoaderMock2.Object);
cb.RegisterInstance(studyLoader).As<IAboutTideEditorService>();
var container = cb.Build();
// now setup the functions of studyLoaderMock1 and studyLoaderMock2
// required for your function `AddAboutTideContent` from `IAboutTideEditorService` to work.
using (var scope = container.BeginLifetimeScope()) {
var component = scope.Resolve<IAboutTideEditorService>(); // changed to IAboutTideEditorService
responseData = component.AddAboutTideContent(applicationUser, aboutTide);
Assert.Equal(ProcessStatusEnum.Invalid, responseData.Status);
}
}
Keep in mind that I am assuming here the order of the parameters required for AboutTideEditorService. For more information on how to setup mocks with Moq take a look here.

Tests using InMemoryDatabase and Identity columns, how to deal?

.Net Core 2.2 / EFC 2.2.3 / Pomelo.EntityFrameworkCore.MySql 2.2.0
Imagine that you have a table called Colors with some predefined data.
public void Configure(EntityTypeBuilder<Color> builder)
{
builder.ToTable("Colors");
builder.HasKey(r => r.Id).UseMySqlIdentityColumn();
builder.Property(r => r.Name).IsRequired().HasMaxLength(255);
builder.Property(v => v.RGB).IsRequired().HasMaxLength(7);
builder.HasData(GetSeed());
}
private ICollection<Color> GetSeed()
{
return new List<Color>()
{
new Color(){Id=1, Name="Black", RGB="#000"},
new Color(){Id=2, Name="White", RGB="#fff"},
}
}
One of my tests is to test the CreateColorCommandHandler. Very straightfoward
var Context = CBERPContextFactory.Create();
var query = new CreateColorCommandHandler(Context);
var command = new CreateColorCommand();
command.Name= "Random color";
command.RGB = "#001122";
var colorId = await query.Handle(command, CancellationToken.None);
//Assert
Assert.IsInstanceOf<long>(colorId);
Assert.NotZero(colorId);
var cor = Context.Colors.Where(p => p.Id == colorId).SingleOrDefault();
Assert.NotNull(cor);
Assert.AreEqual(command.Name, cor.Name);
Assert.AreEqual(command.RGB, cor.RGB);
CBERPContextFactory.Destroy(Context);
//>>> Handle simply add a new entity without informing ID
Handle method
public async Task<long> Handle(CreateColorCommand request, CancellationToken cancellationToken)
{
var entity = new Color
{
Name = request.Name,
RGB = request.RGB,
};
_context.Colors.Add(entity);
await _context.SaveChangesAsync(cancellationToken);
return entity.Id;
}
When I ran this test I get the error An item with the same key has already been added. Key: 1. Which means that InMemoryDatabase do not has auto increment feature.
Am I writing the test wrong?
How can I test case like this? I want to make sure that the command is OK.
Probably I am missing some very basic rule here.

I assume problem is in the following line:
var Context = CBERPContextFactory.Create();
May be you are using the same context instance for multiple tests. According to Testing with InMemory documentation:
Each test method specifies a unique database name, meaning each method has its own InMemory database.
So make sure that your each and every test method has a distinct context instance.
If still does not work then try setting the identity key value manually because InMemory database may does not support auto-increment.

InMemoryDatabase do not have all features yet, and AUTO INCREMENT one of those that need improvements: https://github.com/aspnet/EntityFrameworkCore/issues/6872
Not the answer I wanted, but is the one working for now: clear all seeds before testing.
private static void Clear(this DbContext context)
{
var properties = context.GetType().GetProperties();
foreach (var property in properties)
{
var setType = property.PropertyType;
bool isDbSet = setType.IsGenericType && (typeof(DbSet<>).IsAssignableFrom(setType.GetGenericTypeDefinition()));
if (!isDbSet) continue;
dynamic dbSet = property.GetValue(context, null);
dbSet.RemoveRange(dbSet);
}
context.SaveChanges();
}