How to use AvroSerializer without a schema registry - c#

I am trying to write a unit test that verifies that adding a new property to an Avro schema is backwards compatible.
First I took the Avro generated .cs model and saved it as MyModelOld.cs and renamed the class inside to MyModelOld.
Then I re-ran Avro gen against the avsc file with the new property.
What I'm trying to do is this:
var schemaRegistry = -> something that doesn't require a running docker image <-;
var deserializerOld = new AvroDeserializer<MyModelOld>(schemaRegistry);
var serializerNew = new AvroSerializer<MyModel>(schemaRegistry);
var myModel = new MyModel() {...};
var myModelBytes = await serializerNew.SerializeAsync(myModel, new());
var myModelOld = await deserializerOld.DeserializeAsync(myModelBytes, false, new());
// Check properties...
Then I was going to go the opposite direction and check that the new property uses the specified default value.
The problem I'm having is what to use for the schema registry. I don't want to have a docker image running for these tests because I don't think it shouldn't be necessary.
I've tried a mock of ISchemaRegistry, but it appears to need a fully functional class in order for serialize/deserialize to work.
I could probably walk through the logic for CachedSchemaRegistryClient and try to munge it to work, but before I do so I'd like to find out if someone knows of an ISchemaRegistry implementaion that would work for my use case.
Has anyone tried to write tests to validate backwards compatibility of Avro schema updates?
If so, how did you go about doing so?
Thanks.

I ended up doing it this way:
private ISchemaRegistryClient NewTestRegistry(string topic)
{
// Code to mock SchemaRegistry taken from:
// https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/test/Confluent.SchemaRegistry.Serdes.UnitTests/SerializeDeserialize.cs
Dictionary<string, int> store = new Dictionary<string, int>();
var schemaRegistryMock = new Mock<ISchemaRegistryClient>();
#pragma warning disable CS0618 // Type or member is obsolete
schemaRegistryMock.Setup(x => x.ConstructValueSubjectName(topic, It.IsAny<string>()))
.Returns($"{topic}-value");
schemaRegistryMock.Setup(x => x.RegisterSchemaAsync($"{topic}-value", It.IsAny<string>(), It.IsAny<bool>()))
.ReturnsAsync((string topic, string schema, bool normalize) =>
store.TryGetValue(schema, out int id) ? id : store[schema] = store.Count + 1
);
#pragma warning restore CS0618 // Type or member is obsolete
schemaRegistryMock.Setup(x => x.GetSchemaAsync(It.IsAny<int>(), It.IsAny<string>()))
.ReturnsAsync((int id, string format) =>
new Schema(store.Where(x => x.Value == id).First().Key, null, SchemaType.Avro)
);
return schemaRegistryMock.Object;
}
[TestMethod]
public async Task BackwardsCompatible()
{
var topic = "MyCoolTopic";
var schemaRegistry = NewTestRegistry(topic);
var context = new SerializationContext(MessageComponentType.Value, topic);
var deserializerOld = new AvroDeserializer<MyModelOld>(schemaRegistry);
var serializerNew = new AvroSerializer<MyModel>(schemaRegistry);
var myModel = new MyModel() { /* Set properties */};
var myModelBytes = await serializerNew.SerializeAsync(myModel, context);
var myModelOld = await deserializerOld.DeserializeAsync(myModelBytes, false, context);
// Check properties...
}
[TestMethod]
public async Task ForwardsCompatible()
{
// Similar to the above test.
}

If you want to test schemas, you don't need Kafka-related serializers; just use raw Avro C# library.
Alternatively, look at the existing tests
var config = new SchemaRegistryConfig { Url = "irrelevanthost:8081" };
var src = new CachedSchemaRegistryClient(config);
Assert...(src... );

Related

Mock testing IDistrubutedCache

I am trying to test a function that uses IDistrubtedMemory cache, I've configured the cache mock object's Get and Set and the Get function seems to be functioning, but when I set a new memory object, it never gets returned. Below is the test
-- Test code
[Fact]
public async void Can_AddToMonitoring()
{
var cacheKey = "Simulator";
var tList = new List<string>();
var tNumber = Guid.NewGuid().ToString().Substring(0, 6);
// Setup - Add mock objects that are needed
var mockLogger = new Mock<ILogger<TSimulatorService>>();
var mockCache = new Mock<IDistributedCache>();
mockCache.Setup(cacheMoc => cacheMoc.Get(It.IsAny<string>()))
.Returns(Mock.Of<byte[]>);
mockCache.Setup(cacheMoc => cacheMoc.Set(It.IsAny<string>(), It.IsAny<byte[]>(), null));
var mockEventBus = new Mock<IEventBus>();
var tSimulationService = new TSimulatorService(mockLogger.Object, mockEventBus.Object, mockCache.Object);
await tSimulationService.AddToMonitoring(tNumber);
// Assert - Verify the MemoryCache contains the tag we sent over
var testTag = await tSimulationService.GetTsForPublish();
var tSimulations = testTag as TSimulation[] ?? testT.ToArray();
tSimulations.ShouldNotBeEmpty();
tSimulations.ShouldContain( t => t.TNumber.Equals(tNumber));
}
This is the method I am attempting to test
public async Task AddToMonitoring(string tNumber)
{
var cacheList = await GetMonitoredTListFromCache();
var tList = cacheList.ToList();
if (!tagList.Contains(tNumber))
tList.Add(tNumber);
await _cache.SetStringAsync(tListCacheKey, JsonConvert.SerializeObject(tList));
}
Everything appears to work until I get to the Assert part of the test, when I attempt to pull the object I set, all I get is a null return and i am not entirely sure of why?

How to use hangfire in .net core with mongodb?

I want to use Hangfire for background jobs in registration form process,But I am unable to find Startup.cs file code for Hangfire.mongo.
Just an update to this thread,
As of v0.7.11, MongoMigrationOptions have been updated and no longer contains Strategy. As per the release notes, you now have to use MigrationStrategy instead of Strategy. Also, the values you use for these are different as well. See example below.
var migrationOptions = new MongoMigrationOptions
{
MigrationStrategy = new MigrateMongoMigrationStrategy(),
BackupStrategy = new CollectionMongoBackupStrategy()
};
In Startup class
In ConfigureServices method
Add
//you will use some way to get your connection string
var mongoConnection = Configuration.GetConnectionString("MongoDBAtlasJaken");
var migrationOptions = new MongoMigrationOptions
{
Strategy = MongoMigrationStrategy.Drop,
BackupStrategy = MongoBackupStrategy.Collections
};
services.AddHangfire(config =>
{
config.SetDataCompatibilityLevel(CompatibilityLevel.Version_170);
config.UseSimpleAssemblyNameTypeSerializer();
config.UseRecommendedSerializerSettings();
config.UseMongoStorage(mongoConnection, "Hangfire",new MongoStorageOptions { MigrationOptions = migrationOptions });
});
services.AddHangfireServer();
In Configure method you can add if you want
app.UseHangfireDashboard();
#Graeme's anwser works on the older version of Hangfire. For the new version,
The Migration and Backup strategy are changed to class from enum.
var migrationOptions = new MongoMigrationOptions
{
MigrationStrategy = new DropMongoMigrationStrategy(),
BackupStrategy = new CollectionMongoBackupStrategy()
};
The UseMongoStorage method does not accept the collection name anymore.
services.AddHangfire(config =>
{
config.SetDataCompatibilityLevel(CompatibilityLevel.Version_170);
config.UseSimpleAssemblyNameTypeSerializer();
config.UseRecommendedSerializerSettings();
config.UseMongoStorage(mongoConnection, new MongoStorageOptions { MigrationOptions = migrationOptions, CheckConnection = false });
});
services.AddHangfireServer();

Tests using InMemoryDatabase and Identity columns, how to deal?

.Net Core 2.2 / EFC 2.2.3 / Pomelo.EntityFrameworkCore.MySql 2.2.0
Imagine that you have a table called Colors with some predefined data.
public void Configure(EntityTypeBuilder<Color> builder)
{
builder.ToTable("Colors");
builder.HasKey(r => r.Id).UseMySqlIdentityColumn();
builder.Property(r => r.Name).IsRequired().HasMaxLength(255);
builder.Property(v => v.RGB).IsRequired().HasMaxLength(7);
builder.HasData(GetSeed());
}
private ICollection<Color> GetSeed()
{
return new List<Color>()
{
new Color(){Id=1, Name="Black", RGB="#000"},
new Color(){Id=2, Name="White", RGB="#fff"},
}
}
One of my tests is to test the CreateColorCommandHandler. Very straightfoward
var Context = CBERPContextFactory.Create();
var query = new CreateColorCommandHandler(Context);
var command = new CreateColorCommand();
command.Name= "Random color";
command.RGB = "#001122";
var colorId = await query.Handle(command, CancellationToken.None);
//Assert
Assert.IsInstanceOf<long>(colorId);
Assert.NotZero(colorId);
var cor = Context.Colors.Where(p => p.Id == colorId).SingleOrDefault();
Assert.NotNull(cor);
Assert.AreEqual(command.Name, cor.Name);
Assert.AreEqual(command.RGB, cor.RGB);
CBERPContextFactory.Destroy(Context);
//>>> Handle simply add a new entity without informing ID
Handle method
public async Task<long> Handle(CreateColorCommand request, CancellationToken cancellationToken)
{
var entity = new Color
{
Name = request.Name,
RGB = request.RGB,
};
_context.Colors.Add(entity);
await _context.SaveChangesAsync(cancellationToken);
return entity.Id;
}
When I ran this test I get the error An item with the same key has already been added. Key: 1. Which means that InMemoryDatabase do not has auto increment feature.
Am I writing the test wrong?
How can I test case like this? I want to make sure that the command is OK.
Probably I am missing some very basic rule here.
I assume problem is in the following line:
var Context = CBERPContextFactory.Create();
May be you are using the same context instance for multiple tests. According to Testing with InMemory documentation:
Each test method specifies a unique database name, meaning each method has its own InMemory database.
So make sure that your each and every test method has a distinct context instance.
If still does not work then try setting the identity key value manually because InMemory database may does not support auto-increment.
InMemoryDatabase do not have all features yet, and AUTO INCREMENT one of those that need improvements: https://github.com/aspnet/EntityFrameworkCore/issues/6872
Not the answer I wanted, but is the one working for now: clear all seeds before testing.
private static void Clear(this DbContext context)
{
var properties = context.GetType().GetProperties();
foreach (var property in properties)
{
var setType = property.PropertyType;
bool isDbSet = setType.IsGenericType && (typeof(DbSet<>).IsAssignableFrom(setType.GetGenericTypeDefinition()));
if (!isDbSet) continue;
dynamic dbSet = property.GetValue(context, null);
dbSet.RemoveRange(dbSet);
}
context.SaveChanges();
}

Testing HttpModule wtih Response.Filter and Response.Write

I'm working on an Http Module that simply records the response time and size and then appends the results to the response body.
My module looks something like this:
public override void PreRequestHandlerExecute(HttpContextBase context)
{
// Add a filter to capture response stream
context.Response.Filter = new ResponseSniffer(context.Response.Filter);
}
public override void ApplicationEndRequest(HttpContextBase context)
{
....
context.Response.Write(builder.ToString());
}
I now wish to unit test this module. I'm very new to unit testing. I've adapted code from o2platform to get a moq httpcontext and that works so far. However, the response filter does seem to get set in Pre and the response body is what I initialized it has from the test setup.
I've tried a handful options (and read a lot of stuff) but none of these seemed to work:
public Mock<HttpResponseBase> MockResponse { get; set; }
...
var outputStream = new MemoryStream();
var filter = new MemoryStream();
//MockResponse.Setup(response => response.OutputStream).Returns(GetMockStream(outputStream).Object);
//MockResponse.Setup(response => response.Filter).Returns(GetMockStream(filter).Object);
MockResponse.Setup(response => response.OutputStream).Returns(() => outputStream);
//MockResponse.SetupSet(response => response.OutputStream = It.IsAny<Stream>()).Returns(() => outputStream);
MockResponse.Setup(response => response.Filter).Returns(() => filter);
MockResponse.SetupSet(response => response.Filter = It.IsAny<Stream>());
MockResponse.SetupSet(response => response.Filter = It.IsAny<ResponseSniffer>());
Test Method
[TestMethod]
public void TestMethod1()
{
var mockHttpContext = new MoqHttpContext();
var httpContext = mockHttpContext.HttpContext();
var html = #"<html>
<head></head>
<body>
<h1>Hello World</h1>
</body>
</html>";
httpContext.ResponseWrite(html);
httpContext.StreamWrite(httpContext.Response.Filter, html);
var module = new Module();
module.PreRequestHandlerExecute(mockHttpContext.HttpContext());
module.ApplicationBeginRequest(mockHttpContext.HttpContext());
module.ApplicationEndRequest(mockHttpContext.HttpContext());
var responseRead = httpContext.ResponseRead(); //extension method to get output stream
var b = 1; //put breakpoint here
}
I realize the test needs to Assert rather then the breakpoint. I also realize that the test should be broken up a bit.
Code Repo
Github
Let's see at the following statement in Module.ApplicationEndRequest() method:
context.Response.Write(builder.ToString());
When this code is executed from Unit Test, context.Response is a mock that you set up in MoqHttpContext.CreateBaseMocks():
MockResponse = new Mock<HttpResponseBase>();
// ...
MockContext.Setup(ctx => ctx.Response).Returns(MockResponse.Object);
You can't expect that you call a Write() method on a mock and then can read the same data back. Mock is a fake object. Its default implementation of Write() method does nothing, and passed data is just lost.
To fix the problem, you could setup a callback on Response mock that will write passed data to a stream and then return it back on read. You are actually very close to it.
In MoqHttpContext class declare a stream where you will save the data:
public class MoqHttpContext
{
private readonly MemoryStream _outputStream = new MemoryStream();
// ...
}
Then in CreateBaseMocks() method setup a callback:
public MoqHttpContext CreateBaseMocks()
{
// ...
MockResponse = new Mock<HttpResponseBase>();
MockResponse.Setup(x => x.Write(It.IsAny<string>())).Callback<string>(s =>
{
var data = Encoding.ASCII.GetBytes(s);
_outputStream.Write(data, 0, data.Length);
_outputStream.Flush();
_outputStream.Position = 0;
});
// ...
}
You also should remove a line that sets inputStream position to 0 in MoqHttpContextExtensions.StreamWrite(), so that html data that you write in UnitTest1.TestMethod1() is appended, not overwritten:
public static HttpContextBase StreamWrite(this HttpContextBase httpContextBase, Stream inputStream, string text)
{
if (inputStream == null) inputStream = new MemoryStream();
var streamWriter = new StreamWriter(inputStream);
inputStream.Position = inputStream.Length;
streamWriter.Write(text);
streamWriter.Flush();
// Remove this line
//inputStream.Position = 0;
return httpContextBase;
}
That's it. Now if you check value of responseRead in the test, you will see that data appended by Http module is there.
UPDATE (Fixing problem with a filter)
There are 3 different issues with current code that prevent correct work of a filter from UT.
You tried handful options for mocking Filter property, however none of them seems correct. The correct way to mock property getter with Moq is:
MockResponse.SetupGet(response => response.Filter).Returns(filter);
Remove all other statements for mocking response.Filter, but don't add above statement yet, it's not a final version.
You have following check in Module.ApplicationEndRequest() method:
if (context.Response.Filter is ResponseSniffer filter)
{
// ...
When UT is executed, context.Response.Filter is a MemoryStream not a ResponseSniffer. Setter that is called in Module constructor:
context.Response.Filter = new ResponseSniffer(context.Response.Filter);
will not actually affect value returned by Filter getter because it's a mock that currently always return instance of MemoryStream that you setup with SetupGet. To fix this problem you should actually emulate property behavior: save the value passed to setter and return it in the getter. Here is a final setup of response.Filter property:
Stream filter = new MemoryStream();
MockResponse.SetupSet(response => response.Filter = It.IsAny<Stream>()).Callback<Stream>(value => filter = value);
MockResponse.SetupGet(response => response.Filter).Returns(() => filter);
Make sure you have deleted all other mocks of response.Filter property.
The final problem that you should fix - is the order of Module invocations from UT. Currently the order is the following:
httpContext.StreamWrite(httpContext.Response.Filter, html);
// ...
var module = new Module();
module.PreRequestHandlerExecute(mockHttpContext.HttpContext());
But PreRequestHandlerExecute sets Response.Filter with an instance of ResponseSniffer. So when httpContext.StreamWrite above it is called, httpContext.Response.Filter holds actually instance of MemoryStream, not ResponseSniffer. So the last fix you should make is to change the order of statements in UT body:
// ...
var module = new Module();
module.PreRequestHandlerExecute(mockHttpContext.HttpContext());
httpContext.ResponseWrite(html);
httpContext.StreamWrite(httpContext.Response.Filter, html);
module.ApplicationBeginRequest(mockHttpContext.HttpContext());
module.ApplicationEndRequest(mockHttpContext.HttpContext());
// ...
UPDATE (UT Redesign)
At this point your UT should work. However current test is very cumbersome. The fact that it takes so much time to understand why it does not work proves it. Such tests are very hard to maintain and fix, they become a real pain over time.
Moreover it's rather Integration test than Unit test, because it invokes several of classes with different functionality - ResponseSniffer and Module.
You should strongly consider redesign of current test. And the good start is to make separate tests for ResponseSniffer and Module classes.
Most valuable test for ResponseSniffer is the one that verifies that written data is registered in RecordStream:
[TestClass]
public class ResponseSnifferTests
{
[TestMethod]
public void Write_WritesDataToRecordStream()
{
// Arrange
var inData = new byte[] { 0x01 };
var target = new ResponseSniffer(Mock.Of<Stream>());
// Act
target.Write(inData, 0, inData.Length);
// Assert
target.RecordStream.Position = 0;
var outData = new byte[inData.Length];
int outSize = target.RecordStream.Read(outData, 0, outData.Length);
Assert.AreEqual(inData.Length, outSize);
CollectionAssert.AreEqual(inData, outData);
}
}
As regards Module class, there are several checks that should be made:
PreRequestHandlerExecute() sets Response.Filter with instance of ResponseSniffer.
ApplicationBeginRequest() adds Stopwatch to context.Items dictionary.
ApplicationEndRequest() writes request info to the response.
UT approach implies checking of these facts in separate tests. Here are samples of such 3 tests:
[TestClass]
public class ModuleTests
{
[TestMethod]
public void PreRequestHandlerExecuteShouldSetResponseSnifferAsFilter()
{
// Arrange
Stream filter = null;
Mock<HttpResponseBase> httpResponseMock = new Mock<HttpResponseBase>();
httpResponseMock.SetupSet(response => response.Filter = It.IsAny<Stream>()).Callback<Stream>(value => filter = value);
Mock<HttpContextBase> httpContextStub = new Mock<HttpContextBase>();
httpContextStub.SetupGet(x => x.Response).Returns(httpResponseMock.Object);
var target = new Module();
// Act
target.PreRequestHandlerExecute(httpContextStub.Object);
// Assert
Assert.IsNotNull(filter);
Assert.IsInstanceOfType(filter, typeof(ResponseSniffer));
}
[TestMethod]
public void ApplicationBeginRequestShouldStoreStopwatchInContextItems()
{
// Arrange
var items = new Dictionary<string, object>();
Mock<HttpContextBase> httpContextStub = new Mock<HttpContextBase>();
httpContextStub.SetupGet(x => x.Items).Returns(items);
var target = new Module();
// Act
target.ApplicationBeginRequest(httpContextStub.Object);
// Assert
Assert.IsTrue(items.ContainsKey("X-ResponseTime"));
Assert.IsInstanceOfType(items["X-ResponseTime"], typeof(Stopwatch));
}
[TestMethod]
public void ApplicationEndRequestShouldAddRequestInfoToResponse()
{
// Arrange
Mock<HttpRequestBase> httpRequestMock = new Mock<HttpRequestBase>();
httpRequestMock.SetupGet(x => x.FilePath).Returns("/test");
string writtenData = null;
Mock<HttpResponseBase> httpResponseMock = new Mock<HttpResponseBase>();
httpResponseMock.Setup(x => x.Write(It.IsAny<string>())).Callback<string>(s => writtenData = s);
Mock<HttpContextBase> httpContextStub = new Mock<HttpContextBase>();
httpContextStub.SetupGet(x => x.Request).Returns(httpRequestMock.Object);
httpContextStub.SetupGet(x => x.Response).Returns(httpResponseMock.Object);
httpContextStub.SetupGet(x => x.Items).Returns(new Dictionary<string, object> { ["X-ResponseTime"] = new Stopwatch() });
var target = new Module();
// Act
target.ApplicationEndRequest(httpContextStub.Object);
// Assert
Assert.IsTrue(Regex.IsMatch(writtenData, #"Response Size: \d+ bytes<br/>"));
Assert.IsTrue(Regex.IsMatch(writtenData, #"Module request time: \d+ ms"));
}
}
As you see, the tests are pretty simple and straightforward. You don't need those MoqHttpContext and MoqHttpContextExtensions with a lot of mocks and helpers anymore. Another benefit - if some of the tests get broken, it's much easier to identify the root cause and fix it.
If you are new to Unit Testing and are looking for good source of info on it, I strongly suggest book The Art of Unit Testing by Roy Osherove.

WF 4.5 Compile CSharpValue<T> error. What is the correct method?

I've been uniting testing two simple examples of compiling CSharpValue activities. One works and the other doesn't I can't figure out why. If someone could point out the issue and optionally a change to correct it if possible.
Details:
The first unit test works SequenceActivityCompile() the second CodeActivityCompile fails with a NotSupportedException (Expression Activity type CSharpValue requires compilation in order to run. Please ensure that the workflow has been compiled.)
I heard somewhere this can be related to ForImplementation but CodeActivityCompile has the same error whether its value is true or false.
This example is a basic adaption of the Microsoft example at: https://msdn.microsoft.com/en-us/library/jj591618(v=vs.110).aspx
This example blog post discussing compiling C# expressions in WF 4+ at length. If anyone reaching this question needs a basic introduction to the topic:
http://blogs.msdn.com/b/tilovell/archive/2012/05/25/wf4-5-using-csharpvalue-lt-t-gt-and-csharpreference-lt-t-gt-in-net-4-5-compiling-expressions-and-changes-in-visual-studio-generated-xaml.aspx
Related Code:
[TestMethod]
public void SequenceActivityCompile()
{
Activity sequence = new Sequence
{
Activities = { new CSharpValue<string>("\"Hello World \"") }
};
CompileExpressions(sequence);
var result = WorkflowInvoker.Invoke(sequence);
}
[TestMethod]
public void CodeActivityCompile()
{
var code = new CSharpValue<String>("\"Hello World\"");
CompileExpressions(code);
var result = WorkflowInvoker.Invoke(code);
}
void CompileExpressions(Activity activity)
{
// activityName is the Namespace.Type of the activity that contains the
// C# expressions.
string activityName = activity.GetType().ToString();
// Split activityName into Namespace and Type.Append _CompiledExpressionRoot to the type name
// to represent the new type that represents the compiled expressions.
// Take everything after the last . for the type name.
//string activityType = activityName.Split('.').Last() + "_CompiledExpressionRoot";
string activityType = "TestType";
// Take everything before the last . for the namespace.
//string activityNamespace = string.Join(".", activityName.Split('.').Reverse().Skip(1).Reverse());
string activityNamespace = "TestSpace";
// Create a TextExpressionCompilerSettings.
TextExpressionCompilerSettings settings = new TextExpressionCompilerSettings
{
Activity = activity,
Language = "C#",
ActivityName = activityType,
ActivityNamespace = activityNamespace,
RootNamespace = null,
GenerateAsPartialClass = false,
AlwaysGenerateSource = true,
ForImplementation = false
};
// Compile the C# expression.
TextExpressionCompilerResults results =
new TextExpressionCompiler(settings).Compile();
// Any compilation errors are contained in the CompilerMessages.
if (results.HasErrors)
{
throw new Exception("Compilation failed.");
}
// Create an instance of the new compiled expression type.
ICompiledExpressionRoot compiledExpressionRoot =
Activator.CreateInstance(results.ResultType,
new object[] { activity }) as ICompiledExpressionRoot;
// Attach it to the activity.
System.Activities.Expressions.CompiledExpressionInvoker.SetCompiledExpressionRoot(
activity, compiledExpressionRoot);
}

Categories