We Keep Coding

IndividualAssemblyLoadContext XmlSerializers.dll memory leak

I have a problem with a memory leak in .NET Core 3.1 API. The application is hosted in azure app service.
It is clearly visible on a graph that under constant load the memory is very slowly growing. it will only go down after app restart.
I created two memory dumps. One with high memory and one after restart and it's clearly visible that the reason is the app trying to load XmlSerialization.dll multiple times.
Now we have multiple other APIs that are using almost identical code when it comes to serialization and I'm not exactly sure why the problem occurs only in this one. Potentially because maybe this one has a much higher traffic when using the APIs.
I've read some articles about XmlSerializer class having memory issues but those were listed for some of the constructors we are not using. The only instance of using XmlSerializer directly in code was using an XmlSerializer(Type) constructor.
private static async Task<T> ParseResponseContentAsync<T>(HttpResponseMessage response, Accept accept)
{
try
{
using (Stream contentStream = await response.Content.ReadAsStreamAsync())
{
using (StreamReader reader = new StreamReader(contentStream, Encoding.UTF8))
{
switch (accept)
{
case Accept.Xml:
XmlSerializer serializer = new XmlSerializer(typeof(T));
return (T)serializer.Deserialize(reader);
case Accept.Json:
string stringContent = await reader.ReadToEndAsync();
return JsonConvert.DeserializeObject<T>(stringContent);
default:
throw new CustomHttpResponseException(HttpStatusCode.NotImplemented, $"Unsupported Accept type '{accept}'");
}
}
}
}
catch (Exception ex)
{
throw new InvalidOperationException($"Response content could not be deserialized as {accept} to {typeof(T)}", ex);
}
}
But I'm pretty sure this method is not used in this API anyway .
So another potential problematic place could be somewhere in the Controller serialization of responses.
Startup.cs registration:
services
.AddControllers(options =>
{
options.OutputFormatters.Add(new XmlSerializerOutputFormatter(
new XmlWriterSettings
{
OmitXmlDeclaration = false
}));
options.Filters.Add<CustomHttpResponseExceptionFilter>();
})
.AddNewtonsoftJson(options => options.SerializerSettings.Converters.Add(
new StringEnumConverter(typeof(CamelCaseNamingStrategy))))
.AddXmlSerializerFormatters();
Example of an endpoint:
[Produces(MimeType.ApplicationXml, MimeType.TextXml, MimeType.ApplicationJson, MimeType.TextJson)]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status404NotFound)]
[ProducesResponseType(StatusCodes.Status401Unauthorized)]
[HttpGet("EndpointName")]
[Authorize]
public async Task<ActionResult<ResponseDto>> Get([FromModel] InputModel inputModel)
{
//some code
return responseDto;
}
Dto returned from the API:
[XmlRoot(ElementName = "SomeName")]
public class ResponseDto
{
[XmlElement(ElementName = "Result")]
public Result Result { get; set; }
[XmlAttribute(AttributeName = "Status")]
public string Status { get; set; }
[XmlAttribute(AttributeName = "DoneSoFar")]
public int DoneSoFar { get; set; }
[XmlAttribute(AttributeName = "OfTotal")]
public int OfTotal { get; set; }
}
Now I haven't been able to find any documented cases of .AddXmlSerialization causing these kinds of issues and I'm not sure what the solution or a workaround should be. Any help would be greatly appreciated.
EDIT:
I've run some additional tests as #dbc suggested.
Now it seems that we are not even hitting this line new XmlSerializer(typeof(T) in our scenarios since nothing was logged after logger code was added. We do however use default xml serialization for some of our API endpoints. Now one thing I noticed that might be causing this behavior is that the paths in memory dumps logs don't match the files that actually exist in the root folder.
The paths which are visible in memory dumps are *.Progress.Lib.XmlSerializers.dll or *.Domain.Lib.XmlSerializers.dll
Now I wonder if this isn't the issue documented here - link since I can't see those files in wwwroot directory.
If it is I'm not sure if the solution would be to somehow reference the .dlls directly ?
Edit2:
Adding a screen of how memory looks like after deploying cached serializer suggested by #dbc. There is no constant growth but it seems after few hours memory rises and doesn't go down. It is possible that the main problem is resolved but since it takes a lot of time to notice big differences we will monitor this for now. There is nothing showing in large object heap or any big number of memory is not allocated in managed memory. This API however when first deployed runs around 250 mB and after one day now at 850 mB. When we turn off the load test tool the memory didn't really go down too much.
Edit3:
So we looked closer at some historical data and it seems that the last screen is a normal behavior. It never grows beyond a certain point. Not sure why that happens but this is acceptable.

The assemblies that the new XmlSerializer(typeof(T)) constructor are trying to load are Microsoft XML Serializer Generator assemblies a.k.a Sgen.exe assemblies that might have or might not been created at the time the app was built.
But what are Sgen assemblies? In brief, XmlSerializer works by generating code to serialize and deserialize the type passed into the constructor, then compiling that generated code into a DLL and loading it into the application domain to do the actual serialization. This run-time DLL generation can be time-consuming, but as long as you use the XmlSerializer(Type) or XmlSerializer(Type, String) constructors it will only be done once per type T, with the resulting assembly being cached internally in a static dictionary by XmlSerializer.
As you might imagine this can cause the first call to new XmlSerializer(typeof(T)) to be slow, so (in .NET 2 I believe, this is all very old code) Microsoft introduced a tool to generate those run-time serialization DLLs at application build time: SGen.exe. This tool doesn't work for all types (e.g. generics) and was, if I recall correctly, finicky to use, but when it did work it did speed up serializer construction. Once loaded successfully the Sgen assembly is cached in the same cache used for generated assemblies.
And it seems like you have stumbled across a bug in .NET Core 3.1, 5, and 6 related to this:
The base class method OutputFormatter.CanWriteResult(OutputFormatterCanWriteContext context) of XmlSerializerOutputFormatter tests whether a type can be serialized by calling XmlSerializerOutputFormatter.CanWriteType(Type type). This in turn tests to see whether a type is serializable by XmlSerializer by attempting to construct a serializer for the type and returning false if construction failed because any exception was thrown. The serializer is cached if construction was successful, but nothing is cached if construction failed.
the new XmlSerializer(Type) constructor tries to load an Sgen assembly unless an assembly has already been cached for the type by a previous successful call to the constructor.
But if a type is not serializable by XmlSerializer, the constructor will throw an exception and nothing will be cached. Thus successive attempts to construct a serializer for the same non-serializable type will result in multiple calls to load Sgen assemblies.
As you yourself found, .NET Core itself permanently leaks a small amount of IndividualAssemblyLoadContext memory every time assembly load fails: Failed Assembly.Load and Assembly.LoadFile leaks memory #58093.
Putting all this together, enabling XML serialization when some of your DTOs are not serializable (because e.g. they don't have parameterless constructors) can result in ever-growing IndividualAssemblyLoadContext memory use.
So, what are your options for a workaround?
Firstly, issue #58093 was apparently fixed in .NET 7 with pull #68502 so if you upgrade to this version the problem may resolve itself.
Secondly, you could subclass XmlSerializerOutputFormatter to cache returned XmlSerializer instances even when null. This will prevent multiple attempts to create serializers for non-seializable types.
First, subclass XmlSerializerOutputFormatter and override XmlSerializerOutputFormatter.CreateSerializer(Type) as follows:
public class CachedXmlSerializerOutputFormatter : XmlSerializerOutputFormatter
{
// Cache and reuse the serializers returned by base.CreateSerializer(t). When null is returned for a non-serializable type,
// a null serializer will be cached and returned.
static readonly ConcurrentDictionary<Type, XmlSerializer> Serializers = new ConcurrentDictionary<Type, XmlSerializer>();
public CachedXmlSerializerOutputFormatter() : base() { }
public CachedXmlSerializerOutputFormatter(ILoggerFactory loggerFactory) : base(loggerFactory) { }
public CachedXmlSerializerOutputFormatter(XmlWriterSettings writerSettings) : base(writerSettings) { }
public CachedXmlSerializerOutputFormatter(XmlWriterSettings writerSettings, ILoggerFactory loggerFactory) : base(writerSettings, loggerFactory) { }
protected override XmlSerializer CreateSerializer(Type type) { return Serializers.GetOrAdd(type, (t) => base.CreateSerializer(t)); }
}
Then replace use of XmlSerializerOutputFormatter with your subclassed version as follows:
services
.AddControllers(options =>
{
options.OutputFormatters.Add(new CachedXmlSerializerOutputFormatter (
new XmlWriterSettings
{
OmitXmlDeclaration = false
}));
options.Filters.Add<CustomHttpResponseExceptionFilter>();
})
.AddNewtonsoftJson(options => options.SerializerSettings.Converters.Add(
new StringEnumConverter(typeof(CamelCaseNamingStrategy))))
.AddXmlSerializerFormatters();
This should in theory eliminate the repeated failing calls to load Sgen assemblies.
Notes:
If you have enabled XML model binding and some of your input types are not XML-serializable, you may need to similarly subclass XmlSerializerInputFormatter. Its CreateSerializer(Type type)) also fails to cache failed attempts to construct a serializer.
Demo fiddles:
Demo fiddle showing that that multiple calls to XmlSerializerOutputFormatter.CanWriteType() for a non-serializable DTO result in multiple assembly load failures here: demo #1.
Demo fiddle showing that CachedXmlSerializerOutputFormatter fixes this problem here: demo #2.
Demo that multiple calls to XmlSerializerOutputFormatter.CanWriteType() for a serializable DTO do not result in multiple assembly load failures, and hence don't cause growing IndividualAssemblyLoadContext memory use, here: demo #3.

This might not be feasible, but could you offload the XML generation onto Azure API Management?
https://learn.microsoft.com/en-us/azure/api-management/api-management-transformation-policies#ConvertJSONtoXML

Illogical NullReferenceException

So in my normal course of getting some work done, i came across a generic System.NullReferenceException: 'Object reference not set to an instance of an object.'.
So no problem i'm thinking as I start looking for the culprit. After much debugging, I threw my hands up in the air and wanted to give up for the moment... My strategy for bypassing this was to just instantiate an array instead of calling some service for the data.
So i changed:
var data = service.GetData(someId);
to
var data = new DataType[]{};
to get my code compiling and debuggable so i can come back to this.
The kicker: upon doing so, i'm still getting the exception on that line. So JUST an array instantiation is yielding this exception. The exception has zero data to it. No inner exception or stack trace beyond pointing to this line. The class itself is just a poco with no methods (no constructor either) so nothing is happening internally. (see below for the class)
My next thought is that clearly, the code i'm seeing isn't what my debug session is using.
To test this thought I've
Cleaned in VS
manually deleted the bin
manually deleted obj
removed all package directories
restarted my computer
tried it in the new beta of VS
tried it in JetBrains' Rider
still i'm facing this.
Anyone have thoughts on how i can get more details on why this is happening, or is there something else I should be cleaning?
I've also looked at the output of git clean -xdn to see what kind of ephemeral stuff is hanging around that could be removed and nothing of interest there...
no matter what code is actually on the line, if i completely comment out this line and have completely different code there in it's place (as initially described), then that code throws the exception instead. i just had one of my colleagues grab this branch and run it and it was fine for them
3 hours deep so far... no fun
Edit:
as requested, here's a definition:
using System;
using Dapper.Contrib.Extensions;
namespace some_namespace
{
#pragma warning disable 1591
[Table ("CompanyOptions")]
public class CompanyOption
{
[ExplicitKey]
public Guid PKID { get; set; }
public Guid CompanyPKId { get; set; }
public string OptionName { get; set; }
public string OptionValue { get; set; }
public DateTime CreatedOn { get; set; }
public string CreatedBy { get; set; }
public DateTime? UpdatedOn { get; set; }
public string UpdatedBy { get; set; }
}
}

So it seems there were issues with my installation of some microsoft packages.
I found that running Update-Package Microsoft.CodeDom.Providers.DotNetCompilerPlatform -r in the package management console to reinstall solved my problem. In looking at the diff that that yielded, the only meaningful thing is the following target framework bumps in my packages.config:
These package references haven't changed recently and the project has been targeting 4.7.1 for quite some time now so not sure why the sudden problem. Seems something became corrupted in the roslyn layer perhaps?
Going to slowly back away from my computer now.

Table has no (public) columns only on real device

I have the simplest of apps that I thought I would try on my device before I got too engrossed. However, I am getting the strangest error message when I run it on my iPhone (as apposed to the the emulator on my macbook).
Table has no (public) columns .
I am using the SQLite.Net PCL and I have built it from git hub as I had some problems with it not having the platform dlls for IOS otherwise.
Relevant code.
In my models I have this:
public class Setting
{
[PrimaryKey, AutoIncrement]
public long Id { get; set; }
[Indexed]
public string Key { get; set; }
public string Value { get; set; }
}
The code that throws this error message is the simple:
using (SQLiteConnection db = GetCon ()) {
db.CreateTable<Setting> ();
}
but in my opinion the strangest thing is that this code works fine on the emulator but crashes the application on the iphone itself.
If anyone has some ideas that would be great.
EDIT:
This error is thrown on the SQLite.Net-PCL library on this file line 380 but only on the device and not on the emulator.

For others to whom this may concern, I found the answer to my problem. The issue was with the Type not having any properties (the type in question the simple model class). Knowing that to be rubbish I found the following links that gave more information which I will relate in this post in case the links go dead:
Type.GetProperties returning nothing
NOTE: Be careful with assembly linker
If you're building with linker enabled you may need to use the class
somewhere, so it will not be ripped off at compile time. Sometimes,
only instantiating the class in your code is not enough, the linker
may detect that the instance is never used and will remove it anyway.
http://developer.xamarin.com/guides/ios/advanced_topics/linker/
The linking process can be customized via the linker behavior
drop-down in Project Options. To access this double-click on the iOS
project and browse to iOS Build > Linker Options, as illustrated below
(see link for details)
I have for now left it to be unlinked, however, I will try before release to get the linker to ignore these classes. Thanks for all your help.

I found my problem was just a (not that subtle) programming error. I was working with the TypeInfo class and wanted to use the Sqlite Connection method:
CreateTable (Type type);
What I had in my hand was a TypeInfo instance which I needed to convert back to the System.Type. I accidentally without thinking used the GetType() method instead of AsType() method which is obvious when you think about it. The clue I got was in the exception message along with the OP message was does System.Runtime have public properties?
var type = table.TypeInfo.AsType();
// var type = table.TypeInfo.GetType(); *WRONG*
connection.CreateTable(type);

Protbuf-net Compiling to DLL string[] causing corrupt dll

Here is my code:
using ProtoBuf;
[ProtoContract]
[ProtoInclude(500, typeof(SampleClassDrv))]
public class SampleClass
{
[ProtoMember(1)] public int theInt;
[ProtoMember(2)] public string[] items;
public SampleClass(){}
public SampleClass(int c) {this.theInt = c;}
}
[ProtoContract]
public class SampleClassDrv : SampleClass
{
[ProtoMember(1)] public int theOtherInt;
public SampleClassDrv(){}
public SampleClassDrv(int b):base(1){this.theOtherInt=b;}
}
To compile my DLL I run the following code:
RuntimeTypeModel rModel = TypeModel.Create();
rModel.AllowParseableTypes = true;
rModel.AutoAddMissingTypes = true;
rModel.Add(typeof(SampleClass), true);
rModel.Add(typeof(SampleClassDrv), true);
rModel.Compile("MySerializer", "MySerializer.dll");
Finally I should be able to initialize by RuntimeTypeModel from the dll like so:
MySerializer serializer = new MySerializer();
serializer.Serialize(stream, object);
But Unity throws the following exception
Internal compiler error. See the console log for more information.
[...]
Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Assembly-CSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies.
Interestingly enough if I go back and remove the line
[ProtoMember(2)] public string[] items;
It works as expected...
It is also worth noting the RunTimeModel works as expected if used after adding the classes instead of attempting to use dll.
My environment:
Unity3D 4.3.4
Protobuf-net r668
using protbuf-net.dll in Full/unity
I would greatly appreciate if someone could point out the error in my ways.
EDIT
From the suggestion by Flamy I changed form string[] to List
[ProtoMember(2)] public List<string> items;
Sadly the error still persists.
Another Note
Also I decided to use a dll decompiler to see what is going on. I was not able to decompile the dll until the "string[] items" variable was removed.
SOLVED
I think it is related to some issue with compiling the DLL with Unity3D.
When I created the project in Visual Studios with the code I showed above everything seems to be working as expected. Which is a relief as this seems like it would be a huge issue if protobuf could not serialize string[].
I followed the article provided by BCCode to setup the visual studio project and compile the DLLs.
Now all I need to do is create the dll with my large scale project! Fingers Crossed
Thanks everyone for their help!

Are your references correct?
%project dir%\Library\ScriptAssemblies\Assembly-CSharp.dll
Also take a look here:
http://purdyjotut.blogspot.com/2013/10/using-protobuf-in-unity3d.html?m=1

Serilizing array of strings using Binary serilization or a serilization method that uses binary format (protobuff in this case) will always result in so many issues. The reason is String by itself is an array of chars,which doesnt have a defined size, meaning it dont know where one string ends and the next starts... Usually we serialize string array one by one instead of the whole array. so I advice you to to use a property which does that (hopefully protobuff accepts attributes on properties!) or create a list of strings instead (though i haven't tested yet it should work!)

How can I use MSBuild to update version information only when an assembly has changed?

I have a requirement to install multiple web setup projects (using VS2005 and ASP.Net/C#) into the same virtual folder. The projects share some assembly references (the file systems are all structured to use the same 'bin' folder), making deployment of changes to those assemblies problematic since the MS installer will only overwrite assemblies if the currently installed version is older than the one in the MSI.
I'm not suggesting that the pessimistic installation scheme is wrong - only that it creates a problem in the environment I've been given to work with. Since there are a sizable number of common assemblies and a significant number of developers who might change a common assembly but forget to update its version number, trying to manage versioning manually will eventually lead to massive confusion at install time.
On the flip side of this issue, it's also important not to spontaneously update version numbers and replace all common assemblies with every install, since that could (temporarily at least) obscure cases where actual changes were made.
That said, what I'm looking for is a means to update assembly version information (preferably using MSBuild) only in cases where the assembly constituents (code modules, resources etc) has/have actually changed.
I've found a few references that are at least partially pertinent here (AssemblyInfo task on MSDN) and here (looks similar to what I need, but more than two years old and without a clear solution).
My team also uses TFS version control, so an automated solution should probably include a means by which the AssebmlyInfo can be checked out/in during the build.
Any help would be much appreciated.
Thanks in advance.

I cannot answer all your questions, as I don't have experience with TFS.
But I can recommend a better approach to use for updating your AssemblyInfo.cs files than using the AssemblyInfo task. That task appears to just recreate a standard AssemblyInfo file from scratch, and loses any custom portions you may have added.
For that reason, I suggest you look into the FileUpdate task, from the MSBuild Community Tasks project. It can look for specific content in a file and replace it, like this:
<FileUpdate
Files="$(WebDir)\Properties\AssemblyInfo.cs"
Regex="(\d+)\.(\d+)\.(\d+)\.(\d+)"
ReplacementText="$(Major).$(ServicePack).$(Build).$(Revision)"
Condition="'$(Configuration)' == 'Release'"
/>
There are several ways you can control the incrementing of the build number. Because I only want the build number to increment if the build is completely successful, I use a 2-step method:
read a number from a text file (the only thing in the file is the number) and add 1 without changing the file;
as a final step in the build process, if everything succeeded, save the incremented number back to the text file.
There are tasks such as ReadLinesFromFile, that can help you with this, but I found it easiest to write a small custom task:
using System;
using System.IO;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
namespace CredibleCustomBuildTasks
{
public class IncrementTask : Task
{
[Required]
public bool SaveChange { get; set; }
[Required]
public string IncrementFileName { get; set; }
[Output]
public int Increment { get; set; }
public override bool Execute()
{
if (File.Exists(IncrementFileName))
{
string lines = File.ReadAllText(IncrementFileName);
int result;
if(Int32.TryParse(lines, out result))
{
Increment = result + 1;
}
else
{
Log.LogError("Unable to parse integer in '{0}' (contents of {1})");
return false;
}
}
else
{
Increment = 1;
}
if (SaveChange)
{
File.Delete(IncrementFileName);
File.WriteAllText(IncrementFileName, Increment.ToString());
}
return true;
}
}
}
I use this before the FileUpdateTask to get the next build number:
<IncrementTask
IncrementFileName="$(BuildNumberFile)"
SaveChange="false">
<Output TaskParameter="Increment" PropertyName="Build" />
</IncrementTask>
and as my final step (before notifying others) in the build:
<IncrementTask
IncrementFileName="$(BuildNumberFile)"
SaveChange="true"
Condition="'$(Configuration)' == 'Release'" />
Your other question of how to update the version number only when source code has changed is highly dependent on your how your build process interacts with your source control. Normally, checking in source file changes should initiate a Continuous Integration build. That is the one to use to update the relevant version number.

I have written one custome task you can refer the code below. It will create an utility to which you can pass assemblyinfo path Major,minor and build number. you can modify it to get revision number. Since in my case this task was done by developer i used to search it and again replace whole string.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
namespace UpdateVersion
{
class SetVersion
{
static void Main(string[] args)
{
String FilePath = args[0];
String MajVersion=args[1];
String MinVersion = args[2];
String BuildNumber = args[3];
string RevisionNumber = null;
StreamReader Reader = File.OpenText(FilePath);
string contents = Reader.ReadToEnd();
Reader.Close();
MatchCollection match = Regex.Matches(contents, #"\[assembly: AssemblyVersion\("".*""\)\]", RegexOptions.IgnoreCase);
if (match[0].Value != null)
{
string strRevisionNumber = match[0].Value;
RevisionNumber = strRevisionNumber.Substring(strRevisionNumber.LastIndexOf(".") + 1, (strRevisionNumber.LastIndexOf("\"")-1) - strRevisionNumber.LastIndexOf("."));
String replaceWithText = String.Format("[assembly: AssemblyVersion(\"{0}.{1}.{2}.{3}\")]", MajVersion, MinVersion, BuildNumber, RevisionNumber);
string newText = Regex.Replace(contents, #"\[assembly: AssemblyVersion\("".*""\)\]", replaceWithText);
StreamWriter writer = new StreamWriter(FilePath, false);
writer.Write(newText);
writer.Close();
}
else
{
Console.WriteLine("No matching values found");
}
}
}
}

I hate to say this but it seems that you may be doing it wrongly. Is much easier if you do generate the assembly versions on the fly instead of trying to patch them.
Take a look at https://sbarnea.com/articles/easy-windows-build-versioning/
Why I do think you are doing it wrong?
* A build should not modify the version number
* if you build the same changeset twice you should get the same build numbers
* if you put build number inside what microsoft calls build number (proper naming would be PATCH level) you will eventually reach the 65535 limitation.