AccessViolationException when accessing variable solution value - c#

We've been utilizing the OR tools to solve linear optimizations in a real-time, .NET application. That is, solving linear optimizations regularly using different inputs as time progresses.
Recently we ran into an issue that we haven't seen before while running our application on a server for extended periods of time, in which seemingly random attempts to solve the optimizations were causing AccessViolationExceptions. Specifically,
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
at Google.OrTools.LinearSolver.operations_research_linear_solverPINVOKE.Variable_SolutionValue(System.Runtime.InteropServices.HandleRef)
...
I'm trying to find out more specifically where this is happening in the pipeline, but given the output there I believe it is a section in which we are trying to retrieve the individual variable solution values out of the solver after solving the optimization.
We are using a wide variety of constraints over a decent sized number of variables.
Has anyone seen this before?

Reference github issue link
After some testing we found that what appears to have been happening is that the garbage collector was collecting some of the Variables we were using during the P/Invoke, as per this.
Unfortunately, this seems to be a side effect of the way that SWIG creates its .NET wrappers and their IDisposable implementations, using HandleRefs instead of something like SafeHandles, which 'handle' this as per the documentation:
Platform invoke operations automatically increment the reference count of handles encapsulated by a SafeHandle and decrement them upon completion. This ensures that the handle will not be recycled or closed unexpectedly.
More information here.
Without wanting to get into the business of creating our own SWIG typemap or compiling a new version of SWIG, .NET provides a way of keeping objects 'alive' with regard to the Garbage Collector. That is, calling GC.KeepAlive on all of the objects which we will be accessing values from via P/Invoke (in our case the Solver and our Variables) at the end of the optimization procedure, prevents the garbage collector from thinking that they are collectible until the end of the scope of the KeepAlive method without side effects (as per their documentation).
Preliminary testing has shown this to work, though given that it was already intermittently occurring before, we'll be watching for this happening going forward.
Going forward, I think either making a request of SWIG to use SafeHandles is probably the best idea (it has been discussed before and is still an open issue) or changing the typemap to use SafeHandles directly, is likely the best option. I may try investigating the later option myself, but because this fix ended up only adding 3 lines of code (plus a host of comments) to our code base for what seems like a full fix, it's going to be low priority for me. That said, a fix for this would be nice for an upcoming version.

Related

What is the reason that individual assemblies cannot be unloaded in C#

In C# (or maybe in .NET in general) individual assemblies cannot be unloaded from memory.
Unloading can only occur at the AppDomain level.
I am wondering what are there reasons behind this design? Other languages support this feature (C++ i think)
Here is an MSDN blog post listing some reasons why not. The main issue is:
First off, you are running that code in the app domain (duh!). That means there are potentially call sites and call stacks with addresses in them that are expecting to keep working. Have you ever gotten an access violation where your EIP points to 0x???????? That is an example where someone freed up a DLL, the pages got unmapped by the memory system, and then you tried to branch to it. This typically happens in COM when you have a ref counting error and you make an interface method call. We cannot afford to be as lose with managed code. We must guarantee we know all of the code you are executing and that it is type safe and verifiable. That means explicit tracking of anything that could be using that code, including GC objects and COM interop wrappers. This tracking is handled today around an app domain boundary. Tracking it at the assembly level becomes quite expensive.
I'll summarise this in higher-level language:
Basically, things that go wrong if you simply delete executable code go wrong on the unmanaged level. You would have compiled code that points to other compiled code that is no longer there, so your code would jump into an area that is invalid, and possibly contains arbitrary data.
This is unacceptable in managed code, because things are meant to be safe and have some guarantees around them. One of these guarantees is that your code can't execute arbitrary sections of memory.
To handle this issue properly you'd have to track many more things more closely, and this would be a large overhead. The alternative is to only track these things at appdomain boundaries, which is what is done.

Unknown function CoUninitializeE shows up during profiling

While using SlimTune to profile a C# application, I find that when profiling native functions is enabled there are lots of entries for a function called "CoUninitializeE." CoUninitialize seems to be related to COM objects, however I'm not directly using any Com objects, and Google has no information about the version ending with an E.
Does anyone have knowledge of what this function is/how to reduce the amount of time spent on it? (For instance, is it related to memory management, so that reducing memory allocations or deallocations would help?)
Edit
It appears the function's name is actually "CoUninitializeEx" and that SlimTune is just chopping off a letter for some reason. I still would appreciate knowledge of what leads to this function being called.
CoInitalizeEx() and CoUninitialize() are pretty core in Windows programming. They respectively initialize and shutdown COM on a thread. The CLR calls these functions automatically before and after a Thread runs. It is pretty hard to avoid using COM in a .NET program, it is the basic extensibility model for native Windows code. Quite invisible, thanks to the many wrapper classes in the .NET framework that hides the plumbing.
The generic diagnostic is that you use a lot of threads. Yes, expensive. The thread pool is a workaround.

When to use ReleaseComObject vs FinalReleaseComObject? [duplicate]

This question already has an answer here:
Why use FinalReleaseComObject instead of ReleaseComObject?
(1 answer)
Closed 9 years ago.
When should I use Marshal.FinalReleaseComObject vs Marshal.ReleaseComObject?
Is there any danger in using Marshal.FinalReleaseComObject?
There's some virtue in FinalReleaseComObject, it will crash your program quicker. "COM object that has been separated from its underlying RCW cannot be used" is the CLR telling you that you taking care of COM reference counts yourself instead of leaving it up the CLR was a mistake. Your mileage may vary, you cannot really trust to get it right when it works on your dev machine. Make sure you implement good error reporting when you deploy the code to your customer's machine.
The virtue is that there's only one place in your code where you got it wrong, the FinalReleaseComObject call. It gets much fuzzier when you use ReleaseComObject. Because that will go undetected for a while, crashing your program when the CLR calls the final IUnknown::Release(), the one that destroys the object. Very far removed from an incorrect ReleaseComObject call. But that's the doomsday scenario, the more likely outcome is that the call just doesn't make any difference because you missed the hard ones. Like mumble["foo"], an indexer reference that is so very hard to see being used.
Well, my advice is obvious: don't do this. You are competing with a machine that never gets it wrong. It is merely a bit slow at doing so. A very good "report from real life" is available here. The "silent assassin" section is most relevant.
If it is absolutely essential to get a COM server to exit instantly then let the machine take care of getting all the reference counts to 0. You do so with GC.Collect(). But do keep in mind that you have to place that call correctly if you want this to also work when you debug. It won't work in the same method that uses the COM objects, explained in this answer. Put it in the calling method instead.

What strategies and tools are useful for finding memory leaks in .NET?

I wrote C++ for 10 years. I encountered memory problems, but they could be fixed with a reasonable amount of effort.
For the last couple of years I've been writing C#. I find I still get lots of memory problems. They're difficult to diagnose and fix due to the non-determinancy, and because the C# philosophy is that you shouldn't have to worry about such things when you very definitely do.
One particular problem I find is that I have to explicitly dispose and cleanup everything in code. If I don't, then the memory profilers don't really help because there is so much chaff floating about you can't find a leak within all the data they're trying to show you. I wonder if I've got the wrong idea, or if the tool I've got isn't the best.
What kind of strategies and tools are useful for tackling memory leaks in .NET?
I use Scitech's MemProfiler when I suspect a memory leak.
So far, I have found it to be very reliable and powerful. It has saved my bacon on at least one occasion.
The GC works very well in .NET IMO, but just like any other language or platform, if you write bad code, bad things happen.
Just for the forgetting-to-dispose problem, try the solution described in this blog post. Here's the essence:
public void Dispose ()
{
// Dispose logic here ...
// It's a bad error if someone forgets to call Dispose,
// so in Debug builds, we put a finalizer in to detect
// the error. If Dispose is called, we suppress the
// finalizer.
#if DEBUG
GC.SuppressFinalize(this);
#endif
}
#if DEBUG
~TimedLock()
{
// If this finalizer runs, someone somewhere failed to
// call Dispose, which means we've failed to leave
// a monitor!
System.Diagnostics.Debug.Fail("Undisposed lock");
}
#endif
We've used Ants Profiler Pro by Red Gate software in our project. It works really well for all .NET language-based applications.
We found that the .NET Garbage Collector is very "safe" in its cleaning up of in-memory objects (as it should be). It would keep objects around just because we might be using it sometime in the future. This meant we needed to be more careful about the number of objects that we inflated in memory. In the end, we converted all of our data objects over to an "inflate on-demand" (just before a field is requested) in order to reduce memory overhead and increase performance.
EDIT: Here's a further explanation of what I mean by "inflate on demand." In our object model of our database we use Properties of a parent object to expose the child object(s). For example if we had some record that referenced some other "detail" or "lookup" record on a one-to-one basis we would structure it like this:
class ParentObject
Private mRelatedObject as New CRelatedObject
public Readonly property RelatedObject() as CRelatedObject
get
mRelatedObject.getWithID(RelatedObjectID)
return mRelatedObject
end get
end property
End class
We found that the above system created some real memory and performance problems when there were a lot of records in memory. So we switched over to a system where objects were inflated only when they were requested, and database calls were done only when necessary:
class ParentObject
Private mRelatedObject as CRelatedObject
Public ReadOnly Property RelatedObject() as CRelatedObject
Get
If mRelatedObject is Nothing
mRelatedObject = New CRelatedObject
End If
If mRelatedObject.isEmptyObject
mRelatedObject.getWithID(RelatedObjectID)
End If
return mRelatedObject
end get
end Property
end class
This turned out to be much more efficient because objects were kept out of memory until they were needed (the Get method was accessed). It provided a very large performance boost in limiting database hits and a huge gain on memory space.
You still need to worry about memory when you are writing managed code unless your application is trivial. I will suggest two things: first, read CLR via C# because it will help you understand memory management in .NET. Second, learn to use a tool like CLRProfiler (Microsoft). This can give you an idea of what is causing your memory leak (e.g. you can take a look at your large object heap fragmentation)
Are you using unmanaged code? If you are not using unmanaged code, according to Microsoft, memory leaks in the traditional sense are not possible.
Memory used by an application may not be released however, so an application's memory allocation may grow throughout the life of the application.
From How to identify memory leaks in the common language runtime at Microsoft.com
A memory leak can occur in a .NET
Framework application when you use
unmanaged code as part of the
application. This unmanaged code can
leak memory, and the .NET Framework
runtime cannot address that problem.
Additionally, a project may only
appear to have a memory leak. This
condition can occur if many large
objects (such as DataTable objects)
are declared and then added to a
collection (such as a DataSet). The
resources that these objects own may
never be released, and the resources
are left alive for the whole run of
the program. This appears to be a
leak, but actually it is just a
symptom of the way that memory is
being allocated in the program.
For dealing with this type of issue, you can implement IDisposable. If you want to see some of the strategies for dealing with memory management, I would suggest searching for IDisposable, XNA, memory management as game developers need to have more predictable garbage collection and so must force the GC to do its thing.
One common mistake is to not remove event handlers that subscribe to an object. An event handler subscription will prevent an object from being recycled. Also, take a look at the using statement which allows you to create a limited scope for a resource's lifetime.
This blog has some really wonderful walkthroughs using windbg and other tools to track down memory leaks of all types. Excellent reading to develop your skills.
I just had a memory leak in a windows service, that I fixed.
First, I tried MemProfiler. I found it really hard to use and not at all user friendly.
Then, I used JustTrace which is easier to use and gives you more details about the objects that are not disposed correctly.
It allowed me to solve the memory leak really easily.
If the leaks you are observing are due to a runaway cache implementation, this is a scenario where you might want to consider the use of WeakReference. This could help to ensure that memory is released when necessary.
However, IMHO it would be better to consider a bespoke solution - only you really know how long you need to keep the objects around, so designing appropriate housekeeping code for your situation is usually the best approach.
I prefer dotmemory from Jetbrains
Big guns - Debugging Tools for Windows
This is an amazing collection of tools. You can analyze both managed and unmanaged heaps with it and you can do it offline. This was very handy for debugging one of our ASP.NET applications that kept recycling due to memory overuse. I only had to create a full memory dump of living process running on production server, all analysis was done offline in WinDbg. (It turned out some developer was overusing in-memory Session storage.)
"If broken it is..." blog has very useful articles on the subject.
After one of my fixes for managed application I had the same thing, like how to verify that my application will not have the same memory leak after my next change, so I've wrote something like Object Release Verification framework, please take a look on the NuGet package ObjectReleaseVerification. You can find a sample here https://github.com/outcoldman/OutcoldSolutions-ObjectReleaseVerification-Sample, and information about this sample http://outcoldman.com/en/blog/show/322
The best thing to keep in mind is to keep track of the references to your objects. It is very easy to end up with hanging references to objects that you don't care about anymore.
If you are not going to use something anymore, get rid of it.
Get used to using a cache provider with sliding expirations, so that if something isn't referenced for a desired time window it is dereferenced and cleaned up. But if it is being accessed a lot it will say in memory.
One of the best tools is using the Debugging Tools for Windows, and taking a memory dump of the process using adplus, then use windbg and the sos plugin to analyze the process memory, threads, and call stacks.
You can use this method for identifying problems on servers too, after installing the tools, share the directory, then connect to the share from the server using (net use) and either take a crash or hang dump of the process.
Then analyze offline.
From Visual Studio 2015 consider to use out of the box Memory Usage diagnostic tool to collect and analyze memory usage data.
The Memory Usage tool lets you take one or more snapshots of the managed and native memory heap to help understand the memory usage impact of object types.
one of the best tools I used its DotMemory.you can use this tool as an extension in VS.after run your app you can analyze every part of memory(by Object, NameSpace, etc) that your app use and take some snapshot of that, Compare it with other SnapShots.
DotMemory

GDI+ System.Drawing.Bitmap gives error Parameter is not valid intermittently

I have some C# code in an ASP.Net application that does this:
Bitmap bmp = new Bitmap(1184, 1900);
And occasionally it throws an exception "Parameter is not valid". Now i've been googling around and apparently GDI+ is infamous for throwing random exceptions, and lots of people have had this problem, but nobody has a solution to it! I've checked the system and it has plenty of both RAM and swap space.
Now in the past if i do an 'iisreset' then the problem goes away, but it comes back in a few days. But i'm not convinced i've caused a memory leak, because as i say above there is plenty of ram+swap free.
Anyone have any solutions?
Stop using GDI+ and start using the WPF Imaging classes (.NET 3.0). These are a major cleanup of the GDI+ classes and tuned for performance. Additionally, it sets up a "bitmap chain" that allows you to easily perform multiple actions on the bitmap in an efficient manner.
Find more by reading about BitmapSource
Here's an example of starting with a blank bitmap just waiting to receive some pixels:
using System.Windows.Media.Imaging;
class Program {
public static void Main(string[] args) {
var bmp = new WriteableBitmap(1184, 1900, 96.0, 96.0, PixelFormat.Bgr32, null);
}
}
For anyone who's interested, the solution i'm going to use is the Mono.Cairo libraries from the mono C# distribution instead of using system.drawing. If i simply drag the mono.cairo.dll, libcairo-2.dll, libpng13.dll and zlib1.dll files from the windows version of mono into the same folder as my executable, then i can develop in windows using visual studio 2005 and it all works nicely.
Update - i've done the above, and stress tested the application and it all seems to run smoothly now, and uses up to 200mb less ram to boot. Very happy.
Everything I've seen to date in my context is related to memory leaks / handle leaks. I recommend you get a fresh pair of eyes to investigate your code.
What actually happens is that the image is disposed at a random point in the future, even if you've created it on the previous line of code. This may be because of a memory/handle leak (cleaning some out of my code appears to improve but not completely resolve this problem).
Because this error happens after the application has been in use for a while, sometimes using lots of memory, sometimes not, I feel the garbage collector doesn't obey the rules because of some special tweaks related to services and that is why Microsoft washes their hands of this problem.
http://blog.lavablast.com/post/2007/11/The-Mysterious-Parameter-Is-Not-Valid-Exception.aspx
You not only need enough memory, it needs to be contiguous. Over time memory becomes fragmented and it becomes harder to find big blocks. There aren't a lot of good solutions to this, aside from building up images from smaller bitmaps.
new Bitmap(x, y) pretty much just needs to allocate memory -- assuming that your program isn't corrupted in some way (is there any unsafe code that could corrupt the heap), then I would start with this allocation failing. Needing a contiguous block is how a seemingly small allocation could fail. Fragmentation of the heap is something that is usually solved with a custom allocator -- I don't think this is a good idea in IIS (or possible).
To see what error you get on out of memory, try just allocation a gigantic Bitmap as a test -- see what error it throws.
One strategy I've seen is to pre-allocate some large blocks of memory (in your case Bitmaps) and treat them as a pool (get and return them to the pool). If you only need them for a short period of time, you might be able to get away with just keeping a few in memory and sharing them.
I just got a reply from microsoft support. Apparently if you look here:
http://msdn.microsoft.com/en-us/library/system.drawing.aspx
You can see it says "Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service. Attempting to use these classes from within one of these application types may produce unexpected problems, such as diminished service performance and run-time exceptions."
So they're basically washing their hands of the issue.
It appears that they're admitting that this section of the .Net framework is unreliable. I'm a bit disappointed.
Next up - can anyone recommend a similar library to open a gif file, superimpose some text, and save it again?
Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service
For a supported alternative, see Windows Imaging Components (msdn), a native library which ironically System.Drawing is based on.

Categories