I have noticed inconsistent behavior from the GC when compiling console applications under both 32-bit and 64-bit in .Net 4.0 using VS 2013.
Consider the following code:
class Test
{
public static bool finalized = false;
~Test()
{
finalized = true;
}
}
and in Main() ...
var t = new Test();
t = null;
GC.Collect();
GC.WaitForPendingFinalizers();
if (!Test.finalized)
throw new Exception("oops!");
When running in 64-bit (debug) mode this works every time without fail; however, running in 32-bit mode I cannot force this object to get collected (even if I create more objects and wait a period of time, which I have tried).
Does anyone have any ideas as to why this is? It's causing me trouble when trying to debug some code that must deal with releasing unmanaged proxy data for the 32-bit version of the assemblies. There's a lot of objects in 32-bit mode that just sit there until a long time later (no so in 64-bit).
I'm trying to debug something in 32-bit mode, but the finalizers are not getting called (at least, not by force). The objects just sit there and never get collected (I can see all the weak references still having a value). In 64-bit mode, all the weak references are cleared as expected, and all finalizers get called.
Note: Though the code above is on a very small scale, I have noticed in 32-bit mode many more objects stuck in the GC until more objects get created later (even when "Collect" and "WaitForPendingFinalizers" is called). This is never the case in 64-bit mode. I have one user wondering why so many objects where not getting collected, which caused me to investigate, to which I found out that everything seems to work better in 64-bit mode than 32. Just trying to understand why.
Edit: Here is better code to show the differences:
class Program
{
class Test
{
public static bool Finalized = false;
public int ID;
public Test(int id)
{
ID = id;
}
~Test()
{ // <= Put breakpoint here
Finalized = true;
Console.WriteLine("Test " + ID + " finalized.");
}
}
static List<WeakReference> WeakReferences = new List<WeakReference>();
public static bool IsNet45OrNewer()
{
// Class "ReflectionContext" exists from .NET 4.5 onwards.
return Type.GetType("System.Reflection.ReflectionContext", false) != null;
}
static void Main(string[] args)
{
Console.WriteLine("Is 4.5 or newer: " + IsNet45OrNewer());
Console.WriteLine("IntPtr: " + IntPtr.Size + Environment.NewLine);
Console.WriteLine("Creating the objects ...");
for (var i = 0; i < 10; ++i)
WeakReferences.Add(new WeakReference(new Test(i)));
Console.WriteLine("Triggering collect ...");
GC.Collect();
Console.WriteLine("Triggering finalizers ..." + Environment.NewLine);
GC.WaitForPendingFinalizers();
Console.WriteLine(Environment.NewLine + "Checking for objects still not finalized ...");
bool ok = true;
for (var i = 0; i < 10; ++i)
if (WeakReferences[i].IsAlive)
{
var test = (Test)WeakReferences[i].Target;
if (test != null)
Console.WriteLine("Weak references still exist for Test " + test.ID + ".");
ok = false;
}
if (ok)
Console.WriteLine("All Test objects successfully collected and finalized.");
Console.WriteLine(Environment.NewLine + "Creating more objects ...");
for (var i = 0; i < 10; ++i)
WeakReferences.Add(new WeakReference(new Test(i)));
Console.WriteLine("Triggering collect ...");
GC.Collect();
Console.WriteLine("Triggering finalizers ..." + Environment.NewLine);
GC.WaitForPendingFinalizers();
Console.WriteLine(Environment.NewLine + "Checking for objects still not finalized ...");
ok = true;
for (var i = 0; i < 10; ++i)
if (WeakReferences[i].IsAlive)
{
var test = (Test)WeakReferences[i].Target;
if (test != null)
Console.WriteLine("Weak references still exist for Test " + test.ID + ".");
ok = false;
}
if (ok)
Console.WriteLine("All Test objects successfully collected and finalized.");
Console.WriteLine(Environment.NewLine + "Done.");
Console.ReadKey();
}
}
It works in 64-bit, but not in 32-bit. On my system, "Test #9" never gets collected (a weak reference remains), even after creating more objects, and trying a second time.
FYI: The main reason for asking the question is because I have a \gctest option in my console to test garbage collection between V8.Net and the underlying V8 engine on the native side. It works in 64-bit, but not 32.
No repro on VS2013 + .NET 4.5 on x86 and x64:
class Program
{
class Test
{
public readonly int I;
public Test(int i)
{
I = i;
}
~Test()
{
Console.WriteLine("Finalizer for " + I);
}
}
static void Tester()
{
var t = new Test(1);
}
public static bool IsNet45OrNewer()
{
// Class "ReflectionContext" exists from .NET 4.5 onwards.
return Type.GetType("System.Reflection.ReflectionContext", false) != null;
}
static void Main(string[] args)
{
Console.WriteLine("Is 4.5 or newer: " + IsNet45OrNewer());
Console.WriteLine("IntPtr: " + IntPtr.Size);
var t = new Test(2);
t = null;
new Test(3);
Tester();
Console.WriteLine("Pre GC");
GC.Collect();
GC.WaitForPendingFinalizers();
Console.WriteLine("Post GC");
Console.ReadKey();
}
}
Note that (as noticed by #Sriram Sakthivel), there could be "ghost" copies of your variable around, that have their lifespan extended until the end of the method (when you compile the code in Debug mode or you have a debugger attached (F5 instead of Ctrl+F5), the lifespan of your variables is extended until the end of the method to ease debugging)
For example object initialization (when you do something like new Foo { I = 5 }) creates an hidden local variable.
Related
Still pretty new to threads so I'm sure it is one of those little gotchas and a repeat question, but I have been unable to find the answer browsing the threads.
I have a port scanner app in C#.
I'm using threadpools to spin up a new TcpClient for each port and probe if it's open.
After suffering through the concepts of closures and thread synchronization, I am having an issue where when multiple threads try to save their results to different indexes in the Orchestrator.hosts (List).
I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write, however I'm finding that on some updates, multiple entries are getting the same update.
IE, Thread #1 supposed to update Hosts[0].Ports[0].Status to "Open",
What happens:
Thread #1 updates multiple host with the port result despite passing a specific index for Hosts.
Hosts[0].Ports[0].Status to "Open",
Hosts[1].Ports[0].Status to "Open",
Hosts[2].Ports[0].Status to "Open",
Not sure where my problem is. The Static method I'm calling to perform a probe of a given port
public static void ScanTCPPorts()
{
// Create a list of portsToScan objects to send to thread workers
//List<ScanPortRequest> portsToScan = new List<ScanPortRequest>();
using (ManualResetEvent resetEvent = new ManualResetEvent(false))
{
int toProcess = 0;
for (var i = 0; i < hostCount; i++) // Starting at Begining
{
int currentHostId = i;
// To hold our current hosts ID (Assign outside of threaded function to avoid race-condition)
if (hosts[i].IsAlive || scanDefinition.isForced())
{
int portCount = hosts[i].Ports.Count;
for (int p = 0; p < portCount; p++)
{
// Thread-safe Increment our workQueue counter
Interlocked.Increment(ref toProcess);
int currentPortPosition = p;
// We need to send the arrayIndex in to the thread function
PortScanRequestResponse portRequestResponse = new PortScanRequestResponse(hosts[currentHostId], currentHostId, hosts[currentHostId].Ports[currentPortPosition], currentPortPosition);
ThreadPool.QueueUserWorkItem(
new WaitCallback(threadedRequestResponseInstance => {
PortScanRequestResponse portToScan = threadedRequestResponseInstance as PortScanRequestResponse;
PortScanRequestResponse threadResult = PortScanner.scanTCPPort(portToScan);
// Lock so Thread-safe update to result
lock (Orchestrator.hosts[portToScan.hostResultIndex])
{
if (threadResult.port.status == PortStatus.Open)
{
// Update result
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Open;
//Logger.Log(hosts[currentHostId].IPAddress + " " + hosts[currentHostId].Ports[currentPortPosition].type + " " + hosts[currentHostId].Ports[currentPortPosition].portNumber + " is open");
}
else
{
Orchestrator.hosts[portToScan.hostResultIndex].Ports[portToScan.portResultIndex].status = PortStatus.Closed;
}
// Check if this was the last scan for the given host
if (Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete != true)
{
if (Orchestrator.hosts[portToScan.hostResultIndex].isCompleted())
{
Orchestrator.hosts[portToScan.hostResultIndex].PortScanComplete = true;
// Logger.Log(hosts[currentHostId].IPAddress + " has completed a port scan");
Orchestrator.hosts[portToScan.hostResultIndex].PrintPortSummery();
}
}
}
// Safely decrement the counter
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}), portRequestResponse); // Pass in our Port to scan
}
}
}
resetEvent.WaitOne();
}
}
Here is the worker process in a separate public static class.
public static PortScanRequestResponse scanTCPPort(object portScanRequest) {
PortScanRequestResponse portScanResponse = portScanRequest as PortScanRequestResponse;
HostDefinition host = portScanResponse.host;
ScanPort port = portScanResponse.port;
try
{
using (TcpClient threadedClient = new TcpClient())
{
try
{
IAsyncResult result = threadedClient.BeginConnect(host.IPAddress, port.portNumber, null, null);
Boolean success = result.AsyncWaitHandle.WaitOne(Orchestrator.scanDefinition.GetPortTimeout(), false);
if (threadedClient.Client != null)
{
if (success)
{
threadedClient.EndConnect(result);
threadedClient.Close();
portScanResponse.port.status = PortStatus.Open;
return portScanResponse;
}
}
} catch { }
}
}
catch
{ }
portScanResponse.port.status = PortStatus.Closed;
return portScanResponse;
}
Originally I was pulling the host index from a free variable, thinking this was the problem moved it to inside the delegate.
I tried locking the Hosts object everywhere there was a write.
I have tried different thread sync techniques (CountdownEvent and ManualResetEvent).
I think there is just some fundamental threading principal I have not been introduced to yet, or I have made a very simple logic mistake.
I have multiple threads trying to update a single List results object. My understanding is this is fine as long as I lock the object on write.
I haven't studied your code, but the above statement alone is incorrect. When a List<T>, or any other non-thread-safe object , is used in a multithreaded environment, all interactions with the object must be synchronized. Only one thread at a time should be allowed to interact with the object. Both writes and reads must be enclosed in lock statements, using the same locker object. Even reading the Count must be synchronized. Otherwise the usage is erroneous, and the behavior of the program is undefined.
I was hyper-focused on it being a thread issue because this was my firsts threaded project. Turned out to be that I didn't realize copies of a List<> objects are references to their original object (reference type). I assumed my threads were accessing my save structure in an unpredictable way, but my arrays of ports were all referencing the same object.
This was a "reference type" vs "value type" issue on my List<> of ports.
My problem is this:
If Autodesk Inventor is not running, my app (console app) creates a new instance with Activator.CreateInstance(InventorType); and uses it as a COM object. When my app does not quit Inventor but leaves it open and the user later quits it by hand there is still a process inventor.exe running in TaskManager which can only be killed in TaskManager.
Curiously the problem only arises when these two things are combined. Whenever my app quits Inventor with InventorApp.Quit(); it is closed properly and there is no process left open.
If my app starts Inventor with Process.Start(..); or the user starts Inventor before starting the app and then my app grabs Inventor with Marshal.GetActiveObject(ProgId); there is no problem no matter if the app or the user quits Inventor.
If my app starts Inventor with Activator.CreateInstance(InventorType); then leaves Inventor open, the app is closed and then restarted, it grabs Inventor with Marshal.GetActiveObject(..); and then quits Inventor via InventorApp.Quit(); there is no problem.
So, the problem with the left open process only arises in this specific combination:
start Inventor via Activator.CreateInstance(InventorType);
the user quits Inventor by hand
The left open process is not in the Running Object Table anymore so it can't be handled as a COM object anymore and it has no visible UI, which means it can only be killed in TaskManager.
Using the 'bad combination' as described I even tried to call GC.WaitForPendingFinalizers(); GC.Collect(); several times (I know this is bad but I am just trying everything) in different combinations and before and/or after Marshal.ReleaseComObject(invApp); Marshal.FinalReleaseComObject(invApp);. I even tried a minimal app which literally does nothing else. See below for the code.
So, what is Activator.CreateInstance(InventorType); doing that is causing this? Is there any way to prevent this? Or is this a problem specific to Inventor?
Minimal app example:
Inventor.Application invApp = null;
string ProgId = "Inventor.Application";
try
{
invApp = (Inventor.Application)Marshal.GetActiveObject(ProgId);
}
catch (Exception e1)
{
try
{
Type InventorType = Type.GetTypeFromProgID(ProgId);
invApp = (Inventor.Application)Activator.CreateInstance(InventorType);
}
catch (Exception e2)
{
Console.WriteLine(e1);
Console.WriteLine(e2);
}
}
invApp = invApp as Inventor.Application;
invApp.Visible = true;
Console.Write("Quit Inventor? (y/n) ");
string quit = Console.ReadLine();
if (quit == "y")
{
invApp.Quit();
}
// desperately trying to release the COM object ...
GC.WaitForPendingFinalizers();
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
if (invApp != null)
{
Marshal.ReleaseComObject(invApp);
Marshal.FinalReleaseComObject(invApp);
}
GC.WaitForPendingFinalizers();
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
if (invApp != null)
{
Marshal.ReleaseComObject(invApp);
Marshal.FinalReleaseComObject(invApp);
}
GC.WaitForPendingFinalizers();
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
invApp = null;
This is not specific for Inventor but for any COM object (Excel for example).
Usually I don't use this COM communication for production, because there are many vulnerabilities and some performance issues. I recommend you to use another workflow when possible.
But to your question. You can't release this COM object as you try. I recommend you to wrap Inventor.Application to some IDisposable object and quit them in dispose method, when you create your own instance.
static void Main(string[] args)
{
InventorTest();
//Waiting for dispose message
//Console.ReadKey();
}
private static void InventorTest()
{
using (var invProvider = new InventorDisposableProvider())
{
var invApp = invProvider.InventorApp;
invApp.Visible = true;
Console.Write("Quit Inventor? (y/n) ");
string quit = Console.ReadLine();
if (quit == "y")
{
invApp.Quit();
}
}
}
class InventorDisposableProvider : IDisposable
{
private Application invApp;
private bool startedByMe = false;
/// <summary>
/// Gets running or start new instance of Inventor
/// </summary>
public Application InventorApp
{
get
{
if (invApp == null) GetInventorApp();
return invApp;
}
}
/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
public void Dispose()
{
if (startedByMe && invApp != null)
{
invApp .Quit();
Console.WriteLine("Quit");
}
}
private void GetInventorApp()
{
string ProgId = "Inventor.Application";
try
{
invApp = (Inventor.Application)Marshal.GetActiveObject(ProgId);
startedByMe = false;
}
catch (Exception e1)
{
try
{
Type InventorType = Type.GetTypeFromProgID(ProgId);
invApp = (Inventor.Application)Activator.CreateInstance(InventorType);
startedByMe = true;
}
catch (Exception e2)
{
Console.WriteLine(e1);
Console.WriteLine(e2);
}
}
}
}
I don't know if this is the best solution but it is a good start point.
I found only one issue. When user quit the console application by cross. In this case you can see this article how to solve this case.
capture-console-exit-c-sharp
When I run Execute on a ScriptService run request, it takes some memory and fails to release it. Testing this by running on monodevelop on a Raspberry Pi shows the memory rising at an alarming rate, and will eventually crash the program. The GC.Collect was an attempt at re-claiming this memory. Is there any insight into what I am doing wrong?
public MainWindow() : base(Gtk.WindowType.Toplevel)
{
Build();
while (true)
{
getDashRow();
Thread.Sleep(1000);
Console.WriteLine("Total available memory before collection: {0:N0}", System.GC.GetTotalMemory(false));
System.GC.Collect();
Console.WriteLine("Total available memory collection: {0:N0}", System.GC.GetTotalMemory(true));
}
}
private int getDashRow()
{
ScriptsResource.RunRequest runreq;
DriveService driveservice;
ExecutionRequest exrequest;
Console.WriteLine("getDashRow");
int retval = 0;
exrequest = new ExecutionRequest();
exrequest.Function = "getMacRow";
IList<object> parameters = new List<object>();
parameters.Add(spreadsheetname);
exrequest.Parameters = parameters;
exrequest.DevMode = false;
try
{
// run a Google Apps Script function on the online sheet to find number of rows (more efficient)
runreq = scriptservice.Scripts.Run(exrequest, dashscriptid);
// following line consumes the memory
Operation op = runreq.Execute();
retval = Convert.ToInt16(op.Response["result"]);
parameters = null;
exrequest = null;
op = null;
}
catch (Exception ex)
{
Console.WriteLine("getDashRow: " + ex.Message);
}
return retval;
}
I solved this, installing Mono Preview v6.0.0.277 has resolved this problem, memory is now correctly freed up without manually calling GC.Collect().
Related issue which led to the solution
I am building an application that processes an incoming image from file or buffer and outputs results in the form of an array or doubles of fixed size. The application needs to be relatively quick. I ran into a problem with cycle time. I started recording cycle time while processing one image and it went from the minimum at 65ms and gradually started increasing to all the way 500ms which is way too slow. Sure enough, I checked on the memory usage and it was steadily increasing as well.
I'm running GC after every cycle and dumping unused variable as ofter as possible. I don't create new objects within the processing loop. Image processing is done on its own thread so that all the resources get dumped. It seems the majority of the cycle time increase happens when I'm pulling the image from file. What could I be missing?
here's the rough code, the full thing is pretty large. Main Function
private void button4_Click(object sender, EventArgs e)
{
Stopwatch sw = new Stopwatch();
sw.Start();
cogDisplay1.Image = null;
ImageFromFile.Operator.Open(Properties.Settings.Default.ImageFile, CogImageFileModeConstants.Read);
ImageFromFile.Run();
cogDisplay1.Image = ImageFromFile.OutputImage;
cogDisplay1.AutoFit = true;
Thread t = new Thread(Vision);
t.Start();
textBox3.Clear();
sw.Stop();
textBox3.AppendText(sw.ElapsedMilliseconds.ToString() + Environment.NewLine + "TS: " + t.ThreadState.ToString());
textBox3.AppendText("GC" + GC.GetTotalMemory(true).ToString());
GC.Collect(GC.MaxGeneration , GCCollectionMode.Forced, false);
}
Image Processing
public void Vision()
{
Stopwatch sw = new Stopwatch();
sw.Start();
try
{
AlignmentParams.ApproximateNumberToFind = 1;
AlignmentParams.SearchRegionMode = 0;
AlignmentResult = AlignmentPat.Execute(cogDisplay1.Image as CogImage8Grey, null , AlignmentParams);
Fixture.InputImage = cogDisplay1.Image;
Fixture.RunParams.UnfixturedFromFixturedTransform = AlignmentResult[0].GetPose();
Fixture.Run();
AlignmentResult = null;
#region FindLineTools
#endregion
#region FindEllipse
#endregion
sw.Stop();
SetText("V" + sw.ElapsedMilliseconds.ToString() + Environment.NewLine);
}
catch (Exception err)
{
SetText(Environment.NewLine + "***********Error***********" + Environment.NewLine);
SetText(err.ToString() + Environment.NewLine);
SetText("***************************" + Environment.NewLine);
}
}
First, I would recommend to post a cleaned-up code for better readability (remove all commented-off stuff). Second, focus only on essential part, namely: your problem is memory overuse/leak due to heavy image processing (correct if wrong). Therefore, in your thread named Vision de-reference the image objects and set them to null immediately after processing completion (as mentioned above, GC is not a big help in your case). The concept briefly demonstrated by following code snippet:
public void Vision()
{
Stopwatch sw = new Stopwatch();
sw.Start();
try
{
AlignmentParams.ApproximateNumberToFind = 1;
AlignmentParams.SearchRegionMode = 0;
AlignmentResult = AlignmentPat.Execute(cogDisplay1.Image as CogImage8Grey, null , AlignmentParams);
Fixture.InputImage = cogDisplay1.Image;
Fixture.RunParams.UnfixturedFromFixturedTransform = AlignmentResult[0].GetPose();
Fixture.Run();
AlignmentResult = null;
// your coding stuff
sw.Stop();
SetText("V" + sw.ElapsedMilliseconds.ToString() + Environment.NewLine);
}
catch (Exception err)
{
SetText(Environment.NewLine + "***********Error***********" + Environment.NewLine);
SetText(err.ToString() + Environment.NewLine);
SetText("***************************" + Environment.NewLine);
}
finally{Fixture.InputImage=null}
Don't call GC.Collect. Let the VM decide when it is out of memory and must do the collection. You typically are not in a good position to decide the best time for a GC.Collect unless your running some other heartbeat or idle watching threads.
Secondly, ensure that whatever resources you're receiving from the method calls are being Disposed. Setting variables to NULL do NOT do this, you should be explicitly calling Dispose or within a Using block:
using(SomeResource resource = Class.GiveMeResource("image.png"))
{
int width = resource.Width;
int height = resource.Height;
Console.Write("that image has {0} pixels", width*height);
} //implicitly calls IDisposable.Dispose() here.
You also need to do some memory and call profiling to detect where, if any, leaks exist.
I posted about this a little while ago, but I resolved the other issue and ran into one more. I am about to deploy this program to 28 hosting machines so I want to make sure this is working before I do so.
I wrote a little c# NET application that is basically a wrapper for a Java application, when my app starts, the Java app starts, when my app closes, it closes, and so on.
Everything works properly except that when I close my application, the Java application continues to run. When I create the process, I store the Process var in a variable outside of the methods, and then use that when my application goes to shutdown. For whatever reason though it is not terminating the Java application.
class Program
{
private static Process minecraftProcess;
public static void LaunchMinecraft(String file, String memoryValue)
{
String memParams = "-Xmx" + memoryValue + "M" + " -Xms" + memoryValue + "M ";
String args = memParams + "-jar " + file + " nogui";
ProcessStartInfo processInfo = new ProcessStartInfo("java.exe", args);
processInfo.CreateNoWindow = true;
processInfo.UseShellExecute = false;
try
{
//using (Process minecraftProcess = Process.Start(processInfo))
using (minecraftProcess = Process.Start(processInfo))
{
minecraftProcess.WaitForExit();
}
}
catch
{
// Log Error
}
}
static void Main(string[] args)
{
Arguments CommandLine = new Arguments(args);
// Hook ProcessExit Event
AppDomain.CurrentDomain.ProcessExit += new EventHandler(Current_ProcessExit);
if (CommandLine["file"] != null && CommandLine["memory"] != null)
{
// Launch the Application (Command Line Parameters)
LaunchMinecraft(CommandLine["file"], CommandLine["memory"]);
}
else
{
// Launch the Application (Default Parameters)
LaunchMinecraft("minecraft_server.jar", "1024");
}
}
static void Current_ProcessExit(object sender, EventArgs e)
{
System.Threading.Thread.Sleep(10000);
// If we have an active Minecraft Service, Shut it down
if (minecraftProcess != null)
{
minecraftProcess.Kill();
}
}
}
You can't Sleep in a ProcessExit handler.
The documentation states:
The total execution time of all
ProcessExit event handlers is limited,
just as the total execution time of
all finalizers is limited at process
shutdown. The default is two seconds.
An unmanaged host can change this
execution time by calling the
ICLRPolicyManager::SetTimeout method
with the OPR_ProcessExit enumeration
value.
Nevermind, I just realized the minecraftProcess variable is static.
Don't know if you did not solve this issue by yourself but:
You should be aware that there are Start methods for instances (returning bool) and static (returning a object).
You should not use using with something other than using-local variables!
Just this should work fine:
minecraftProcess = Process.Start(processInfo)
minecraftProcess.WaitForExit();