I am using a native COM server from a .Net C# application
Everything works fine except that callbacks from the COM server into the .Net application gradually becomes slower. The server and .Net application is always running on the same machine.
Calling from .Net to the COM server is always fast.
The strange thing is that it doesn't happen on all computers even if they run the same binaries.
I have spent a lot of time on this issue. Compared environments where the callbacks are fast with ones where it is slow without finding anything special.
The callbacks start fast but get exponentially slower over time.
Doesn't matter if the callbacks are assigned to .Net methods or not.
(There is a server switch to turn all callbacks off. That is how I know the problem is with the callbacks)
The slow computers use Window 7 64-bit but the same configuration is fast on other computers.
There are slow and fast computers on the same domain and network
Doesn't matter if the user is local admin or standard user
I have monitored Disk/Network activity but there is no difference between slow and fast
There is no noticeable difference in memory consumption
Looked at CLR memory from WinDbg but found nothing strange
Some things I have noted:
The server process use 100% CPU when the callbacks are slow.
Looked at the call stack with process explorer and the server is most of the time in one of the RPC Ndr* functions, i.e. NdrClientCall2.
I am now out of ideas and need some help to solve this.
Related
I have a windows service written in C# using Selfhost technology that act like a little server that can handle some webservice's requests.
This server need to be run continuosly and actually has no policy of autoreload or explicit memory cleanup. In some tests i have noticed that after few days (a dozen) it become "instable" (what i mean is that it starts to fail accesses to DB and the memory allocated by process drammatically increase).
What i would to ask is if it's a known issue of this technology that need to be reload or a clean up memory afert certainly time, or if i need to investigate better in my code.
I smell of memory leak but it seems too strange because i only use managed objects.
As the load on our Azure website has increased (along with the complexity of the work that it's doing), we've noticed that we're running into CPU utilization issues. CPU utilization gradually creeps upward over the course of several hours, even as traffic levels remain fairly steady. Over time, if the Azure stats are correct, we'll somehow manage to get > 60 seconds of CPU per instance (not quite clear how that works), and response times will start increasing dramatically.
If I restart the web server, CPU drops immediately, and then begins a slow creep back up. For instance, in the image below, you can see CPU creeping up, followed by the restart (with the red circle) and then a recovery of the CPU.
I'm strongly inclined to suspect that this is a problem somewhere in my own code, but I'm scratching my head as to how to figure it out. So far, any attempts to reproduce this on my dev or testing environments have proven ineffectual. Nearly all the suggestions for profiling IIS/C# performance seem to presume either direct access to the machine in question or at least a "Cloud Service" instance rather than an Azure Website.
I know this is a bit of a long shot, but... any suggestions, either for what it might be, or how to troubleshoot it?
(We're using C# 5.0, .NET 4.5.1, ASP.NET MVC 5.2.0, WebAPI 2.2, EF 6.1.1, Azure System Bus, Azure SQL Database, Azure redis cache, and async for every significant code path.)
Edit 8/5/14 - I've tried some of the suggestions below. But when the website is truly busy, i.e., ~100% CPU utilization, any attempt to download a mini-dump or GC dump results in a 500 error, with the message, "Not enough storage." The various times that I have been able to download a mini-dump or GC dump, they haven't shown anything particularly interesting, at least, so far as I could figure out. (For instance, the most interesting thing in the GC dump was a half dozen or so >100KB string instances - those seem to be associated with the bundling subsystem in some fashion, so I suspect that they're just cached ScriptBundle or StyleBundle instances.)
Try remote debugging to your site from visual studio.
Try https://{sitename}.scm.azurewebsites.net/ProcessExplorer/ there you can take memory dumps an GC dumps of your w3wp process.
Then you can compare 2 GC dumps to find memory leaks and open the memory dump with windbg/VS for further "offline" debugging.
I was working on a windows 8 app in C#/XAML. I was using MVVM pattern, SQLite DB, multiple language support, etc. I did not pay much attention to how much memory the app uses when it runs. It ran reasonably fast. At a certain point I felt that the app rans significantly slower. I was trying to go through the submission process too -- creating the upload packages(for X64, X86, ARM), running the certification test. The app passed the certification test. But I was a little bit concerned about the speed and I was trying to check its memory usage and found that it use several hundred MB memory and at some point the memory can go up to 1GB (based on the number reported in Task Manager). So I tried some debugging and found that before it reaches the second line of the code in App.xaml.cs, it already uses around 150 MB. I tried to load the project into a different machine and ran it there, the memory usage is usually less than 100MB and the speed is what I experienced before it slowed down. So that is normal to me.
So does any of you have some similar experience? Do you have any idea how to make the app work normally on my original machine? My impression is that it has nothing to do what I have in the code. It could be related to some setting I had for the project on my original machine. But I don't know what settings. I tried restart the machine and that did not solve the problem.
After the app became slower, it also crashed more. In the event viewer, I saw the message mentioned the vrfcore.dll. I did some search on that and saw that it is related to application verifier and I did remember trying to run app verifier before. I also tried the debug location in the toolbars too and tried simulating suspension. But the memory is high even when I am not aware of doing those. This problem seemed only affect my app on my machine, but not all the apps.
I have a problem with an application that I wrote in .NET/C#. It consists of a server which manages a few other machines, and runs tests on them. It is a windows forms application. In order to run tests with proper error handling, for each machine I have two threads: one for running tests and one that pings it continuously. Each machine has a running queue, in which tasks are stored, tasks that will be run on that particular machine.
The issue is that after some time, when more than a few tasks are present in the queue, the memory it consumes(process explorer, task manager) gradually increases from about 50-100MB to 1.6-1.8 GB. At about this limit almost every transaction(file copy on share, remote WMI access) with the remote machines fails with either "Not enough storage" or "Out of memory". I tried some tools in order to localize the string and the closest I got was .Net Memory Profiler. That wasn't of great help, because the largest amount of memory was residing in "Private Data - Unidentified". This I'm guessing it's unmanaged data, because I can evaluate every other data(from my program) down to each string and int, and every instance of it.
Can anyone suggest a tool I can use in order to properly localize the leak and maybe fix it. It would help me a lot if I would know the DLL(from my app)/Thread that uses that memory, or at least if I can view somehow what is in that memory.
Note: A lot of posts are out there about the two exceptions: Not enough storage, and Out of memory. Most of them suggest increasing the IRPStackSize on the 'server' machine(in my case, clients). I have IRPStackSize of 50(0x32) on all of the machines, including the server.
EDIT
Regarding the comments: yes, I do maintain a log, but nothing strange happens. Using a memory profiler I discovered that my application, the .NET side uses about 20MB of memory when the unmanaged part is well over 1GB. With the help of WinDbg I found out what resides in that extra memory(in most of it). In order to access the machines and run different tests on them I use WMI, for which I have a wrapper. Everything I use is being disposed(using statements, and for some actually calling the Dispose method. Strangely though, the memory is filled with clones of this class. Does anyone know why a class would clone itself in memory.
Note: the rate at which the memory usage increases in about 5MB/s, so it's not really over a long period of time. I also wonder why it is not being freed by the garbage collector. I am using C# classes to work with WMI, not COM, nor unmanaged code. Also, among the objects on the heap I see a lot of data belonging to wmiutils, CWbemError. Oddly enough, google doesn't even know the word(no results for CWbemError)
I have a C# service written for .NET 2.0 that uses the Oracle data access provider for .NET 2.102.2.20. The service runs multiple threads and runs lots of queries to an Oracle 9.2 database. I am using NHibernate.
What I am seeing is that when it starts up it runs fast, then gets slower and slower. The CPU usage starts low then goes up and up. After a few minutes it is crawling and the CPU is at 100%. I have looked in my code and find nothing that could be doing this. Percent time in GC is < 5%. I have tried varying the ODP.NET parameters to no avail.
Anyone have an idea what could be doing this?
More detail:
The threads are worker threads and need to keep running all the time. I have no runaway threads, they are doing real work. I have profiled the program and it looks like it is spending a lot of its time inside the Oracle provider, which you would expect, but why would it use so much CPU? It's as if it's spinning waiting for resultsets or something, but it doesn't happen right away, only after a while.
Update:
It may be something to do with the .NET CLR on the server.
A multi-threaded test program that does a lot of string manipulation also shows the same behavior on this machine, starting out fast, then slowing down over the course of 15 minutes to about 1/3 of the speed. The test program does not show this slowdown behavior on a another identical server with the same OS version and the same (we think) .NET CLR version.
Update:
It is now looking like this is a problem with the server overheating and thermal protection kicking in and slowing down the CPUs' frequency.
Update:
A firmware update for the server fixed this. I still think it was an overheating problem because if I stopped running the process and let the server "rest" for a little while it would start running fast when I restarted the process, but if I killed the process and restarted it right away the process would start already running slow. My guess is that the firmware control for the fans was malfunctioning so the CPUs would heat up and slow down. If I stopped running for a little while they would cool down and run fast again, until they heated up again.
Sounds like your threads aren't terminated and keep running all the time or you've some serious memory leaks. Without more information, this could be anything.
If CPU is at 100% you're probably facing a runaway thread. You can diagnose this using WinDbg + SoS. Attach to the process and use the !runaway command to get an overview of how much CPU each thread is using. Then use !clrstack to find out what the runaway thread is doing. Let me know if you need additional details.
I'm getting the exact same issue. The OracleConnection gets progressively slower and slower. What's interesting is that if I call:
cn.Close();
OracleConnection.ClearPool(cn);
every time, it never slows down.
It must have something to do with the oracle connection (caching??)