I have a large-ish Winforms application written in C# which is periodically unresponsive. The issue seems to occur once the application has been use for an hour or so. Exact timings are difficult to gather as users often go off to work on something selse, get back to it and find it has become unresponsive.
I think a memory leak can be ruled out as I'm not seeing excessive memory usage (I've asked users to send a screenshot of the task manager and memory usagage is the same as I would see when the application is runnning normally)
Similarly, CPU usage is normal (single digit %)
As I've so far been unable to recreate the issue on mydevelopment PC I am planning on sitting next to one of the affected users and mirror every action the user performs in order to recreate this. (I'll be setting up a laptop to RDP in to my main PC)
Recreating the issue is one thing, but I'll need to find out what is actually going on in the application.
Could anyone tell me if running in debug mode (through visual studio) will be sufficient or will I need to do something different?
I've searched through a few posts and I've seen mention of profiling software, however I'm not sure if this would only help with general performance issues or memory management issues.
Alternatively, if anyone has come across similar freezing issues then do you have any suggestions of the kind of causes for this?
Some technical details: Aplication is C#, compiled against .NET 3.5, winforms GUI. There are a few external libraries (most significant is ComponentFactory Krypton Suite). Data access is to a Microsoft SQL Server 2005 database. The solution contains 39 projects, I'm not sure if that might have something to do with it?
Any suggestions/pointers would be greatly appreciated.
The application is working much more reliably now, freezing issues still occur on occasion but nowhere near as often as before.
The issue appears to be related to the endpoint security (in this case, Cisco Security Agent) present in the environment I'm working in, application has been whitelisted and has has significantly rediced the instances of application hangs. The development system I work on does not have this endpoint security present, so it didn't show up in early stages of testing.
Thanks for all your feedback, I think there are still threading and garbage collection issues that need cleaning up, hopefully this should sort out the last few issues.
Related
I apologize for the length of the question, but I believe it is difficult to understand the “why” without the background.
Background: I have two applications running in a Windows Embedded Standard 7 environment. They should be the only two applications running on the machine. One, called “Controller”, is written in C++ the other, “DBconnector”, is written in c#. This is not new code. It has been in active use and development for almost 20 years.
The purpose of the software is to run a manufacturing machine for producing parts. These machines are big and dangerous if the program crashes. Long ago, I discovered that if the network went down for some reason, all the threads in the application would stall – not just the network thread. This was disastrous since leaving the controller in a state with the wrong relays on in extremely rare circumstances could cause the machine to literally explode. Note: Several things have been added to the software and hardware to prevent this now. While this danger doesn’t really exist anymore, stability is still extremely important. I never want the operator to be stuck in a state where they can’t hit the reset button. My solution at the time was to move the networking tasks into a separate application. The OS was windows XP based at the time. I have no idea if the problem still exists in windows 10 since I really don’t want to rewrite hundreds of thousands of lines of code to try and merge the two programs now.
The development of the two programs diverged such that the one that controlled the machine, Controller, was designed for extreme stability and the other, DBconnector, was where dangerous things like networking and most file I/O happened. Communication between the two programs is facilitated using a memory mapped file that they both can access. I have no problem sharing window handles or process id’s or any other data that might be needed between the two programs.
Here is my question. How can I make the Controller application display the GUI of DBconnector? For example, I have started to add functionality to Controller that requires DBconnector to display the quality control sheets that are held on a web site on company servers. I want for an operator to be able to pull up the quality control sheet directly on the machine. The operator currently only interacts with the Controller application. I don’t want Controller to be able to access the network. Also, C# has some tools to make displaying a web page easy. It seems to me that the place to do this is DBconnector. The problem is that DBconnector runs in the background and cannot currently be seen or accessed by a user. So, the question is how to solve this.
First option I have tried is to tell DBconnector to come forward and put Controller in the background. Then, when the user is done, Controller comes back to the front. I have made this to work using some hacks, but it is inconsistent. The trick I used was to minimize and then maximize DBconnector which seems to bring it to the front most of the time and try to hold focus on one or the other. There still might be a way to do it this way, but it needs to be something that is consistent.
The second option is to run the DBconnector application inside of one of Controller’s windows. I have no idea how to do this. I thought about using ATL or COM, but I think these run as threads within Controllers process rather than as a separate application.
The third option I’ve considered is to create a window inside Controller that intercepts and passes all user input messages directly to Dbconnector using a windows message handle and takes a screenshot of DBconnector whenever the it is invalidated and passes it through the memory mapped file. Currently, this is what I am swaying towards.
Are there any suggestions on how to do the first and last option better, or how to do the second option at all, or another solution that I have missed? Keep in mind that our current hardware is running Windows Embedded Standard 7. The project is currently in visual studio 2015. The C++ window technology is MFC implemented using libraries originally from around 2003 I think. DBconnector is in .NET framework 4 in C#.
Goal
I'm searching for tools / frameworks which allow my C# to be monitored against failures and especially crashes.
Background information
I've a C# application using the Chromium Embedded Framework for hosting a web application with access to OS functions. Thus the application is a mix of C# and native C/C++ code. Furthermore the application requires running several processes simultaneously to proper operation.
In case of problems in some of the processes the applcation easily becomes disfunctional. This can be due to
- memory faults in C/C++
- out of memory situations
- programming errors
- etc.
To a) know about problems occuring in the wild and to b) being able to fix problems I'd like to have some kind of crash reporter, which
ideally includes not only silent reports, but also has some feedback system where the user can leave comments
collects as much information on the crash as possible (i.e. exceptions, callstack of every process, optionally a memory dump)
since the most serious problems lead to crashing processes, the crash guard/reporter should not be hosted within the process, but be an external programm monitoring the software
I'm willing to try tools which are not separate processes, if they perform some magic to circumvent being killed together with the whole process
Investigations so far
So far, I've looked at sentry.io, but unfortunately Sentry-enabled apps simply quit on memory faults without any reporting or crash detection.
I've checked for alternatives to sentry.io, but there's a huge number of frameworks I'm not able to evaluate all.
I've checked on StackOverflow for similar threads, found
https://stackoverflow.com/questions/755847/crash-reporting-watchdog-for-when-my-application-locks-up-on-a-customers-machin
https://stackoverflow.com/questions/32502190/how-can-i-monitor-an-application-crash-from-a-separate-process
https://stackoverflow.com/questions/78048/best-way-to-detect-an-application-crash-and-restart-it?noredirect=1&lq=1
But these don't seem to help me out.
I'd be very happy if someone had experience with this kind of problem and pointed out tools/frameworks for it.
Best regards,
cd_
So this is a weird one.
I created a WPF application using MahApps for the GUI. So far my testing indicates that the app works fine on several different machines. Of course this is not the case on the client's machine.
The client makes use of Terminal Services and Windows Server 2008R2. Several users can be logged into their own version of the server at anytime. The app starts up fine once or twice, but after a day or so, it no longer opens up.
The app doesn't show up in the Application tab of Task Manager, but its process can be seen to be running in Processes Tab of Task Manager.
To be honest, I'm completely stumped. I had a look at the event manager log and couldn't find anything indicative of a problem. (Of course I might have missed something). I saw another SO question suggesting to disable hardware acceleration, but I'm not if that would help.
Any and all ideas would be greatly appreciated.
EDIT:
I thought I might mention the only thing that helps is if we restart the client machine.
EDIT:
I think I have isolated the issue to integration with Twain (should probably have mentioned that as another possible factor). I think the Twain library (unmanaged code) somehow stalls without sending back an error. Disabling it has "fixed" the issue.
This somehow relates to Twain and multi-session setups. I'm almost sure of it.
First you can analyze the wait chain in Windows Resource Monitor to check if there are any resources the process is waiting for. (You can find more information about the wait chain here or here.)
If you don't find any viable suspects there, you can create a memory dump of the hanging process and analyze the call stacks. If you don't know how to create one, you can read about it here. If you want to use Windows Task Manager and your OS is 64-bit then please be aware that you need to use the same bitness of Task Manager as the application.
That is: If your application is 64-bit then you have to use C:\Windows\System32\taskmgr.exe and if it's 32-bit you have to use C:\Windows\SysWOW64\taskmgr.exe. If you forget this important step you'll just get an unusable dump full of gibberish.
After you got the memory dump you can either load it into WinDbg (using the same bitness as the application) or Visual Studio (best to use 2015 or later) and analyze the call stacks of all running threads.
You can download WinDbg here and read about the necessary WinDbg configuration here. For the list of all threads you need to use this SOS command.
If you need help in loading memory dumps into Visual Studio you can find more information here.
After you've looked at the call stacks you most definitely find the answer what is waiting on what resources and is thus preventing the shutdown or startup of the application. It can either be a classic deadlock or an external resource like writing/reading of a file or some other waiting without a timeout like accessing a database or an URL that can't be reached at the moment. And of course it can also be just an infinite loop - if it doesn't consume much CPU then perhaps with some kind of DoEvents in between.
And last but very not least: If you are really interested what can be analyzed if an application hangs you can read about an example analysis done by the absolutely awesome great Mark Russinovich here.
The company I work for uses Visual Studio to develop its website and all of its features, and there is also a separate site that's been developed for testing the site. This 'testing' site can run individual test cases against the website, and must be run for each possible case.
Everything is written in VB.NET and each time the program is run a single thread is created to run the test. However, at the 'end' of the test the thread seems to still lingers. The stop button in Visual Studio must be manually clicked in order to terminate the application. Also, a process icon lingers in the task bar long after the application has closed.
It appears to me that the program is not correctly terminating all threads run during the tests, but I'm not sure if this is an issue worth brining up in the office, so I ask the following question...
What is the purpose of properly closing an application and all threads running on it, and what are the consequences, if any, of not doing so?
Well it's probably a small problem now, but it's not a good practice, IMHO. Imagine what would happen if the same code was now being executed by a continuous integration server, for instance, TeamCity (or Jenkins, or...), and the unit tests are being run continuously and automatically, by said build server.
What would happen to the build status when those threads fail to close down cleanly? We often face this problem due to bad design decisions in threading, or due to simple (and possibly, idiotic) mistakes in our unit testing code. The net effect though, is a hung build process.
I've seen CI servers hang for almost half a day before someone (mercifully) killed the build process. Essentially, this indicates a problem in our code that may or may not become a huge issue. If this was server-side code, there is potential for this code to lead to a pretty bad situation. My advice would be to dig out your introspection toolkits (memory profiling, perf profiling, etc) and see what exactly is going on, and resolve it.
We had a similar problem with an application that is being called to index SPA pages on our application server. It was throwing an exception in some cases and threads were not closing. The biggest downside is that it will consume the servers memory which is bad
Another downside as it runs as a web application that it will consume available ports and stop running when it run out of available ports.
The code should be modified to peacefully kill the thread after finishing or on exceptions and of course report any.
We have a 2 x Quad Core Xeon server with 8GB of RAM and Windows Server 2003 Enterprise installed on it. We installed our application server which is based on .NET Framework 3.5 on it. The server uses SQL Server 2005 as its database server.
When we installed the application server, it used to have ultra fast performance and everything was fine. Once we joined it into our domain, its performance decreased dramatically. For example a task that took 1 sec to complete, now takes about 30 sec. This is very strange since only .NET based applications' performance got this performance hit but the other applications still run at their normal speed.
Does anyone have any idea about why is this happening? Any help or suggestion is much appreciated.
Unfortunately, more is probably needed to answer your question. There are a host of possible reasons why this is occurring, and most of them involve your code.
Based on the symptom that you joined the domain and then things started causing trouble, I'd say you've got a lot of networking that you're doing that previously was able to be done locally on your machine and the latency is now actually causing trouble.
But that's a wild guess based on not nearly enough information.
I'd suggest you profile your code. Find out where the majority of your time is spent during execution and then post the code or a sanitized version of it here so we can help you optimize it.
I did find the answer to my question so i thought it might be good to share it here. The CLR want generate publisher evidence for assemblies with authenticode signature when it tries to load the assemblies. In our case CLR was trying to connect to clr.microsoft.com but our server's internet access was blocked so it caused huge delay whenever the application server tries to load a new assembly.
The following post describes how you can disable this feature:
Bypassing the Authenticode Signature Check on Startup
I'm going to make a guess here and think that you're talking about a web application. If this is correct, you might want to take a look at the application pools you have setup on the webserver. Your application might be getting confused about which pool to set itself in when it starts running.
Another thing to check might be your data connections and make sure that you're closing everything that's been opened.
The last thing, like Randolpho said, you're just really going to have to follow your code execution with some kind of profiler and see where things are getting tied up.
Good luck!