The application in question is written in C#. We are late in the development cycle, close to launch on our application. One of my coworkers is seeing the following issue:
When he logs out of his Windows 7 session while the application is running, he gets a "csc.exe - Application Error" popup window that says "The application was unable to start correctly (0xc0000142). Click OK to close the application."
I believe that I have tracked this down to the fact that we update the application's XML config file on exit, and the code uses XmlSerializer. According to this question, XmlSerializer launches csc.exe to compile serialization assemblies dynamically on an as-needed basis, at run time. My suspicion is bolstered by the fact that, if I remove the update to the config file at exit time, then my coworker no longer sees the error message in question.
Can someone explain to me in more detail what is happening here? Why does csc.exe fail to start properly when executed at system logout? Is there some low-risk solution that I can put in place to mitigate the problem?
Things I have considered:
Use sgen to generate the serialization assemblies and deploy them with the application. This sounds promising, but my experiments with it were pretty dismal. It seems to only be able to generate a DLL either for an entire assembly or for a single class, no way to specify a list of classes. Also, when I point it to one of my assemblies, it starts complaining about classes in the assembly with duplicate names.
Use another means to read / write the XML. I'm not confident about implementing this at our current stage of development. We are hoping to launch soon, and this feels like too much of a risk.
I have a large-ish Winforms application written in C# which is periodically unresponsive. The issue seems to occur once the application has been use for an hour or so. Exact timings are difficult to gather as users often go off to work on something selse, get back to it and find it has become unresponsive.
I think a memory leak can be ruled out as I'm not seeing excessive memory usage (I've asked users to send a screenshot of the task manager and memory usagage is the same as I would see when the application is runnning normally)
Similarly, CPU usage is normal (single digit %)
As I've so far been unable to recreate the issue on mydevelopment PC I am planning on sitting next to one of the affected users and mirror every action the user performs in order to recreate this. (I'll be setting up a laptop to RDP in to my main PC)
Recreating the issue is one thing, but I'll need to find out what is actually going on in the application.
Could anyone tell me if running in debug mode (through visual studio) will be sufficient or will I need to do something different?
I've searched through a few posts and I've seen mention of profiling software, however I'm not sure if this would only help with general performance issues or memory management issues.
Alternatively, if anyone has come across similar freezing issues then do you have any suggestions of the kind of causes for this?
Some technical details: Aplication is C#, compiled against .NET 3.5, winforms GUI. There are a few external libraries (most significant is ComponentFactory Krypton Suite). Data access is to a Microsoft SQL Server 2005 database. The solution contains 39 projects, I'm not sure if that might have something to do with it?
Any suggestions/pointers would be greatly appreciated.
The application is working much more reliably now, freezing issues still occur on occasion but nowhere near as often as before.
The issue appears to be related to the endpoint security (in this case, Cisco Security Agent) present in the environment I'm working in, application has been whitelisted and has has significantly rediced the instances of application hangs. The development system I work on does not have this endpoint security present, so it didn't show up in early stages of testing.
Thanks for all your feedback, I think there are still threading and garbage collection issues that need cleaning up, hopefully this should sort out the last few issues.
So I'm trying to speed up our applications startup times -- and I've identified a major bottleneck to work on. Each of our webservice client classes takes forever and a day to instantiate. Some investigation revealed this is entirely due to the SoapHttpClientProtocol running GenerateXMLMappings. I started searching for information on this and found this SO post Slow SoapHttpClientProtocol constructor
I was ready to sound the trumpets since my issues mirrored what was talked about there to the letter. I went through every step listed in the first post to use sgen to pre-generate a serializer dll, and then removed the various tags from the code and built that into a normal dll which I referenced in the applciation as a normal reference (as opposed to a web reference). However after all this, I don't see any difference when profiling the application. Tons of time is still soaked up doing GenerateXMLMappings as part of the SoapHttpClientProtocol constructor.
I have verified that it is in fact using my custom webservice client dll. I have also verified that it is at least looking for the XmlSerializers dll (if I do not include the file I can see a filenotfound is spit up about it).
Does anyone have detailed info about how the SoapHttpClientProtocol constructor decides what it needs to do? This is a really frustrating problem because the whole process seems to be blackboxed with no good way to see what is actually going on internally.
Thanks in advance for any help -- I'm completely against a wall on this one.
I hit this every so often. I'll be happy to guess, but guesses are usually wrong.
To find what the problem really is I just run the app under the IDE and pause it a few times while it's being slow, to see what it's doing. That's this technique.
OK, here are the guesses, which I've seen but for you are probably wrong.
Fetching strings from resources during load.
Notifications gone mad while building data structure.
Initializing 3rd-party grids/controls, even with empty data.
Parsing/Writing XML more than you thought.
Zipping/Unzipping more than you thought.
I have this strange issue with some third party DLL's. The third party provider references some open source DLL's that have a memory exception whenever I try to use a certain method. This issue does not appear when the app is run on a single core machine, but obviously we cannot assume a user will have that.
Is there a way to force an app, or even better yet a referenced DLL to run on a single core? Any other way to possibly fix this? Getting the third party to rebuild the OS dll's is apparently out of the question (its a bit of a sore spot with me currently :) ) so I have to handle it myself or just forget about providing this functionality.
By the way, the error message being thrown from the OS DLL's is "Attempting to access corrupt or protected memory".
What you want to do is achieved by using Process.ProcessorAffinity. Note that this will make your entire application run single-core.
Edit: your problem may be a result of the DLL expecting to have single-processor affinity, but it can also be a threading issue (e.g. race condition) that is very unlikely to happen when you only have a single core. If the last one is true, you can't really do anything except cross your fingers and pray (and maybe consider dropping the functionality to keep your application stable).
Personally I'd drop that functionality (you said this is an option). Multi-threading is a very touchy subject and it's obvious the 3rd party DLL isn't written very well.
You say the issue doesn't appear if you run it on a single core but not seeing a problem doesn't mean you don't HAVE a problem (and threading issues are only rarely seen anyway), so chances are your product might fail because of this every once in a while.
I once had some strange problems when referencing DLLs that were 32-bit but the .NET application was built as 64-bit. Since you mentioned that it doesn't happen on the single core machines, I'm assuming they're 32-bit and the multi-core machines are 64-bit?
The only difference was I was getting a BadImageFormatException, which you didn't mention. Anyway, the way I solved it was to set the "Platform target" of my application to x86 and everything worked after that.
I'm looking for a few talking points I could use to convince coworkers that it's NOT OK to run a 24/7 production application by simply opening Visual Studio and running the app in debug mode.
What's different about running a compiled console application vs. running that same app in debug mode?
Are there ever times when you would use the debugger in a live setting? (live: meaning connected to customer facing databases)
Am I wrong in assuming that it's always a bad idea to run a live configuration via the debugger?
You will suffer from reduced performance when running under the debugger (not to mention the complexity concerns mentioned by Bruce), and there is nothing to keep you from getting the same functionality as running under the debugger when compiled in release mode -- you can always set your program up to log unhandled exceptions and generate a core dump that will allow you to debug issues even after restarting your app.
In addition, it sounds just plain wrong to be manually managing an app that needs 24/7 availability. You should be using scheduled tasks or some sort of automated process restarting mechanism.
Stepping back a bit, this question may provide some guidance on influencing your team.
Just in itself there's no issue in running it in debugging if the performance is good enough. What strikes me as odd is that you are running business critical 24/7 applications as users, perhaps even on a workstation. If you want to ensure robustness and avaliability you should consider running this on dedicated hardware that no one uses besides the application. If you are indeed running this on a users machine, accidents can be easily made, such as closing down the "wrong" visual studio, or crashing the computer etc.
Running in debug should be done in the test environment. Where I've work/worked we usually have three environments, Production, Release and Test.
Production
Dedicated hardware
Limited access, usually only the main developers/technology
Version control, a certain tagged version from SVN/CVS
Runs the latest stable version that has been promoted to production status
Release
Dedicate hardware
Full access to all developers
Version control, a certain tagged version from SVN/CVS
Runs the next version of the product, not yet promoted to production status, but will probably be. "Gold" if you like.
Test
Virtual machine or louse hardware
Full access
No version control, could be the next, next version, or just a custom build that someone wanted to test out on "near prod environment"
This way we can easily test new version in Release, even debug them there. In Test environment it's anything-goes. It's more if someone want to test something involving more than one box (your own).
This way it will protect you against quick-hacks that wasn't tested enough by having dedicated test machines, but still allow you to release those hacks in an emergency.
Speaking very generically, when you run a program under a debugger you're actually running two processes - the target and the debugger - and tying them together pretty intimately. So the opportunities for unexpected influences and errors (that aren't in a production run) exist. Of course, the folks who write the debuggers do their best to minimize these effects, but running that scenario 24/7 is likely to expose any issues that do exist.
If you're trying to track down a particular failure, sometimes running under a debugger is the best solution; but even there, often enabling tracing of one sort or another is a lower-impact solution that is just as effective.
The debugger is also using up resources - depending on the machine and the app, that could be an issue. If you need more specific examples of things that could go wrong using a debugger 24/7 let me know.
Ask them if they'd like to be publicly mocked on The Daily WTF. (Because with enough details in the write up, this would qualify.)
I can't speak for everyone's experience, but for me Visual Studio crashes a lot. It not only crashes itself, but it crashes explorer. This is exacerbated by add-ons and plugins. I'm not sure if its ever been tested to run for 24/7 over days and days and days the same way the OS has.
Your essentially putting the running of your app at the mercy of this huge behemoth of a second app that sounds like its easily orders-of-magnitude larger and more complex than your app. Youre just going to get bug reports and most of them are going to involve visual studio crashing.
Also, are you paying for visual studio licenses for production machines?
You definitely don't want an application that needs to be up 24/7 to be run manually from the debugger, regardless of the performance issues. If you have to convince your co-workers of that, find a new job.
I have sometimes used the debugger live (i.e. against live customer data) to debug data-related application problems in situations where I couldn't exactly reproduce the production data in a test environment.
Simple answer: you will almost certainly reduce performance (most likely considerably) and you will vastly increase your dependencies. In one step you've added the entire VS stack including the IDE and every other little bit to your dependencies. Smart people keep the dependencies of high-uptime services as tight as possible.
If you want to run under a debugger then you should use a lighter weight debugger like ntsd, this is just madness.
We never run it via the debugger. There are compiler options which may accidentally be turned on/off. Optimizations aren't turned on, and running it in production is a huge security risk.
Aside from the debug code possibly having different code paths (#ifdef, Debug.Assert(), etc) code-wise it will run the same.
A little scary mind you - set breakpoints, set the next line of code you want to execute, interactive exceptions popup and the not-as-stable running under visual studio.There are also debugger options that allow you to break always when an exception occurs. Even inspecting classes can cause side-effects if you haven't written code properly... It sure isn't something i'd want to do as the normal 24x7 process.
The only reason to run from the debugger is to debug the application. If you're doing that on a regular basis in production, it's a big red flag that your code and your process need help.
To date I've never had to run debug mode interactively in production. The rare time we switched over to a debug version for extra logging, but never sat there with visual studio open.
I would ask them what is the advantage of running it via Visual Studio?
There are plenty of disadvantages that have been listed in the replies. I can't think of any advantages.