I have updated my:
Ubuntu server to 16.04.1 LTS and
MONO to v4.6.2
...from official repository.
Since the update, the websites are still running fine, but after about a day or two, some of the MONO processes go crazy and take 100% of the CPU. I have different websites; mostly plain HTML with just a little bit of code. It happens randomly, and on different websites each time. It's totally random.
I then receive an email alert of high CPU usage, connect via SSH, type "htop", and kill the process and it's back to normal ... for a day or two.
This definitely looks like a bug in this version of MONO. Any way to fix it? Anyone else had this problem? Or perhaps I should switch to a different version that doesn't have this corruption?
Thanks
Edit: After 2 days, EVERY MONO process is taking up the full CPU.
Looking into the Apache2 log file, I could find this related to MONO
WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)
Also, "service apache2 restart" does not solve the problem. I must manually kill the processes, or reboot.
After trying all options, it seems MONO just doesn't work well with Apache2 with mod_mono. The only solution I found is to switch Apache2 from prefork to worker mode, where the MONO server needs to be started manually and Apache2 simply forwards the requests to it -- and thus Apache2 doesn't directly touch MONO at all. There is very little documentation on how to do this, but since NGINX works in that mode, you can find instructions on how to set it up for NGINX and translate the app config file for Apache2.
These are good places to start
http://www.mono-project.com/docs/web/fastcgi/nginx/
http://epmjunkie.com/mono-fastcgi-startup-script/
I have played around with various MONO versions, and typing "service apache2 reload" to reproduce the high CPU usage problem.
In MONO 4.8, it seems to happen to happen a bit less often but the problem is still there.
In MONO 4.2.3.4, the problem is also there.
In MONO 4.2.1 that comes by default on Ubuntu, this problem doesn't happen.
As for .NET Core, some have tried it and highly recommended me to avoid it until it becomes more stable.
So for now, the only solution is to stick to MONO 4.2.1
This also confirms that this is related to MONO and not to my code or the server configuration.
Related
So this is a weird one.
I created a WPF application using MahApps for the GUI. So far my testing indicates that the app works fine on several different machines. Of course this is not the case on the client's machine.
The client makes use of Terminal Services and Windows Server 2008R2. Several users can be logged into their own version of the server at anytime. The app starts up fine once or twice, but after a day or so, it no longer opens up.
The app doesn't show up in the Application tab of Task Manager, but its process can be seen to be running in Processes Tab of Task Manager.
To be honest, I'm completely stumped. I had a look at the event manager log and couldn't find anything indicative of a problem. (Of course I might have missed something). I saw another SO question suggesting to disable hardware acceleration, but I'm not if that would help.
Any and all ideas would be greatly appreciated.
EDIT:
I thought I might mention the only thing that helps is if we restart the client machine.
EDIT:
I think I have isolated the issue to integration with Twain (should probably have mentioned that as another possible factor). I think the Twain library (unmanaged code) somehow stalls without sending back an error. Disabling it has "fixed" the issue.
This somehow relates to Twain and multi-session setups. I'm almost sure of it.
First you can analyze the wait chain in Windows Resource Monitor to check if there are any resources the process is waiting for. (You can find more information about the wait chain here or here.)
If you don't find any viable suspects there, you can create a memory dump of the hanging process and analyze the call stacks. If you don't know how to create one, you can read about it here. If you want to use Windows Task Manager and your OS is 64-bit then please be aware that you need to use the same bitness of Task Manager as the application.
That is: If your application is 64-bit then you have to use C:\Windows\System32\taskmgr.exe and if it's 32-bit you have to use C:\Windows\SysWOW64\taskmgr.exe. If you forget this important step you'll just get an unusable dump full of gibberish.
After you got the memory dump you can either load it into WinDbg (using the same bitness as the application) or Visual Studio (best to use 2015 or later) and analyze the call stacks of all running threads.
You can download WinDbg here and read about the necessary WinDbg configuration here. For the list of all threads you need to use this SOS command.
If you need help in loading memory dumps into Visual Studio you can find more information here.
After you've looked at the call stacks you most definitely find the answer what is waiting on what resources and is thus preventing the shutdown or startup of the application. It can either be a classic deadlock or an external resource like writing/reading of a file or some other waiting without a timeout like accessing a database or an URL that can't be reached at the moment. And of course it can also be just an infinite loop - if it doesn't consume much CPU then perhaps with some kind of DoEvents in between.
And last but very not least: If you are really interested what can be analyzed if an application hangs you can read about an example analysis done by the absolutely awesome great Mark Russinovich here.
My code tries to compile a big XSLT2 transform (not complex, just lots of simple - about 24,000 lines from MapForce) in C# (.NET 4.5 on 64-bit Win7) in Saxon HE (9.5 latest).
When I run this from a console application, it works fine (albeit slow). Executing the transform takes 200-300ms and I get the output I'm expecting.
When I run the same code wrapped in a WCF service in IIS (7.5), or as a http handler in IIS, I get a StackOverflowException shortly after executing the compile command (the next line is never executed).
If I try with a small transform, my code works in IIS.
The event and IIS logs don't show anything that appears useful.
Other than building Saxon from source (apparently a bit hard - any pointers welcome) to see if that helps, does anyone have any ideas where to start with this one?
After much fiddling, it turns out that the IIS worker process has, by default, a much smaller stack than a standalone application does, and this was the cause of the problem. You can modify the .exe to change this, but it was simpler for us to create a new thread and specify the thread's stack size on creation. Problem solved instantly. One to remember!
I developed a program (in C# Winforms) and distributed it through a Google site I created.
I got a comment from someone that it doesn't work without the DEP disabled (he has Windows 7).
I read a little about the DEP thing and I understand that it blocks any program that tries to run with the RAM that suppose to save to the windows system...
Is this something I did when I developed the program? I made a setup project for the program so it creates a msi file. Is there is a way to prevent my program from running those forbidden pieces on the RAM (if i understand that correctly of course)?
the link to my site if it helps -
https://sites.google.com/site/chessopeningmaster/
All .NET programs, at least since .NET 2.0 but possibly before, declare themselves DEP compatible. That's done by a flag in the header of the executable. You can see it when you run dumpbin.exe /headers on the EXE from the Visual Studio Command Prompt:
...
2 subsystem (Windows GUI)
8540 DLL characteristics
Dynamic base
NX compatible // <=== here
No structured exception handler
Terminal Server Aware
100000 size of stack reserve
....
"NX" means Never eXecute, a data execution prevention mechanism implemented in hardware by the processor. The Wikipedia article about it is pretty good.
This is enforced by any modern version of Windows (XP SP2 and later) and any modern processor. You can safely assume that your program is in fact DEP compatible if it executes properly on your machine.
So this user probably saw your program crash, for whatever reason, and started tinkering with the tools available to him. Like turning DEP enforcement off. Technically it is possible that this stopped the crash. That however doesn't mean that the program is operating correctly. It is most certainly doesn't mean that you should turn this option off. Which is technically possible by running editbin.exe with the /nxcompat:no option.
If you want to pursue this then you should ask the user for a minidump of the crashed process.
I have this strange issue with some third party DLL's. The third party provider references some open source DLL's that have a memory exception whenever I try to use a certain method. This issue does not appear when the app is run on a single core machine, but obviously we cannot assume a user will have that.
Is there a way to force an app, or even better yet a referenced DLL to run on a single core? Any other way to possibly fix this? Getting the third party to rebuild the OS dll's is apparently out of the question (its a bit of a sore spot with me currently :) ) so I have to handle it myself or just forget about providing this functionality.
By the way, the error message being thrown from the OS DLL's is "Attempting to access corrupt or protected memory".
What you want to do is achieved by using Process.ProcessorAffinity. Note that this will make your entire application run single-core.
Edit: your problem may be a result of the DLL expecting to have single-processor affinity, but it can also be a threading issue (e.g. race condition) that is very unlikely to happen when you only have a single core. If the last one is true, you can't really do anything except cross your fingers and pray (and maybe consider dropping the functionality to keep your application stable).
Personally I'd drop that functionality (you said this is an option). Multi-threading is a very touchy subject and it's obvious the 3rd party DLL isn't written very well.
You say the issue doesn't appear if you run it on a single core but not seeing a problem doesn't mean you don't HAVE a problem (and threading issues are only rarely seen anyway), so chances are your product might fail because of this every once in a while.
I once had some strange problems when referencing DLLs that were 32-bit but the .NET application was built as 64-bit. Since you mentioned that it doesn't happen on the single core machines, I'm assuming they're 32-bit and the multi-core machines are 64-bit?
The only difference was I was getting a BadImageFormatException, which you didn't mention. Anyway, the way I solved it was to set the "Platform target" of my application to x86 and everything worked after that.
We have a 2 x Quad Core Xeon server with 8GB of RAM and Windows Server 2003 Enterprise installed on it. We installed our application server which is based on .NET Framework 3.5 on it. The server uses SQL Server 2005 as its database server.
When we installed the application server, it used to have ultra fast performance and everything was fine. Once we joined it into our domain, its performance decreased dramatically. For example a task that took 1 sec to complete, now takes about 30 sec. This is very strange since only .NET based applications' performance got this performance hit but the other applications still run at their normal speed.
Does anyone have any idea about why is this happening? Any help or suggestion is much appreciated.
Unfortunately, more is probably needed to answer your question. There are a host of possible reasons why this is occurring, and most of them involve your code.
Based on the symptom that you joined the domain and then things started causing trouble, I'd say you've got a lot of networking that you're doing that previously was able to be done locally on your machine and the latency is now actually causing trouble.
But that's a wild guess based on not nearly enough information.
I'd suggest you profile your code. Find out where the majority of your time is spent during execution and then post the code or a sanitized version of it here so we can help you optimize it.
I did find the answer to my question so i thought it might be good to share it here. The CLR want generate publisher evidence for assemblies with authenticode signature when it tries to load the assemblies. In our case CLR was trying to connect to clr.microsoft.com but our server's internet access was blocked so it caused huge delay whenever the application server tries to load a new assembly.
The following post describes how you can disable this feature:
Bypassing the Authenticode Signature Check on Startup
I'm going to make a guess here and think that you're talking about a web application. If this is correct, you might want to take a look at the application pools you have setup on the webserver. Your application might be getting confused about which pool to set itself in when it starts running.
Another thing to check might be your data connections and make sure that you're closing everything that's been opened.
The last thing, like Randolpho said, you're just really going to have to follow your code execution with some kind of profiler and see where things are getting tied up.
Good luck!