I have this strange issue with some third party DLL's. The third party provider references some open source DLL's that have a memory exception whenever I try to use a certain method. This issue does not appear when the app is run on a single core machine, but obviously we cannot assume a user will have that.
Is there a way to force an app, or even better yet a referenced DLL to run on a single core? Any other way to possibly fix this? Getting the third party to rebuild the OS dll's is apparently out of the question (its a bit of a sore spot with me currently :) ) so I have to handle it myself or just forget about providing this functionality.
By the way, the error message being thrown from the OS DLL's is "Attempting to access corrupt or protected memory".
What you want to do is achieved by using Process.ProcessorAffinity. Note that this will make your entire application run single-core.
Edit: your problem may be a result of the DLL expecting to have single-processor affinity, but it can also be a threading issue (e.g. race condition) that is very unlikely to happen when you only have a single core. If the last one is true, you can't really do anything except cross your fingers and pray (and maybe consider dropping the functionality to keep your application stable).
Personally I'd drop that functionality (you said this is an option). Multi-threading is a very touchy subject and it's obvious the 3rd party DLL isn't written very well.
You say the issue doesn't appear if you run it on a single core but not seeing a problem doesn't mean you don't HAVE a problem (and threading issues are only rarely seen anyway), so chances are your product might fail because of this every once in a while.
I once had some strange problems when referencing DLLs that were 32-bit but the .NET application was built as 64-bit. Since you mentioned that it doesn't happen on the single core machines, I'm assuming they're 32-bit and the multi-core machines are 64-bit?
The only difference was I was getting a BadImageFormatException, which you didn't mention. Anyway, the way I solved it was to set the "Platform target" of my application to x86 and everything worked after that.
Related
I have updated my:
Ubuntu server to 16.04.1 LTS and
MONO to v4.6.2
...from official repository.
Since the update, the websites are still running fine, but after about a day or two, some of the MONO processes go crazy and take 100% of the CPU. I have different websites; mostly plain HTML with just a little bit of code. It happens randomly, and on different websites each time. It's totally random.
I then receive an email alert of high CPU usage, connect via SSH, type "htop", and kill the process and it's back to normal ... for a day or two.
This definitely looks like a bug in this version of MONO. Any way to fix it? Anyone else had this problem? Or perhaps I should switch to a different version that doesn't have this corruption?
Thanks
Edit: After 2 days, EVERY MONO process is taking up the full CPU.
Looking into the Apache2 log file, I could find this related to MONO
WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)
Also, "service apache2 restart" does not solve the problem. I must manually kill the processes, or reboot.
After trying all options, it seems MONO just doesn't work well with Apache2 with mod_mono. The only solution I found is to switch Apache2 from prefork to worker mode, where the MONO server needs to be started manually and Apache2 simply forwards the requests to it -- and thus Apache2 doesn't directly touch MONO at all. There is very little documentation on how to do this, but since NGINX works in that mode, you can find instructions on how to set it up for NGINX and translate the app config file for Apache2.
These are good places to start
http://www.mono-project.com/docs/web/fastcgi/nginx/
http://epmjunkie.com/mono-fastcgi-startup-script/
I have played around with various MONO versions, and typing "service apache2 reload" to reproduce the high CPU usage problem.
In MONO 4.8, it seems to happen to happen a bit less often but the problem is still there.
In MONO 4.2.3.4, the problem is also there.
In MONO 4.2.1 that comes by default on Ubuntu, this problem doesn't happen.
As for .NET Core, some have tried it and highly recommended me to avoid it until it becomes more stable.
So for now, the only solution is to stick to MONO 4.2.1
This also confirms that this is related to MONO and not to my code or the server configuration.
We have a C# .Net application using WCF services. And the application is deployed in our production server under a Windows Service Application. One part of the module is responsible for creating shape files ((*.shp, *.dbf) for a smaller area the workers will be working today and send them down to a PDA.
To write the shape files, we use a third party dll, NetTopologySuite
GisSharpBlog.NetTopologySuite.IO.ShapefileWriter
which is also in C#. (I am not sure whether any dll it reference use unmanaged code.)
The system might work fine for a while say for a week. Then suddenly we get an exception saying
Attempted to read or write protected memory.
This is often an indication that other memory is corrupt.
from the Write method, where we write the geometry collection to shape files.
sfw.Write(FileName, new GeometryCollection(gc.ToArray()));
(GeometryCollection is also from a third party dll, GeoAPI.dll)
This error brings down the whole service and makes it unfunctional. Then we would just restart the service and try to run the same data again, it would work fine for another week till it crash again. It happens only in production and at random times. We were not able to find the cause of the issue.
Many forums suggest that it might be because of memory leaks in some unmanaged code. But we couldn't find which one.
We are also ready to rewrite the part that create new shape files.
Please help me to resolve this issue.
Let me know if more details are required. Thanks in advance.
In my experience, that message was a result of a memory leak. This is what I'd do if I am in your situation especially since you are working on a third-party DLL.
1) Monitor your WCF server and see what is going on with the DLLHost.exe and the aspnet services in the task manager. I have a feeling that your third-party DLL has a memory leak that causes these 2 services to bloat and reach the limit of your servers memory. This is the reason why it works for a while and then suddenly just stopped working.
2) Identify a good schedule on when you can recycle your servers memory and application pool. Since the issue is rampant, you might want to do this every midnight or when no one is actively using it.
3) Write a good error logging code to know exactly what is happening during the time it bogged down. I would put the following information on the error logs: The parameters that you are passing, the user who encountered that problem etc. This is so you will know exactly what is happening.
4) Check the Event Viewer as maybe there is some information in there that can pinpoint the problem.
4) After doing 1, 2, and 3 and I will call your third-party DLL vendor and see what they can do to help you. You might need to provide the information that you collected from 1, 2, 3 and 4 items from above.
Good luck and I hope this will help.
I think you have some unmanaged code in the third libraries that is getting an address protected by the system or used by other applications.
You have an Access Violation (pointer to memory not belonging to your application space, including null/mass - 0x0 - address) in one of your third-party DLLs.
Or else, it's maybe some unmanaged COMObject you're using that causes this error.
The random nature of this error, would suggest to me that it may be a matter of threads. Specifically the Write method of ShapefileWriter might have been called, got delayed in a thread then you call Close. The delayed Write method then tries to write over a closed (and protected) file, which could result in the error you see.
This is purely speculation since there's not much code to make a better guess, but I've experienced this issue using video writing libraries, so it might be the same in your case.
Check to make sure you don't have threads within threads. That is what happened when I encountered this error. See this link for more information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt
I have a large-ish Winforms application written in C# which is periodically unresponsive. The issue seems to occur once the application has been use for an hour or so. Exact timings are difficult to gather as users often go off to work on something selse, get back to it and find it has become unresponsive.
I think a memory leak can be ruled out as I'm not seeing excessive memory usage (I've asked users to send a screenshot of the task manager and memory usagage is the same as I would see when the application is runnning normally)
Similarly, CPU usage is normal (single digit %)
As I've so far been unable to recreate the issue on mydevelopment PC I am planning on sitting next to one of the affected users and mirror every action the user performs in order to recreate this. (I'll be setting up a laptop to RDP in to my main PC)
Recreating the issue is one thing, but I'll need to find out what is actually going on in the application.
Could anyone tell me if running in debug mode (through visual studio) will be sufficient or will I need to do something different?
I've searched through a few posts and I've seen mention of profiling software, however I'm not sure if this would only help with general performance issues or memory management issues.
Alternatively, if anyone has come across similar freezing issues then do you have any suggestions of the kind of causes for this?
Some technical details: Aplication is C#, compiled against .NET 3.5, winforms GUI. There are a few external libraries (most significant is ComponentFactory Krypton Suite). Data access is to a Microsoft SQL Server 2005 database. The solution contains 39 projects, I'm not sure if that might have something to do with it?
Any suggestions/pointers would be greatly appreciated.
The application is working much more reliably now, freezing issues still occur on occasion but nowhere near as often as before.
The issue appears to be related to the endpoint security (in this case, Cisco Security Agent) present in the environment I'm working in, application has been whitelisted and has has significantly rediced the instances of application hangs. The development system I work on does not have this endpoint security present, so it didn't show up in early stages of testing.
Thanks for all your feedback, I think there are still threading and garbage collection issues that need cleaning up, hopefully this should sort out the last few issues.
We've recently started seeing an issue where log4net.dll is corrupted - specifically, it is the right size but contains all zeroes. This causes BadImageFormatExceptions (ultimately wrapped in TypeInitializationExceptions since we tend to initialize our ILog instances statically in each class). This happens sporadically and without a discernible pattern. Uninstalling and reinstalling our app seems to "fix" it, in that we haven't (yet) seen the issue twice on the same machine - but that could just be coincidence.
Our application consists of a Windows service and a "regular" Windows app which communicate via named pipes (nothing too unusual). We've used log4net in the regular app for a long time (before I joined the project), but only recently added log4net to the service. We initialize it using the same mechanism in both applications, specifically:
[assembly: log4net.Config.XmlConfigurator(Watch = true)]
And we typically instantiate ILog instances statically in each class, i.e.:
private static readonly ILog _log = LogManager.GetLogger(typeof (SomeClass));
So far, we have considered the following but without any hard evidence to lead us to a conclusion:
An installer issue. We're using a pretty standard MSI installer, and this file is simply copied normally as part of the install. It's not the first or last file copied. Furthermore, we've seen at least one instance where the issue didn't manifest until a day or so after the install (whereas most of the occurrences have happened shortly or immediately after install). We have, however, recently cleaned up the installer so there have been changes there (unfortunately I can't speak to them in great detail, as the installer work was done by someone else and MSI is not something I'm deeply familiar with).
Interaction with anti-virus software. All the machines that we've seen the issue on (4-5) have the same AV software (Sophos), but there's no indication in any Sophos logs of log4net getting flagged or quarantined.
Something related to using log4net in the Windows service application.
As you can see, we have reasons to doubt #1 and #2, but haven't yet ruled anything out, and have very little evidence for or against #3. Since this is our first release where we're using log4net in the service, it obviously seems suspicious that we're now seeing the DLL get corrupted - but we also had our dev, QA, and beta testers using early iterations of this release for several weeks before this started occurring, so there isn't an obvious correlation with respect to the timing of the change.
We've also tried automating the uninstall/reinstall process to run repeatedly to see if we could shake anything out (nothing after several thousand installs). We've got all users in our office running with ProcMon so that if we see it again, we can hopefully see what touches the file, but that means at this point we're in a waiting game.
Any thoughts on how we could more definitively rule some of our possible causes in or out, or any info from someone who's seen this issue specifically would be greatly appreciated!
UPDATE:
We haven't seen this as much recently, though that's probably due at least in part to lower usage of the installer. However, one key change that we made recently was turning off publisher evidence for the service application (see here: http://msdn.microsoft.com/en-us/library/bb629393%28v=VS.90%29.aspx)
We made this change (as the above link recommends for service applications) to improve our startup times since we were occasionally seeing the service time out and fail to start. However, we also haven't seen the log4net.dll corruption issue since we made this change. Is it possible that by disabling this feature, we are bypassing some sort of operation that could potentially mangle the log4net.dll (which is a strongly-named assembly)? We don't want to assume we've fixed the issue just because we haven't seen it in a few weeks, since we don't have any hard evidence and thus it could just be coincidence
Again, any thoughts would be much appreciated.
Is it possible to update an application to a new version without closing it?
Or is there a good way to do that without user noticing it was closed?
Typically applications notice on startup that an update is available, then ask the user whether it's okay to update. They then start the update process and exit. The update process replaces the files, then launches the new version.
In some cases you may be able to get away with updating some pieces of an application without a restart - but the added complexity is significant, and frankly it's better not to try in 99% of cases, IMO.
Of course, you haven't said what kind of app you're writing - if you could give more information, that would help.
The application needs to be closed before updating it, because updating an application generally means replacing the executable files (.exe, .dlls, etc.) with their newer versions, and this can't be done without closing the application.
As Jon said, in some cases, you can upgrade the application without closing it. But, this is not advisable, as it might cause failure in the updater, and the whole update might rollback.
Updater can be another executable which will first close the main application, then download the updates, apply them, start the main application, and exit (An example of this is Skype, FireFox, etc.)
You could separate the backend into a separate process/module and update the the backend by restarting it without the user realizing it.
Updating the front end will be a bit trickier, but could be avoided or delayed, if necessary.
A nice and clean way to achieve this would be using dynamic plugins.
You can code your application heavily plugin-based. When an update is needed, unload the plugin that needs to be updated, update the .dll file and load it back into the application.
However, making this invisible to the user may be a tough job, therefore it depends heavily on your design and coding.
I remember InTime having the ability to swap exe's live, however that had to be carefully coded. I know it's possible but as Jon Skeet said, you're likely better off not trying.
Unless you're doing some kind of automation or something very serious... even then, you should consider a failover so you can shut one down / restart if needed.
If you has some some sort of skeletal framework which launched your application and dlls, you could look at CreateDomain. It will take serious design efforts on your part though. Good luck!