I have several 3rd party DLLs which are super flaky. They can sometimes hang and never return or sometimes throw weird exceptions which can bring down the whole process.
I want to move these DLLs and load then in a separate Child process. That way instead of having to do nasty Thread.Abort I can just bring down the process cleanly and later re-spawn it when required.
So my parent application receives a list of files that need to be processed by certain third party DLLs. I essentially need to pass these to the new child process, let it process the file and then communicate back to the parent that it was successful. I must also be able to bring down the process if sh*t hits the fan and re-spawn. These files come as constant stream so spawning a process every-time i get a file is not possible, id want it to hang around and just accept requests.
Right now i'm spawning the child process from the parent and then attempting to use Memory Mapped Files to share the files/work. Would it be easier just passing the location of said file and somehow getting a response when its processed?
What would be a good strategy here...
I would....
Create a WCF service, using PerCall instancing that hosts the dlls and does the file process - this would spawn a new instance for each call and if any goes down it should not affect the other. You could host it even as part of your main app but maybe as a separate Windows service, and as its probably going to be on the same machine, use named pipes transport.
Fire each request at it from your main app.
If you don't get a successful response (as long as its not a wcf exception - i.e. endpoint not found) just retry the request for x number of times
Related
We use a third party library to manipulate Pdf's. Our application runs as a Windows service and handles thousands of files every month. Once in a while someone uploads a malformed Pdf, which makes the library run amok and eventually throw a StackOverflowException.
The library manufacturer has not fixed the error over the last 2 years, and we can't have our production crash when someone feels like it.
Automatically restarting the service does not seem like an option, as the application would then retry the malformed file. Since we process many files in parallel, we cannot know which is the malformed when starting.
Since stackoverflows can't be catched by default, I would like to know if I can tweak the CLR of the service to catch the exception anyway.
You could re-architect the application to create new child processes, each with its own instance of the library, to perform the work.
Most importantly, with this approach the failing instance doesn't crash the entire application or take the other child processes down with it. You also have the advantage that the manager process can keep track of which files are in progress (and on which process) so it would know which files to not retry after a failure.
The title of my question might already give away the fact that I'm not sure about what I want, as it might not make sense.
For a project I want to be able to run executables within my application, while redirecting their standard in and out so that my application can communicate with them through those streams.
At the same time, I do not want to allow these executables to perform certain actions like use the network, or read/write outside of their own working directory (basically I only want to allow them to write and read from the standard in and out).
I read on different places on the internet that these permissions can be set with PermissionStates when creating an AppDomain in which you can then execute the executables. However, I did not find a way to then communicate with the executables through their standard in and out, which is essential. I can however do this when starting a new Process (Process.Start()), though then I cannot set boundaries as to what the executable is allowed to do.
My intuition tells me I should somehow execute the Process inside the AppDomain, so that the process kind of 'runs' in the domain, though I cannot see a way to directly do that.
A colleague of mine accomplished this by creating a proxy-application, which basically is another executable in which the AppDomain is created, in which the actual executable is executed. The proxy-application is then started by a Process in the main application. I think this is a cool idea, though I feel like I shouldn't need this step.
I could add some code containing what I've done so far creating a process and appdomain, though the question is pretty long already. I'll add it if you want me to.
The "proxy" application sounds like a very reasonable approach (given that you only ever want to run .NET assemblies).
You get the isolation of different processes which allows you to communicate via stdin/stdout and gives the additional robustness that the untrusted executable cannot crash your main application (which it could if it was running in an AppDomain inside your main application's process.
The proxy application would then setup a restricted AppDomain and execute the sandboxed code, similar to the approach described here:
How to: Run Partially Trusted Code in a Sandbox
In addition, you can make use of operation system level mechansims to reduce the attack surface of a process. This can be achieved e.g. by starting the proxy process with lowest integrity which removes write access to most resources (e.g. allow writing files only in AppData\LocalLow). See here for an example.
Of course, you need to consider whether this level of sandboxing is sufficient for you. Sandboxing, in general, is hard, and the level of isolation always will be to a certain degree only.
I have a ASP.NET (C#) website that uses a third party DLL to process the data that the users POST via a web form. The call is pretty straightforward:
string result = ThirdPartyLib.ProcessData(myString);
Once in a blue moon this library hangs and (according to my hosting provider logs) consumes 100% of CPU. The website is hosted on a shared hosting, so I have no access to the IIS or event logs. When this happens, my website is automatically stopped by the hosting provider performance monitor, and I have manually switch it back on.
Now, I know that the right thing to to is investigate the problem and fix (or replace) the DLL. But as it's third-party software, I am unuable to fix it, and their support is not helpful at all. Moreover, I can't reproduce the problem. Replacing the library is a pain too.
Is there a way in C# to detect when this DLL starts consuming 100%CPU and kill the process automatically from my ASP.NET code?
You cannot "detect" if the current process is hanging because as the caller of a method (third party or not) you're simply not in control until it returns.
What you can do is move the call to the third party library into a separate executable and have it output its result via the standard output (you can simply use Console.WriteLine(string) for this).
Once you've done that, you can start a separate Process that runs this executable, read the result via StandardOutput and use WaitForExit(int) to wait a certain amount of time (maybe a few seconds) for the process to finish. The return value of WaitForExit() tells you if the process actually exited. In case it didn't, you can Kill() it and move on without IIS worker process hanging as a whole.
I have two separate programs, one is a console application, and the other one is a windows application.
My windows application:
Has a graphic interface, buttons, and others functions.
One of the buttons, named "research": when I click on it, I launch the console application with this line of code:
string strResult = ProcessHelper.LaunchProcessWaitForPipedResult("MyExecFile.exe", strArguments, 10 * 60 * 1000, true); // 10 mins max
My console Application:
do a query on all existing files in a directory.
My problem:
I want to create a progress-bar on the windows application to show the progress of the console application. The problem is I don't know how to pass this information between the two processes. The only restriction is to not use a database or file.
Given two processes in the same user session, and wanting to avoid any communication outside that session I would look at three options:
1. Using named pipes.
The parent process creates a named pipe using a random name (and confirms that name is not in use by opening it). It passes that name to the child process. A simple protocol is used that allows the child to send updates.
There are a number of challenges to overcome:
Getting the logic to ensure the name is unique right (named pipe names are global).
Ensuring no other process can connect (the default named pipe ACL limits connections to the session: this might be enough).
Handling the case where a different parent process does not support progress updates.
Handling the child or parent crashing.
Avoiding getting too clever with the communication protocol, but allowing room for growth (what happens when more than a simple progress bar is wanted?)
2. Using Shared Memory
In this case names of objects are, by default, local to the session. By default this is more secure.
The parent process creates a sufficiently large amount of shared memory (for a simple progress update: not much), a mutex and an event.
The parent process then, concurrently with the GUI waits for the event to be signalled, when it is it enters the mutex and reads the content of shared memory. It then unsets the event and leaves the mutex.
Meanwhile to send an update the child enters the mutex, updates and memory and sets the event before leaving the mutex.
The challenges here include:
Defining the layout of the shared memory. Without a shared assembly this is likely to be error prone.
Avoiding others using the shared memory and synchronisation objects. .NET makes things harder here: in Win32 I would make the handles inheritable thus not needing to name the objects (except for debugging) and pass to the child directly.
Getting the sequencing of shared memory, mutex and event correct is critical. Memory corruption and more subtle bugs await any errors.
It is harder to do variable length data with shared memory, not an issue for a simple progress count but customers always want more.
Summary
I would probably look at named pipes in the first place (or perhaps custom WMI types if I wanted greater flexibility). BUT I would do that only after trying everything to avoid needing multiple processes in the first place. A shared library plus console wrapper for others, while I use the library directly would be a far easier option.
I have created a TelNet server for a project I need to do which is working fine, however when a client connects to the server it needs to connect to a database, again this works fine when the connection information is correct and/or calls to the database do not take too long.
If the database call takes a long time (usually due to incorrect credentials or a badly optimised stored procedure) the server will crash with a Windows error message (i.e. not debuggable), which I understand is the underlying TCP system kicking in, which is fine. To resolve this I am putting all the database calls into BackgroundWorkers, so the server (and clients) continue to work, however I need to kill off this process if it is obviously taking too long.
I know about using BackgroundWorker.CancellationPending, but as this is a single method call to the database (via and external DLL), it will never get checked. Same issue with a self-made approach that I have seen elsewhere. The other option I have seen is using Thread.Abort(), but I also know that is unpredictable and unsafe, so probably best not to use that.
Does anyone have any suggestions how to accomplish this?
The problem here is that an external DLL is controlling the waiting. Normally, you could cancel ADO.NET connections or socket connections but this doesn't work here.
Two reliable approaches:
Move the connection into a child process that you can kill. Kill is safe (in contrast to Thread.Abort!) because all state of that process is gone at the same time.
Structure the application so that in case of cancellation the result of the connection attempt is just being ignored and that the app continues running something else. You just let the hanging connection attempt "dangle" in the background and throw away its result when it happens to return later.