3dparty component leaves dead threads - c#

While an app tries to connect to database server and retrieve some data, it sometimes raise an exception and it seems like it leaves dead threads when expcetion raised even when it's handled. So there are about 300 threads that getting service down.
Here is code invoked periodically on timer:
Parallel.ForEach(dbs, pair =>
{
db l = pair.Value;
if (String.IsNullOrEmpty(l.city))
l.city = l.configCity;
using (OracleConnection conn = new OracleConnection(l.connString))
{
try
{
conn.Open();
}
catch (Exception exc)
{
Console.WriteLine(String.Format("({0}, {1}): {2}{3}", l.connAlias, l.lid, exc.Message, Environment.NewLine));
}
try
{
if ((conn != null) && (conn.State == ConnectionState.Open))
{
// This method just call stored procedure and then set received data to 'l' object
if (!DbConnection.SetBadicData(conn, ref l))
{
Console.WriteLine(String.Format("Couldn't refresh basic data on ({0}, {1})", l.connAlias, l.id));
}
// This method also just call procedure and set received data to object
if (!DbConnection.SetExtendedData(conn, ref l))
{
Console.WriteLine(String.Format("Couldn't refresh advanced data on ({0}, {1})", l.connAlias, l.lid));
}
}
}
catch (Exception exc)
{
Console.WriteLine(String.Format("({0}, {1}): {2}{3}", l.connAlias, l.lid, exc.Message, Environment.NewLine));
}
}
});
Exceptions are:
Attempted to read or write protected memory. This is often an
indication that other memory is corrupt
Internal exception in Oracle client
SEHException - External component has thrown an exception
Component that used to connect to database is devArt dotConnect for Oracle.
How can I manage it guys? Does BeginConnect and then forced breaking by EndConnect will help?

Get a fixed library :-)
But seriously. If you have a third party library that you have to use, cannot change it and which is buggy, the only way I see is to run it in a separate AppDomain. Communication between domains is more difficult than just calling a method but still relatively easy. You can for example use a WCF service (with named pipe) to communicate.
Once you have your code handling nasty library in a separate AppDomain, you can recycle (destroy and recreate) that domain periodically or under other conditions. That will kill off all hanging threads, unreleased objects etc.
It is a workaround type of solution but it should give you at least a way out of this.

Related

System.NullReferenceException On WCF call

I'm having quite a difficulty trying to trace this bug. At first I thought it was linq SQL data that's causing NRE because every time I look at the tracer log, it gives me the .cs line 37 which is pointed to my sql (stored proc) call. The reason I suspect this, the result of this sproc is null (not NULL) but there's no row return for all the fields. What I did was to put data on it, and now comes the frustration; I still get the same error. Maybe it's worth mentioning that when I use the WCF as reference(not servicereferences) a DLL straight to bin then it works perfect, only when I try to use the client (mywebservice.client) one.
Any ideas where I should be looking? It is on app.config? Also, I'm using a console app to access the WCF.
This is my code:
public static List<usp_GetPaymentsResult> GetScheduledPayment(DateTime DateRun, int Fee)
{
try
{
PaymentDataContext DBContext = new PaymentDataContext();
return DBContext.usp_GetPayments(DateRun, Fee).ToList(); //line 37
}
catch (SqlException ex)
{
throw ex;
}
catch (Exception ex)
{
throw ex;
}
}
public static List<usp_GetPaymentsResult> GetScheduledPayment(DateTime DateRun, int Fee)
{
using(var context = new PaymentDataContext())
{
var payments = DBContext.usp_GetPayments(DateRun, Fee);
if (payments == null)
{
// bad database! handle error here
}
return payments.ToList();
}
}
Please note the following:
A DBContext is IDisposable. Put it in a using block, or it will
not be disposed in a timely fashion.
Your exception handling blocks did nothing but trash the exception.
If you want to rethrow an exception, use throw; without the
variable. It will rethrow the exception. Using the variable ex again
will overwrite it's stacktrace and valuable information will be lost.
However, as your blocks would have been nothing but rethrowing an
exception, you can just not catch it. Same result.
Check the result for null before you call an extention method like
.ToList() on it.
Finally, WCF Webservices are not magic. You can put breakpoints there and debug them. If you have problems doing so, maybe it would be a good idea to ask a question about that with more details about your webservice setup (VS internal or IIS Express or IIS?) and your project structure. Guessing the error based on line number is so 80s :)
After a frustrating 30 mins trying to debug a WCF NullReferenceException, turned out I had corrected an error in the WCF method structure, but hadn't refreshed the service in the calling client.

The I/O operation has been aborted because of either a thread exit or an application request

My application is working as a client application for a bank server. The application is sending a request and getting a response from the bank. This application is normally working fine, but sometimes
The I/O operation has been aborted because of either a thread exit or
an application request
error with error code as 995 comes through.
public void OnDataReceived(IAsyncResult asyn)
{
BLCommonFunctions.WriteLogger(0, "In :- OnDataReceived",
ref swReceivedLogWriter, strLogPath, 0);
try
{
SocketPacket theSockId = (SocketPacket)asyn.AsyncState;
int iRx = theSockId.thisSocket.EndReceive(asyn); //Here error is coming
string strHEX = BLCommonFunctions.ByteArrToHex(theSockId.dataBuffer);
}
}
Once this error starts to come for all transactions after that same error begin to appear, so
please help me to sort out this problem. If possible then with some sample code
Regards,
Ashish Khandelwal
995 is an error reported by the IO Completion Port. The error comes since you try to continue read from the socket when it has most likely been closed.
Receiving 0 bytes from EndRecieve means that the socket has been closed, as does most exceptions that EndRecieve will throw.
You need to start dealing with those situations.
Never ever ignore exceptions, they are thrown for a reason.
Update
There is nothing that says that the server does anything wrong. A connection can be lost for a lot of reasons such as idle connection being closed by a switch/router/firewall, shaky network, bad cables etc.
What I'm saying is that you MUST handle disconnections. The proper way of doing so is to dispose the socket and try to connect a new one at certain intervals.
As for the receive callback a more proper way of handling it is something like this (semi pseudo code):
public void OnDataReceived(IAsyncResult asyn)
{
BLCommonFunctions.WriteLogger(0, "In :- OnDataReceived", ref swReceivedLogWriter, strLogPath, 0);
try
{
SocketPacket client = (SocketPacket)asyn.AsyncState;
int bytesReceived = client.thisSocket.EndReceive(asyn); //Here error is coming
if (bytesReceived == 0)
{
HandleDisconnect(client);
return;
}
}
catch (Exception err)
{
HandleDisconnect(client);
}
try
{
string strHEX = BLCommonFunctions.ByteArrToHex(theSockId.dataBuffer);
//do your handling here
}
catch (Exception err)
{
// Your logic threw an exception. handle it accordinhly
}
try
{
client.thisSocket.BeginRecieve(.. all parameters ..);
}
catch (Exception err)
{
HandleDisconnect(client);
}
}
the reason to why I'm using three catch blocks is simply because the logic for the middle one is different from the other two. Exceptions from BeginReceive/EndReceive usually indicates socket disconnection while exceptions from your logic should not stop the socket receiving.
In my case, the request was getting timed out. So all you need to do is to increase the time out while creating the HttpClient.
HttpClient client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(5);
I had the same issue with RS232 communication. The reason, is that your program executes much faster than the comport (or slow serial communication).
To fix it, I had to check if the IAsyncResult.IsCompleted==true. If not completed, then IAsyncResult.AsyncWaitHandle.WaitOne()
Like this :
Stream s = this.GetStream();
IAsyncResult ar = s.BeginWrite(data, 0, data.Length, SendAsync, state);
if (!ar.IsCompleted)
ar.AsyncWaitHandle.WaitOne();
Most of the time, ar.IsCompleted will be true.
I had this problem. I think that it was caused by the socket getting opened and no data arriving within a short time after the open. I was reading from a serial to ethernet box called a Devicemaster. I changed the Devicemaster port setting from "connect always" to "connect on data" and the problem disappeared. I have great respect for Hans Passant but I do not agree that this is an error code that you can easily solve by scrutinizing code.
In my case the issue was caused by the fact that starting from .NET 5 or 6 you must either call async methods for async stream, or sync methods for sync strem.
So that if I called FlushAsync I must have get context using GetContextAsync
What I do when it happens is Disable the COM port into the Device Manager and Enable it again.
It stop the communications with another program or thread and become free for you.
I hope this works for you. Regards.
I ran into this error while using Entity Framework Core with Azure Sql Server running in Debug mode in Visual Studio. I figured out that it is an exception, but not a problem. EF is written to handle this exception gracefully and complete the work. I had VS set to break on all exceptions, so it did. Once I unchecked the check box in VS to not break on this exception, my C# code, calling EF, using Azure Sql worked every time.

While(true) loop get stuck in Windows Service without any log or eventLog

I have a windows Service with a main method that includes 5 tasks:
4 System.Timers.Timer()
1 method with infinit loop While(true) handled by a separat thread
N.B: before each method execution a description like "Send command starting now ..." is written in .log file
Yesterday when I checked the log, I noted that all 4 timers methods logs into log file and execute the code, but the method with the infinite loop While(true) logged nothing.
Could you provide me your suggestion about what can cause the infinite loop?
private void StartThread(){
if (_Thread != null){
if (_Thread.IsAlive){
return;
}
}
Log.Write("thread started.");
_Thread = new Thread(SchedulerWorker);
_Thread.IsBackground = true;
_Thread.Name = "scheduler thread";
_Thread.Priority = ThreadPriority.Lowest;
_Thread.Start();
}
private void SchedulerWorker(){
while (true){
try{
DoScheduleWork();
}
catch (Exception ex){
Log.Write("Worker exception : " + ex);
}
Thread.Sleep(TIMER_INTERVAL);
}
}
First, you need to check Log object for multithread support. If it works in main thread does not mean that it works in another thread well at same time. When work with multiple threads you always need to keep in mind about concurrency and other important rules. At least, may be you should place Log into lock statement (I don't know internal structure of your Log object).
Second, don't think that Log not throws exceptions or exceptions don't exists inside catch block. Is quite possible that thread crashes here:
catch (Exception ex)
{
Log.Write("Worker exception : " + ex);
}
Third, try simpliest and safest logging first for your debug purposes. Usually Windows Services logs their events into system journal. But it not well suited for debugging, it is rather a part of Service "interface". Try to use Trace class from System.Diagnostics.
using System.Diagnostics;
...
catch (Exception ex)
{
Trace.WriteLine("Worker exception : " + ex);
}
For multithreaded applications you need more accurately write, verify and debug every step of your code before think that it works as expected. Multithreading in most cases significantly increases the complexity of development.
Creation of Service applications is also not a trivial task. For example using of forms in Services is strongly discouraged and complicates debugging.

Exception in "using" statement with WCF not closing connections properly. How does one close faulted WCF client connections or those with exceptions?

There are several questions on StackOverflow regarding closing WCF connections, however the highest ranking answers refers to this blog:
http://marcgravell.blogspot.com/2008/11/dontdontuse-using.html
I have a problem with this technique when I set a breakpoint at the server and let the client hang for more than one minute. (I'm intentionally creating a timeout exception)
The issue is that the client appears to "hang" until the server is done processing. My guess is that everything is being cleaned up post-exception.
In regard to the TimeOutException it appears that the retry() logic of the client will continue to resubmit the query to the server over and over again, where I can see the server-side debugger queue up the requests and then execute each queued request concurrently. My code wan't expecting WCF to act this way and may be the cause of data corruption issues I'm seeing.
Something doesn't totally add up with this solution.
What is the all-encompassing modern way
of dealing with faults and exceptions
in a WCF proxy?
Update
Admittedly, this is a bit of mundane code to write. I currently prefer this linked answer, and don't see any "hacks" in that code that may cause issues down the road.
This is Microsoft's recommended way to handle WCF client calls:
For more detail see: Expected Exceptions
try
{
...
double result = client.Add(value1, value2);
...
client.Close();
}
catch (TimeoutException exception)
{
Console.WriteLine("Got {0}", exception.GetType());
client.Abort();
}
catch (CommunicationException exception)
{
Console.WriteLine("Got {0}", exception.GetType());
client.Abort();
}
Additional information
So many people seem to be asking this question on WCF that Microsoft even created a dedicated sample to demonstrate how to handle exceptions:
c:\WF_WCF_Samples\WCF\Basic\Client\ExpectedExceptions\CS\client
Download the sample:
C# or VB
Considering that there are so many issues involving the using statement, (heated?) Internal discussions and threads on this issue, I'm not going to waste my time trying to become a code cowboy and find a cleaner way. I'll just suck it up, and implement WCF clients this verbose (yet trusted) way for my server applications.
Optional Additional Failures to catch
Many exceptions derive from CommunicationException and I don't think most of those exceptions should be retried. I drudged through each exception on MSDN and found a short list of retry-able exceptions (in addition to TimeOutException above). Do let me know if I missed an exception that should be retried.
Exception mostRecentEx = null;
for(int i=0; i<5; i++) // Attempt a maximum of 5 times
{
try
{
...
double result = client.Add(value1, value2);
...
client.Close();
}
// The following is typically thrown on the client when a channel is terminated due to the server closing the connection.
catch (ChannelTerminatedException cte)
{
mostRecentEx = cte;
secureSecretService.Abort();
// delay (backoff) and retry
Thread.Sleep(1000 * (i + 1));
}
// The following is thrown when a remote endpoint could not be found or reached. The endpoint may not be found or
// reachable because the remote endpoint is down, the remote endpoint is unreachable, or because the remote network is unreachable.
catch (EndpointNotFoundException enfe)
{
mostRecentEx = enfe;
secureSecretService.Abort();
// delay (backoff) and retry
Thread.Sleep(1000 * (i + 1));
}
// The following exception that is thrown when a server is too busy to accept a message.
catch (ServerTooBusyException stbe)
{
mostRecentEx = stbe;
secureSecretService.Abort();
// delay (backoff) and retry
Thread.Sleep(1000 * (i + 1));
}
catch(Exception ex)
{
throw ex; // rethrow any other exception not defined here
}
}
if (mostRecentEx != null)
{
throw new Exception("WCF call failed after 5 retries.", mostRecentEx );
}
Closing and Disposing a WCF Service
As that post alludes to, you Close when there were no exceptions and you Abort when there are errors. Dispose and thus Using shouldn't be used with WCF.

c# wcf - exception thrown when calling open() on proxy class

I have the following problem, basically i have a WCF service which operates fine in small tests. However when i attempt a batch/load test i get an InvalidOperationException with the message when the open() method is called on the proxy class:
"The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be modified while it is in the Opened state."
I have searched google, but cannot find anyone else really quoting this exception message.
I guess some further info on the service may be necessary for a diagnosis - when the service receives data through one of it's exposed methods, it basically performs some processing and routes the data to a service associated with the data (different data will result in different routing). To ensure that the service runs as quickly as possible, each cycle of receiving, processing and routing of the data is handled by a seperate thread in the threadpool. could this be a problem arising from one thread calling proxyClass.Open() whilst another is already using it? would a lock block eliminate this problem, if indeed this is the problem?
thanks guys, - ive been workikng on this project for too long, and finally want to see the back of it - but this appears to be the last stumbling block, so any help much appreciated :-)
=========================================================================
thanks for highlighting that i shouldn't be using the using construct for WCF proxy classes. However the MSDN article isn't the most clearly written piece of literature ever, so one quick question: should i be using a proxy as such:
try
{
client = new proxy.DataConnectorServiceClient();
client.Open();
//do work
client.Close();
}
.................. //catch more specific exceptions
catch(Exception e)
{
client.Abort();
}
How are you using proxy? Creating new proxy object for each call. Add some code regarding how you use proxy.
Desired way of using proxy is for each call you create new proxy and dispose it once completed. You are calling proxy.open() for opened proxy that is wrong. It should be just called once.
Try using something like below in finally, as wcf does not dispose failed proxy and it piles up. Not sure it would help but give it a shot.
if (proxy.State == CommunicationState.Faulted)
{
proxy.Abort();
}
else
{
try
{
proxy.Close();
}
catch
{
proxy.Abort();
}
}
Why to do this?
http://msdn.microsoft.com/en-us/library/aa355056.aspx
Code you posted above would work but you will alway be eating exception. So handle wcf related exception in seperate catch and your generic catch with Excelion would abort then throw exception.
try
{
...
client.Close();
}
catch (CommunicationException e)
{
...
client.Abort();
}
catch (TimeoutException e)
{
...
client.Abort();
}
catch (Exception e)
{
...
client.Abort();
throw;
}
Also if you still want to use convenience of using statement then you can override dispose method in your proxy and dispose with abort method in case of wcf error.
And do not need to call .Open() as it will open when required with first call.
I'm assuming you're using .NET 3.5 or later. In .NET 3.5, the WCF ClientBase'1 class (base class for generated client proxies) was updated to use cached ChannelFactories/Channels. Consequently, unless you're using one of the Client use/creation strategies which disables caching (Client constructor that takes in a Binding object, or accessing one of a few certain properties before the backing channel is created), even though you're creating a new Client instance, it could very well still be using the same channel. In other words, before calling .Open(), always ensure you're checking the .Created status.
It definitely sounds like you've called Open() multiple times on the same object.
we hit the same roadblock as you sometime ago.
The issue with the using statement , is that if you get to a faulted state, it will still try to close at the end of the block. Another consideration, which was critical for us, is the cost of creating the proxy everytime.
We learned a lot from those blog posts:
http://blogs.msdn.com/wenlong/archive/2007/10/26/best-practice-always-open-wcf-client-proxy-explicitly-when-it-is-shared.aspx
and
http://blogs.msdn.com/wenlong/archive/2007/10/27/performance-improvement-of-wcf-client-proxy-creation-and-best-practices.aspx
Hopefuly it will help you as well.
Cheers, Wagner.

Categories