We Keep Coding

How can I determine if a Network Stream timed out?

I have a TCP client and I set the network stream timeout as follows.
stream.ReadTimeout = 60000;
It works. But I would like to know how to test if the stream timed out. The class doesn't provide this method.
A little more detail to the question.
I am sending data to a TCPListener, about 33KB every 30 minutes. Typically, the transmission lasts about 10s and the client issues a manual "DISCONNECT" command to causes the Listener to start again. The client is an embedded system using a 3G module and sometimes the network connectivity causes the link to break. Right now, I am simply setting a read timeout of 60s. If we do not get data during that time, we simply restart the listener and wait for the next connection.
I am logging the performance of the system and would like to know how many timeouts typically occur in, say, one week. It'd have been good for the listener to simply check if the read operation time out, but I do not see a way of doing it easily in C#.
Will appreciate any help.

I do not really understand the problem about logging. I would look for the Read operation's return value, because if that is 0, a timeout occured for sure. And before I reinitialized the listener I would put a logging logic that logs the fact of the timeout. Please tell me if I misunderstood the concept of your program.

Detect and handle unresponsive RabbitMQ in .NET application

We recently had an outage where one of our APIs became unresponsive due to our rabbit cluster being given artificially high load. We where running out of threads in mono (.NET) and requests to the API failed. Although this is unlikely to happen again we would like to put some protection in against this. Ideally we would have calls to bus.Publish() timeout after a set amount of time but we can't workout how.
We then came across the blocked connections notification feature of RabbitMQ and thought this might help. However we can't figure out how to get at the connection object that is in the IServiceBus. So far we have tried
_serviceBus = serviceBus;
var connection =
((MassTransit.Transports.RabbitMq.RabbitMqEndpointAddress) _serviceBus.Endpoint.Address)
.ConnectionFactory.CreateConnection();
connection.ConnectionBlocked += Connection_ConnectionBlocked;
connection.ConnectionUnblocked += Connection_ConnectionUnblocked;
But when we do this we get a BrokerUnreachableException which I don't understand.
My questions are, is this the right approach to detect timeouts and fail (we have a backup mechanism to collect the data in the message and repost later) and if this is correct, how do we make it work?

I think you can manage this by combining System.Timer or Observable.Timer to schedule checks, and the check, which use request-response. Consumer for the request should be inside the same process. You can specify a cancellation token with reasonable timeout for the Request call and it you get a timeout - your messaging infrastructure is down or too busy, or your endpoint is too busy.

How long is a WCF connection held open?

I'm running a small WCF client application that connects to an IIS server every few minutes to download data. There are about 500 of these clients for 2 or 3 servers, and my basic code is something like this:
Client connection = null;
try
{
connection = new Client();
List<TPointer> objects = connection.GetList();
// Some work on List<T>
foreach (TPointer pointer in objects)
{
T data = GetDataFromStream(pointer, connection);
// Some additional processing on T
}
connection.SendMoreData();
// More work
}
catch (...)
{
// Exception handling for various exceptions
}
finally
{
// Handle Close() or Abort()
if (connection != null)
connection.Close();
}
When I simulate running all the clients at once for large amounts of TPointers, I start encountering the following error:
System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00.
That seems like one of those errors that can occur for any number of reasons. For all I know the server could just be swamped, or I could be requesting too large/too many objects and it's taking too long to download (a whole minute though?). Increasing the timeout is an option, but I'd like to understand the actual problem instead of fixing the symptom.
Given I have no control over the server, how can I streamline my client?
I'm not actually sure what the "request channel" mentioned in the timeout refers to. Does the timeout start ticking from when I create new Client() until I call Client.Close()? Or does each specific request I'm sending to the server (e.g. GetList or GetData) get another minute? Is it worth my while to close Client() in between each call to the server? (I'm hoping not... that would be ugly)
Would it be helpful to chunk up the amount of data I'm receiving? The GetList() call can be quite large (running into the thousands). I could try obtaining a few objects at a time and jobbing off the post-processing for later...
Edit:
Since a few people mentioned streaming:
The Client binding uses TransferMode.StreamedResponse.
GetDataFromStream() uses a Stream derived from TPointer, and SendMoreData()'s payload size is more or less negligible.
Only GetList() actually returns a non-stream object, but I'm unclear as to whether or not that affects the method of transfer.

Or does each specific request I'm sending to the server (e.g. GetList or GetData) get another minute?
The timeout property applies to each and every operation that you're doing. It's reset. If your timeout is one minute, then it starts the moment you invoke that method.
What I'd do is implement a retry policy and use an async version of the client's method and use a CancellationToken or call Abort() on your client when it's taking too long. Alternatively, you can increment or set your timeouts on the InnerChannel on the operation timout.
client.InnerChannel.OperationTimeout = TimeSpan.FromMinutes(10);
You can use that during your operation and in your retry policy you can abort entirely and reset your timeout after your retries have failed or succeeded.
Alternatively, you can try to stream your results and see if you can operate individually on them, but I don't know if keeping that connection open will trip the timeout. You'll have to hold off on operating on your collection until you have everything.
Also, set TransferMode = TransferMode.StreamedResponse in your binding.

I believe the timeout you are hitting is time to first response. In your scenario here first response is the whole response since you are returning the list, more data more time. You might want to consider streaming the data instead of returning a full list.

I suggest to modify both your web.config file (wcf side) and also app.config (client side), adding binding section like this (i.e. timeout of 25 minutes in stead of 1 minute which is default value):
<bindings>
<wsHttpBinding>
<binding name="WSHttpBinding_IYourService"
openTimeout="00:25:00"
closeTimeout="00:25:00"
sendTimeout="00:25:00"
receiveTimeout="00:25:00">
</binding>
</wsHttpBinding>
</bindings>

Given I have no control over the server, how can I streamline my client?
Basically you can not do this when you only have control over the client. It seems like the operations return no Stream (unless the pointers are types which derive from Stream).
If you want to know more about how to generally achieve streaming just read up on this MSDN article.
Everything you can do on the client is scratching on the surface of the problem. Like #The Anathema proposed in his answer you can create a retry logic and/or set the timeout to a higher value. But to eradicate the root of the problem you'd need to investigate the source of the service itself so that it can handle a higher amount of requests. Or have instances of the service running on multiple servers with a load balancer in front.

I ended up going with a combination of the answers here, so I'll just post an answer. I chunked GetList() to a certain size to avoid keeping the connection open so long (it also had a positive effect on the code in general, since I was keeping less in memory temporarily.) I already have a retry policy in place, but will also plan on messing with the timeout, as The Anathema and a couple others suggested.

suppressing WCF error

I have written a WCF service. The client is built up in Silverlight.
It works fine, but as soon as the internet connection is lost even for a second, my application throws an error in a message box "the remote server returned an error not found"
Also, it sometimes thorws WCF Request Timed Out exception. All these exceptions are displayed in a message box. I wanted to know if there is a way to suppress these exceptions, as-in I don't want a message box to be popped up everytime with these messages.
Please give me some leads.
Thanks

Not sure about your first issue.
The second issue can be solved by two ways: 1.set up client side binding timeout to longer time in web.config 2.write code to set up binding timeout. The following example sets up BasicHttpBinding at code behind.
BasicHttpBinding binding = new BasicHttpBinding();
binding.ReceiveTimeout = System.TimeSpan.Parse("00:10:00");
binding.SendTimeout = System.TimeSpan.Parse("00:10:00");

you have to catch the Exception in the Catch block and there you can keep the process for the ideal state for some time.
After some time again try to access the remote services. This will how you can handle this.

TcpChannel registration problem

I've read here: Error 10048 when trying to open TcpChannel
I am having what I thought to be a similar problem - apparently not. I took the advice of the first respondant to reset winsock (how does the winsock get corrupted, anyhow?) Anyway, here is my channel registration:
channel = new TcpChannel(channelPort);
ChannelServices.RegisterChannel(channel, false);
and the client call:
// Create a channel for communicating w/ the remote object
// Notice no port is specified on the client
TcpChannel channel = new TcpChannel();
ChannelServices.RegisterChannel(channel, false);
// Create an instance of the remote object
CommonDataObject obj = Activator.GetObject( typeof(CommonDataObject) ,
"tcp://localhost:49500/CommonDataObject") as CommonDataObject;
This seems all too straightforward to be such a hassle to use. But, the problem seems to be with the server's ChannelServices.RegisterChannel(...). Now, the reason I included the client portion is because the client instances, checks for the server object. If it can't find it, then it 'nudges' the server to instance itself. What I was wondering is if checking for the object's available first (a la: Activator.GetObject(...) ) would cause the ChannelServices to 'think' this tcp channel is already registered? It sounds dumb, but that is my only possible explanation. I have turned off the firewall, anti-fungal app, and rebooted. Still receive this
The channel 'tcp' is already
registered.
I looked at my stack trace and did notice:
at System.Runtime.Remoting.Channels.ChannelServices.RegisterChannelInternal(IChannel chnl, Boolean ensureSecurity)
at System.Runtime.Remoting.Channels.ChannelServices.RegisterChannel(IChannel chnl, Boolean ensureSecurity)
I wondered if the RegisterChannelInternal(...) might be what is causing the 'already registerd' issue. So, other than that, I am at a loss...
It's possible that the call I'm making to check for that Channel is causing it. If that is the consensus, then my question changes to: How can I poll for the Channel?
UPDATE:
After removing the initial check for the server from the client and 'assuming' that the server needs to be instanced, I did discover that the client checking is causing the problem. I've managed to get the server going, and the client did get a 'transparent proxy' object. But the question still remains: "How can I poll to discover if the server is instanced?"

The answer is evidently, yes...when the client is registering the channel, it keeps the server from registering another Tcp channel. I have removed the client instancing of a Tcp channel and the registration.
Since I haven't gotten an answer on pinging, I'm going through with a try/catch block on the obj = Activator.GetObject(...). If obj is returned null, then I 'nudge' the server, it fires up...and then the client connects with the CommonDataObject (derived from MarshalByRefObject).
So, in a sense, that is the polling technique I'm using. I'd like something more elegant - that is, an implementation that didn't work by causing a failure. To me, that's more of a hack work-around than a solution.
I found the answer here. Thanks to Abhijeet for the inadvertent solution!!! Btw...don't forget to declare:
using System.Linq;