I have a curious behaviour in some code I inherited - simplified example below which demonstrates the issue if built into a plain Console App.
WhoIs hits its usage allowance on the 5th call - and returns a message + shuts the socket. Using ReadLineAsync this produces a SocketException and some IOExceptions - sometimes these are caught in the catch block and everything is as it should be, most times they are not caught and the program simply hangs - Break all in the VS debugger shows me on one of the calls to Console.WriteLine on the main thread. This behaviour persists when running the .exe file directly outside the debugger.
Can anyone see what/why this is happening?
I can fix my problem practically by using Peek() but I'd like to know what is going on with the exception not being caught - and the "deadlock". Presumably it is some kind of threading or context issue. If it's something I'm doing, I'd like to know what so I can avoid it elsewhere!
using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;
namespace AsyncIssueConsoleApplication
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.WriteLine(Task.Run(() => LookupAsync("elasticsearch.org")).Result);
Console.ReadLine();
}
private static async Task<string> LookupAsync(string domain)
{
StringBuilder builder = new StringBuilder();
TcpClient tcp = new TcpClient();
await tcp.ConnectAsync("whois.pir.org", 43).ConfigureAwait(false);
string strDomain = "" + domain + "\r\n";
byte[] bytDomain = Encoding.ASCII.GetBytes(strDomain.ToCharArray());
try
{
using (Stream s = tcp.GetStream())
{
await s.WriteAsync(bytDomain, 0, strDomain.Length).ConfigureAwait(false);
using (StreamReader sr = new StreamReader(s, Encoding.ASCII))
{
try
{
//This is fine
/*while (sr.Peek() >= 0)
{
builder.AppendLine(await sr.ReadLineAsync());
}*/
//This isn't - produces SocketException which usually isn't caught below
string strLine = await sr.ReadLineAsync().ConfigureAwait(false);
while (null != strLine)
{
builder.AppendLine(strLine);
strLine = await sr.ReadLineAsync().ConfigureAwait(false);
}
}
catch (Exception e)
{
//Sometimes the SocketException/IOException is caught, sometimes not
return builder.ToString();
}
}
}
}
catch (Exception e)
{
return builder.ToString();
}
return builder.ToString();
}
}
}
Suggested duplicate Q&A may relate but doesn't answer this query that I can see, certainly not fully: i.e. what I would need to do about the SynchronizationContext - I am already using ConfigureAwait(false).
When the code is deadlocked as described above, the stack trace is:
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) Unknown
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Unknown
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout = -1, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout = -1, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Threading.Tasks.Task<string>.GetResultCore(bool waitCompletionNotification = true) Unknown
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.Result.get() Unknown
AsyncIssueConsoleApplication.exe!AsyncIssueConsoleApplication.Program.Main(string[] args = {string[0]}) Line 18 C#
The IOException is: {"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."}
The SocketException is: {"An established connection was aborted by the software in your host machine"}
I can reproduce it now. In order to see where it is hanging I switched to synchronous IO. It is a common debugging issue with async IO that you cannot see what IOs are currently pending. This is a key reason why you might not want to use async IO in the first place.
It's hanging because the remote server does not close the connection. The ReadLine call would only end if the remote side closed the connection.
This could be a bug in the rate limiting code. It also could be an expected behavior of the protocol. Maybe you are meant to send the next request now? Or maybe you are supposed to detect the rate limiting from the previous line and shut down yourself.
This is not a threading issue. There is no concurrency going on at all. All LookupAsync instances are running sequentially.
I also tried properly closing the TcpClients in case the remote server behaves differently in the face of multiple connections. There was no effect. You should dispose your resources in any case, though. This is a severe leak.
Related
I have a TCP/IP server that is supposed to allow a connection to remain open as messages are sent across it. However, it seems that some clients open a new connection for each message, which causes the CPU usage to max out. I tried to fix this by adding a time-out but still seem to have the problem occasionally. I suspect that my solution was not the best choice, but I'm not sure what would be.
Below is my basic code with logging, error handling and processing removed.
private void StartListening()
{
try
{
_tcpListener = new TcpListener( IPAddress.Any, _settings.Port );
_tcpListener.Start();
while (DeviceState == State.Running)
{
var incomingConnection = _tcpListener.AcceptTcpClient();
var processThread = new Thread( ReceiveMessage );
processThread.Start( incomingConnection );
}
}
catch (Exception e)
{
// Unfortunately, a SocketException is expected when stopping AcceptTcpClient
if (DeviceState == State.Running) { throw; }
}
finally { _tcpListener?.Stop(); }
}
I believe the actual issue is that multiple process threads are being created, but are not being closed. Below is the code for ReceiveMessage.
private void ReceiveMessage( object IncomingConnection )
{
var buffer = new byte[_settings.BufferSize];
int bytesReceived = 0;
var messageData = String.Empty;
bool isConnected = true;
using (TcpClient connection = (TcpClient)IncomingConnection)
using (NetworkStream netStream = connection.GetStream())
{
netStream.ReadTimeout = 1000;
try
{
while (DeviceState == State.Running && isConnected)
{
// An IOException will be thrown and captured if no message comes in each second. This is the
// only way to send a signal to close the connection when shutting down. The exception is caught,
// and the connection is checked to confirm that it is still open. If it is, and the Router has
// not been shut down, the server will continue listening.
try { bytesReceived = netStream.Read( buffer, 0, buffer.Length ); }
catch (IOException e)
{
if (e.InnerException is SocketException se && se.SocketErrorCode == SocketError.TimedOut)
{
bytesReceived = 0;
if(GlobalSettings.IsLeaveConnectionOpen)
isConnected = GetConnectionState(connection);
else
isConnected = false;
}
else
throw;
}
if (bytesReceived > 0)
{
messageData += Encoding.UTF8.GetString( buffer, 0, bytesReceived );
string ack = ProcessMessage( messageData );
var writeBuffer = Encoding.UTF8.GetBytes( ack );
if (netStream.CanWrite) { netStream.Write( writeBuffer, 0, writeBuffer.Length ); }
messageData = String.Empty;
}
}
}
catch (Exception e) { ... }
finally { FileLogger.Log( "Closing the message stream.", Verbose.Debug, DeviceName ); }
}
}
For most clients the code is running correctly, but there are a few that seem to create a new connection for each message. I suspect that the issue lies around how I handle the IOException. For the systems that fail, the code does not seem to reach the finally statement until 30 seconds after the first message comes in, and each message creates a new ReceiveMessage thread. So the logs will show messages coming in, and 30 seconds in it will start to show multiple messages about the message stream being closed.
Below is how I check the connection, in case this is important.
public static bool GetConnectionState( TcpClient tcpClient )
{
var state = IPGlobalProperties.GetIPGlobalProperties()
.GetActiveTcpConnections()
.FirstOrDefault( x => x.LocalEndPoint.Equals( tcpClient.Client.LocalEndPoint )
&& x.RemoteEndPoint.Equals( tcpClient.Client.RemoteEndPoint ) );
return state != null ? state.State == TcpState.Established : false;
}
You're reinventing the wheel (in a worse way) at quite a few levels:
You're doing pseudo-blocking sockets. That combined with creating a whole new thread for every connection in an OS like Linux which doesn't have real threads can get expensive fast. Instead you should create a pure blocking socket with no read timeout (-1) and just listen on it. Unlike UDP, TCP will catch the connection being terminated by the client without you needing to poll for it.
And the reason why you seem to be doing the above is that you reinvent the standard Keep-Alive TCP mechanism. It's already written and works efficiently, simply use it. And as a bonus, the standard Keep-Alive mechanism is on the client side, not the server side, so even less processing for you.
Edit: And 3. You really need to cache the threads you so painstakingly created. The system thread pool won't suffice if you have that many long-term connections with a single socket communication per thread, but you can build your own expandable thread pool. You can also share multiple sockets on one thread using select, but that's going to change your logic quite a bit.
I have two different applications communicating with pipes. Most requests work fine, but from time to time, I get following exception when trying to send a string through pipe:
System.InvalidOperationException : Pipe hasn't been connected yet.
With the following stack:
at System.IO.Pipes.PipeStream.CheckWriteOperations()
at System.IO.Pipes.PipeStream.Flush()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Dispose(Boolean disposing)
at System.IO.TextWriter.Dispose()
Last line points to the line in client pipe code, I marked it with ">"
using (var pipeClient =
new NamedPipeClientStream("PipeName"))
{
try
{
using (var streamWriter = new StreamWriter(pipeClient))
{
pipeClient.Connect(20000);
streamWriter.AutoFlush = true;
streamWriter.WriteLine(dataToSend);
pipeClient.WaitForPipeDrain();
>}
}
catch (Exception x)
{
Log.Error("ExceptionText");
}
}
On server, this is how pipes are being handled:
using (var streamReader = new StreamReader(pipeServer))
{
pipeServer.WaitForConnection();
string data;
while ((data = streamReader.ReadLine()) != null)
{
//Process the data here...
}
pipeServer.Disconnect();
}
According to MSDN, PipeStream.Flush() isn't even supposed to do anything, yet it somehow fails? I'm confused about the whole situation.
What does it all mean? Does it try to dispose the pipe that somehow became disconnected in process?
If yes, what can I do about it?
Or am I doing something wrong?
Edit:
According to reference, during PipeStream.Flush() pipe state is checked. If state happens to be WaitingToConnect, it returns the exception I'm getting.
How can this happen though? If pipeClient.Connect(20000); happened without exceptions, then the state has to be different, whether it is Connected, Broken or something else.
I'm trying to communicate between a web browser client and an ASP.NET server using WebSockets.
I make a set of requests, of different sizes and with some seconds of elapsed time between each of them. The three first ones pass correctly, but a precise one, with nothing in particular from the other, close the WebSocket connection, throw an exception on server side.
The error message and stack trace of this exception look like this :
FATAL ERROR: Cannot access a disposed object.
Object name: 'System.Web.WebSockets.AspNetWebSocket'.
at System.Web.WebSockets.AspNetWebSocket.ThrowIfDisposed()
at System.Web.WebSockets.AspNetWebSocket.SendAsyncImpl(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken, Boolean performValidation)
at System.Web.WebSockets.AspNetWebSocket.SendAsync(ArraySegment 1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken)
at [my code path here...]
It may be a threading problem, because I'm using async methods everywhere from functions that communicate with websockets.
I know the exception is throwed from this code (at socket.SendAsync):
public class SocketTranslater
{
private WebSocket socket;
private JavaScriptSerializer serializer;
// [...]
private ArraySegment<byte> Encode(Object data)
{
string json = serializer.Serialize(data);
return (new ArraySegment<byte>(Encoding.UTF8.GetBytes(json)));
}
public async Task Send(Object packet)
{
ArraySegment<byte> encoded = this.Encode(packet);
await socket.SendAsync(encoded, WebSocketMessageType.Text, true, CancellationToken.None);
}
}
The socket is created from another class :
public class EventSender : IHttpHandler
{
private static List<SocketTranslater> socketTranslaters = new List<SocketTranslater>();
public void ProcessRequest(HttpContext context)
{
this.id = context.Request.UserHostName + ":" + context.Request.UserAgent;
if (context.IsWebSocketRequest)
{
context.AcceptWebSocketRequest(ProcessSocket);
}
}
private async Task ManageSocket(AspNetWebSocketContext context)
{
WebSocket socket = context.WebSocket;
SocketTranslater translater = new SocketTranslater(socket);
translaters.add(translater);
while (true)
{
// Manage messages, using the other class to send responses.
translater.Send(/* Any struct here */);
}
}
Unhopefully, my project is too big to put all the code here.
Any idea of error source, or additional information that you would require ?
UPDATE:
After some more tests, I don't have this exception from time to time. I always do the same thing, but the server seems to have a random comportment.
That makes my problem even weirder...
Finally, after some more tests and interesting questions and answer from here (like this one), I understood:
My problem was that I was stocking WebSockets in a Dictionary linked with hostnames. So on the first connection of a client on my server, everything worked correctly. But if I refresh the page, the websocket was closed by the server (because there was no chance to use it again) et another one was created.
But as I used the same key for both sockets, the deprecated and the new one, I was trying to answer to the new client with the previous socket, that was closed. (disposed = closed for ASP.NET).
So the only thing that I had to do is to remove a socket from the list on the client disconnection (at the end of ManageSocket). And forbid a client to connect twice with the same hostname.
I didn't mention the part where I was linking sockets with hostnames, so I admit you couldn't really help me... I apologize.
I have made a game that connects to a server to get some high scores. It works fine in the first connection (and then I close the datawriter/reader, and the streamsocket as the connection is supposed to retrieve data in one shot).
But after I attempt a second connection I get a "InvalidOperationException was unhandled by user code". A method was called at a unexpected time.
I used the template for the client connection from the following:
StreamSocket sample
So how can I kill all of the all the threads in the thread pool after this operation, since I think that is what is blocking the second connection?
Also this seems to be a timing issue since it will work on the second connection if I set a break point at the loadasync methods, and step through the code?
Thanks
public async void ScoreUpdate(string input)
{
DataWriter writer = new DataWriter(socket.OutputStream);
DataReader reader = new DataReader(socket.InputStream);
string stringToSend = input;
writer.WriteString(stringToSend);
try
{
uint returnLength = await writer.StoreAsync();
uint sizeFieldCount = await reader.LoadAsync(4);
if (sizeFieldCount != sizeof(uint))
{
// The underlying socket was closed before we were able to read the whole data.
return;
}
uint stringLengthSize = reader.ReadUInt32();
uint stringLength = await reader.LoadAsync((uint)stringLengthSize -4);
UpdateScore = reader.ReadString(stringLength);
socket.Dispose();
socket = null;
}
catch (Exception exception)
{
// If this is an unknown status it means that the error if fatal and retry will likely fail.
if (SocketError.GetStatus(exception.HResult) == SocketErrorStatus.Unknown)
{
throw;
}
socket.Dispose();
socket = null;
}
}
}
I try to build a simple async net tcp wcf tool which will open connection, send command, receive answer (a List with 0-10 string sentences), close connection.
The problem is, I get on (self-hosted) service side always - no matter what I try - "The I/O operation has been aborted because of either a thread exit or an application request", on client side of course the corresponding errors like "Existing connection was closed by remote host" and timeouts and so on.
I tried alot for the past days but I can't get rid of it.
Client Side (running on .NET 4.0, called around once a sec):
void callservice(string mykey) {
ServiceReference1.Client c = new ServiceReference1.Client();
c.GetDataCompleted += c_GetDataCompleted;
try {
c.GetDataAsync(mykey);
}
catch (FaultException aa)
{
c.Abort();
}
}
private void c_GetDataCompleted(object sender, ServiceReference1.GetDataCompletedEventArgs e)
{
ServiceReference1.Client c = (ServiceReference1.Client)sender;
c.GetDataCompleted -= c_GetDataCompleted;
try
{
if (e.Result != null && e.Result.Length > 0)
{
... }
c.Close();
}
catch (Exception) {
c.Abort();
}
}
Server Side (running on .NET4.5):
[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple,
InstanceContextMode=InstanceContextMode.PerCall,IncludeExceptionDetailInFaults=true)]
public class Service1 : IMyService
{
public async Task<List<string>> GetData(string whatkey)
{
List<string> mydatalist = new List<string>();
mydatalist= await Task.Run<List<string>>(() =>
{
...
});
return mydatalist;
}
What is going wrong there? Could it be that it is something not having to do with WCF at all? What could it be?
Server Side Exception:
System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
The I/O operation has been aborted because of either a thread exit or an application request
at System.ServiceModel.Channels.SocketConnection.HandleReceiveAsyncCompleted()
at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
System.Net.Sockets.SocketException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
3E3
One more interesting fact:
SVCLogs show me that the I/O Exeption occurs after a timespan I can define in the
<connectionPoolSettings groupName="default" leaseTimeout="00:03:00"
idleTimeout="00:02:39" maxOutboundConnectionsPerEndpoint="20" />
settings.
In this example it will occur the first time after 00:02:39.
My interpretation: It closes open connections due to the settings there and that causes the Exception since the ReceiveAsync operation i.ex. was still open.
My question is so far why does client.close() not close it completely and why isn't it finished yet when it is calling the c_getdatacompleted-event? Why does the operation "hang out" for 02:39 minutes and does not come to an end?
(If I would not force the close down via the connectionpool settings I end up with hundreds of open operations if I use netstat i.ex. to display)
Async WCF operations (AsyncPattern=true) are implemented with the Asynchronous Programming Model. That is, you implement an operation ("Operation") with two asynchronous operations ("BeginOperation" and "EndOeration"). The client can wrap those operations with a Task (presumably with the FromAsync overload)
For example:
[ServiceContract]
public interface ISampleTaskAsync
{
[OperationContract(AsyncPattern = true)]
IAsyncResult BeginDoWork(int count, AsyncCallback callback, object state);
int EndDoWork(IAsyncResult result);
}
The WCF contract does not return a Task<T>
Then, on the client you could do something like:
var proxy = new Services.SampleTaskAsyncClient();
object state = "This can be whatever you want it to be";
var task = Task<int>.Factory.FromAsync(proxy.BeginDoWork,
proxy.EndDoWork, 10, state);
For more information see:
http://msdn.microsoft.com/en-us/library/system.servicemodel.operationcontractattribute.asyncpattern.aspx
http://blogs.msdn.com/b/rjacobs/archive/2011/06/30/how-to-implement-a-wcf-asynchronous-service-operation-with-task-lt-t-gt.aspx
If you want to use Task<T>, I believe you don't need AsyncPattern=true.