I have a library that is connected to some network service using TCP sockets and randomly receives a data from it. I need to process these data line by line and for that I have 2 options:
Create a new thread (I don't want to do that) in which I have never ending loop which calls StreamReader.ReadLine() (I don't want to spawn new threads as this is a library and threads should be fully under control of main program)
Using async callback which gets called every time some data arrives to stream buffer. I currently use this option, but I am having troubles getting lines only. I hacked out this simple solution:
// This function reset the callback after it's processed
private void resetCallback()
{
if (this.networkStream == null)
return;
if (!string.IsNullOrEmpty(this.lineBuffer) && this.lineBuffer.EndsWith("\n"))
{
this.processOutput(this.lineBuffer);
this.lineBuffer = "";
}
AsyncCallback callback = new AsyncCallback(OnReceive);
this.networkStream.BeginRead(buffer, 0, buffer.Length, callback, this.networkStream);
}
// This function gets called every time some data arrives to buffer
private void OnReceive(IAsyncResult data)
{
if (this.networkStream == null)
return;
int bytes = this.networkStream.EndRead(data);
string text = System.Text.Encoding.UTF8.GetString(buffer, 0, bytes);
if (!text.Contains("\n"))
{
this.lineBuffer += text;
}
else
{
List<string> parts = new List<string>(text.Split('\n'));
while (parts.Count > 0)
{
this.lineBuffer += parts[0];
if (parts.Count > 1)
{
this.processOutput(this.lineBuffer + "\n");
this.lineBuffer = "";
}
parts.RemoveAt(0);
}
}
this.resetCallback();
}
As you can see I am using very nasty solution where I am basically checking in every "packet" of data that are received on buffer:
Whether data in buffer are whole line (ends with new line)
Whether data in buffer contains more than 1 line (new line is somewhere in middle of data, or there are more than 1 new line)
Data in buffer contains only a part of a line (no new line in text)
The problem here is that callback function can be called any time when some data are received, and these data can be a line, part of a line, or even multiple lines.
Based on the new line I am storing data in another buffers and when I finally get a new line, I process it somehow.
This is actually working just fine, but I am still wondering if there isn't a better solution that is more clean and doesn't require such a hacking in order to read the stream line by line without using threads?
Please note commenter Damien_The_Unbeliever's point about the issue with partial UTF8 characters. As he says, there's nothing in TCP that would guarantee that you only receive whole characters; a sequence of bytes in the stream can be interrupted at any point, including mid-character.
The usual way to address this would be to using an instance of a Decoder (which you can retrieve from the appropriate Encoding subclass, e.g. Encoding.UTF8.GetDecoder()). A decoder instance will buffer characters for you, returning only whole characters as they are available.
But in your case, there is a much easier way: use the TextReader.ReadLineAsync() method.
For example, here's an asynchronous method which will process each line of text read from the stream, with the returned task for the method completing only when the stream itself has reached the end (i.e. graceful closure of the socket):
async Task ProcessLines()
{
using (StreamReader reader = new StreamReader(
this.networkStream, Encoding.UTF8, false, 1024, true))
{
string line;
while ((line = await reader.ReadLineAsync()) != null)
{
this.processOutput(line);
}
}
// Clean up here. I.e. send any remaining response to remote endpoint,
// call Socket.Shutdown(SocketShutdown.Both), and then close the
// socket.
}
You would call that (preferably awaiting the result in another async method, though that would depend on the exact context of the caller) from wherever you call resetCallback() now. Given the lack of a good, minimal, complete code example a more specific explanation than that can't be provided.
The key is that, being an async method, the method will return as soon as you call ReadLineAsync() (assuming the call doesn't complete immediately), and will resume execution later once that operation completes, i.e. a complete line of text is available and can be returned.
This is the standard idiom now in C# for dealing with this kind of asynchronous operation. It allows you to write the code practically as if you are doing everything synchronously, while the compiler rewrites the code for you to actually implement it asynchronously.
(As an aside: you may want to consider using the usual .NET conventions, i.e. Pascal casing, for method names, instead of the Java-style camel-casing. That will help readers more readily understand your code examples).
Related
I am new here and by no means an expert at c# programming.
I am writing an application that connects to a device over TCP. It sends the device a command and the device responds. Sometimes The device will send another message after it has responded to my command. For example if I say "Read Tag" It will respond with the tag value "Tag: abcdefg". But sometimes, after a couple of hundred milliseconds, it will respond with something like "Buffer Low: 14" telling me the size of its buffer.
Here is how I am currently receiving data:
public Task<string> ReceiveDataAsync()
{
receiveBuffer = new byte[receiveBufferSize];
Task<int> streamTask = _networkstream.ReadAsync(receiveBuffer, 0, receiveBufferSize);
// Since the read is async and data arrival is unknown, the event
// must sit around until there is something to be raised.
var resultTask = streamTask.ContinueWith<String>(antecedent =>
{
Array.Resize(ref receiveBuffer, streamTask.Result); // resize the result to the size of the data that was returned
var result = Encoding.ASCII.GetString(receiveBuffer);
OnDataReceived(new TCPEventArgs(result));
return result;
});
return resultTask;
}
I am confused about reading the network stream. When I use the ReadAsync method, and then I get something back, how do I handle the delay? In my mind, I get the first response of the tag data, then I start to work on that task. Even though I work on the task ".ContinueWith" will my stream continue to receive data? Will the task automatically go back and process more data as it comes in the stream? Do I need to call the ReceiveDataAsync method every time I think some data should be arriving or will it remain open until Dispose of the stream?
Yes, you need to call ReceiveDataAsync repeatedly, usually call it in callback of ContinueWith, or just put it in a loop if you use async/await, so that you read some data, process it and then go back to read (or wait) the next bytes.
Like this:
private static void OnContinuationAction(Task<string> text)
{
Console.WriteLine(text);
ReceiveDataAsync().ContinueWith(OnContinuationAction);
}
...
ReceiveDataAsync().ContinueWith(OnContinuationAction);
Or with async/await:
private async void ReceiveDataContinuously()
{
while(true)
{
var text = await ReceiveDataAsync();
Console.WriteLine(text);
}
}
If you don't call ReadAsync on the stream repeatedly, as long as the underlying TCP connection is open it will continue receiving data into the buffer, but your program cannot get them.
My current named pipe implementation reads like this:
while (true)
{
byte[] data = new byte[256];
int amount = pipe.Read(data, 0, data.Length);
if (amount <= 0)
{
// i was expecting it to go here when a client disconnects but it doesnt
break;
}
// do relevant stuff with the data
}
how can I correctly detect when a client disconnects?
Set a read timeout and poll the NamedPipeClientStream.IsConnected flag when a timeout occurs.
A Read Timeout will cause reads that are idle for the timeout duration to throw InvalidOperationException
If you are not reading, and want to detect disconnections, call this method on a worker thread for the lifetime of your pipe connection.
while(pipe.IsConnected && !isPipeStopped) //use a flag so that you can manually stop this thread
{
System.Threading.Thread.Current.Sleep(500);
}
if(!pipe.IsConnected)
{
//pipe disconnected
NotifyOfDisconnect();
}
One easy way to tell if your pipe has been broken (remotely) is to always use asynchronous reads instead of sync ones, and to always have at least one read submitted asynchronously. That is, for every successful read you get, post another async read, whether you intend to read another or not. If you close the pipe, or the remote end closes it, you'll see the async read complete, but with a null read size. You can use this to detect a pipe disconnection. Unfortunately, the pipe will still show IsConnected, and you still need to manually close it, I think, but it does allow you to detect when something went wonky.
Use WaitForPipeDrain() method after Writing to the Pipe (using WriteByte() or Write()) and catch the exception which is "Pipe is Broken".
You may want to put that in a while loop and keep writing to the pipe.
in the case of Synchronous call you track the -1 return by ReadByte of Stream abstract class, which is inherited by NamedPipeServerStream:
var _pipeServer = new NamedPipeServerStream(PipeConst._PIPE_NAME, PipeDirection.InOut);
int firstByte = _pipeServer.ReadByte();
const int END_OF_STREAM = -1;
if (firstByte == END_OF_STREAM)
{
return null;
}
The docs states indeed:
//
// Summary:
// Reads a byte from a pipe.
//
// Returns:
// The byte, cast to System.Int32, or -1 indicates the end of the stream (the pipe
// has been closed).
public override int ReadByte();
Only after a first failed read your IsConnected property will be correctly set to false:
_pipeServer.IsConnected
You might observe that even on the
official illustration of Microsoft (and more precisely in the StreamString class)
this check is not done:
Do not forget to Vote for this answer and visit my Youtube channel. More info on my profile.
Regards !
I am pretty new to coding with some experience in ASM and C for PIC. I am still learning high level programming with C#.
Question
I have a Serial port data reception and processing program in C#. To avoid losing data and knowing when it was coming, I set a DataReceived event and loop into the handling method until there were no more bytes to read.
When I attempted this, the loop continued endlessly and blocked my program from other tasks (such as processing the retrieved data) when I continuously received data.
I read about threading in C#, I created a thread that constantly checks for SerialPort.Bytes2Read property so it will know when to retrieve available data.
I created a second thread that can process data while new data is still being read. If bytes have been read and ReadSerial() has more bytes to read and the timeout (restarted every time a new byte is read from the serial) they can still be processed and the frames assembled via a method named DataProcessing() which reads from the same variable being filled by ReadSerial().
This gave me the desired results, but I noticed that with my solution (both ReadSerial() and DataProcessing() threads alive), CPU Usage was skyrocketed all the way to 100%!
How do you approach this problem without causing such high CPU usage?
public static void ReadSerial() //Method that handles Serial Reception
{
while (KeepAlive) // Bool variable used to keep alive the thread. Turned to false
{ // when the program ends.
if (Port.BytesToRead != 0)
{
for (int i = 0; i < 5000; i++)
{
/* I Don't know any other way to
implement a timeout to wait for
additional characters so i took what
i knew from PIC Serial Data Handling. */
if (Port.BytesToRead != 0)
{
RxList.Add(Convert.ToByte(Port.ReadByte()));
i = 0;
if (RxList.Count > 20) // In case the method is stuck still reading
BufferReady = true; // signal the Data Processing thread to
} // work with that chunk of data.
BufferReady = true; // signals the DataProcessing Method to work
} // with the current data in RxList.
}
}
}
I can not understand completely what you are meaning with the "DataReceived" and the "loop". I am also working a lot with Serial Ports as well as other interfaces. In my application I am attaching to the DataReceived Event and also reading based on the Bytes to read, but I dont use a loop there:
int bytesToRead = this._port.BytesToRead;
var data = new byte[bytesToRead];
this._port.BaseStream.Read(data , 0, bytesToRead);
If you are using a loop to read the bytes I recommend something like:
System.Threading.Thread.Sleep(...);
Otherwise the Thread you are using to read the bytes is busy all the time. And this will lead to the fact that other threads cannot be processed or your CPU is at 100%.
But I think you don't have to use a loop for polling for the data if you are using the DataReceived event. If my undertanding is not correct or you need further information please ask.
I have a problem with the SocketAsyncEventArgs class..the problem is when I try to send 8K of data
for example over internet, the socket sometimes only sends 1K or 2K, I know that this is normal for
TCP socket and that one send doesn't guarantee one receive.
now for this to work for me I modified my code to resend the remaining data, for example
when I the SocketAsyncEventArgs.SendAsync completes, in the callback i check whether it sent all the 8K or not if it's not, I call the SocketAsyncEventArgs.SendAsync again with the remaining data untill
I send it all.
Now, when I looked at some SocketAsyncEventArgs code.. I saw that most people don't do so!
and they just clean up when the send complete without checking if it sent all the data or not!
also when I looked at Microsoft's example, they were saying the ONE call to SocketAsyncEventArgs.SendAsync guarantees that all the data will be sent.
I mean I tested it myself and NO all data will not be sent in one call to SocketAsyncEventArgs.SendAsync.
What I'm doing wrong ?
thanks in advance.
Edit:
Here is the code which doesn't send all data(exactly like microsoft's)
the SendAsyncComplete will be called when the socket sends for example 1Kb of data
not all 8K!
public virtual void Send(byte[] packet, int offset, int length)
{
if (_tcpSock != null && _tcpSock.Connected)
{
var args = SocketHelpers.AcquireSocketArg();
if (args != null)
{
args.Completed += SendAsyncComplete;
args.SetBuffer(packet, offset, length);
args.UserToken = this;
var willRaiseEvent = _tcpSock.SendAsync(args);
if (!willRaiseEvent)
{
ProcessSend(args);
}
unchecked
{
_bytesSent += (uint)length;
}
Interlocked.Add(ref _totalBytesSent, length);
}
else
{
log.Error("Client {0}'s SocketArgs are null", this);
}
}
}
private static void ProcessSend(SocketAsyncEventArgs args)
{
args.Completed -= SendAsyncComplete;
SocketHelpers.ReleaseSocketArg(args);
}
private static void SendAsyncComplete(object sender, SocketAsyncEventArgs args)
{
ProcessSend(args);
}
There are many things I would change there. As a preamble, read this.
First of all, whenever you sent any data on the socket, you must process that event: either you stop the whole send process, or you issue another socket send operation to send the remaining data.
So it makes sense to have 3 methods, like this:
// This is the Send() to be used by your class' clients
1) public void Send(byte[] buffer);
This method will take care to apply any data formatting that you need, create (retrieve) a SocketAsyncEventArgs object, set the token to hold your buffer, and call the next method bellow:
2) private void Send(SocketAsyncEventArgs e);
This one actually calls
Socket.SendAsync(SocketAsyncEventArgs e)
and copies the contents from the token (the buffer, remember?) to the SAEA object. Now that is why because method number (2) might be called several times to send the remaining data that couldn't be sent in one operation by the socket. So here you copy the remaining data from the token to the SAEA buffer.
3) private void ProcessSent(SocketAsyncEventArgs e);
This last method will examine the data that has been sent by the socket. If all the data has been sent, the SAEA object will be released. If not, method (2) will be called again for the rest of the data. In order to keep track of sent data, you use SAEA.BytesTransferred. You should add this value to a value stored in the custom token I advise you to create (so do not use "this" as a token).
This is where you also check for SocketError on the SAEA parameter.
This last method will be called in two places:
in the 2nd method, like this:
// Attempt to send data in an asynchronous fashion
bool isAsync = this.Socket.SendAsync(e);
// Something went wrong and we didn't send the data async
if (!isAsync)
this.ProcessSent(e);
This bit is important any many people missed it even when using the more traditional Begin/EndXXX pattern (in that case, via IAsyncResult). If you don't place it, once in a while (quite rare), a StackOverflow exception will pop out of nowhere and will keep you puzzled for long time.
in the Completed event handler:
private void Completed(object sender, SocketAsyncEventArgs e)
{
// What type of operation did just completed?
switch (e.LastOperation)
{
case SocketAsyncOperation.Send:
{
ProcessSent(e);
break;
}
}
}
The tricky thing is to use one SocketAsyncEventArgs object per 1st Send(byte[]) operation, and release it in 3rd operation, if all data has been sent.
For that, you must create a custom token (class or immutable struct) to place in SocketAsyncEventArgs.UserToken inside the 1st method, and then keep track of how much data you have transferred on each Socket.SendAsync() operation.
When you are reading the article provided in the beginning, notice how the writer reuses the same SAEA object when at the end of Send() operation proceeds with a Receive() operation, if all the data has been sent. That is because his protocol is: each one of the parties (server and client) talk to each other in turns.
Now, should multiple calls to the 1st Send() method occur at the same time, there is no rule in which order they will be handled by the OS. If that is likely to happen and message order is important, and since any call to Send(byte[]) from an "outside entity" result in a Socket.SendAsync(), I suggest that the 1st method actually writes down the received bytes in an internal buffer. As long as this buffer is not empty, you keep sending this data internally. Think of it like a producer-consumer scenario, where the "outside entity" is the producer and the internal send op is the consumer. I prefer an optimistic concurrency scenario here.
The documentation on the matter is rather shallow, with the exception of this article, and even in this case, when you start implementing your own application, some things turn out to be different. This SocketAsyncEventArgs model is arguably a bit counter-intuitive.
Let me know if you need more help, I had my times of struggle with this a while ago when I developed my own library.
Edit:
If I were you, I would move
unchecked
{
_bytesSent += (uint)length;
}
Interlocked.Add(ref _totalBytesSent, length);
to your ProcessSend(SAEA), and use args.BytesTransferred instead of "length".
When using C# NamedPipeServerStream, in case a client doesn't send any message-end-pattern (like \r\n when server reads with ReadLine()) NamedPipeServerStream Read methods will wait forever and no Abort() or Interupt() methods will work on that thread.
Since:
1) Stream.ReadTimeout not supported for NamedPipeServerStream
2) Abort() or Interupt() doesn't work on thread
3) NamedPipeServerStream.Disconnect() nether work
It is unclear, how to setup timeout on NamedPipeServerStream read operations?
Let me introduce an example. The specification of IPC we have require an exchange of \0-terminated strings. A client sends message, the server processes the message and as 'a must' sends a response.
If the client doesn't send \0 in the end (client is not ours so we can't guarantee correctness of its working), the Read method will wait forever and client (since we don't control it) may wait forever for a response too.
Next is a simplified example of an implementation:
public void RestartServer()
{
_pipeServerThread.Interrupt(); //doesn't affect Read wait
_pipeServerThread.Abort(); //doesn't affect Read wait
}
private void PipeServerRun(object o) //runs on _pipeServerThread
{
_pipeServer = new NamedPipeServerStream(_pipeName, InOut, 100,
PipeTransmissionMode.Message, PipeOptions.WriteThrough);
//_pipeServer.ReadTimeout = 100; //System.InvalidOperationException: Timeouts are not supporte d on this stream.
// Wait for a client to connect
while (true)
{
_pipeServer.WaitForConnection();
string request = ReadPipeString();
//... process request, send response and disconnect
}
}
/// <summary>
/// Read a \0 terminated string from the pipe
/// </summary>
private string ReadPipeString()
{
StringBuilder builder = new StringBuilder();
var streamReader = new StreamReader(_pipeServer);
while (true)
{
//read next byte
char[] chars = new char[1];
streamReader.Read(chars, 0, 1); // <- This will wait forever if no \0 and no more data from client
if (chars[0] == '\0') return builder.ToString();
builder.Append(chars[0]);
}
}
So how to set timeout on NamedPipeServerStream read operations?
Since you are running the pipe in message mode, you should first read the whole message into a byte[] buffer or a memory stream and then decide whether it's valid and decode it. Pipe messages have a definite length. It cannot be retrieved explicitly, but it shows up when you are reading from a message-mode pipe. Win32 ReadFile fails with ERROR_MORE_DATA if there still are unread bytes in the message, then it returns TRUE to indicate that the message is over. After this, a call to ReadFile will block until a new message is available. StreamReader naturally doesn't know any of this and blocks your thread.
Update: to implement timeouts, use asynchronous I/O (Stream.BeginRead). StreamReader does not support this directly. If you absolutely must use it, write a wrapper stream which will implement Read in terms of BeginRead on the underlying stream and support timeouts, cancellation etc.
Try setting NamedPipeServerStream.ReadMode and/or .TransmissionMode to Byte.
Regardless of these you should use the available BeginRead / EndRead methods with NamedPipeServerStream.
This way you can implement the timeout logic yourself.