I am using System.IO.Pipelines to read from a web socket and bumped into a problem where PipeReader.ReadAsync is not getting unblocked, with the example code I can clearly see that the full messages are read from the web socket and flushed BUT the reader is not unblocked.
Could someone point what could be wrong with my implementation
protected async Task FillTask(CancellationToken cancellationToken = default)
{
var writer = _input.Writer;
try
{
IsReceiving = true;
while (_webSocket.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested)
{
//get buffer for writing
var buffer = writer.GetMemory(1024);
//get the array segment from the buffer
MemoryMarshal.TryGetArray(buffer, out ArraySegment<byte> segment);
//write data received from socket into the buffer
var result = await _webSocket.ReceiveAsync(segment, cancellationToken).ConfigureAwait(false);
Debug.WriteLine($"Read result {result.MessageType}");
Debug.WriteLine($"Read {result.Count} from socket");
//advance reader
writer.Advance(result.Count);
//if full message received flush the data to the reader
if (result.EndOfMessage)
{
writes++;
Debug.WriteLine($"Writes {writes}");
var flushTask = await writer.FlushAsync(cancellationToken).ConfigureAwait(false);
if (flushTask.IsCompleted)
{
break;
}
}
}
}
catch (Exception ex)
{
RaiseExceptionEvent(ex);
}
finally
{
await writer.CompleteAsync().ConfigureAwait(false);
IsConnected = false;
IsReceiving = false;
}
}
protected async Task ReadTask(PipeReader reader, CancellationToken cancellationToken = default)
{
//loop and read data from reader as long as its required
while (true && !cancellationToken.IsCancellationRequested)
{
//read from reader
ReadResult result = await reader.ReadAsync(cancellationToken).ConfigureAwait(false);
//get reader buffer
var buffer = result.Buffer;
//check if any data where read
if (buffer.Length > 0)
{
//rent byte array
var array = _pool.Rent((int)buffer.Length);
try
{
//copy the buffer to newly created array
result.Buffer.CopyTo(array);
//advance reader
reader.AdvanceTo(result.Buffer.End,result.Buffer.End);
receives++;
Debug.WriteLine($"Receives {receives}");
//raise event and process the data
RaiseDataReceivedEvent(default, array, 0, (int)result.Buffer.Length);
Debug.WriteLine($"Receives {receives} - Sends {sends}");
BytesReceived += (ulong)buffer.Length;
}
catch
{
throw;
}
finally
{
//return the array to the pool
_pool.Return(array);
}
}
// stop reader once there is no more data to read
if (result.IsCompleted)
break;
}
await reader.CompleteAsync().ConfigureAwait(false);
}
Edit
Setting the PipeOptions to PipeScheduler.Inline for reader resolves the problem, would be interesting to know why.
Related
I am testing a very simple web server code (either copied it from MSDN sample or somewhere I forgot), I am running it on Raspberry Pi 3 Model B with IoT Core v.10.0.16299.309. I did everything properly including disposing the socket on each request and the code works most of the time.
I only have ONE client, sending 3 very simple HTTP GET requests to this web service every 30 seconds, just a simple URL like this: http://serverIP/get?id=123 It returns the result properly but RANDOMLY during the hour, the request object is empty for some strange reason. I couldn't figure it out why.
private StreamSocketListener _listener;
private int _bufferSize = 8192;
public async void Run(IBackgroundTaskInstance taskInstance)
{
taskInstance.GetDeferral();
_listener = new StreamSocketListener();
_listener.ConnectionReceived += HandleRequest;
await _listener.BindServiceNameAsync("9080");
}
public async void HandleRequest(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
{
try
{
StringBuilder request = new StringBuilder()
using (IInputStream input = args.Socket.InputStream)
{
byte[] data = new byte[_bufferSize];
IBuffer buffer = data.AsBuffer();
uint bytesRead = _bufferSize;
while (bytesRead == _bufferSize)
{
await input.ReadAsync(buffer, _bufferSize, InputStreamOptions.Partial);
request.Append(Encoding.UTF8.GetString(data, 0, data.Length));
bytesRead = buffer.Length;
// Some other code here to process the request object
// and respond it back to the client. They are irrelevant.
} // while
} // using
} // catch
catch (Exception ex)
{
// Log here
}
finally
{
args.Socket.Dispose();
}
} // method
I thought that there is something wrong with Threading, so I disabled most of the async and await. Interestingly, the result is A LOT BETTER. The request object is 99% fine. Within 12 hours, there were only 3-4 times where the request was empty. I don't think this is the right solution... but I really get stuck on this one and couldn't figure out why. Any help is appreciated.
private StreamSocketListener _listener;
private int _bufferSize = 8192;
public async void Run(IBackgroundTaskInstance taskInstance)
{
taskInstance.GetDeferral();
_listener = new StreamSocketListener();
_listener.ConnectionReceived += HandleRequest;
_listener.BindServiceNameAsync("9080"); // ** No "await"
}
// ** No "async"
public void HandleRequest(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
{
try
{
StringBuilder request = new StringBuilder()
using (IInputStream input = args.Socket.InputStream)
{
byte[] data = new byte[_bufferSize];
IBuffer buffer = data.AsBuffer();
uint bytesRead = _bufferSize;
while (bytesRead == _bufferSize)
{
// ** No "await"
input.ReadAsync(buffer, _bufferSize, InputStreamOptions.Partial);
request.Append(Encoding.UTF8.GetString(data, 0, data.Length));
bytesRead = buffer.Length;
// Some other code here to process the request object
// and respond it back to the client. They are irrelevant.
} // while
} // using
} // catch
catch (Exception ex)
{
// Log here
}
finally
{
args.Socket.Dispose();
}
} // method
The IInputStream.ReadAsync() documentation has some interesting information that might help:
Always read data from the buffer returned in the
IAsyncOperationWithProgress(IBuffer, UInt32). Don't assume that the
input buffer contains the data. Depending on the implementation, the
data that's read might be placed into the input buffer, or it might be
returned in a different buffer. For the input buffer, you don't have
to implement the IBuffer interface. Instead, you can create an
instance of the Buffer class.
The provided buffer might not contain the data.
You can either use the return buffer and extract the data from it or use DataReader.ReadBuffer() (or DataReader.ReadString() in your case) to get the data from the stream.
In general, it is easier to use a DataReader to read the data from a stream than using the low level API.
You code becomes
byte[] data = new byte[_bufferSize];
IBuffer buffer = data.AsBuffer();
uint bytesRead = _bufferSize;
while (bytesRead == _bufferSize)
{
var readFromStreamBuffer = await input.ReadAsync(buffer, _bufferSize, InputStreamOptions.Partial);
var readBytes = readFromStreamBuffer.ToArray();
request.Append(Encoding.UTF8.GetString(readBytes, 0, readBytes.Length));
bytesRead = readFromStreamBuffer.Length;
}
Or with DataReader.ReadBuffer():
byte[] data = new byte[_bufferSize];
IBuffer buffer = data.AsBuffer();
uint bytesRead = _bufferSize;
var dataReader = new DataReader(input);
while (bytesRead == _bufferSize)
{
await dataReader.LoadAsync(_bufferSize);
var readFromStreamBuffer = dataReader.ReadBuffer();
var readBytes = readFromStreamBuffer.ToArray();
request.Append(Encoding.UTF8.GetString(readBytes, 0, readBytes.Length));
bytesRead = readFromStreamBuffer.Length;
}
Or with DataReader.ReadString():
byte[] data = new byte[_bufferSize];
IBuffer buffer = data.AsBuffer();
uint bytesRead = _bufferSize;
var dataReader = new DataReader(input);
dataReader.UnicodeEncoding = UnicodeEncoding.Utf8;
while (bytesRead == _bufferSize)
{
await dataReader.LoadAsync(_bufferSize);
request.Append(dataReader.ReadString(dataReader.UnconsumedBufferLength));
bytesRead = readFromStreamBuffer.Length;
}
Ran out of idea, I changed my code to use Socket.InputStream.AsStreamForRead() instead, then use .ReadLineAsync() just to read the first line only for testing. It seemed to be stable, passing 12+ hours there wasn't an empty request!!!
For what I need, this is actually fine because I am not really building a full function web server, so I don't need to get the complete request. All I need is to get the HTTP GET QueryString and pass the parameters to the GPIO of my Raspberry Pi.
As Vincent suggested, .ReadAsync() has some very interesting behavior. I will definitely try his suggestion later.
public async void HandleRequest(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
{
string requestString;
try
{
Stream inStream = args.Socket.InputStream.AsStreamForRead();
StreamReader reader = new StreamReader(inStream);
requestString = await reader.ReadLineAsync();
// Some other code here to process the request object
// and respond it back to the client. They are irrelevant
} // catch
catch (Exception ex)
{
// Log here
}
finally
{
args.Socket.Dispose();
}
}
I'm trying to receive packets and if no bytes are received continue with the code below. await ReadAsync blocks until a new packet is received. Is there any way to just read the current bytes received?
If I don't use await messages aren't received.
byte[] data = new byte[BufferSize];
IInputStream inputStream = socket.InputStream;
IBuffer buffer = data.AsBuffer();
socketInformation.GetStopwatchPingIdleTime().Start();
while (socketInformation.open)
{
try
{
inputStream.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
data = buffer.ToArray();
}
catch (Exception)
{
break;
}
while (true)
{
// Wait for payload size
if (buffer.Length >= 4)
{
int commandType = (int)BitConverter.ToInt16(data, 0);
int payloadSize = (int)BitConverter.ToInt16(data, 2);
int packetSize = PacketHeaderSize + payloadSize;
// Wait for a full message
if (buffer.Length >= packetSize)
{
byte[] packet = new byte[packetSize];
System.Buffer.BlockCopy(data, 0, packet, 0, packetSize);
ParsePacketSequence(socket, socketInformation, packet);
if (buffer.Length > packetSize)
{
int bufferLength = (int)buffer.Length - packetSize;
byte[] newData = new byte[BufferSize];
System.Buffer.BlockCopy(data, packetSize, newData, 0, bufferLength);
data = newData;
buffer.Length = (uint)bufferLength;
}
else if (buffer.Length == packetSize)
{
break;
}
else
{
break;
}
}
else if (buffer.Length == packetSize)
{
break;
}
}
else
{
break;
}
}
if (host)
{
// Send ping to player
if (socketInformation.waitingPong == false &&
socketInformation.GetStopwatchPingIdleTime().ElapsedMilliseconds > 5000)
{
byte[] pingPacket = CreatePacket(6, null);
SendPacket(socket, socketInformation, pingPacket);
socketInformation.waitingPong = true;
}
}
await Task.Delay(33, tokenSource.Token);
}
inputStream.Dispose();
socket.Dispose();
tokenSource.Cancel();
It looks to me you are receiving a stream of messages. When a message is there you want to process it potentially at some later time or at a different place in the code.
A good approach for that is to have one Task continuously reading messages from the socket and putting them into a queue. You can then pull complete messages from the queue whenever you like.
That way you can get rid of most of the logic here. You never have to abort a read request and you never need to check for timeouts.
I'm trying to save a zip file stream that I've downloaded from a server.
Now I've the Stream But I'm not able to save to a file. here is my attempt =>
private async Task DownloadCompleted(Stream inputStream, CancellationToken ct)
{
var file = await _downloadFolder.CreateFileAsync(_productDescription.ProductFileName, CreationCollisionOption.ReplaceExisting, ct);
using (Stream str = await file.OpenAsync(FileAccess.ReadAndWrite, ct))
{
await inputStream.CopyToAsync(str);
}
}
I'm trying to do it Xamarin.Android Project, I'm not good at streams, Also some good pointer are highly appreciated.
Edit- here I got the stream
private async Task DownloadFileFromUrl(string url, CancellationToken ct)
{
try
{
var receivedBytes = 0;
using (var client = new WebClient())
using (var stream = await client.OpenReadTaskAsync(url))
{
var buffer = new byte[4096];
var totalBytes = int.Parse(client.ResponseHeaders[HttpResponseHeader.ContentLength]);
for (;;)
{
var bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length, ct);
if (bytesRead == 0)
{
await Task.Yield();
break;
}
receivedBytes += bytesRead;
if (_downloadProgressHandler != null)
{
_downloadProgressHandler((int)(((double)receivedBytes / totalBytes) * 100), false);
}
}
await DownloadCompleted(stream, ct);
}
}
catch (System.OperationCanceledException)
{
throw;
}
catch (Exception ex)
{
Analytics.AddHandledExceptionEvent(ex, "Ex-ProductDownloader-DownloadFileFromUrl");
throw new NetworkNotAvailableException("");
}
}
I'm looking to read data off the wire and send it to a function to parse the contents. Due to the potential varying size of the message (xml), I could read a whole message, more than one message, or a message portion.
I am trying to implement code using the BlockingCollection where I would call TryAdd when I read the data off the wire and use a consumer thread to pull the data off the BlockingCollection to parse. The examples seem pretty straight-forward, but they only seem to work once, then exit. I want the consumer to continuously parse as messages come in. See code below for what i am currently doing.
handling of messages:
private static BlockingCollection<byte[]> queue = new BlockingCollection<byte[]>();
public XmlHandler()
{
CancellationTokenSource cts = new CancellationTokenSource();
Task.Factory.StartNew(() =>
{
if (Console.ReadKey().KeyChar == 'c')
cts.Cancel();
});
Task.Factory.StartNew(() => ParserWorker(queue, cts.Token));
}
//run producer
public void AddData(byte[] data, int bytesRead)
{
bool success = false;
try
{
success = queue.TryAdd(data);
}
catch (OperationCanceledException)
{
Console.WriteLine("Add loop canceled.");
queue.CompleteAdding();
}
if (success)
{
Console.WriteLine(" Add:{0}", data);
}
else
{
Console.Write(" AddBlocked");
}
System.Console.WriteLine("queue count = " + queue.Count);
}
private static void ParserWorker(BlockingCollection<byte[]> bc, CancellationToken ct)
{
ASCIIEncoding encoder = new ASCIIEncoding();
String xmlString;
while (!bc.IsCompleted)
{
byte[] nextItem;
try
{
if (!bc.TryTake(out nextItem, 0, ct))
{
Console.WriteLine(" Take Blocked");
}
else
{
xmlString = encoder.GetString(nextItem, 0, nextItem.Length);
System.Console.WriteLine(xmlString);
}
}
catch (OperationCanceledException)
{
Console.WriteLine("Taking canceled.");
break;
}
}
}
Reading off the wire (this runs in a thread):
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[8192];
int bytesRead;
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, 4096);
byte[] temp = new byte[bytesRead];
Array.Copy(message, temp, bytesRead);
/*CODE WILL HANG HERE...*/
ASCIIEncoding encoder = new ASCIIEncoding();
String xmlString = encoder.GetString(message, 0, message.Length);
System.Console.WriteLine(xmlString);
/*DOES NOT GO BEYOND LINE ABOVE */
handler.AddData(message, bytesRead); //xmlhandler
}
catch (Exception e)
{
System.Console.WriteLine(e.ToString());
break;
}
if (bytesRead == 0)
{
break;
}
}
}
So can anyone tell me what I am doing wrong here?
Im trying to set up a named pipe server and client to send data between two programs.
My issue is that when i data is recived eg. BeginRead command om server triggers after i have serialized an object from the client it triggers the callback like 20 times for the same message. The goal is that the client program will send commands to the server program. And when the server processes tasks it will send status updates back to the client when there is one connected.
Here is my current test program.
class Program
{
static void Main(string[] args)
{
var server = new PipeServer();
server.Init();
var client = new PipeClient();
if (client.Connect())
{
Console.WriteLine("Connected to server.");
}
else
{
Console.WriteLine("Connection failed.");
return;
}
while (true)
{
Console.Write(" \\> ");
string input = Console.ReadLine();
if (string.IsNullOrEmpty(input)) break;
var arr = input.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
int value = 0;
if (arr.Length != 2) break;
if (!int.TryParse(arr[1], out value)) break;
var obj = new PipeObject { Name = arr[0], Value = value };
client.Send(obj);
//string result = f.Deserialize(client) as string;
//Console.WriteLine(result);
}
}
}
internal class PipeServer
{
IFormatter Formatter = new BinaryFormatter();
public NamedPipeServerStream Instance { get; internal set; }
public bool IsConnected { get; internal set; }
byte[] buffer = new byte[65535];
public object Message { get; set; }
StreamReader sr;
StreamWriter sw;
internal PipeServer()
{
IsConnected = false;
}
public void Init()
{
var ps = new PipeSecurity();
ps.AddAccessRule(new PipeAccessRule(WindowsIdentity.GetCurrent().User, PipeAccessRights.FullControl, AccessControlType.Allow));
ps.AddAccessRule(new PipeAccessRule(new SecurityIdentifier(WellKnownSidType.AuthenticatedUserSid, null), PipeAccessRights.ReadWrite, AccessControlType.Allow));
Instance = new NamedPipeServerStream("Levscan4Pipe", PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.Asynchronous | PipeOptions.WriteThrough, 65535, 65535, ps);
sr = new StreamReader(Instance);
sw = new StreamWriter(Instance);
Instance.BeginWaitForConnection(OnClientConnected, Instance);
Thread t = new Thread(Run);
t.Start();
}
void Run()
{
int index = 0;
if (IsConnected)
{
try
{
Instance.BeginRead(buffer, 0, buffer.Length, OnRead_Completed, Instance);
//index += Instance.Read(buffer, 0, buffer.Length);
//try
//{
// using (var ms = new MemoryStream(buffer))
// {
// Message = Formatter.Deserialize(ms);
// index = 0;
// }
//}
//catch (Exception e)
//{
// Debug.WriteLine(e.Message);
// Debug.WriteLine(e.StackTrace);
//}
}
catch (IOException)
{
IsConnected = false;
Instance.Disconnect();
}
}
Thread.Sleep(Timeout.Infinite);
//Instance.WaitForConnection();
//Thread t = new Thread(Run);
//t.Start();
}
void OnClientConnected(IAsyncResult ar)
{
Instance.EndWaitForConnection(ar);
IsConnected = true;
}
void OnRead_Completed(IAsyncResult ar)
{
var bytes = Instance.EndRead(ar);
Debug.WriteLine("{1} > Read completed - bytes read: {0}".FormatWith(bytes, DateTime.Now.ToString()));
//try
//{
// using (var ms = new MemoryStream(buffer))
// {
// Message = Formatter.Deserialize(ms);
// }
//}
//catch (Exception e)
//{
// Debug.WriteLine(e.Message);
// Debug.WriteLine(e.StackTrace);
//}
}
}
internal class PipeClient
{
IFormatter f = new BinaryFormatter();
public NamedPipeClientStream Instance { get; internal set; }
StreamWriter sw;
StreamReader sr;
public PipeClient()
{
Instance = new NamedPipeClientStream(".", "Levscan4Pipe", PipeDirection.InOut, PipeOptions.Asynchronous | PipeOptions.WriteThrough);
sr = new StreamReader(Instance);
sw = new StreamWriter(Instance);
}
public bool Connect()
{
try
{
Instance.Connect(5000);
Instance.ReadMode = PipeTransmissionMode.Message;
Instance.WaitForPipeDrain();
return true;
}
catch
{
return false;
}
}
public void Send(object obj)
{
f.Serialize(Instance, obj);
Instance.Flush();
Instance.WaitForPipeDrain();
}
}
Edit
Changed the while loop to a if, to start the BeginRead. This solves the multiple callbacks but i stil dont get a complete message.
If the server is writing to the stream like:
write field 1
write field 2
write field 3
etc.
There is some time between the writes, and the receiver (your program) can be reading the first three fields while the server is still writing the others. The pipe stream doesn't know when the server is finished writing, so it can't buffer everything and send it to you all in one big chunk.
When the server writes everything to a memory stream first and then copies the memory stream to the pipe stream, your program can get it all at once. Maybe. If the server is sending a very large packet, you might read just part of it.
The pipe stream is just a stream of bytes. It doesn't impose any format on the data. It doesn't have any concept of records or anything like that. So you have to treat it like a stream of bytes and do your own composing of records, etc.
If you need to know the size of the record sent from the server, the server has to put that information in the stream for you. Typically, the server will write the length and then the data. The receiver can then read the length, convert it to an integer, and then read that many bytes from the stream. And, yes, it might take multiple reads in order to get all of the bytes. That's just the nature of a byte stream.
The other way to handle this is to have an end-of-record marker. So the server sends its data and your program reads until it finds the byte sequence that signifies the end of the record. You have to be careful, though, because the server could be sending multiple records and your read could grab the end of one record as well as the beginning of the next.
Working with byte streams can be a lot of work because you have to reconstruct records after reading the bytes. It's much easier to use an existing framework (like WCF, as mentioned in one of the comments) if you can.