I have a callback function in which I am trying to write the data that I read in an overriden ReadAsync().
private void StreamCallback(byte[] bytes)
{
Console.WriteLine("--> " + Encoding.UTF8.GetString(bytes)); // the whole application is blocked here, why?
if (OnDataReceived != null)
{
string data = Encoding.UTF8.GetString(bytes);
OnDataReceived(data);
}
}
The overriden ReadAsync() looks as follows.
public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, System.Threading.CancellationToken cancellationToken)
{
var read = await _originalStream.ReadAsync(buffer, offset, count, cancellationToken);
_readCallback(buffer);
return read;
}
What I actually want to achieve is to monitor a network stream just before it gets parsed by an XmlReader. This relates to my other question > Reading from same SslStream simultaneously? <. How would I do that?
UPDATE:
It is actually Encoding.UTF8.GetString(bytes) that is blocking the application. In order for the question to be more complete I am listing the code for reading the XML stream.
using (XmlReader r = XmlReader.Create(sslStream, new XmlReaderSettings() { Async = true }))
{
while (await r.ReadAsync())
{
switch (r.NodeType)
{
case XmlNodeType.XmlDeclaration:
...
break;
case XmlNodeType.Element:
...
Based on the code you posted, StreamCallback() will block until that stream ends. You pass a byte pointer to Encoding.UTF8.GetString(bytes); So, it needs to keep querying bytes until it reaches the end. It will never reach the end since bytes comes from a stream until that stream is closed.
You need to either process your stream a certain number of bytes at a time or until a certain character is seen.
Related
I came across IAsyncEnumerable while I am testing C# 8.0 features. I found remarkable examples from Anthony Chu (https://anthonychu.ca/post/async-streams-dotnet-core-3-iasyncenumerable/). It is async stream and replacement for Task<IEnumerable<T>>
// Data Access Layer.
public async IAsyncEnumerable<Product> GetAllProducts()
{
Container container = cosmosClient.GetContainer(DatabaseId, ContainerId);
var iterator = container.GetItemQueryIterator<Product>("SELECT * FROM c");
while (iterator.HasMoreResults)
{
foreach (var product in await iterator.ReadNextAsync())
{
yield return product;
}
}
}
// Usage
await foreach (var product in productsRepository.GetAllProducts())
{
Console.WriteLine(product);
}
I am wondering if this can be applied to read text files like below usage that read file line by line.
foreach (var line in File.ReadLines("Filename"))
{
// ...process line.
}
I really want to know how to apply async with IAsyncEnumerable<string>() to the above foreach loop so that it streams while reading.
How do I implement iterator so that I can use yield return to read line by line?
Exactly the same, however there is no async workload, so let's pretend
public async IAsyncEnumerable<string> SomeSortOfAwesomeness()
{
foreach (var line in File.ReadLines("Filename.txt"))
{
// simulates an async workload,
// otherwise why would be using IAsyncEnumerable?
// -- added due to popular demand
await Task.Delay(100);
yield return line;
}
}
or
This is just an wrapped APM workload, see Stephen Clearys comments for clarification
public static async IAsyncEnumerable<string> SomeSortOfAwesomeness()
{
using StreamReader reader = File.OpenText("Filename.txt");
while(!reader.EndOfStream)
yield return await reader.ReadLineAsync();
}
Usage
await foreach(var line in SomeSortOfAwesomeness())
{
Console.WriteLine(line);
}
Update from Stephen Cleary
File.OpenText sadly only allows synchronous I/O; the async APIs are
implemented poorly in that scenario. To open a true asynchronous file,
you'd need to use a FileStream constructor passing isAsync: true or
FileOptions.Asynchronous.
ReadLineAsync basically results in this code, as you can see, it's only the Stream APM Begin and End methods wrapped
private Task<Int32> BeginEndReadAsync(Byte[] buffer, Int32 offset, Int32 count)
{
return TaskFactory<Int32>.FromAsyncTrim(
this, new ReadWriteParameters { Buffer = buffer, Offset = offset, Count = count },
(stream, args, callback, state) => stream.BeginRead(args.Buffer, args.Offset, args.Count, callback, state), // cached by compiler
(stream, asyncResult) => stream.EndRead(asyncResult)); // cached by compiler
}
I did some performance tests and it seems that a large bufferSize is helpful, together with the FileOptions.SequentialScan option.
public static async IAsyncEnumerable<string> ReadLinesAsync(string filePath)
{
using var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read,
FileShare.Read, 32768, FileOptions.Asynchronous | FileOptions.SequentialScan);
using var reader = new StreamReader(stream);
while (true)
{
var line = await reader.ReadLineAsync().ConfigureAwait(false);
if (line == null) break;
yield return line;
}
}
The enumeration in not trully asynchronous though. According to my experiments (.NET Core 3.1) the xxxAsync methods of the StreamReader class are blocking the current thread for a duration longer than the awaiting period of the Task they return. For example reading a 6 MB file with the method ReadToEndAsync in my PC blocks the current thread for 120 msec before returning the task, and then the task is completed in just 20 msec. So I am not sure that there is much value at using these methods. Faking asynchrony is much easier by using the synchronous APIs together with some Linq.Async:
IAsyncEnumerable<string> lines = File.ReadLines("SomeFile.txt").ToAsyncEnumerable();
.NET 6 update: The implementation of the asynchronous filesystem APIs has been improved on .NET 6. For experimental data with the File.ReadAllLinesAsync method, see here.
I have an arduino device that reads data and sends it in 64Byte chunks. At times it could be reading upto 4MB
Serial.write(data, 64);
I'm trying to read this data using a UWP app following the Microsoft example:
private async Task ReadAsync(CancellationToken cancellationToken, uint readBufferLength)
{
Task<UInt32> loadAsyncTask;
// Don't start any IO if we canceled the task
lock (ReadCancelLock)
{
cancellationToken.ThrowIfCancellationRequested();
// Cancellation Token will be used so we can stop the task operation explicitly
// The completion function should still be called so that we can properly handle a canceled task
DataReaderObject.InputStreamOptions = InputStreamOptions.Partial;
loadAsyncTask = DataReaderObject.LoadAsync(readBufferLength).AsTask(cancellationToken);
}
UInt32 bytesRead = await loadAsyncTask;
while (bytesRead == 0)
{
bytesRead = await loadAsyncTask;
}
if (bytesRead > 0)
{
dataRead = new byte[bytesRead];
DataReaderObject.ReadBytes(dataRead);
ReadBytesCounter += bytesRead;
}
I don't as the amount of data returned from the device is different each time I don't know the read buffer length. When I set the read buffer length to higher than 4MB it fails to return any data. When I set it lower than 4MB (say 32KB) it just returns that 32KB.
My questions are:
How do I get data to return when I don't know the read buffer length.
Is there no other way to read data from a serial device in UWP other than DataReaderObject.LoadAsync()? i.e a while loop where it continues to read data until no more is found? Or something like RS232_PollComport()?
How do I get data to return when I don't know the read buffer length.
I don't have a device for testing but try to update your code snippet like follows:
UInt32 bytesRead = await loadAsyncTask;
while (bytesRead == ReadBufferLength)
{
bytesRead = await loadAsyncTask;
}
if (DataReaderObject.UnconsumedBufferLength > 0)
{
ReadBytesTextBlock.Text += DataReaderObject.ReadString(DataReaderObject.UnconsumedBufferLength);
}
When I set the read buffer length to higher than 4MB it fails to return any data. When I set it lower than 4MB (say 32KB) it just returns that 32KB.
The readBufferLength parameter is setting for DataReader.LoadAsync method that means the count of bytes to load into the intermediate buffer. So that if you set it with a higher value than the data length, it should work.But avoid to set the buffer too big. If you set the readBufferLength less than the data length, await loadAsyncTask will return the buffer length, and still left data need load. But in your code snippet, you only continue load with while (bytesRead == 0), so that in this case you only load once with data length same to readBufferLength. You may need to continue LoadAsync with while (bytesRead == ReadBufferLength).
I don't know the read buffer length
Also you may try to use StreamReader which looks more simple. For example:
Stream streamIn = EventHandlerForDevice.Current.Device.InputStream.AsStreamForRead();
StreamReader reader = new StreamReader(streamIn);
string response = await reader.ReadLineAsync();
I need a serial port program to read data coming in at 4800 baud. Right now I have a simulator sending 15 lines of data every second. The output of it seems to get "behind" and can't keep up with the speed/amount of data coming in.
I have tried using ReadLine() with a DataReceieved event, which did not seem to be reliable, and now I am using an async method with serialPort.BaseStream.ReadAsync:
okToReadPort = true;
Task readTask = new Task(startAsyncRead);
readTask.Start();
//this method starts the async read process and the "nmeaList" is what
// is used by the other thread to display data
public async void startAsyncRead()
{
while (okToReadPort)
{
Task<string> task = ReadLineAsync(serialPort);
string line = await task;
NMEAMsg tempMsg = new NMEAMsg(line);
if (tempMsg.sentenceType != null)
{
nmeaList[tempMsg.sentenceType] = tempMsg;
}
}
public static async Task<string> ReadLineAsync(
this SerialPort serialPort)
{
// Console.WriteLine("Entering ReadLineAsync()...");
byte[] buffer = new byte[1];
string ret = string.Empty;
while (true)
{
await serialPort.BaseStream.ReadAsync(buffer, 0, 1);
ret += serialPort.Encoding.GetString(buffer);
if (ret.EndsWith(serialPort.NewLine))
return ret.Substring(0, ret.Length - serialPort.NewLine.Length);
}
}
This still seems inefficient, does anyone know of a better way to ensure that every piece of data is read from the port and accounted for?
Generally speaking, your issue is that you are performing IO synchronously with data processing. It doesn't help that your data processing is relatively expensive (string concatenation).
To fix the general problem, when you read a byte put it into a processing buffer (BlockingCollection works great here as it solves Producer/Consumer) and have another thread read from the buffer. That way the serial port can immediately begin reading again instead of waiting for your processing to finish.
As a side note, you would likely see a benefit by using StringBuilder in your code instead of string concatenation. You should still process via queue though.
I am trying to read/write bytes to/from a Bluetooth printer using Xamarin Android in C#. I am making use of System.IO.Stream to do this. Unfortunately, whenever I try to use ReadTimeout and WriteTimeout on those streams I get the following error:
Message = "Timeouts are not supported on this stream."
I don't want my Stream.Read() and Stream.Write() calls to block indefinitely. How can I solve this?
You probably would like to expose an method with cancellation token so your api can be easliy consumed.
One of the CancellationTokenSource constructors takes TimeSpan as a parameter. CancellationToken on other hand exposes Register method which allows you close the stream and the reading operation should stop with an exception being thrown.
Method call
var timeout = TimeSpan.Parse("00:01:00");
var cancellationTokenSource = new CancellationTokenSource(timeout);
var cancellationToken = cancellationTokenSource.Token;
await ReadAsync(stream, cancellationToken);
Method implementation
public async Task ReadAsync(Stream stream, CancellationToken cancellationToken)
{
using (cancellationToken.Register(stream.Dispose))
{
var buffer = new byte[1024];
var read = 0;
while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
// do stuff with read data
}
}
}
The following code will dispose stream only if it times out
More to can be found here.
Edit:
Changed .Close() to .Dispose() since it is no longer available in some PCLs .Close() vs .Dispose()
You must do the reads on another thread; in this case if you must stop reading you can close the stream from other thread and the read will finish with an exception.
Another easy way is to use a System.Threading.Timer to dispose the stream:
Stream str = //...
Timer tmr = new Timer((o) => str.Close());
tmr.Change(yourTimeout, Timeout.Infinite);
byte[] data = new byte(1024);
bool success = true;
try{ str.Read(data, 0, 1024); }
catch{ success = false, }
finally{ tmr.Change(Timeout.Inifinite, Timeout.Infinite); }
if(success)
//read ok
else
//read timeout
Currently I have a working live stream using webapi. By receiving a flv stream directly from ffmpeg and sending it straight to the client using PushStreamContent. This works perfectly fine if the webpage is already open when the stream starts. The issue is when I open another page or refresh this page you can no longer view the stream (the stream is still being sent to the client fine). I think it is due to something missing from the start of the stream but I am not sure what to do. Any pointers would be greatly appreciated.
Code for client reading stream
public class VideosController : ApiController
{
public HttpResponseMessage Get()
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent(WriteToStream, new MediaTypeHeaderValue("video/x-flv"));
return response;
}
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
//I think metadata needs to be written here but not sure how
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
}
Code for receiving stream and then sending to client
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
}
I've never used FLV or studied video formats closely
Most file formats are structured, especially video formats. They contain frames (i.e. a complete or partial screen shots depending on the compression format).
You should be really lucky if you manage to hit a specific frame when you start streaming to the new subscriber. Hence when they start receiving the stream they cannot identify the format as frame is partial.
You can read more FLV frames in wikipedia article. This is most likely your problem.
A simple attempt would be to try to save the initial header that you receive from the streaming server when the first subscriber connects.
Something like:
static byte _header = new byte[9]; //signature, version, flags, headerSize
public void YourStreamMethod()
{
int bytesRec = handler.Receive(bytes);
if (!_headerIsStored)
{
//store header
Buffer.BlockCopy(bytes, 0, _header, 0, 9);
_headerIsStored = true;
}
}
.. which allows you to send the header to the next connecting subscriber:
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
// send the FLV header
arg1.Write(_header, 0, 9);
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
Once done, pray that the receiver will ignore partial frames. If it doesn't you need to analyze the stream to identify where the next frame is.
To do that you need to do something like this:
Create a BytesLeftToNextFrame variable.
Store the received packet header in a buffer
Convert the "Payload size" bits to an int
Reset the BytesLeftToNextFrame to the parsed value
Countdown until the next time you should read a header.
Finally, when a new client connects, do not start streaming until you know that the next frame arrives.
Pseudo code:
var bytesLeftToNextFrame = 0;
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
if (subscriber.IsNew && bytesLeftToNextFrame < bytesRec)
{
//start from the index where the new frame starts
await theSubscriber.WriteAsync( bytes, bytesLeftToNextFrame, bytesRec - bytesLeftToNextFrame);
subscriber.IsNew = false;
}
else
{
//send everything, since we've already in streaming mode
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
//TODO: check if the current frame is done
// then parse the next header and reset the counter.
}
I'm not a expert in streaming, but looks like you should close stream then all data will be writed
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
Like it mentions in WebAPI StreamContent vs PushStreamContent
{
// After save we close the stream to signal that we are done writing.
xDoc.Save(stream);
stream.Close();
}
I LIKE THIS CODE BECAUSE IT DEMONSTRATES A FUNDAMENTAL ERROR when dealing with async programming
while (true)
{
}
this is a synced loop, that loops itself as fast as possible.. every second it can execute thousands of times (depending on availabe software and hardware resources)
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
this is an async command (if that wasn't clear enough) that execute in a DIFFERENT thread (while loop representes the main thread execution)
now... in order to make the while loop to wait to the async command we use await... sounds good (or else the while loop will execute thousands of times, executing countless async commands)
BUT because the loop (of subscribers) need to transmit the stream for all subscribers simulatanly it get stucked by the await keyword
THAT IS WHY RELOAD / NEW SUBSCRIBER FREEZE THE WHOLE THING (new connection = new subscriber)
conclusion: the entire for loop should be inside a Task. the Task need to wait until the server send the stream to all subscribers. ONLY THEN it should continue to the while loop with ContinueWith (that is why it called like that, right?)
so... the write command need to get execute without await keyword
theSubscriber.WriteAsync
the foreach loop should use a task that continue with the while loop after it is done