We Keep Coding

The C# app hangs up when reading from and writing to a NamedPipeServerStream *from the same thread*

For the past month or so I've been debugging an apparent deadlock caused by a thread reading from and writing to a NamedPipeServerStream, like this:
NamedPipeServerStream pipestream = new NamedPipeServerStream(#"\\.\foo.bar", PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
//Wait for client connection here
byte[] buffer = new byte[4096];
while (true)
{
int n = await pipestream.ReadAsync(buffer, 0, n);
if (n > 0)
{
await pipestream.WriteAsync(buffer, 0, n);
await pipestream.FlushAsync();
}
else
{
break;
}
}
This code is part of a WPF app, and when the "deadlock" happens, the entire app including the GUI freezes as if the message loop itself is somehow stopped.
Before the deadlock happens, the app works for some arbitrary amount of time (including the echoback loop) so the implementation seems to be correct except for the deadlock.
(The app runs on x86-64 Windows)
I tried many workarounds including making the calls to Read() and Write() synchronous, calling WaitForPipeDrain(), changing the PipeTransmissionMode to PipeTransmissionMode.Byte, removing the Asynchronous flag, changing the buffer size, not flushing after writing, checking for CanWrite and CanRead results, waiting until IsMessageComplete etc. but they didn't work at all.
What did work in the end was running two threads, one for reading and the other for writing, and echoing the data through a System.IO.Pipelines.Pipe instance, like this (I don't believe the fact that I used System.IO.Pipelines specifically is relevant for the fix though):
Main thread:
System.IO.Pipelines.Pipe pipe = new Pipe();
NamedPipeServerStream pipestream = new NamedPipeServerStream(#"\\.\foo.bar", PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.Asynchronous);
//Wait for client connection here
//Start thread 1 and 2
Thread 1:
byte[] buffer = new byte[4096];
while (true)
{
int n = await pipestream.ReadAsync(buffer, 0, 4096);
if (n > 0)
{
await pipe.Writer.AsStream().WriteAsync(buffer);
await pipe.Writer.AsStream().FlushAsync();
}
else
{
break;
}
}
Thread 2:
byte[] buffer = new byte[4096];
while (true)
{
int n = await pipe.Reader.AsStream().ReadAsync(buffer, 0, 4096);
if (n > 0)
{
await pipestream.WriteAsync(buffer, 0, n);
await pipestream.FlushAsync();
}
else
{
break;
}
}
I did some stress tests on this implementation and the deadlock seems to be gone forever which is bizarre to me. What I just did was basically writing the exact same thing but separated into two threads. The only possible explanation that I could come up with is that the Named Pipe implementation on Windows uses some thread-local data structures, but I can't think of any reasonable reason why.
Does anyone have an idea how this happens? Is it simply that I'm using Named Pipes wrong?
I would be grateful if anyone could point me in the right direction. Thank you in advance.

How to implement timeout on a .NET Stream when timeouts are not supported on this stream

I am trying to read/write bytes to/from a Bluetooth printer using Xamarin Android in C#. I am making use of System.IO.Stream to do this. Unfortunately, whenever I try to use ReadTimeout and WriteTimeout on those streams I get the following error:
Message = "Timeouts are not supported on this stream."
I don't want my Stream.Read() and Stream.Write() calls to block indefinitely. How can I solve this?

You probably would like to expose an method with cancellation token so your api can be easliy consumed.
One of the CancellationTokenSource constructors takes TimeSpan as a parameter. CancellationToken on other hand exposes Register method which allows you close the stream and the reading operation should stop with an exception being thrown.
Method call
var timeout = TimeSpan.Parse("00:01:00");
var cancellationTokenSource = new CancellationTokenSource(timeout);
var cancellationToken = cancellationTokenSource.Token;
await ReadAsync(stream, cancellationToken);
Method implementation
public async Task ReadAsync(Stream stream, CancellationToken cancellationToken)
{
using (cancellationToken.Register(stream.Dispose))
{
var buffer = new byte[1024];
var read = 0;
while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
// do stuff with read data
}
}
}
The following code will dispose stream only if it times out
More to can be found here.
Edit:
Changed .Close() to .Dispose() since it is no longer available in some PCLs .Close() vs .Dispose()

You must do the reads on another thread; in this case if you must stop reading you can close the stream from other thread and the read will finish with an exception.
Another easy way is to use a System.Threading.Timer to dispose the stream:
Stream str = //...
Timer tmr = new Timer((o) => str.Close());
tmr.Change(yourTimeout, Timeout.Infinite);
byte[] data = new byte(1024);
bool success = true;
try{ str.Read(data, 0, 1024); }
catch{ success = false, }
finally{ tmr.Change(Timeout.Inifinite, Timeout.Infinite); }
if(success)
//read ok
else
//read timeout

Live FLV streaming in C# WebApi

Currently I have a working live stream using webapi. By receiving a flv stream directly from ffmpeg and sending it straight to the client using PushStreamContent. This works perfectly fine if the webpage is already open when the stream starts. The issue is when I open another page or refresh this page you can no longer view the stream (the stream is still being sent to the client fine). I think it is due to something missing from the start of the stream but I am not sure what to do. Any pointers would be greatly appreciated.
Code for client reading stream
public class VideosController : ApiController
{
public HttpResponseMessage Get()
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent(WriteToStream, new MediaTypeHeaderValue("video/x-flv"));
return response;
}
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
//I think metadata needs to be written here but not sure how
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
}
Code for receiving stream and then sending to client
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
}

I've never used FLV or studied video formats closely
Most file formats are structured, especially video formats. They contain frames (i.e. a complete or partial screen shots depending on the compression format).
You should be really lucky if you manage to hit a specific frame when you start streaming to the new subscriber. Hence when they start receiving the stream they cannot identify the format as frame is partial.
You can read more FLV frames in wikipedia article. This is most likely your problem.
A simple attempt would be to try to save the initial header that you receive from the streaming server when the first subscriber connects.
Something like:
static byte _header = new byte[9]; //signature, version, flags, headerSize
public void YourStreamMethod()
{
int bytesRec = handler.Receive(bytes);
if (!_headerIsStored)
{
//store header
Buffer.BlockCopy(bytes, 0, _header, 0, 9);
_headerIsStored = true;
}
}
.. which allows you to send the header to the next connecting subscriber:
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
// send the FLV header
arg1.Write(_header, 0, 9);
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
Once done, pray that the receiver will ignore partial frames. If it doesn't you need to analyze the stream to identify where the next frame is.
To do that you need to do something like this:
Create a BytesLeftToNextFrame variable.
Store the received packet header in a buffer
Convert the "Payload size" bits to an int
Reset the BytesLeftToNextFrame to the parsed value
Countdown until the next time you should read a header.
Finally, when a new client connects, do not start streaming until you know that the next frame arrives.
Pseudo code:
var bytesLeftToNextFrame = 0;
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
if (subscriber.IsNew && bytesLeftToNextFrame < bytesRec)
{
//start from the index where the new frame starts
await theSubscriber.WriteAsync( bytes, bytesLeftToNextFrame, bytesRec - bytesLeftToNextFrame);
subscriber.IsNew = false;
}
else
{
//send everything, since we've already in streaming mode
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
//TODO: check if the current frame is done
// then parse the next header and reset the counter.
}

I'm not a expert in streaming, but looks like you should close stream then all data will be writed
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
Like it mentions in WebAPI StreamContent vs PushStreamContent
{
// After save we close the stream to signal that we are done writing.
xDoc.Save(stream);
stream.Close();
}

I LIKE THIS CODE BECAUSE IT DEMONSTRATES A FUNDAMENTAL ERROR when dealing with async programming
while (true)
{
}
this is a synced loop, that loops itself as fast as possible.. every second it can execute thousands of times (depending on availabe software and hardware resources)
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
this is an async command (if that wasn't clear enough) that execute in a DIFFERENT thread (while loop representes the main thread execution)
now... in order to make the while loop to wait to the async command we use await... sounds good (or else the while loop will execute thousands of times, executing countless async commands)
BUT because the loop (of subscribers) need to transmit the stream for all subscribers simulatanly it get stucked by the await keyword
THAT IS WHY RELOAD / NEW SUBSCRIBER FREEZE THE WHOLE THING (new connection = new subscriber)
conclusion: the entire for loop should be inside a Task. the Task need to wait until the server send the stream to all subscribers. ONLY THEN it should continue to the while loop with ContinueWith (that is why it called like that, right?)
so... the write command need to get execute without await keyword
theSubscriber.WriteAsync
the foreach loop should use a task that continue with the while loop after it is done

Why does Console.WriteLine() block in callback from Stream.ReadAsync()?

I have a callback function in which I am trying to write the data that I read in an overriden ReadAsync().
private void StreamCallback(byte[] bytes)
{
Console.WriteLine("--> " + Encoding.UTF8.GetString(bytes)); // the whole application is blocked here, why?
if (OnDataReceived != null)
{
string data = Encoding.UTF8.GetString(bytes);
OnDataReceived(data);
}
}
The overriden ReadAsync() looks as follows.
public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, System.Threading.CancellationToken cancellationToken)
{
var read = await _originalStream.ReadAsync(buffer, offset, count, cancellationToken);
_readCallback(buffer);
return read;
}
What I actually want to achieve is to monitor a network stream just before it gets parsed by an XmlReader. This relates to my other question > Reading from same SslStream simultaneously? <. How would I do that?
UPDATE:
It is actually Encoding.UTF8.GetString(bytes) that is blocking the application. In order for the question to be more complete I am listing the code for reading the XML stream.
using (XmlReader r = XmlReader.Create(sslStream, new XmlReaderSettings() { Async = true }))
{
while (await r.ReadAsync())
{
switch (r.NodeType)
{
case XmlNodeType.XmlDeclaration:
...
break;
case XmlNodeType.Element:
...

Based on the code you posted, StreamCallback() will block until that stream ends. You pass a byte pointer to Encoding.UTF8.GetString(bytes); So, it needs to keep querying bytes until it reaches the end. It will never reach the end since bytes comes from a stream until that stream is closed.
You need to either process your stream a certain number of bytes at a time or until a certain character is seen.

How to write file chunk by chunk in WinRT?

I am trying to write buffer data into a file. I receive buffer data in a callback function continuously. I need to read the buffer and save it in a file as it is received. This will be repeated till i get complete file,i get data chunk of 4k in size. But below code either throws an exception or output file is corrupted. Please let me know how to do this in winRT.
StorageFile file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync(strFileName, Windows.Storage.CreationCollisionOption.ReplaceExisting);
public async void Receive(byte[] buffer)
{
using (var ostream = await file.OpenStreamForWriteAsync())
{
await ostream.WriteAsync(buffer, 0, buffer.Length);
}
}

The problem is in the signature of Receive. Because it's void, it is not awaited and you can run to writting processes in the same time (that's probably what causes the exception, and/or corrupted datas).
I suggest to use this instead :
StorageFile file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync(strFileName, Windows.Storage.CreationCollisionOption.ReplaceExisting);
public async Task Receive(byte[] buffer)
{
using (var ostream = await file.OpenStreamForWriteAsync())
{
await ostream.WriteAsync(buffer, 0, buffer.Length);
}
}
And call with
await Receive(b);