I am trying to write buffer data into a file. I receive buffer data in a callback function continuously. I need to read the buffer and save it in a file as it is received. This will be repeated till i get complete file,i get data chunk of 4k in size. But below code either throws an exception or output file is corrupted. Please let me know how to do this in winRT.
StorageFile file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync(strFileName, Windows.Storage.CreationCollisionOption.ReplaceExisting);
public async void Receive(byte[] buffer)
{
using (var ostream = await file.OpenStreamForWriteAsync())
{
await ostream.WriteAsync(buffer, 0, buffer.Length);
}
}
The problem is in the signature of Receive. Because it's void, it is not awaited and you can run to writting processes in the same time (that's probably what causes the exception, and/or corrupted datas).
I suggest to use this instead :
StorageFile file = await Windows.Storage.ApplicationData.Current.LocalFolder.CreateFileAsync(strFileName, Windows.Storage.CreationCollisionOption.ReplaceExisting);
public async Task Receive(byte[] buffer)
{
using (var ostream = await file.OpenStreamForWriteAsync())
{
await ostream.WriteAsync(buffer, 0, buffer.Length);
}
}
And call with
await Receive(b);
Related
I'm working on a C# application that communicates with the clangd language server.
clangd is started as a separate process managed by my C# program and communication works via I/O redirection.
My program is exchanging language server protocol request-response pairs with clangd. Requests are sent to clangd via its process' StandardInput stream and responses are read using its process' StandardOutput stream. clangd emits debug information using its process' StandardError stream.
I am using async methods for reading and writing in order to keep the user interface responsive.
However, after sending the third textDocument/didOpen message, my program freezes while trying to read the response.
Making use of the StandardError stream I found out, that clangd processes the third textDocument/didOpen message correctly, as it emits debug messages, meaning the response should be available on StandardOutput.
I saved the requests in a file and sent those to a clangd instance running on the command line, which worked like a charm. I attached that file in case you need it.
Furthermore, the debug messages which read at the bottom of the SendRequest method indicate that the file was opened:
I[09:55:53.552] <-- textDocument/didOpen(26)
I[09:55:56.512] Updating file C:\Temp\crossrail\src\ClLogic.cpp with command [C:\Temp\crossrail\src] clang C:\Temp\crossrail\src\ClLogic.cpp -resource-dir=C:\Program Files (x86)\LLVM\bin\..\lib\clang\8.0.0
Below you can see the LSP client code for reading and writing the responses. I marked the location that gets blocked.
private Process languageServer;
private StreamWriter requestWriter;
private StreamReader responseReader;
private StreamReader errorReader;
// ...
public void Connect(String workingDirectory)
{
if (this.Connected == false)
{
this.currentMessageID = LSP_FIRST_MESSAGE_ID;
this.languageServer.StartInfo.WorkingDirectory = workingDirectory;
this.languageServer.Start();
this.Connected = true;
this.requestWriter = this.languageServer.StandardInput;
this.responseReader = this.languageServer.StandardOutput;
this.errorReader = this.languageServer.StandardError;
}
}
public async Task<String> Query<T>(JsonRpcRequest<T> request)
{
await mutex.WaitAsync();
try
{
await this.SendRequest(request);
return await this.ReadResponse();
}
finally
{
mutex.Release();
}
}
private async Task SendRequest<T>(JsonRpcRequest<T> request)
{
request.ID = this.currentMessageID;
++this.currentMessageID;
String requestBody = request.ToString();
Console.WriteLine(requestBody);
await this.requestWriter.WriteAsync(requestBody.ToCharArray(), 0, requestBody.Length);
await this.requestWriter.FlushAsync();
if (request.ID == 26) // ID of the third textDocument/didOpen message
{
//await this.ReadErrors(); // the debug messages following the third textDocument/didOpen request are printed correctly
}
}
private async Task<String> ReadResponse()
{
String contentLengthHeader = await this.responseReader.ReadLineAsync(); // blocks after the third textDocument/didOpen message
int responseLength = Int32.Parse
(
contentLengthHeader.Substring(contentLengthHeader.IndexOf(LSP_HEADER_KEY_VALUE_DELIMITER) + LSP_HEADER_VALUE_OFFSET)
.Trim()
);
await this.responseReader.ReadLineAsync();
char[] buffer = new char[BUFFER_SIZE];
StringBuilder response = new StringBuilder();
int totalReadBytes = 0;
while (totalReadBytes < responseLength)
{
int readBytes = await this.responseReader.ReadAsync(buffer, 0, BUFFER_SIZE);
response.Append(buffer, 0, readBytes);
totalReadBytes += readBytes;
}
Console.WriteLine(response.ToString());
return response.ToString();
}
public async Task SendFileCloseMessage(DocumentCloseRequest request)
{
await mutex.WaitAsync();
try
{
await this.SendRequest(request);
this.responseReader.DiscardBufferedData();
}
finally
{
mutex.Release();
}
}
Here is my code using the LSP client's methods for sending the textDocument/didOpen message:
private async Task InitializeLanguageServer(bool highlightFunctionDeclarations)
{
if (this.languageServer.Connected == false)
{
this.languageServer.Connect(this.workingDirectory);
await this.SendInitializationMessage();
}
await this.SendOpenFileMessage();
await this.LoadSymbolDeclarationLocations(highlightFunctionDeclarations);
await this.LoadSymbolUsages();
}
private async Task SendInitializationMessage()
{
InitializationRequest request = new InitializationRequest
(
this.workingDirectory,
System.Diagnostics.Process.GetCurrentProcess().Id,
false,
ApplicationSettings.LANGUAGE_SERVER_PROTOCOL_SUPPORTED_SYMBOLS
);
Console.WriteLine(await this.languageServer.Query(request));
}
private async Task SendOpenFileMessage()
{
DocumentOpenRequest request = new DocumentOpenRequest(this.filePath, "cpp", 1, this.SourceCode);
Console.WriteLine(await this.languageServer.Query(request));
}
InitializeLanguageServer is called in the constructor without await, but that shouldn't be a problem, as clangd is fast enough to process every source code file in a maximum of 2.5 seconds.
The languageServer member is retrieved using TinyIoC:
public SourceViewerVM()
{
// ...
this.languageServer = TinyIoCContainer.Current.Resolve<LanguageServerProtocolClient>();
#pragma warning disable CS4014
this.InitializeLanguageServer(highlightFunctionDeclarations);
#pragma warning restore CS4014
}
Edit:
The reading really blocks and isn't just waiting for a new line character. If I put the follwing code at my breakpoint which usually reads the StandardError, execution is blocked too:
if (request.ID == 26) // 26 is the ID of the third textRequest/didOpen message
{
char[] buffer = new char[BUFFER_SIZE];
int readBytes = await this.responseReader.ReadAsync(buffer, 0, BUFFER_SIZE); // blocking
Console.WriteLine(new String(buffer, 0, readBytes));
//await this.ReadErrors();
}
I am trying to read/write bytes to/from a Bluetooth printer using Xamarin Android in C#. I am making use of System.IO.Stream to do this. Unfortunately, whenever I try to use ReadTimeout and WriteTimeout on those streams I get the following error:
Message = "Timeouts are not supported on this stream."
I don't want my Stream.Read() and Stream.Write() calls to block indefinitely. How can I solve this?
You probably would like to expose an method with cancellation token so your api can be easliy consumed.
One of the CancellationTokenSource constructors takes TimeSpan as a parameter. CancellationToken on other hand exposes Register method which allows you close the stream and the reading operation should stop with an exception being thrown.
Method call
var timeout = TimeSpan.Parse("00:01:00");
var cancellationTokenSource = new CancellationTokenSource(timeout);
var cancellationToken = cancellationTokenSource.Token;
await ReadAsync(stream, cancellationToken);
Method implementation
public async Task ReadAsync(Stream stream, CancellationToken cancellationToken)
{
using (cancellationToken.Register(stream.Dispose))
{
var buffer = new byte[1024];
var read = 0;
while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
// do stuff with read data
}
}
}
The following code will dispose stream only if it times out
More to can be found here.
Edit:
Changed .Close() to .Dispose() since it is no longer available in some PCLs .Close() vs .Dispose()
You must do the reads on another thread; in this case if you must stop reading you can close the stream from other thread and the read will finish with an exception.
Another easy way is to use a System.Threading.Timer to dispose the stream:
Stream str = //...
Timer tmr = new Timer((o) => str.Close());
tmr.Change(yourTimeout, Timeout.Infinite);
byte[] data = new byte(1024);
bool success = true;
try{ str.Read(data, 0, 1024); }
catch{ success = false, }
finally{ tmr.Change(Timeout.Inifinite, Timeout.Infinite); }
if(success)
//read ok
else
//read timeout
Currently I have a working live stream using webapi. By receiving a flv stream directly from ffmpeg and sending it straight to the client using PushStreamContent. This works perfectly fine if the webpage is already open when the stream starts. The issue is when I open another page or refresh this page you can no longer view the stream (the stream is still being sent to the client fine). I think it is due to something missing from the start of the stream but I am not sure what to do. Any pointers would be greatly appreciated.
Code for client reading stream
public class VideosController : ApiController
{
public HttpResponseMessage Get()
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent(WriteToStream, new MediaTypeHeaderValue("video/x-flv"));
return response;
}
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
//I think metadata needs to be written here but not sure how
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
}
Code for receiving stream and then sending to client
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
}
I've never used FLV or studied video formats closely
Most file formats are structured, especially video formats. They contain frames (i.e. a complete or partial screen shots depending on the compression format).
You should be really lucky if you manage to hit a specific frame when you start streaming to the new subscriber. Hence when they start receiving the stream they cannot identify the format as frame is partial.
You can read more FLV frames in wikipedia article. This is most likely your problem.
A simple attempt would be to try to save the initial header that you receive from the streaming server when the first subscriber connects.
Something like:
static byte _header = new byte[9]; //signature, version, flags, headerSize
public void YourStreamMethod()
{
int bytesRec = handler.Receive(bytes);
if (!_headerIsStored)
{
//store header
Buffer.BlockCopy(bytes, 0, _header, 0, 9);
_headerIsStored = true;
}
}
.. which allows you to send the header to the next connecting subscriber:
private async Task WriteToStream( Stream arg1, HttpContent arg2, TransportContext arg3 )
{
// send the FLV header
arg1.Write(_header, 0, 9);
Startup.AddSubscriber( arg1 );
await Task.Yield();
}
Once done, pray that the receiver will ignore partial frames. If it doesn't you need to analyze the stream to identify where the next frame is.
To do that you need to do something like this:
Create a BytesLeftToNextFrame variable.
Store the received packet header in a buffer
Convert the "Payload size" bits to an int
Reset the BytesLeftToNextFrame to the parsed value
Countdown until the next time you should read a header.
Finally, when a new client connects, do not start streaming until you know that the next frame arrives.
Pseudo code:
var bytesLeftToNextFrame = 0;
while (true)
{
bytes = new byte[8024000];
int bytesRec = handler.Receive(bytes);
foreach (var subscriber in Startup.Subscribers.ToList())
{
var theSubscriber = subscriber;
try
{
if (subscriber.IsNew && bytesLeftToNextFrame < bytesRec)
{
//start from the index where the new frame starts
await theSubscriber.WriteAsync( bytes, bytesLeftToNextFrame, bytesRec - bytesLeftToNextFrame);
subscriber.IsNew = false;
}
else
{
//send everything, since we've already in streaming mode
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
}
}
catch
{
Startup.Subscribers.Remove(theSubscriber);
}
}
//TODO: check if the current frame is done
// then parse the next header and reset the counter.
}
I'm not a expert in streaming, but looks like you should close stream then all data will be writed
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
Like it mentions in WebAPI StreamContent vs PushStreamContent
{
// After save we close the stream to signal that we are done writing.
xDoc.Save(stream);
stream.Close();
}
I LIKE THIS CODE BECAUSE IT DEMONSTRATES A FUNDAMENTAL ERROR when dealing with async programming
while (true)
{
}
this is a synced loop, that loops itself as fast as possible.. every second it can execute thousands of times (depending on availabe software and hardware resources)
await theSubscriber.WriteAsync( bytes, 0, bytesRec );
this is an async command (if that wasn't clear enough) that execute in a DIFFERENT thread (while loop representes the main thread execution)
now... in order to make the while loop to wait to the async command we use await... sounds good (or else the while loop will execute thousands of times, executing countless async commands)
BUT because the loop (of subscribers) need to transmit the stream for all subscribers simulatanly it get stucked by the await keyword
THAT IS WHY RELOAD / NEW SUBSCRIBER FREEZE THE WHOLE THING (new connection = new subscriber)
conclusion: the entire for loop should be inside a Task. the Task need to wait until the server send the stream to all subscribers. ONLY THEN it should continue to the while loop with ContinueWith (that is why it called like that, right?)
so... the write command need to get execute without await keyword
theSubscriber.WriteAsync
the foreach loop should use a task that continue with the while loop after it is done
I have so many files that i have to download. So i try to use power of new async features as below.
var streamTasks = urls.Select(async url => (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream()).ToList();
var streams = await Task.WhenAll(streamTasks);
foreach (var stream in streams)
{
using (var fileStream = new FileStream("blabla", FileMode.Create))
{
await stream.CopyToAsync(fileStream);
}
}
What i am afraid of about this code it will cause big memory usage because if there are 1000 files that contains 2MB file so this code will load 1000*2MB streams into memory?
I may missing something or i am totally right. If am not missed something so it is better to await every request and consume stream is best approach ?
Both options could be problematic. Downloading only one at a time doesn't scale and takes time while downloading all files at once could be too much of a load (also, no need to wait for all to download before you process them).
I prefer to always cap such operation with a configurable size. A simple way to do so is to use an AsyncLock (which utilizes SemaphoreSlim). A more robust way is to use TPL Dataflow with a MaxDegreeOfParallelism.
var block = new ActionBlock<string>(url =>
{
var stream = (await WebRequest.CreateHttp(url).GetResponseAsync()).GetResponseStream();
using (var fileStream = new FileStream("blabla", FileMode.Create))
{
await stream.CopyToAsync(fileStream);
}
},
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 100 });
Your code will load the stream into memory whether you use async or not. Doing async work handles the I/O part by returning to the caller until your ResponseStream returns.
The choice you have to make dosent concern async, but rather the implementation of your program concerning reading a big stream input.
If I were you, I would think about how to split the work load into chunks. You might read the ResponseStream in parallel and save each stream to a different source (might be to a file) and release it from memory.
This is my own answer chunking idea from Yuval Itzchakov and i provide implementation. Please provide feedback for this implementation.
foreach (var chunk in urls.Batch(5))
{
var streamTasks = chunk
.Select(async url => await WebRequest.CreateHttp(url).GetResponseAsync())
.Select(async response => (await response).GetResponseStream());
var streams = await Task.WhenAll(streamTasks);
foreach (var stream in streams)
{
using (var fileStream = new FileStream("blabla", FileMode.Create))
{
await stream.CopyToAsync(fileStream);
}
}
}
Batch is extension method that is simply as below.
public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> source, int chunksize)
{
while (source.Any())
{
yield return source.Take(chunksize);
source = source.Skip(chunksize);
}
}
I have a callback function in which I am trying to write the data that I read in an overriden ReadAsync().
private void StreamCallback(byte[] bytes)
{
Console.WriteLine("--> " + Encoding.UTF8.GetString(bytes)); // the whole application is blocked here, why?
if (OnDataReceived != null)
{
string data = Encoding.UTF8.GetString(bytes);
OnDataReceived(data);
}
}
The overriden ReadAsync() looks as follows.
public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, System.Threading.CancellationToken cancellationToken)
{
var read = await _originalStream.ReadAsync(buffer, offset, count, cancellationToken);
_readCallback(buffer);
return read;
}
What I actually want to achieve is to monitor a network stream just before it gets parsed by an XmlReader. This relates to my other question > Reading from same SslStream simultaneously? <. How would I do that?
UPDATE:
It is actually Encoding.UTF8.GetString(bytes) that is blocking the application. In order for the question to be more complete I am listing the code for reading the XML stream.
using (XmlReader r = XmlReader.Create(sslStream, new XmlReaderSettings() { Async = true }))
{
while (await r.ReadAsync())
{
switch (r.NodeType)
{
case XmlNodeType.XmlDeclaration:
...
break;
case XmlNodeType.Element:
...
Based on the code you posted, StreamCallback() will block until that stream ends. You pass a byte pointer to Encoding.UTF8.GetString(bytes); So, it needs to keep querying bytes until it reaches the end. It will never reach the end since bytes comes from a stream until that stream is closed.
You need to either process your stream a certain number of bytes at a time or until a certain character is seen.