I have azure function, which accepts huge bson object. It binds to http request and then try to deserialize it using stream with the following code:
using (var stream = new MemoryStream())
{
await request.Content.CopyToAsync(stream);
using (var reader = new BsonDataReader(stream))
{
var serializer = new JsonSerializer();
var readings =
serializer.Deserialize<IEnumerable<ProviderReading>>(reader);
}
}
readings object is always null.
I tested it using the standard ReadAsAsync method:
var test = await request.Content.ReadAsAsync<List<ProviderReading>>(
new[]{new BsonMediaTypeFormatter()});
in that case it deserialize the collection of readings correctly.
Any suggestions?
Using CopyTo (or its async variant) advances both the source's and target's position. That means that by the time you construct the BsonDataReader the input stream is already at its end.
You should reset the stream's position:
stream.Position = 0;
Related
I am trying to read request from httpcontext,but it is changing, not same with original request.So it creates problem during hashing on SHA256.When I try to create sha256 on online tools with original request it is okey,but when I take request after reading it from httpcontext.request its is not same hash I create with original request.
What is the exact solution to read request as same as with original request without changing it and convert to string to compute SHA256?
using (var ms = new MemoryStream())
{
await httpContext.Request.Body.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
using (var sr = new StreamReader(ms))
{
using (var jsonTextReader = new JsonTextReader(sr))
{
var bodyContent = serializer.Deserialize(jsonTextReader);
//hashing starts here with bodyContent.ToString()
}
}
}
I want to write export/download functionality for files from external API.
I've created separate Action for it. Using external API I can get stream for that file.
When I am saving that stream to local file, everything is fine, file isn't empty.
var exportedFile = await this.GetExportedFile(client, this.ReportId, this.WorkspaceId, export);
// Now you have the exported file stream ready to be used according to your specific needs
// For example, saving the file can be done as follows:
string pathOnDisk = #"D:\Temp\" + export.ReportName + exportedFile.FileSuffix;
using (var fileStream = File.Create(pathOnDisk))
{
await exportedFile.FileStream.CopyToAsync(fileStream);
}
But when I return exportedFile object that contains in it stream and do next:
var result = await this._service.ExportReport(reportName, format, CancellationToken.None);
var fileResult = new HttpResponseMessage(HttpStatusCode.OK);
using (var ms = new MemoryStream())
{
await result.FileStream.CopyToAsync(ms);
ms.Position = 0;
fileResult.Content = new ByteArrayContent(ms.GetBuffer());
}
fileResult.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = $"{reportName}{result.FileSuffix}"
};
fileResult.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
return fileResult;
Exported file is always empty.
Is it problem with stream or with code that try to return that stream as file?
Tried as #Nobody suggest to use ToArray
fileResult.Content = new ByteArrayContent(ms.ToArray());
the same result.
Also tried to use StreamContent
fileResult.Content = new StreamContent(result.FileStream);
still empty file.
But when I'm using StreamContent and MemmoryStream
using (var ms = new MemoryStream())
{
await result.FileStream.CopyToAsync(ms);
ms.Position = 0;
fileResult.Content = new StreamContent(ms);
}
in result I got
{
"error": "no response from server"
}
Note: from 3rd party API I get stream that is readonly.
you used GetBuffer() to retrieve the data of the memory stream.
The function you should use is ToArray()
Please read the Remarks of the documentation of these functions.
https://learn.microsoft.com/en-us/dotnet/api/system.io.memorystream.getbuffer?view=net-6.0
using (var ms = new MemoryStream())
{
ms.Position = 0;
await result.FileStream.CopyToAsync(ms);
fileResult.Content = new ByteArrayContent(ms.ToArray()); //ToArray() and not GetBuffer()
}
Your "mistake" although it's an obvious one is that you return a status message, but not the actual file itself (which is in it's own also a 200).
You return this:
var fileResult = new HttpResponseMessage(HttpStatusCode.OK);
So you're not sending a file, but a response message. What I'm missing in your code samples is the procedure call itself, but since you use a HttpResonseMessage I will assume it's rather like a normal Controller action. If that is the case you could respond in a different manner:
return new FileContentResult(byteArray, mimeType){ FileDownloadName = filename };
where byteArray is ofcourse just a byte[], the mimetype could be application/octet-stream (but I suggest you'd actually find the correct mimetype for the browser to act accordingly) and the filename is the filename you want the file to be named.
So, if you were to stitch above and my comment together you'd get this:
var exportedFile = await this.GetExportedFile(client, this.ReportId, this.WorkspaceId, export);
// Now you have the exported file stream ready to be used according to your specific needs
// For example, saving the file can be done as follows:
string pathOnDisk = #"D:\Temp\" + export.ReportName + exportedFile.FileSuffix;
using (var fileStream = File.Create(pathOnDisk))
{
await exportedFile.FileStream.CopyToAsync(fileStream);
}
return new FileContentResult(System.IO.File.ReadAllBytes(pathOnDisk), "application/octet-stream") { FileDownloadName = export.ReportName + exportedFile.FileSuffix };
I suggest to try it, since you still report a 200 (and not a fileresult)
Currently I am using a two step approach to fetch data from the Web Api and
deserialize the CSV records into objects.
var response = await httpClient.GetAsync(queryString);
using (var reader = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
var csvr = new CsvReader(reader);
var responseValue = csvr.GetRecords<TrainingDataSetData>().ToList();
result.Readings.AddRange(responseValue);
}
How do I optimize this code?
If you're trying to avoid creating an intermediate MemoryStream - you could use the GetStreamAsync method on HttpClient, which should return the raw NetworkStream for you pass straight into CsvHelper, instead of ReadAsStreamAsync, which will default to reading the full response into a MemoryStream before returning.
using (var reader = new StreamReader(await httpClient.GetStreamAsync(queryString)))
{
var csvr = new CsvReader(reader);
var responseValue = csvr.GetRecords<TrainingDataSetData>().ToList();
result.Readings.AddRange(responseValue);
}
If you still need access to the HttpResponseMessage, you could achieve the same effect by using HttpCompletionOption.ResponseHeadersRead, which won't buffer the response before returning.
var response = await httpClient.GetAsync(queryString, HttpCompletionOption.ResponseHeadersRead);
As to whether or not this is actually more efficient - this is something that would require benchmarking in your particular environment to decide, since it may be conditional on the size of the response, speed of the network etc.
I have a class MyData which is Json serializable by using Json.Net JsonSerializer.Serialize(TextWriter, object). I want to send this data (as json) to a web service via HttpClient.PostAsync.
Because converting the json to string and then sending it as StringContent is (probably) not performant, I want to do it with streams.
I found the class StreamContent, which takes a stream in its constructor. And serializing json into streams should be possible as well. So I tried this:
MyData data = ...; // already filled
string uri = ...; // already filled
HttpClient client = new HttpClient();
JsonSerializer serializer = new JsonSerializer();
using (MemoryStream ms = new MemoryStream())
{
using (StreamWriter sw = new StreamWriter(ms))
using (JsonWriter jw = new JsonTextWriter(sw))
{
serializer.Serialize(sw, data);
ms.Flush();
ms.Position = 0;
}
HttpResponseMessage response = client.PostAsync(uri, new StreamContent(ms)).Result;
}
But running this code gives me two exceptions in the line HttpResponseMessage response = ...:
HttpRequestException: Error when copying content into a stream.
ObjectDisposedException: Could not access closed stream.
What am I doing wrong?
If you serialize the object into a MemoryStream, the entire JSON data will be written in the buffer, so there is no significant performance benefit over just serializing into a string and using StringContent.
Your StremWriter disposes the memory stream before the request is sent, that is why you get the exceptions.
You can either move your using statements to be in the same scope as the MemoryStream, or use the StreamWriter's constructor that accepts a boolean parameter to leave the stream open after the writer is disposed.
StreamWriter constructor:
Unless you set the leaveOpen parameter to true, the StreamWriter object calls Dispose() on the provided Stream object when StreamWriter.Dispose is called.
var incomingStream = ...
var outgoingStream = ...
await incomingStream.CopyToAsync(outgoingStream);
The above code is simple enough, and copies a incoming stream to the outgoign stream. Both streams being chunked transfers coming/going over the interet.
Now, lets say i wanted to Transform the stream with something like Func<Stream,Stream,Task> how would I do that without reading all data in.
Ofcause I could just do
var ms = new MemoryStream();
incomingStream.CopyTo(ms);
--- do transform of streams and seek
ms.CopyTo(outgoingStream)
but that would read the hole thing into the ms, is there any build in stuff that allows me to read from incoming stream and write to a new stream that dont buffer everything up but instead just keep a small internal stream for buffered data and it wont read from incoming stream before data is pulled off it again.
What I am trying to do is:
protected async Task XmlToJsonStream(Stream instream, Stream outStream)
{
XmlReaderSettings readerSettings = new XmlReaderSettings();
readerSettings.IgnoreWhitespace = false;
var reader = XmlReader.Create(instream, readerSettings);
var jsonWriter = new JsonTextWriter(new StreamWriter(outStream));
jsonWriter.WriteStartObject();
while (await reader.ReadAsync())
{
jsonWriter.writeReader(reader);
}
jsonWriter.WriteEndObject();
jsonWriter.Flush();
}
protected async Task XmlFilterStream(Stream instream, Stream outStream)
{
XmlReaderSettings readerSettings = new XmlReaderSettings();
readerSettings.IgnoreWhitespace = false;
var reader = XmlReader.Create(instream, readerSettings);
var writer = XmlWriter.Create(outStream, new XmlWriterSettings { Async = true, CloseOutput = false })
while (reader.Read())
{
writer.writeReader(reader);
}
}
but i dont know how to hook it up.
var incomingStream = ...
var outgoingStream = ...
var temp=...
XmlFilterStream(incomingStream,temp);
XmlToJsonStream(temp,outgoingstream);
because if I use a MemoryStream as temp, would it not just at the end have it all stored in the stream. Looking for at stream that throws away the data again when it has been read.
All of the above is just example code, missing some disposes and seeks ofcause, but I hope I managed to illustrate what i am going for. To be able to based on settings to plug and play between just copying stream, doing xml filtering and optional transform it to json.
Streams are sequences of bytes, so a stream transformation would be something like Func<ArraySegment<byte>, ArraySegment<byte>>. You can then apply it in a streaming way:
async Task TransformAsync(this Stream source, Func<ArraySegment<byte>, ArraySegment<byte>> transform, Stream destination, int bufferSize = 1024)
{
var buffer = new byte[bufferSize];
while (true)
{
var bytesRead = await source.ReadAsync(buffer, 0, bufferSize);
if (bytesRead == 0)
return;
var bytesToWrite = transform(new ArraySegment(buffer, 0, bytesRead));
if (bytesToWrite.Count != 0)
await destination.WriteAsync(bytesToWrite.Buffer, bytesToWrite.Offset, bytesToWrite.Count);
}
}
It's a bit more complicated than that, but that's the general idea. It needs some logic to ensure WriteAsync writes all the bytes; and there's also usually a "flush" method that is required in addition to the transform method, which is called when the source stream finishes, so the transform algorithm has a last chance to return its final data to write to the output stream.
If you want streams of other things, like XML or JSON types, then you're probably better off going with Reactive Extensions.
I'm not sure I understand your question fully, but I think you're asking how you would operate on an input stream without loading it entirely into memory first.
In this case, you wouldn't want do do something like this:
var ms = new MemoryStream();
incomingStream.CopyTo(ms);
This does load the entire input stream incomingStream into memory -- into ms.
From what I can see, your XmlFilterStream method seems to be redundant, i.e. XmlToJsonStream does everything that XmlFilterStream does anyway.
Why not just have:
protected async Task XmlToJsonStream(Stream instream, Stream outStream)
{
XmlReaderSettings readerSettings = new XmlReaderSettings();
readerSettings.IgnoreWhitespace = false;
var reader = XmlReader.Create(instream, readerSettings);
var jsonWriter = new JsonTextWriter(new StreamWriter(outStream));
jsonWriter.WriteStartObject();
while (await reader.ReadAsync())
{
jsonWriter.writeReader(reader);
}
jsonWriter.WriteEndObject();
jsonWriter.Flush();
}
And call it like this:
var incomingStream = ...
var outgoingStream = ...
XmlToJsonStream(incomingStream ,outgoingstream);
If the answer is that you have omitted some important details in XmlFilterStream then, without seeing those details, I would recommend that you just integrate those into the one XmlToJsonStream function.