I'm trying to create a middleware that changes the request in a certain way. I am able to read it and change the content but I cannot figure out how to correctly setup the stream writers to create a new body. When I call normalized.WriteToAsync(jsonWriter) the MemoryStream remains empty and consequently I receive the A non-empty request body is required. exception. What am I missing here? This is what I have so far:
public async Task Invoke(HttpContext context)
{
if (context.Request.ContentType == "application/json" && context.Request.ContentLength > 0)
{
using var scope = _logger.BeginScope("NormalizeJson");
try
{
using var requestReader = new HttpRequestStreamReader(context.Request.Body, Encoding.UTF8);
using var jsonReader = new JsonTextReader(requestReader);
var json = await JToken.LoadAsync(jsonReader);
var normalized = _normalize.Visit(json); // <-- Modify json and return JToken
// Create new Body
var memoryStream = new MemoryStream();
var requestWriter = new StreamWriter(memoryStream);
var jsonWriter = new JsonTextWriter(requestWriter);
await normalized.WriteToAsync(jsonWriter); // <-- At this point the MemoryStream has still 0 length.
var content = new StreamContent(memoryStream.Rewind()); // <-- Use helper extension to Seek.Begin = 0
context.Request.Body = await content.ReadAsStreamAsync();
}
catch (Exception e)
{
_logger.Scope().Exceptions.Push(e);
}
}
await _next(context);
}
Demo for LINQPad etc.:
async Task Main()
{
var token = JToken.FromObject(new User { Name = "Bob" });
var memoryStream = new MemoryStream();
var requestWriter = new StreamWriter(memoryStream);
var jsonWriter = new JsonTextWriter(requestWriter);
await token.WriteToAsync(jsonWriter);
memoryStream.Length.Dump(); // <-- MemoryStream.Length = 0
}
public class User
{
public string Name { get; set; }
}
You need to properly flush and close your JsonTextWriter and StreamWriter in order to fully populate the memoryStream, like so:
var memoryStream = new MemoryStream();
// StreamWriter implements IAsyncDisposable
// Leave the underlying stream open
await using (var requestWriter = new StreamWriter(memoryStream, leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter); // But JsonTextWriter does not implement IAsyncDisposable, only IDisposable!
try
{
await token.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
}
Demo fiddle #1 here.
Or, since you're writing to a MemoryStream, there's really no nead to use async at all, and instead you can do:
var memoryStream = new MemoryStream();
using (var requestWriter = new StreamWriter(memoryStream, leaveOpen: true)) // Leave the underlying stream open
using (var jsonWriter = new JsonTextWriter(requestWriter))
{
token.WriteTo(jsonWriter);
}
Demo fiddle #2 here.
Notes:
Note the use of await using for the StreamWriter. This syntax guarantees that the StreamWriter will be flushed and closed asynchronously, and can be used on any object that implements IAsyncDisposable. (This only really matters if you were writing to a file stream or other non-memory stream.)
It seems that neither JsonTextWriter nor the base class JsonWriter implement IAsyncDisposable, so I had to asynchronously close the JSON writer manually rather than via a using statement. The outer await using should ensure that the underlying StreamWriter is not left open in the event of an exception.
JSON RFC 8259 specifies that Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text. Thus, when constructing a StreamWriter, it is recommended to pass an encoding such as new UTF8Encoding(false) that does not prepend a BOM. Alternatively, if you just want UTF-8, the StreamWriter constructors will create a StreamWriter with UTF-8 encoding without a Byte-Order Mark (BOM) if you do not specify one yourself and leave a default value for that parameter as is shown in the code above.
Related
I use code like this to replace default json serialize method:
private readonly static RecyclableMemoryStreamManager _recyclableMemoryStreamManager =
new RecyclableMemoryStreamManager(blockSize: 128 * 1024, largeBufferMultiple: 1024 * 1024, maximumBufferSize: 128 * 1024 * 1024);
private ByteArrayContent Serialize(object content, JsonSerializerSettings serializerSettings, Encoding encoding, string mediaType)
{
var jsonSerializer = Newtonsoft.Json.JsonSerializer.Create(serializerSettings);
using (var memoryStream = _recyclableMemoryStreamManager.GetStream())
{
using (var textWriter = new StreamWriter(memoryStream, encoding, 1024, true))
{
using (var jsonTextWriter = new JsonTextWriter(textWriter) { CloseOutput = false })
{
jsonSerializer.Serialize(jsonTextWriter, content);
jsonTextWriter.Flush();
var arraySegment = new ArraySegment<byte>(memoryStream.GetBuffer(), 0, (int)memoryStream.Length);
var resContent = new ByteArrayContent(arraySegment.Array, arraySegment.Offset, arraySegment.Count);
resContent.Headers.ContentType = new MediaTypeHeaderValue(mediaType);
return resContent;
}
}
}
}
But sometimes, http response json with sytanx error:
{
"code": 0,
"msg": null,
"data": [
// ....
]
}
')","foo":"","bar":"baz","flag":0,')","foo":"","bar":"baz","flag":0,
')","foo":"","bar":"baz","flag":0,')","foo":"","bar":"baz","flag":0,
How to fix this?
I think it maybe reuse buffer error,
Maybe can change the values of RecyclableMemoryStreamManager ?
_recyclableMemoryStreamManager.AggressiveBufferReturn = true;
The buffer from GetBuffer() is only well-defined for the lifetime of the stream; you dispose the stream when the method exits the using block for memoryStream, which means those buffers are now up for grabs for re-use.
You may wish to use StreamContent instead; this accepts a Stream of the payload, and disposes it when sent: I'd use that; that would give you the exact semantics you want here. Note: don't dispose memoryStream yourself - remove that using (perhaps adding a catch block that does memoryStream?.Dispose(); throw;).
Note also that GetBuffer() is not necessarily the optimal API for RecyclableMemoryStream, since it may use multiple discontiguous buffers internally; there should be a ReadOnlySequence<byte> GetReadOnlySequence() API which allows that usage - however, this still has the same lifetime limitations impacting buffer re-use, so: it wouldn't change anything here.
Untested, but for consideration:
private HttpContent Serialize(object content, JsonSerializerSettings serializerSettings, Encoding encoding, string mediaType)
{
var jsonSerializer = JsonSerializer.Create(serializerSettings);
var memoryStream = _recyclableMemoryStreamManager.GetStream();
try
{
using (var textWriter = new StreamWriter(memoryStream, encoding, 1024, true))
{
using var jsonTextWriter = new JsonTextWriter(textWriter) { CloseOutput = false };
jsonSerializer.Serialize(jsonTextWriter, content);
jsonTextWriter.Flush();
}
memoryStream.Position = 0; // rewind
var resContent = new StreamContent(memoryStream);
resContent.Headers.ContentType = new MediaTypeHeaderValue(mediaType);
return resContent;
}
catch
{
memoryStream?.Dispose();
throw;
}
}
However, I would expect it would be better to serialize directly to the output via the inbuilt JSON media encoder, rather than using an intermediate buffer.
I'm calling an API that return a response of type application/json. The response can be very tiny but it can be very huge, 500mb to 700mb. I would like to format the content (indentation, new lines, etc) and write the response to a json file with the help of System.Text.Json so the file can be read easely by an humain.
This code write the response stream directly to a file very efficiently on memory and speed but it doesn't format the json content. Since that the stream is directly written to the file, it doesn't take memory at all.
var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();
using (var fileStream = File.Open(filePath, FileMode.Create))
{
await responseStream.CopyToAsync(fileStream);
}
I tried this code to add the formatting but it doesn't seems right since that it use over 1gb of memory.
var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();
var jsonDocument = System.Text.Json.JsonDocument.Parse(responseStream);
var jsonWriterOptions = new System.Text.Json.JsonWriterOptions()
{
Indented = true
};
using (var fileStream = File.Open(filePath, FileMode.Create))
using (var jsonTextWriter = new System.Text.Json.Utf8JsonWriter(fileStream, jsonWriterOptions))
{
jsonDocument.WriteTo(jsonTextWriter);
}
Is there a more optimized way, that use less memory, to deal with huge content?
It seems that there's no easy way to do it with System.Text.Json because you cannot use a Stream object with the System.Text.Json.Utf8JsonReader directly. To bypass this limitation, you need to put the file content into memory with the System.Text.Json.JsonDocument object so obviously, it will take up a lot of memory.
For now, with what a read on the web, the only solution that is memory efficient is to use Newtonsoft.Json library. Since that there's no parsing involved with this logic, no memory is used to make the conversion.
In this example, the stream come from an HttpClient response.
var response = await new HttpClient().GetAsync("an url", HttpCompletionOption.ResponseHeadersRead);
var responseStream = await response.Content.ReadAsStreamAsync();
using (var streamReader = new StreamReader(responseStream))
using (var jsonReader = new JsonTextReader(streamReader))
using (var streamWriter = File.CreateText(destinationFilePath))
using (var jsonTextWriter = new JsonTextWriter(streamWriter))
{
jsonTextWriter.Formatting = Formatting.Indented;
while (jsonReader.Read())
{
jsonTextWriter.WriteToken(jsonReader);
}
}
This example show how to read an existing file
using (var streamReader = new StreamReader(sourceFilePath))
using (var jsonReader = new JsonTextReader(streamReader))
using (var streamWriter = File.CreateText(destinationFilePath))
using (var jsonTextWriter = new JsonTextWriter(streamWriter))
{
jsonTextWriter.Formatting = Formatting.Indented;
while (jsonReader.Read())
{
jsonTextWriter.WriteToken(jsonReader);
}
}
I currently have a method like this
public void BusinessMethod(object value, StreamWriter sw)
{
//Calls a private method that converts the data in `value` to a custom formatted string variable `str`
string str = myPrivateMethod(value);
//write the string to stream
sw.Write(str);
}
I am trying to test this method using the approach mentioned here and have done exactly the same thing. However, my result string comes back as an empty string. I cannot change the method signature. How does one test a method like this? I am using Nunit for testing.
This is my test method
[Test]
public void My_Test()
{
MyPoco dto = new MyPoco ();
//set up the dto properties here
using (var stream = new MemoryStream())
using (var writer = new StreamWriter(stream))
{
sut.BusinessMethod(dto, writer);
string result = Encoding.UTF8.GetString(stream.ToArray());
}
}
You need to Close/Flush/Dispose writer so it actually commits changes to stream:
using (var stream = new MemoryStream())
{
using (var writer = new StreamWriter(stream))
{
sut.BusinessMethod(dto, writer);
}
// moved outside of inner using to ensure writer stored content to stream
string result = Encoding.UTF8.GetString(stream.ToArray());
}
I want to use the DataContractJsonSerializer to serialize to file in JsonFormat.
The problem is that the WriteObjectmethod only has 3 options XmlWriter, XmlDictionaryWriter and Stream.
To get what I want I used the following code:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var ms = new MemoryStream())
{
js.WriteObject(ms, item);
ms.Position = 0;
using (var sr = new StreamReader(ms))
{
using (var writer = new StreamWriter(path, false))
{
string jsonData = sr.ReadToEnd();
writer.Write(jsonData);
}
}
}
Is this the only way or have I missed something?
Assuming you're just trying to write the text to a file, it's not clear why you're writing it to a MemoryStream first. You can just use:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var stream = File.Create(path))
{
js.WriteObject(stream, item);
}
That's rather simpler, and should do what you want...
I am actually quite terrified to claim to know something that Jon Skeet doesn't, but I have used code similar to the following which produces the Json text file and maintains proper indentation:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var stream = File.Create(path))
{
using (var writer = JsonReaderWriterFactory.CreateJsonWriter(stream, Encoding.UTF8, true, true, "\t"))
{
js.WriteObject(writer, item);
writer.Flush();
}
}
(as suggested here.)
I'm having trouble when creating a StreamWriter object in windows-8, usually I just create an instance just passing a string as a parameter, but in Windows 8 I get an error that indicates that it should recieve a Stream, but I noticed that Stream is an abstract class, Does anybody knows how will be the code to write an xml file?, BTW I'm using .xml because I want to save the serialized object, or does anyone knows how to save to a file a serialized object in Windows 8?.
Any ideas?
Currently using Windows 8 Consumer Preview
Code:
StreamWriter sw = new StreamWriter("person.xml");
Error:
The best overloaded method match for 'System.IO.StreamWriter.StreamWriter(System.IO.Stream)' has some invalid arguments
Instead of StreamWriter you would use something like this:
StorageFolder folder = ApplicationData.Current.LocalFolder;
StorageFile file = await folder.CreateFileAsync();
using (IRandomAccessStream fileStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (IOutputStream outputStream = fileStream.GetOutputStreamAt(0))
{
using (DataWriter dataWriter = new DataWriter(outputStream))
{
//TODO: Replace "Bytes" with the type you want to write.
dataWriter.WriteBytes(bytes);
await dataWriter.StoreAsync();
dataWriter.DetachStream();
}
await outputStream.FlushAsync();
}
}
You can look at the StringIOExtensions class in the WinRTXamlToolkit library for sample use.
EDIT*
While all the above should work - they were written before the FileIO class became available in WinRT, which simplifies most of the common scenarios that the above solution solves since you can now just call await FileIO.WriteTextAsync(file, contents) to write text into file and there are also similar methods to read, write or append strings, bytes, lists of strings or IBuffers.
You can create a common static method which you can use through out application like this
private async Task<T> ReadXml<T>(StorageFile xmldata)
{
XmlSerializer xmlser = new XmlSerializer(typeof(List<myclass>));
T data;
using (var strm = await xmldata.OpenStreamForReadAsync())
{
TextReader Reader = new StreamReader(strm);
data = (T)xmlser.Deserialize(Reader);
}
return data;
}
private async Task writeXml<T>(T Data, StorageFile file)
{
try
{
StringWriter sw = new StringWriter();
XmlSerializer xmlser = new XmlSerializer(typeof(T));
xmlser.Serialize(sw, Data);
using (IRandomAccessStream fileStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (IOutputStream outputStream = fileStream.GetOutputStreamAt(0))
{
using (DataWriter dataWriter = new DataWriter(outputStream))
{
dataWriter.WriteString(sw.ToString());
await dataWriter.StoreAsync();
dataWriter.DetachStream();
}
await outputStream.FlushAsync();
}
}
}
catch (Exception e)
{
throw new NotImplementedException(e.Message.ToString());
}
}
to write xml simply use
StorageFile file = await ApplicationData.Current.LocalFolder.CreateFileAsync("data.xml",CreationCollisionOption.ReplaceExisting);
await writeXml(Data,file);
and to read xml use
StorageFile file = await ApplicationData.Current.LocalFolder.GetFileAsync("data.xml");
Data = await ReadXml<List<myclass>>(file);